Desktop data/settings replication

In the last UDS, there were some talks about UbuntuOne, the technologies it uses, and how it could be well integrated into the Desktop. Also, there were discussions about how it could be integrated painlessly into upstream projects. So, here’s an idea on how this could be done.

First, it must be said that the easiest (and quickest) way of achieving UbuntuOne integration in Ubuntu would be to just patch/extend applications so that they supported accessing the UbuntuOne server, and have Ubuntu packages use that as default for users with UbuntuOne accounts. That would make most Ubuntu users happy, but it would not benefit at all users of other distributions, and worst, the upstream projects.

Now, if we look at the technologies being used in UbuntuOne, there is one awesome thing, called CouchDB, a project supported by the Apache Foundation, which provides databases (of JSON documents) that can be replicated (and 2-way synchonized) to other hosts. So, what if we had Linux Desktop applications use this for storage of files and settings?

couchdb-in-the-desktop

Well, what would happen is that we’d gain data / settings replication and synchronization for free. And also, if we could come up with standard formats / locations for common information (accounts, notes, mails, calendars, etc, etc), we’d also gain a shared storage for all applications to use, solving the problem of incompatible formats / locations used by similar free software applications.

And other advantages:

  • CouchDB knows already how to deal with conflicts, as this is included in the automatic replication / syncing features it provides.
  • While normal documents in CouchDB are JSON, you can attach any kind of file to any JSON document (even to empty JSON documents), so any kind of files can be stored. Also, it allows users to create as many databases as needed, so storage for different needs can be easily separated.
  • CouchDB provides a sort of revision history, so it could be used for nice stuff like Zeitgeist.
  • This, not being an Ubuntu-only solution, could benefit every Linux Desktop user.
  • UbuntuOne would be a service built on top of this that users can subscribe to. But others could just setup a CouchDB server on their home / company network and use that by just pointing their local CouchDB to their remote CouchDB replication server.

To continue my investigations/playing on this, I’m going to try writing a gvfs backend to manage files in the CouchDB instances. Once that’s done, applications could start just writing their files to couchdb://… URIs instead of file://… ones and enter the replication/synchronization world with just a single change. Next, a GConf/d-conf backend could be added for replicating/sync’ing settings, and so on.

14 Responses to “Desktop data/settings replication”

  1. Jef Spaleta says:

    How does CouchDB handle authentication? It doesn’t have openid support on its own does it?

    So, you know what would be fun… getting the Tomboy/Snowy developers to take a good hard look at CouchDB. Do they feel like it solves their sync scenario adequately?

    Are you going to be going to GUADEC to talk about CouchDB?

    -jef

  2. Matt Jones says:

    I look forward to seeing your results.

    Make sure to test with syncing large files (300 MB +). I would hate to see something break because it tried to put a 1 GB json object into a string (think iso files), and ran out of ram.

  3. Killerkiwi says:

    Isn’t this what conduit is for??

    Seems like a conduit provider for ubuntuone could do this and more?

    Plus then we could sync settings to a phone / usb drive …

  4. ethana2 says:

    Don’t forget installed apps. Take apt’s list of installed applications and optionally add or remove them from different machines.

  5. ethana2 says:

    ..because that’s the only thing Windows and OS X can’t do with such a service, because it requires a package manager and software repositories.

  6. Natan Yellin says:

    This seems interesting, but how would it fit in with apps that use sqlite databases? Would they be forced to switch to CouchDB?

  7. I am interested by CouchDB for a few projects, but I am wondering about the footprint.
    From what I understand you need to run a CouchDB server on your machine (no embedded version) and communicate with it via an HTTP interface. This does not sound very lightweight. Have you made any measurement on this?

  8. john says:

    Would this also be a good use for Midgard2?

    http://www.midgard-project.org/updates/midgard2-9-3-0-vinland-released/

    I know that there is somebody trying to port tomboy to midgard to do just what you are talking about.

  9. Thanks for the post! Very interesting. Man, it was so good to see you at UDS. Hope the time frames between our next real life meeting are not 2 years again! (-:

  10. Joel Reed says:

    I don’t think you’ll be able to use CouchDB out of the box to maintain a revision history. See, http://wiki.apache.org/couchdb/Compaction and https://issues.apache.org/jira/browse/COUCHDB-36.

  11. […] Rodrigo Moya From lost to the river « Desktop data/settings replication […]

  12. Patcito says:

    @Aurélien Gâteau: one of the main couchdb dev is working on an erlang interface for couchdb called hovercraft, no need for http and it’s super fast: http://github.com/jchris/hovercraft/tree/master

    This one could be used for local install of couchdb.

  13. […] with my CouchDB on the desktop series, here’s the 1st […]