Looks like we are getting some attention from Linux Magazine. Dmitri Popov has written a 3 page article on Conduit and Unison. Some lovely screenshots (looks like he’s an Ubuntu user too :D). Click here for the article.
Posts Tagged ‘Conduit’
So Conduit has moved to GNOME SVN at last If you are running SVN conduit, you can now get your fix from:
This is great news for Conduit, and will hopefully help us spread the syncable desktop vision throughout GNOME and the free desktop in general.
Just checked in some more code for automatic background syncing. The user story is now like this:
- Create a Conduit with a Tomboy and Folder dataprovider
- Sync manually (optional, and may happen automatically in future…)
- Make changes without thinking about conduit anymore
- Changes are synced automatically
The first sync will always use the change calculation code to make sure all changes are detected. But from then until conduit is closed the sync engine will only use the changes detected “on the fly”. In most cases this means the built in DeltaProvider code (which has to get() every object and compare to a previous state file) is bypassed, so big savings can be made.
Fresh, fresh, SVN only code. There no doubt be dragons
Just switched on one of the “always up to date” mechanisms in Conduit. Simply, a dataprovider can now request a sync when it detects a change. The sync will happen straight away, so the data wil always be up to date.
I’ve also cobbled together an AutoSync mixin for the common cases to use. This buffers changes and provides a way to feed a list changes to the sync engine. This means the sync engine can avoid a “calculate what has changed” step. The code deals with situations such as added and deleting an object before it is even synced anywhere transparently, making custom dataproviders with change detection super easy. The code is there and ready for dataproviders to use, though I still need to implement some glue before the sync engine will be able to fully take advantage.
So far, Tomboy, Folders and the experimental GConf preferences sync are geared up to use autosync. F-spot and Evolution are on the list to get this functionality, in both cases we need to implement some code in our dependencies (in f-spot directly, and in the python-evolution support libary). PC to PC needs more thinking as XML-RPC isn’t geared up for two-way communication…
This behaviour should integrate well with Online Desktop, which i hope to blog about in the near future..
So they implemented sync. Good for you guys. The whole approach does, as Owen suggested, sound a tad over-engineered. But I don’t see that as a problem. In this case I agree with Havoc, and the possibility for alternative backends is quite exciting.
What does upset me is that the whole approach is very familiar. As far as i’m concerned, online-prefs-sync-daemon is redundant. They should have got Conduit SVN and contributed a GConf twoway dataprovider. Conduit is online-prefs-sync-daemon for the whole desktop, not just one use case.
The sooner people grok that sync is something that shouldn’t be implemented 60 different times the better!
Who wants to see Conduit Network Sync in action then? Here it is! The quality is a bit crap – if you want to submit a better version of this vid you’d make me a very happy boy.
In it you’ll see both of my laptops. The bigger (and older) one I’ll call Fred. And (purely for the purpose of this post) the little one is Barney. The scenario is that Fred has all my data on and I want to get it to Barney and then keep both in sync. As you’ll see in the video, Conduit’s network code is getting to the point where its ready to take on this challenge.
You’re about to witness how data gets pushed from Fred to Barney (2 contacts and a note). You’ll then see how changes are made on Barney and how they get pulled back down to Fred (2 deletions and 2 modifications are made on Barney, and these are synced to Fred).
I really like how the change to the tomboy note I made on Barney is reflected on Fred immediately after the sync – i don’t have to reopen the note or anything! And with the change detection stuff thats starting to land.. imagine as soon as you change a note it could potentially be pushed back to a server or to your other machine!
The network code currently is split in to 3 main modules. Server.py is the bit that actually makes a dataprovider network accessible. It is implemented using Twisted XML-RPC and is a big PITA because it needs to be threaded. Client.py consumes a remote shared dp and plugs it in to the GUI. It is just a standard xmlrpclib client with a few special cases for relaying Exceptions. All the avahi helper foo is hidden in Peers.py.
The next major focus (other than working on the test framework to hammer the bugs out of the network code) is actually making it possible to set which dataproviders are shared from the GUI – at the moment it is hard coded to only share Evolution Contacts and Tomboy Notes.
I haven’t written in a while so thought it would be a good idea to As the majority of my posts are about Conduit, I thought i’d start there.
Conduit 0.3 was released about a week ago. We’ve had about a 1,000 downloads and a few people have been gleeming with love for it. The past weeks have been quite mad, fixing bugs and enabling dps that are maturing. But 0.3.1 is the work of just two of us, with a couple of 3rd party patches and one contribution of a dataprovider. We’ve had lots of suggestions via the bug tracker, including N800 support. Yes, hadn’t thought of that one at all. Patches welcome, although in this case I think i’d prefer a donation of an N800
The most interesting change we are hoping to have ready for 0.3.1 would have to be the network sync stuff i’ve been doing. This would be packaged as an option, as its still early alpha and doesn’t have decent test coverage yet. It represents a Friday afternoon and weekend of me learning Twisted whilst waving my fist and cursing. Up until Friday the extent of my Twisted experience was running a Finger server inside Conduit, be warned. I’ve managed to inject a REST-ish Twisted web server into a DataProviderFactory as a server. Without going into any detail, the result of my efforts includes a successful two way sync of Tomboy to Tomboy between two Ubuntu Feisty laptops. It’s still flakey, but its getting there.
is coming soon. There should be a final release within the next week and then we can crack on with some big new toys for 0.4 – for me that means my network sync stuff…
Alex, just wanted to throw in my thoughts as an outsider.
I like to keep up with the developments in the free and open source software world. I track a fair few blogs and planets and wade through around 80 posts a day in my aggregator (maybe more). In that time i’ve read a lot about gimmie. I was impressed. I tried it, and was even more impressed! Despite hearing lots about Gimmie, I haven’t heard anything about BigBoard. And it would seem it’s developers are just as much a part of “Planet Gnome” as yourself? Anyway, I just wanted to say that I can’t wait to see what happens in Gimmie next
The discussions about accessing web services made me consolidate a few points i’ve have for a while and i’d like to share with you (and the community in general). On the desktop we have lots of data we want to share, to sync, to, well, use!
- Banshee has to learn how to talk to each type of MP3 player to sync it’s library
- So does Rhythmbox, but in a different language.
- Beagle and Tracker have to learn how each application stores its data so you can search it
- Gimmie has to learn how to talk to generic web app X, Y and Z and how to get any interesting and relevant desktop data
- Conduit is a sync tool and so has to do all this and some more!
- Tomboy wants to be able to sync too
I think it’s great how useful these applications are, but I think a lot more could be done to share data. It’s not really about whacking together a library so two apps can talk to an iPod. That’s bad because each app using it has a layer of glue to link it in to their plugin system. I think instead we as a community need to think in terms of objects that people are using. Have a common way to share a photo between different applications and devices and webservices. Sort of like a higher level VFS.
My utopian vision then.
- It would be nice if Tomboy officially used Conduit to do it’s syncing. Right now that means that you would be able to sync Tomboy to any folder that VFS can see, to an iPod and (i believe) BackPackit. And google notebook wouldn’t be difficult. As more dataproviders are developed for Conduit then this list will increase greatly. The todo list includes my network code which would allow Tomboy to Tomboy over the network. The same code will enable F-Spot to F-Spot etc etc. Why reimplement from scratch when we can do this now?
- More work would be done on the Conduit Web Services stuff and Gimmie would be able to use our DataProvider framework to query these different kinds of sites. And because the DP’s are responding with generic Photo or Contact objects its trivial for Gimmie to handle other DP’s too. The same code could show a Photo from Flickr or a photo from a folder, just need to pick the DP! Because Conduit is in python too then Gimmie could probably directly reference Conduit code rather than going through the DBus API even. If Gimmie can consume one dataprovider, it can pretty much consume them all!
- Banshee and Rhythmbox would have Conduit plugins so that they can use the Conduit DataProvider framework to work with music devices. This is instead of them writing glue around each generic-music-player-support.so. Conduit would provide one set of glue. Both Banshee and Rhythmbox would then support every DAP Conduit ever does (them ALL >: ).. I dream..)
- Tracker and Beagle would be able to use our DataProviders to build their indexes. Why have two different sets of glue code for reading the same data out of evolution and f-spot? You can talk about speed I suppose, but I want a desktop that’s integrated. When I want to scrape out every last MhZ out of a box i’ll give Xubuntu a whirl!
I don’t know how well i’m making my point here. But I don’t see why all of these projects can’t work together more. If they could all pass around these common desktop objects then surely everyone would benefit?
With my last commit Conduit now happily supports hot unplug of dataproviders, in particular the iPod support is taking advantage of this. If you eject your iPod while Conduit is open it will disappear from the list of available dataproviders and any active Conduits you have will be set to “Pending”. Plugging the iPod back in works fine too, the Pending dataproviders are replaced with the real thing once more!
Now to write some tests and make sure everything is cleared down properly – 1000 add / remove cycles should give me some clues about any nasty leaks lurking in there It would be nice (or rather, terrifying!!) to see what happens if I were to eject mid synchronization…