ZeroConf support for Bazaar

When at conferences and sprints, I often want to see what someone else is working on, or to let other people see what I am working on. Usually we end up pushing up to a shared server and using that as a way to exchange branches. However, this can be quite frustrating when competing for outside bandwidth when at a conference.

It is possible to share the branch from a local web server, but that still means you need to work out the addressing issues.

To make things easier, I wrote a simple Bazaar/Avahi plugin. It provides a command “bzr share“, which does the following:

  • Scan the directory for any Bazaar branches it contains.
  • Start up the Bazaar server to listen on a TCP port and share the given directory.
  • Advertise each of the branches via mDNS using Avahi. They are shared using the branch nickname.

For the client side, the plugin implements a “bzr browse” command that will list the Bazaar branches being advertised on the local network (the name and the bzr:// URL). Using the two commands together, it is trivial to share branches locally or find what branches people are sharing.

I am not completely satisfied with how things work, and have a few ideas for how to improve things:

  1. Provide a dummy transport that lets people pull from branches by their advertised service name. This would essentially just redirect from scheme://$SERVICE/ to bzr://$HOST:$PORT/$PATH.
  2. Maybe provide more control over the names the branches get advertised with. Perhaps this isn’t so important though.
  3. Make “bzr share” start and stop advertising branches as they get added/removed, and handle branch nicknames changing (at this point, it is pretty much blue sky though).
  4. Perhaps some form of access control. I’m not sure how easy this is within the smart server protocol, but it should be possible to query the user over whether to accept a connection or not.

It will be interesting to see how well this works at the next sprint or conference.

gnome-vfs-obexftp 0.1 released

I put out a tarball release of gnome-vfs-obexftp here:

http://download.gnome.org/sources/gnome-vfs-obexftp/0.1/

This includes a number of fixes since the work I did in October:

  • Fix up some error handling in the dbus code.
  • Mark files under the obex:/// virtual root as being local. This causes Nautilus to process the desktop entries and give us nice icons.
  • Ship a copy of osso-gwobex, built statically into the VFS module. This removes the need to install another shared library only used by one application.

As well as the standard Gnome and D-BUS libraries, you will need OpenOBEX >= 1.2 and Bluez-Utils >= 3.7. The hcid daemon must be started with the -x flag to enable the experimental D-BUS interfaces used by the VFS module. You will also need a phone or other device that supports OBEX FTP :)

Last but not least, a screenshot:

Update

There was a build issue in the previous release caused by the osso-gwobex headers not being found. I hadn’t noticed it because I had installed the library standalone. An updated tarball is available here:

http://download.gnome.org/sources/gnome-vfs-obexftp/0.2/

UTC+9

Daylight saving started yesterday: the first time since 1991/1992 summer for Western Australia. The legislation finally passed the upper house on 21st November (12 days before the transition date). The updated tzdata packages were released on 27th November (6 days before the transition). So far, there hasn’t been an updated package released for Ubuntu (see bug 72125).

One thing brought up in the Launchpad bug was that not all applications used the system /usr/share/zoneinfo time zone database. So other places that might need updating include:

  • Evolution has a database in /usr/share/evolution-data-server-$version/zoneinfo/ that is in iCalendar VTIMEZONE format.
  • Java has a database in /usr/lib/jvm/java-$version/jre/lib/zi. This uses a different binary file format.
  • pytz (used by Zope 3 and Launchpad among others) has a database consisting of generated Python source files for its database.

All the above rules time zone databases are based on the same source time zone information, but need to be updated individually and in different ways.

In a way, this is similar to the zlib security problems from a few years back: the same problem duplicated in many packages and needing to be fixed over and over again. Perhaps the solution is the same too: get rid of the duplication so that in future only one package needs updating.

As a start, I put together a patch to pytz so that it uses the same format binary time zone files as found in /usr/share/zoneinfo (bug 71227). This still means it has its own time zone database, but it goes a long way towards being able to share the system time zone database. It’d be nice if the other applications and libraries with their own databases could make similar changes.

For people using Windows, there is an update from Microsoft. Apparently you need to install one update now, and then a second update next year — I guess Windows doesn’t support multiple transition rules like Linux does. The page also lists a number of applications that will malfunction and not know about the daylight saving shift, so I guess that they have similar issues of some applications ignoring the system time zone database.

Building obex-method

I published a Bazaar branch of the Nautilus obex method here:

http://bazaar.launchpad.net/~jamesh/+junk/gnome-vfs-obexftp

This version works with the hcid daemon included with Ubuntu Edgy, rather than requiring the btcond daemon from Maemo.

Some simple instructions on building it:

  1. Download and build the osso-gwobex library:

    svn checkout https://stage.maemo.org/svn/maemo/projects/connectivity/osso-gwobex/trunk osso-gwobex

    The debian/ directory should work fine to build a package using debuild.

  2. Download and build the obex module:

    bzr branch http://bazaar.launchpad.net/~jamesh/+junk/gnome-vfs-obexftp

    There is no debian packaging for this — just an autogen.sh script.

It should work on other distributions, but I haven’t tried it. The main requirement is bluez-utils 3.7, and that -x is passed to hcid to enable the org.bluez.RFCOMM interface.

Still to do is cleaning up the obex:/// listing of bonded devices, so that it serves desktop entries rather than symlinks, so that it is usable in Nautilus. This would also make it easier to show the device names to the user and get nice icons.

Playing Around With the Bluez D-BUS Interface

In my previous entry about using the Maemo obex-module on the desktop, Johan Hedberg mentioned that bluez-utils 3.7 included equivalent interfaces to the osso-gwconnect daemon used by the method. Since then, the copy of bluez-utils in Edgy has been updated to 3.7, and the necessary interfaces are enabled in hcid by default.

Before trying to modify the VFS code, I thought I’d experiment a bit with the D-BUS interfaces via the D-BUS python bindings. Most of the interesting method calls exist on the org.bluez.Adapter interface. We can easily get the default adapter with the following code:

import dbus

bus = dbus.SystemBus()
manager = dbus.Interface(
    bus.get_object('org.bluez', '/org/bluez'),
    'org.bluez.Manager')

adapter = dbus.Interface(
    bus.get_object('org.bluez', manager.DefaultAdapter()),
    'org.bluez.Adapter')

At this point, it is possible to perform discovery:

import dbus.glib
import gtk

def remote_device_found(addr, class_, rssi):
    print 'Found:', addr
def discovery_complete():
    gtk.main_quit()

adapter.connect_to_signal('RemoteDeviceFound', remote_device_found)
adapter.connect_to_signal('DiscoveryCompleted', discovery_completed)

adapter.DiscoverDevices()
gtk.main()

It is also possible to configure periodic discovery, which will send signals about devices that get found, disappear, or change name, so we could easily implement the obex: directory listing that shows all the devices found that support OBEX-FTP. One thing that isn’t clear from the API documentation is what happens if multiple programs try to start or stop discovery at the same time. It looks like the second program will get a org.bluez.Error.InProgress error when it tries to begin discovery. Ideally discovery would stay active til the last program interested in the results closed. Maybe I am misunderstanding it a bit and you can actually use the interface in this mode.

When we want to actually do OBEX-FTP with the device, we can establish the rfcomm connection:

rfcomm = dbus.Interface(
    bus.get_object('org.bluez', manager.DefaultAdapter()),
    'org.bluez.RFCOMM')

# will return e.g. /dev/rfcomm0
devname = rfcomm.Connect(bluetooth_address, 'ftp')

# communicate with the phone via the new rfcomm device

rfcomm.Disconnect(devname)

So it should be possible to modify obex-method to function with only the daemons included in Ubuntu Edgy. All that’s left is to do the actual work :).

OBEX in Nautilus

When I got my new laptop, one of the features it had that my previous one didn’t was Bluetooth support. There are a few Bluetooth related utilities for Gnome that let you send and receive SMS messages and a few other things, but a big missing feature is the ability to transfer files to and from the phone easily.

Ideally, I’d be able to browse the phone’s file system using Nautilus. Luckily, the Maemo guys have already done the hard work of writing a gnome-vfs module that speaks the OBEX FTP protocol. I had a go at compiling it on my laptop (running Ubuntu Edgy), and you can see the result below:


Browsing files and viewing images stored on my phone

There are a few rough edges:

  • While I can get a list of discovered devices at the location obex:///, it displays the raw bluetooth addresses rather than device names. Furthermore, the files displayed here are symlinks to the appropriate obex://[$ADDRESS]/ URLs, which aren’t that useful given that gnome-vfs does not support symlinks pointing to other schemes or authorities. This could be fixed by generating .desktop files instead, which would make it possible to provide nice icons too.
  • Can’t rename files. This might be a limitation of the OBEX FTP protocol though: the man page for the command line obexftp client says moves only work with Siemens phones.
  • Doesn’t seem to handle devices disappearing very well — when I tried turning off Bluetooth on my phone and told Nautilus to reload the window, Nautilus hung and stopped redrawing til I turned Bluetooth on again.

I don’t have any ready made binaries for others to try at this point. Below are some notes for anyone else who wants to try building it:

  • You’ll need the osso-gwconnect, osso-gwobex and osso-gnomevfs-extra modules. I grabbed them from Maemo Subversion.
  • When trying to build a debian package for osso-gwconnect, I removed the libosso-dev and mce-dev build dependencies, and made an equivalent change to the configure arguments in debian/rules. The configure script also asks for BlueZ 3.2, while Edgy only has 3.1. The package built fine when I decreased the minimum version requirement.
  • You’ll need to build osso-gwconnect and osso-gwobex before osso-gnomevfs-extra. There are a few build problems with this last module:
    1. The autogen.sh script asks for automake 1.8.x specifically, but works fine with the current 1.9.x releases.
    2. I had to change a dbus_connection_disconnect() call to dbus_connection_close() in obex-module/src/om-dbus.c.
    3. You only need to build the obex-utils and obex-module directories. There are other bits in this module that you probably don’t want, and some bits like the replacement GTK filesystem backend didn’t build for me.

With a little bit of work, this would fit into the main Gnome desktop quite well. When talking to Bastien a while back, he said that the extra dbus daemons shouldn’t really be necessary, so it might be worth trying to bypass them.

Gnome-gpg 0.5.0 Released

Over the weekend, I released gnome-gpg 0.5.0. The main features in this release is support for running without gnome-keyring-daemon (of course, you can’t save the passphrase in this mode), and to use the same keyring item name for the passphrase as Seahorse. The release can be downloaded here:

http://download.gnome.org/sources/gnome-gpg/0.5/

I also switched over from Arch to Bazaar. The conversion was fairly painless using bzr baz-import-branch, and means that I have both my revisions and Colins revisions in a single tree. The branch can be pulled from:

bzr branch http://www.gnome.org/~jamesh/bzr/gnome-gpg/devel gnome-gpg

All of the converted revisions authored by me have been signed with my PGP key. As signatures can’t get moved over in the conversion process, none of Colin’s revisions are signed. Note that the signatures in Bazaar are for particular tree states rather than changes between two tree states, so it doesn’t affect the trust of the current revisions.

While I was at it, I also converted the other branches I had in my www.gnome.org Arch archive over to bzr. The only other branch that people might find useful is the http-resource code, which I’ve updated to compile with the latest libsoup.

Vote Counting and Board Expansion

Recently one of the Gnome Foundation directors quit, and there has been a proposal to expand the board by 2 members. In both cases, the proposed new members have been taken from the list of candidates who did not get seats in the last election from highest vote getter down.

While at first this sounds sensible, the voting system we use doesn’t provide a way of finding out who would have been selected for the board if a particular candidate was removed from the ballot.

The current voting system gives each foundation member N votes to assign to N candidates (where N is the number of seats on the board). The votes are then tallied for each candidate, and the N candidates with the most votes get the seats.

If we look at last year’s results, there were 119 people who voted for Luis. If Luis had not been a candidate, then those 119 people would have used that vote to pick other candidates. The difference in the number of votes received by Vincent (the board member receiving the least votes) and Quim (the unsuccessful candidate with the most votes) was just 16, so those extra 119 votes could easily have affected the ordering of the remaining candidates.

Furthermore, if the election was for nine seats rather than seven then each foundation member would have had an additional two votes to cast.

This particular problem would not be an issue with a preferential voting system where each foundation member lists all the candidates in their order of preference. If a board member drops out, it is trivial to recalculate the results with that candidate removed: the relative orderings of the other candidates on the ballot are preserved. It is also possible to calculate the results for a larger number of seats.

Of course, all the candidates from the last election would make great board members so it isn’t so much of an issue in this case, but it might be worth considering for next time.

JHBuild Updates

The progress on JHBuild has continued (although I haven’t done much in the last week or so). Frederic Peters of JhAutobuild fame now has a CVS account to maintain the client portion of that project in tree.

Perl Modules (#342638)

One of the other things that Frederic has been working on is support for building Perl modules (which use a Makefile.PL instead of a configure script). His initial patchworked fine for tarballs, but by switching over to the new generic version control code in jhbuild it was possible to support Perl modules maintained in any of the supported version control systems without extra effort.

Speed Up Builds (#313997)

One of the other suggestions for jhbuild that came up a while ago was to make it “eleventy billion times faster”. In essence, adding a mode where it would only rebuild modules that had changed. While the idea has merrit, the proposed implementation had some problems (it used the output of “cvs update” to decide whether things had changed).

I’d like to get something like this implemented, preferably with three possible behaviours:

  1. Build everything (the current behaviour).
  2. Build only modules that have changed.
  3. Build only modules that have changed, or have dependencies that have changed.

The second option is obviously the fastest, and is a useful option for collections of modules that should be API stable. The third option is essentially an optimisation of the first option. For both the second and third option, it is necessary to be able to tell if the code in a module has been updated. The easiest way to do this is to record an identifier for the tree state, and the identifier is different after an update.

The identifier should be cheap to calculate too, so will probably be dependent on the underlying version control system:

  • CVS – a hash of the names and versions of all files in the tree. Something like this, maybe (can be constructed by reading the CVS/Root, CVS/Repository and CVS/Entries files in the tree).
  • Subversion – a combination of (a) the repository UUID, (b) the path of the tree inside the repository and (c) the youngest revision for this subtree in the repository.
  • Arch – the output of “baz tree-id“.
  • Bzr – the working tree’s revision ID.
  • Darcs – a hash of the sequence of patches representing the tree, maybe?
  • Tarballs – the version number for the tarball.

On a successful build, the ID for the tree would be recorded. On subsequent builds, the ID gets recalculated after updating the tree. The new and old IDs are then used to decide on whether to build the module or not, according to the chosen policy.

JHBuild Improvements

I’ve been doing most JHBuild development in my bzr branch recently. If you have bzr 0.8rc1 installed, you can grab it here:

bzr branch http://www.gnome.org/~jamesh/bzr/jhbuild/jhbuild.dev

I’ve been keeping a regular CVS import going at http://www.gnome.org/~jamesh/bzr/jhbuild/jhbuild.cvs using Tailor, so changes people make to module sets in CVS make there way into the bzr branch. I’ve used a small hack so that merges back into CVS get recorded correctly in the jhbuild.cvs branch:

  1. Apply the diff between jhbuild.cvs and jhbuild.dev to my CVS checkout and commit.
  2. Modify tailor to pause before committing the to jhbuild.cvs.
  3. While tailor is paused, run bzr revert followed by a merge of the same changes from jhbuild.dev.
  4. Let tailor complete the commit.

It’s a bit of a hack, but it allows me to do repeated merges from the CVS import to my development branch (and back again). It also means that any file moves I do in my bzr branch are reflected in the CVS import when merged.

So now when filing bug reports on jhbuild, you can submit fixes in the form of bzr branches as well as patches.

So, on to the improvements:

Generic Version Control Interface

Previously, to add support for a new version control system the following additions were needed:

  • Some code to invoke the version control utility to make checkouts and update working trees.
  • Code to implement the build state machine for modules using the version control system (these classes would generally derive from AutogenModule which implemented most of the build logic).
  • Code to create instances of the above module type when parsing .modules files.

This was quite a bit of work, and in the end would only help if the code in question was set up to build the same way as most Gnome modules (i.e. with a autogen.sh script and autotools). If you wanted to build a module using Python distutils out of Subversion, a new module type would be needed. If you wanted to build a distutils module from a tarball, that would be another module type again.

With the new system, the different version control support modules provide a common interface. This means that a single module type is capable of implementing the build state machine for any version control system. Similarly, it should now be possible to implement distutils module support such that it will work with any supported version control system.

This work is not yet finished though. A bit more work needs to be done to parse version control system agnostic module definitions from .modules files. When this is done, a fair bit of the current syntax can be deprecated and eventually removed. When this is done, adding support for a new version control system shouldn’t take more than 100-200 lines.

Module Type Simplifications

As well as reducing the number of module types that need to be maintained in JHBuild, I’ve been working on simplifying the code in these module types. Previously, each stage of a module build was represented by a method call on the module type. The return value of the method was used to say (a) whether the stage succeeded or not, (b) what the next state would be and (c) if an error occurred some alternative next states to go to (e.g. offer to rerun autogen.sh).

With the new system, the next state and error states are declared as attributes on the method object. The method can indicate a failure by raising a particular exception. This greatly simplifies the cases where a build stage involves a number of separate actions that could each fail individually, since the exception cuts processing short without the error checks getting in the way of the code.

There are still a few module build stages not converted to the new system since their next state depends on various config settings (e.g. if running “make check” has been enabled or not). Since these generally involve skipping a stage based on some criteria, the plan is to move the logic to the stage being skipped, which should simplify things further.