In Florida

This week I am in Florida for a Launchpad sprint. I was meant to arrive on Sunday night, but I fell asleep in the boarding lounge and missed the San Francisco → Orlando flight (the flight out of Perth was an early morning one, and I didn’t get enough sleep on the plane). The earliest alternative fligh was the same time the next day, so I ended up ariving on Monday night.

I can’t say I was impressed with United’s customer service though. I was directed to the customer service centre in the airport and queued up behind about 10 other people. After a short while, the one staff member at the desk announced that her shift was over and that her replacements would not be arriving for another hour. It seems like really bad management to leave the desk unattended for an hour, particularly when they knew that there were people waiting.

They had a bunch of check-in computers which were supposed to let you change your flight details, so I gave one of these a try.  Unfortunately, the computers directed me to pick up the phone to talk to a representative, and the representative ended up directing me to talk to someone at the customer service centre.  After waiting for the next shift, things got sorted out okay though, which was good.

This was also my first experience with SSSS screening.  In fact I got to experience it twice: once when checking in for the flight I missed, and again for the later flight.  On my way back to Australia, I’ll have two more flights leaving from US airports so it’ll be interesting to see what happens then.

Canonical Shop Open

The new Canonical Shop was opened recently which allows you to buy anything from Ubuntu tshirts and DVDs up to a 24/7 support contract for your server.

One thing to note is that this is the first site using our new Launchpad single sign-on infrastructure. We will be rolling this out to other sites in time, which should give a better user experience to the existing shared authentication system currently in place for the wikis.

Bazaar Bundles

This article follows on from the series of tutorials on using Bazaar that I have neglected for a while. This article is about the bundle feature of Bazaar. Bundles are to Bazaar branches what patches are to tarballs or plain source trees.

Context/unified diffs and the patch utility are arguably one of most important inventions that enable distributed development:

  • The patch is a self contained text file, making it easy to send as an email attachment or attach to a bug report.
  • The size of the patch is proportional to the size of the changes rather than the size of the source tree. So submitting a one line fix to the Linux kernel is as easy as a one line fix for a small one person project.
  • Even if the destination source tree has moved forward since the patch was created, the patch utility does a decent job of applying the changes using heuristics to match the surrounding context. Human intervention is only needed if the edits are to the same section of code.
  • As patches are human readable text files, they are a convenient form to review the code changes.

Of course, patches do have their limitations:

  • The unified diff format doesn’t convey file moves, instead showing the entire file content being removed and then added again. If the file was changed in addition to being moved, the change can easily be missed when reviewing the patch.
  • Changes to binary files are omitted from the patch. While we can’t expect such changes to be represented in a human readable form, it’d be nice for them to be represented in a way that they can be applied at the other end.
  • The patch doesn’t record any intermediate steps in the creation of the change. This can be worked around by sending a sequence of patches that each build on the previous one, but this requires a fair bit of attentiveness on the part of the patch creator.
  • If the project in question is using some form of version control, the changes in the patch will likely be attributed to the person who applied the patch rather than the person who made the patch.

Using distributed version control solves these limitations, but simply publishing a branch and telling someone to pull from it does not provide all the benefits of a patch. For one, the person reviewing the changes needs to be online to merge the branch and evaluate the changes.

Second, the contributor of the change needs somewhere to host the branch. Even though finding a place to host the branch may not be difficult (for example, anyone can host their branches on Launchpad), uploading the branch may be more effort than the contributor cares for (uploading a branch the size of the Linux kernel will take a while, for instance). That branch would need to remain available until the changes were accepted.

For Bazaar, bundles provide a solution to this problem. A bundle is effectively a “branch diff”, which can then be used to integrate a set of revisions into a repository assuming it contains the revisions from the target branch. At this point, those changes can be merged or pulled.

So how do we produce a bundle? Lets start by creating a branch of the project we want to contribute to. For this example, we’ll create a branch of Mailman to make our changes. As Mailman is using Launchpad to host its branches, I can use the shorthand implemented by the Launchpad Bazaar plugin to create my branch:

bzr branch lp:mailman mailman.jamesh
cd mailman.jamesh
# make my changes here
bzr commit

After I am happy with my changes, I can create a bundle of those changes:

bzr bundle > my-changes.diff

As mentioned earlier, a bundle is essentially a diff between two branches. As I did not specify any branch in the above command, Bazaar uses the parent branch, which in this case will be the upstream Mailman branch. If we look at my-changes.diff, we will see a text file with three general sections:

  1. A short header identifying the file as a bundle and giving the last commit message, author and date
  2. A unified diff made between the last common revision with the parent and the head of our branch (this bit is also convenient to review).
  3. Some extra book keeping data. If I’d made multiple commits, this would include data needed to reconstruct the other revisions in the bundle.

I can now submit this bundle in the same way that I’d submit a patch: as an email attachment or in the bug tracker.

To merge the bundle, a developer simply needs to save the bundle to disk and use “bzr merge” on it:

bzr merge my-changes.diff
bzr commit

This will have the same effect as if they merged a branch with those changes. The “bzr log” output will show the merged revisions and “bzr annotate” will credit the changes to the person who made them rather than the person who merged it.

So next time you want to submit a patch to a project that uses Bazaar, consider submitting a bundle instead.

Storm Released

This week at the EuroPython conference, Gustavo Niemeyer announced the release of Storm and gave a tutorial on using it.

Storm is a new object relational mapper for Python that was developed for use in some Canonical projects, and we’ve been working on moving Launchpad over to it. I’ll discuss a few of the nice features of the package:

Loose Binding Between Database Connections and Classes

Storm has a much looser binding between database connections and the classes used to represent records in particular tables. The standard way of querying the database uses a store object:

for obj in store.find(SomeClass, conditions):
    # do something with obj (which will be a SomeClass instance)

Some things to note about this syntax:

  • The class used to represent rows in the table is passed to find(), so it is possible to have multiple classes representing a single table. This can be useful with large tables where you are only interested in a few columns in some cases.
  • The class used to represent the table is not bound to a particular connection. So instances of it can come from different stores.

Lockstep Iteration

As well as iterating over a single table, a Storm result set can iterate over multiple tables together. For instance, if we have a table representing people and a table representing email addresses (where each person can have multiple email addresses), it is possible to iterate over them in lockstep:

for person, email in store.find((Person, Email), == Email.person):
    print, email.address

Automatic Flushing Before Queries

One of the gotchas when using SQLObject was the way it locally cached updates to tables. This is a great way to reduce the number of updates sent to the database, but could result in unexpected results when performing subsequent SELECT queries. It was up to the programmer to remember to flush changes before doing a query.

With Storm, the store will flush pending changes automatically before performing the query.

Easy To Execute Raw SQL

An ORM can really help when developing a database driven application, but sometimes plain old SQL is a better fit. Storm makes it easy to execute raw SQL against a particular store with the store.execute() method. This method returns an object that you can iterate over to get the tuples from the result set. It also makes sure that any local changes have been flushed before executing the query.

Nice Clean Code

After working with SQLObject for a while, Storm has been a breath of fresh air. The internals are clean and nicely laid out, which makes hacking on it very easy. It was developed using test-driven development methodology, so there is an extensive test suite that makes it easy to validate changes.

gnome-vfs-obexftp 0.3

I’ve just released a new version of gnome-vfs-obexftp, which includes the features discussed previously. It can be downloaded from:

The highlights of the release include:

  • Sync osso-gwobex and osso-gnome-vfs-extras changes from Maemo Subversion.
  • Instead of asking hcid to set up the RFCOMM device for communication, use an RFCOMM socket directly. This is both faster and doesn’t require enabling experimental hcid interfaces. Based on work from Bastien Nocera.
  • Improve free space calculation for Nokia phones with multiple memory types (e.g. phone memory and a memory card). Now the free space for the correct memory type for a given directory should be returned. This fixes various free-space dependent operations in Nautilus such as copying files.

Any bug reports should be filed in Launchpad at:

Launchpad 1.0 Public Beta

Today we unveiled the Launchpad 1.0 User Interface, which has been in private beta for the last few months. As well as the improved visual appearance, there are a number of new features such as the ability to add your own branding to Launchpad pages associated with your project (for example, Ubuntu’s pages).

As mentioned in the press release, we’ve got two new high profile projects using us for bug tracking: The Zope 3 Project and The Silva Content Management System. As part of their migration, we imported all their old bug reports (for Zope 3, and for Silva). This was done using the same import process that we used for the SchoolTool import. Getting this process documented so that other projects can more easily switch to Launchpad is still on my todo list.

As Launchpad is a user of Zope 3 technology, it is great being able to provide the bug tracking infrastructure for the project. It should also make it easier to collaborate on bugs that affect both Launchpad and Zope.

SchoolTool Moves to Launchpad

Recently, the SchoolTool project has migrated to Launchpad for their bug tracker.

We performed an import of all their previous bug reports using a new bug importer I wrote. This was the third Launchpad bug importer I’d written (the previous ones being for the Ubuntu Bugzilla import and a SourceForge importer), so I wanted this one to be the last. So the design of this importer was to have a simple XML format as an intermediate step. That way we only need to target the XML format to support a new bug tracker. This will also make it possible for projects to provide their bug data in a form we can consume for the cases where they want to migrate their bugs to Launchpad but Canonical doesn’t have the resources to do the migration.

For the SchoolTool migration, Jean-François Roche put together a simple Roundup exporter.

We should have some documentation about the intermediate XML format available at some point for projects interested in moving to Launchpad.


Daylight saving started yesterday: the first time since 1991/1992 summer for Western Australia. The legislation finally passed the upper house on 21st November (12 days before the transition date). The updated tzdata packages were released on 27th November (6 days before the transition). So far, there hasn’t been an updated package released for Ubuntu (see bug 72125).

One thing brought up in the Launchpad bug was that not all applications used the system /usr/share/zoneinfo time zone database. So other places that might need updating include:

  • Evolution has a database in /usr/share/evolution-data-server-$version/zoneinfo/ that is in iCalendar VTIMEZONE format.
  • Java has a database in /usr/lib/jvm/java-$version/jre/lib/zi. This uses a different binary file format.
  • pytz (used by Zope 3 and Launchpad among others) has a database consisting of generated Python source files for its database.

All the above rules time zone databases are based on the same source time zone information, but need to be updated individually and in different ways.

In a way, this is similar to the zlib security problems from a few years back: the same problem duplicated in many packages and needing to be fixed over and over again. Perhaps the solution is the same too: get rid of the duplication so that in future only one package needs updating.

As a start, I put together a patch to pytz so that it uses the same format binary time zone files as found in /usr/share/zoneinfo (bug 71227). This still means it has its own time zone database, but it goes a long way towards being able to share the system time zone database. It’d be nice if the other applications and libraries with their own databases could make similar changes.

For people using Windows, there is an update from Microsoft. Apparently you need to install one update now, and then a second update next year — I guess Windows doesn’t support multiple transition rules like Linux does. The page also lists a number of applications that will malfunction and not know about the daylight saving shift, so I guess that they have similar issues of some applications ignoring the system time zone database.

bzr branch

One of the things we’ve been working on for Launchpad is good integration with Bazaar. Launchpad provides a way to register or host Bazaar branches, and nominate a Bazaar branch as representing a particular product series.

For each registered branch, there is a branch information page. This leads to a bit of confusion since Bazaar uses URLs to identify branches, so people try running bzr branch on a branch information page. We also get people trying to branch the product or product series pages.

There were two ways we could address this problem: (1) do more to encourage people not to do this, or (2) make bzr branch do what the user meant it to do. The second option is the more user friendly, so that’s what we chose. This just left the problem of how to implement it efficiently.

The obvious way to get this to work is with a simple HTTP redirect for files under the .bzr/ directory. Unfortunately this would result in Bazaar hitting the Launchpad webapp for every file in the branch which is not desirable (each request results in a number of database accesses, which would add unnecessary load to the system). There is a Bazaar bug about improving this so that it would use the new location after the first redirect. This bug was the main reason for not implementing the feature previously.

While playing around with things a bit, I realised that Bazaar already had the features needed to implement the redirects and they have been there since 0.8.

In Bazaar 0.8, the concept of a lightweight checkout was introduced. This is just a working tree plus a reference to a branch stored elsewhere. You can perform most operations on the branch from the checkout directory that you can do from the real branch directory. So what happens if you publish the “branch reference” part of a lightweight checkout on the web? It turns out that they work fine, so that’s what I used for

So with the change in place, the following commands will all give you a copy of the Bazaar webserve plugin:

bzr branch
bzr branch
bzr branch
bzr branch

Furthermore, they will all record the URL as the parent branch, so future bzr pull commands will go directly to the branch.

Microsummaries in Firefox 2

One of the new features in Firefox 2 is Microsummaries, which essentially allows dynamic bookmark titles. This is useful when bookmarking volatile pages, since the title can reflect the current state of the document rather than the state when the bookmark was created.

The system works by registering XSLT transformations that generate a simple text string from the page content. The registrations are either done via a <link> element, or matched via regular expressions. The system is designed to target users (who can register their own microsummary generators), website owners (who can suggest a generator through a <link> tag) and 3rd parties (who can provide generators for other sites to users).

For Launchpad, I’d fall into the second category. It would be nice to provide microsummary generators for bug pages, so you’d get an indication of the status of the bug, plus an up to date bug title. Now while all this information is available in the page content, we can provide it in a much more efficient manner (if we know the user is only interested in generating a microsummary, why send them the 100 comments on the bug every time the bookmark title is to be updated?).

Being able to specify a transformation for the bookmarked URL that would be used instead for generating the summary would be one way of solving this. This would reduce the bandwidth requirements and processing time on our end. Another way would be for Firefox to include something in its request that would allow a site to know that the page was being retrieved for microsummary generation so it could ommit information.

Overall it looks like a useful feature, but I do wonder if it will suffer from the “RSS effect” and cause lots of needless traffic to web sites until people work out how to achieve the same effects in a less resource intensive fashion.