In Hobart

Today was the first day of the mini-conferences that lead up to linux.conf.au later on this week.  I arrived yesterday after an eventful flight from Perth.

I was originally meant to fly out to Melbourne on the red eye leaving on Friday at 11:35pm, but just before I checked in they announced that the flight had been delayed until 4:00am the following day.  As I hadn’t had a chance to check in, I was able to get a pair of taxi vouchers to get home and back.  I only got about 2 hours of sleep though, as they said they would turn off the baggage processing system at 3am.  When I got back to the airport, I could see all the people who had stayed at the terminal spread out with airplane blankets.  A little before the 4:00am deadline, another announcement was made saying the plane would now be leaving at 5:00am.  Apparently they had needed to fly a replacement component in from over east to fix a problem found during maintenance.  Still, it seems it wasn’t the most delayed Qantas flight for that weekend and it did arrive in one piece.

As I had planned to spend a day in Melbourne visiting relatives, it didn’t cause any problems with the flight on to Hobart.  I had been invited to the “Ghosts” dinner, which was to start about an hour after my flight landed, so it was a bit of a rush to get to the university accommodation and then walk down the hill to the restaurant.

The dinner was pretty good, with organisers from all the previous LCA conferences plus the people organising the 2010 conference.  Unfortunately, I was the only one from the 2003 organisers able to attend.  It sounds like the 2010 organisers have things in hand, and the location should be great.

Getting “bzr send” to work with GMail

One of the nice features of Bazaar is the ability to send a bundle of changes to someone via email.  If you use a supported mail client, it will even open the composer with the changes attached.  If your client isn’t supported, then it’ll let you compose a message in your editor and then send it to an SMTP server.

GMail is not a supported mail client, but there are a few work arounds listed on the wiki.  Those really come down to using an alternative mail client (either the editor or Mutt) and sending the mails through the GMail SMTP server.  Neither solution really appealed to me.  There doesn’t seem to be a programatic way of opening up GMail’s compose window and adding an attachment (not too surprising for a web app).

What is possible though is connecting via IMAP and adding messages to the drafts folder (assuming IMAP support is enabled).  So I wrote a small plugin to do just that.  It can be installed with the following command:

bzr branch lp:~jamesh/+junk/bzr-imapclient ~/.bazaar/plugins/imapclient

And then configure the IMAP server, username and mailbox according to the instructions in the README file.  You can then use “bzr send” as normal and then complete and send the draft at your leisure.

One nice thing about the plugin implementation is that it didn’t need any GMail specific features: it should be useful for anyone who has their drafts folder stored on an IMAP server and uses an unsupported mail client.

The main area where this could be improved would be to open up the compose screen in the web browser.  However, this would require knowing the internal message ID for the new message, which I can’t see how to access via IMAP.

Using Twisted Deferred objects with gio

The gio library provides both synchronous and asynchronous interfaces for performing IO.  Unfortunately, the two APIs require quite different programming styles, making it difficult to convert code written to the simpler synchronous API to the asynchronous one.

For C programs this is unavoidable, but for Python we should be able to do better.  And if you’re doing asynchronous event driven code in Python, it makes sense to look at Twisted.  In particular, Twisted’s Deferred objects can be quite helpful.

Deferred

The Twisted documentation describes deferred objects as “a callback which will be put off until later”.  The deferred will eventually be passed the result of some operation, or information about how it failed.

From the consumer side, you can register one or more callbacks that will be run:

def callback(result):
    # do stuff
    return result

deferred.addCallback(callback)

The first callback will be called with the original result, while subsequent callbacks will be passed the return value of the previous callback (this is why the above example returns its argument). If the operation fails, one or more errbacks (error callbacks) will be called:

def errback(failure):
    # do stuff
    return failure

deferred.addErrback(errback)

If the operation associated with the deferred has already been completed (or already failed) when the callback/errback is added, then it will be called immediately. So there is no need to check if the operation is complete before hand.

Using Deferred objects with gio

We can easily use gio’s asynchronous API to implement a new API based on deferred objects.  For example:

import gio
from twisted.internet import defer

def file_read_deferred(file, io_priority=0, cancellable=None):
    d = defer.Deferred()
    def callback(file, async_result):
        try:
            in_stream = file.read_finish(async_result)
        except gio.Error:
            d.errback()
        else:
            d.callback(in_stream)
    file.read_async(callback, io_priority, cancellable)
    return d

def input_stream_read_deferred(in_stream, count, io_priority=0,
                               cancellable=None):
    d = defer.Deferred()
    def callback(in_stream, async_result):
        try:
            bytes = in_stream.read_finish(async_result)
        except gio.Error:
            d.errback()
        else:
            d.callback(bytes)
    # the argument order seems a bit weird here ...
    in_stream.read_async(count, callback, io_priority, cancellable)
    return d

This is a fairly simple transformation, so you might ask what this buys us. We’ve gone from an interface where you pass a callback to the method to one where you pass a callback to the result of the method. The answer is in the tools that Twisted provides for working with deferred objects.

The inlineCallbacks decorator

You’ve probably seen code examples that use Python’s generators to implement simple co-routines. Twisted’s inlineCallbacks decorator basically implements this for generators that yield deferred objects. It uses the enhanced generators feature from Python 2.5 (PEP 342) to pass the deferred result or failure back to the generator. Using it, we can write code like this:

@defer.inlineCallbacks
def print_contents(file, cancellable=None):
    in_stream = yield file_read_deferred(file, cancellable=cancellable)
    bytes = yield input_stream_read_deferred(
        in_stream, 4096, cancellable=cancellable)
    while bytes:
        # Do something with the data.  For this example, just print to stdout.
        sys.stdout.write(bytes)
        bytes = yield input_stream_read_deferred(
            in_stream, 4096, cancellable=cancellable)

Other than the use of the yield keyword, the above code looks quite similar to the equivalent synchronous implementation.  The only thing that would improve matters would be if these were real methods rather than helper functions.

Furthermore, the inlineCallbacks decorator causes the function to return a deferred that will fire when the function body finally completes or fails. This makes it possible to use the function from within other asynchronous code in a similar fashion. And once you’re using deferred results, you can mix in the gio calls with other Twisted asynchronous calls where it makes sense.