a warning about glib

I just wrote an email to the gtk-devel-list with an update on the potentially-dangerous work that has gone into GLib since the Boston Summit (where we planned quite a lot of changes).

One item in particular needs to be mentioned.

A few moments ago I just landed a relatively small patch that refuses attempts to add an interface to a class that has already been initialised (which usually occurs at first instantiation), emitting a g_warning() instead.

This functionality was used in a dirty way by pygobject until earlier today when Martin landed a rewritten version of the code. libsoup also made theoretical use of this feature, although it’s unclear if the code in question was ever executed in the real world (and is certainly never used by any module in jhbuild).

The new warning looks something like this:

(process:30660): GLib-GObject-WARNING **: attempting to add an interface (TestIface1) to class (BaseObject) after class_init

The main upshot of this is that if you are building glib in jhbuild and you fail to also build the new pygobject then you may encounter problems. If you are making packages for a distribution then you should also take care to make sure that you manage the upgrades to glib and pygobject in lock-step.

The patch was written in a way that is intentionally minimal and easy to back out. At this point we’re playing wait-and-see to find out if other modules are broken by this change. If you spot any other instances of this warning in the wild (apart from those mentioned above), please let me know.

“dconf update”

dconf changes landing. watch for falling rocks.

I’ve been putting a lot of work into dconf lately. I just reached a landmark today by landing that work on master. There have been over 100 commits touching 108 files with 8109 insertions and 3023 removals. That may seem like a rather large increase in code size, but nearly half of the additions come in the form of tests (which there were almost none of before) and the rest is inflated by a relatively liberal commenting of the newly-introduced code.

The story behind that is that the first pass at writing dconf aimed at efficiency and speed at the cost of… almost everything else. There has been a lot of renewed interested in adding new features to dconf lately:

  • an API for writing to system-level databases
  • an API for managing lockdown
  • an API for managing lists of items (aka GSettingsList)
  • removing some cruft from the way GSettings dealt with “delayed” writes
  • proper NFS support
  • some stuff I forget

The code was in too brittle of a state to get away with adding any of these features in a clean way (and not for lack of trying by various people on both the GSettingsList and NFS fronts).

One of the reasons that I’m blogging is because with such a massive refactoring of code (almost a rewrite, I’d say) there are bound to be some issues. The new code will be going out with GNOME 3.5.4 next week, so keep your eyes peeled and let me know if you notice anything.

some thoughts on (unit) testing

The other reason I’m blogging is to talk about the major focus of the rewrite: testing and testability.

dconf was the nightmare situation for testability. It’s a relatively complicated system that requires complex interactions between client programs and the service via the filesystem, shared memory and D-Bus. This is not your typical dbus-test-runner situation. To make matters worse, it’s a component that tries to read and write files in /etc and the user’s home directory. People tend to get upset when they end up having their personal data written to by a testcase…

I decided to focus on the client library for testing. The service is a relatively simple component and it operates in relative isolation, safely on the other side of D-Bus, receiving simple requests. The client library is used in a far more interesting set of situations.

The first step was to identify the interactions that dconf on the client side has with the world around it:

  1. environment variables like DCONF_PROFILE and XDG_CONFIG_HOME
  2. reading profile files from /etc/dconf/profile/ via fopen()
  3. reading database files using gvdb
  4. communicating with the service via the shared memory region (shm)
  5. communicating with the service via D-Bus

The first item is easy to deal with: testcases can easily set environment variables. The second item is solved in a bit of a dirty way using the classic trick of ELF symbol interposition to replace the implementation of fopen().

The remaining three items are solved by turning each of them into a separate module, compiled as an in-tree static library that is used as an intermediate step to making the final library. In addition to the standard niceness you feel about “good software engineering” from using modules, each static library can now be used in separate unit tests — and dconf now has a unit test for each of gvdb, shm and its various D-Bus backends.

The most powerful part of doing this is the ability to compile the core dconf code (called “engine”) as a static library too, excluding these modules that it depends on. The dconf testsuite now features “mock” versions of each of the underlying components components that can be linked with the dconf engine library for unit-testing of the engine code. We don’t have to interact with the real filesystem or talk on the real D-Bus — it can all be mocked.

Of course, there is a lot to be said for proper integration testing. Again, this is a hard problem, but I have some creative ideas about how I could address those situations…

test coverage reporting is easy

One tool that I’ve found to be absolutely invaluable during this process is lcov. Coverage reports make testing seem a whole lot more purposeful (it’s fun to try to get to 100%!) and they point out areas that you may have missed testing. Sometimes you failed to test particular cases because they’re actually impossible (in which case they help you remove dead code).

Getting coverage reporting setup was a lot easier than I imagined it would be. There are basically only two requirements to doing that. First you need to build the project with the proper CFLAGS and LDFLAGS. These couple of lines in configure.ac will do (linked to instead of inlined due to WordPress’ inability to do most things properly):

http://git.gnome.org/browse/dconf/tree/configure.ac#n59

The next step is to have some way of actually generating the report. Here’s a fragment from my toplevel Makefile.am that I stole from glib’s gtester Makefile and heavily modified:

http://git.gnome.org/browse/dconf/tree/Makefile.am#n14

One more thing to mention: lcov often generates rather silly coverage results: marking lines like ‘g_assert_not_reached();’ as untested, or marking lines like ‘g_assert();’ as only having tested one of two possible branches. There is a small script that I wrote (now living in dconf’s git) that attempts to deal with some of those situations:

http://git.gnome.org/browse/dconf/tree/trim-lcov.py

glib mainloop sources in python (e.g. for irclib)

Rick Spencer pulled me over today to help with an IRC client that he’s working on. He’s using python irclib to talk to IRC and Gtk/Webkit for the UI. The trouble with that combination is that the two are not using the same mainloop.

We sat down for a while and finally figured out that the ‘IRC’ object in irclib has three hooks for user-provided functions to help with this:

  1. add fd watch
  2. remove fd watch
  3. add timeout

The fd watch functions are passed a python socket object to add or remove a watch for. The intent is that you will watch for the socket becoming readable. As far as I can tell, irclib always performs blocking writes on the assumption that it won’t be a problem.

GLib lacks functionality for easily hooking up watches for fds (although we have some proposals for that in bug #658020 which I will be looking at more closely soon). You can use GIOChannel but that’s always somewhat annoying and as far as I can tell cannot be used from Python with a unix fd (possibly due to a binding issue?). The remaining solution is to implement a GSource from python, which is tricky. Rick made me promise I’d blog about the code that I came up with to help with that. Here it is:

class SocketSource(GLib.Source):
    def __init__(self, callback):
        GLib.Source.__init__(self)
        self.callback = callback
        self.pollfds = []

    def prepare(self):
        return False

    def check(self):
        for pollfd in self.pollfds:
            if pollfd.revents:
                return True

        return False

    def dispatch(self, callback, args):
        self.callback()
        return True

    def add_socket(self, socket):
        pollfd = GLib.PollFD(socket.fileno(), GLib.IO_IN)
        self.pollfds.append(pollfd)
        self.add_poll(pollfd)

    def rm_socket(self, socket):
        fd = socket.fileno()
        for pollfd in self.pollfds:
            if pollfd.fd == fd:
                self.remove_poll(pollfd)
                self.pollfds.remove(pollfd)

The callback function provided to the constructor is called when one of the sockets becomes ready for reading. That maps nicely for irclib’s process_once function. The add_socket and rm_socket fit nicely with irclib’s fn_to_add_socket and fn_to_remove_socket. Using the code looks something like so:

simple = irclib.SimpleIRCClient()
source = Socketsource(simple.ircobj.process_once)
simple.ircobj.fn_to_add_socket = source.add_socket
simple.ircobj.fn_to_remove_socket = source.rm_socket
source.attach()

Timeouts are left as an exercise to the reader.

arduino remote programming with the bluetooth mate (RN-42)

I went downtown today to grab a tea with Behdad and pick up some electronics for a fun weekend project: a Mini Arduino Pro and a Bluetooth Mate. It turns out that although these two components are designed to work together, they don’t work very *well* together. I’m writing this so that anyone else who tries to plug these two devices into each other can save some time.

[[ Some background for those unfamiliar with the way the Arduino is programmed: The Arduino contains a bootloader that runs for the first moment after the device is powered on or reset. It’s possible to communicate with this bootloader using a serial protocol during this time. After the timeout, the software that you loaded onto the board takes over. The tricky part is the timing of sending your code while the bootloader is running. To help with this, the reset pin on the Arduino is wired to a pullup and a capacitor that connects to the DTR input. This means that pulling the DTR active (low) is enough to briefly pulse the reset line (until the pullup recharges the capacitor). This causes the Arduino to reset when a serial connection is made which helps ensure that the computer sending the code over will find a bootloader on the other side. ]]

The Bluetooth Mate (made by Sparkfun) attempts to come close to the pinout of a standard FTDI cable (for compatibility with various Arduino boards that pair with this cable). Unfortunately, it falls short in a major way: where the DTR pin is supposed to be there is instead a RTS output. This wouldn’t be such a big deal (since both are often turned on at the same time anyway) except that the RTS pin is wired through to the local UART for the RN-42 rather than the signal from the host computer.

This means two things. The first is that if your Arduino is sending data to the serial port faster than it can be transmitted out over Bluetooth, the RN-42 will attempt hardware flow control by dropping the RTS line, causing the reset signal to pulse. Your board will reboot in the middle of whatever it was doing.

Possibly worse is that the host is prevented from signalling the start of the programming sequence which pretty much makes programming over Bluetooth impossible unless your timing at manually pressing the reset switch is very good.

Fortunately, the hardware flow control signals are available as output pins on the RN-42 itself and the problem is merely that they aren’t wired through to the pin header. The RN-42 datasheet lists pins 33 and 34 as “PIO10″ and “PIO11″. It also incorrectly labels them as “Input to RN42 with weak pulldown”, so it’s best to ignore that part if you’re reading the datasheet.

The RN-42 AT Command Set reference linked from the Sparkfun product page mentions in Sections 6.1 and 6.2 that PIO10 and PIO11 can be used to output the host DTR and RTS if the device is in DUN-DCE mode. It also mentions that the DTR signal is active high, which is a problem for us, because the DTR pin needs to be pulled low in order to cause the Arduino to reset.

Fortunately the RTS pin is active low and is active at the same time as the DTR when programming so using RTS will work okay. It’s not a perfect solution because it’s possible that the host computer could attempt hardware flow control and accidentally reset the board instead but that’s unlikely as long as your software can keep up.

A more correct solution would probably involve using a transistor to flip the DTR signal. There’s a FET on the board (Q2 in the schematic) that’s used to raise the RTS signal from 3.3V to 5V. Since it’s quite likely that you don’t want to use that line anymore, cutting some traces, removing the input pullup resistor and soldering on some new connections (DTR to gate, ground to source) would be enough to repurpose it, and as a bonus it’s even already connected to the correct output pin. I didn’t do this because I didn’t want to make too many modifications to my board.

One last note: the Arduino I used is a 16MHz atmega328. After some trial and error I was able to determine that it likes to be programmed at 57600 instead of the more usual 115200 baud.

All that said, here’s the way to get a Mini Arduino Pro working with a Bluetooth Mate:

  1. despite both boards having a pin header with 6 positions, solder only a 5-pin connector on the Bluetooth Mate (skipping the RTS pin)
  2. solder a lead on to pin 33 of the RN-42 module (see picture) and plug that into the DTR input on the Arduino board instead
  3. enable the RTS pinout by setting the RN-42 to DUN-DCE mode: enter command mode (‘$$$‘ within 60 seconds of power on) and send ‘S~,1
  4. change the baud rate of the RN-42 to 57600: ‘SU57.6‘ in command mode

After all of that it should now be possible to program the board over Bluetooth and the board should stop spontaneously resetting itself when sending large amounts of data.

On the software side of things, the “official” Arduino IDE seems to have some pretty serious problems when dealing with Bluetooth devices. It seems to enjoy opening all the serial ports on your computer every time you click on the menubar(!) which causes the entire application to freeze as the kernel attempts to establish the Bluetooth rfcomm connection. Programming is also problematic since the IDE claims that the device is already in use (and when you check which process is using it, is the IDE itself). Using avrdude directly is a good substitute. In order to do that, pair with the device (pin is 1234) and then:

  1. rfcomm bind 0 aa:bb:cc:dd:ee:ff
  2. avrdude -v -patmega328p -carduino -P/dev/rfcomm0 -b57600 (programming commands here)
    1. If you happen across this page and find that I’ve made a mistake somewhere, please comment below so that I can fix it.

a gentle introduction to gobject construction

A GObject is a GTypeInstance.

GTypeInstance has its own system of constructing and initialising instances. It’s very simple: you call g_type_create_instance(). GType allocates the necessary memory for the instance and then calls each _init() function, from the base type to the most derived type. There are no properties here.

But we want to create a GObject. That’s always done with g_object_new(). The first thing that g_object_new() does is to sort the properties that you pass to it into construct and non-construct properties. The list of construct properties is in the order that you passed them. Added to this list (at the end) are any construct properties that you did not pass to g_object_new(), with their default values.

g_object_new() then calls the constructor() virtual function for your class with the list of construct properties. This is usually the internal function g_object_construct, but you can override it (but you must chain up). This is the first place that you can possibly intercept construction of a GObject.

g_object_construct()
calls g_type_create_instance() which results in all the _init() functions being called (as described above). It then sets the construct properties in the order that they were given to g_object_new() (with the default values at the end). This results in you getting a lot of _set_property() calls.

Then it returns. Your constructor() override gets another chance to do things here if you want.

Back in g_object_new(), the constructed() virtual function is called to let you know that we’re (almost) done. If you implement this then you need to chain up.

Finally, the non-construct properties that you passed to g_object_new() are set in the order passed. More _set_property() calls for you.

Then g_object_new() returns. A lot of people take this chance to do some extra things in their C API constructor function (but binding authors and those who want to subclass you may get angry if you do that).

The long and the short of it is that there are a bunch of different parts of your code that you can hook into to do various things during object construction (in order of when they are called):

  • constructor() override (before chainup) of each class from most derived to base
  • GType _init() function of each class from base to most derived
  • _set_property() function for each construct property given to g_object_new(), in order
  • _set_property() function for each construct property not given to g_object_new()
  • constructor() override (after chainup) of each class from base to most derived
  • constructed() override (before chainup) of each class from most derived to base
  • constructed() override (after chainup) from each class from base to most derived
  • _set_property() for the remaining properties passed to g_object_new()
  • whatever code you do in your _new() function after g_object_new()

There are a few conclusion to draw from all of this that a lot of people first using GObject are often surprised about:

  • you do not have access to values of properties passed to g_object_new() (even construct properties) from your GType _init() function. You should think of _init() as doing the work necessary to make it safe for _set_property() calls to begin happening to your object.
  • _init() is not a great place to set values for properties on your parent class. If those properties are construct properties then they will later be reset to their default values by the later call to _set_property() (even if the property was not passed to g_object_new()). Consider using constructed() for this instead.
  • none-the-less, as a potential parent class, you should expect that your subclasses may attempt to set non-construct properties from their GType _init() functions which means that you may end up seeing _set_property() for non-construct properties before you see it for construct properties (delivered by GObject in the usual way).

And a couple more notes and general advice:

  • since _init() doesn’t have access to any properties, you should limit it to simple always-the-same initialisation tasks such as setting up your ->priv pointer or allocating internal arrays and hashtables or so on.
  • overriding constructor() is difficult and ugly. Attempting to manipulate the property array that gets passed through is even more complicated. You almost certainly don’t want to do this unless you are doing something exceptionally evil.
  • constructed() is easy to override and at this point all your construct properties have been set. This is a good place to do the ‘heavy lifting’ of initialising your object that is dependent on construct properties. This is where GSettings opens its schemas, for example.
  • constructed() is somewhat less useful than it would have been if it was called after all properties (not just construct-only ones) were set, but we probably can’t change it at this point.

Sprout and the Bean

i returned from the gtk+ hackfest in brno a few days ago.

the biggest thing we dealt with is touch. you can probably read about that on various other blogs. we’re taking a somewhat minimalist approach this cycle with big plans for the future: event controllers. those are bits of code that treat sequences of events as higher level ideas. for example, a press-release mouse button sequence without substantial pointer movement is a “click”. this is somewhat different from our current model where it’s left to individual widgets to determine what constitutes this sort of interaction based on separate delivery of the individual events. naturally, this is almost the same idea as the philosophy of utouch: that most interesting touch interactions are gestures. this should make future integration of gtk and utouch rather natural.

i had a chance to meet with marek about dconf working better with nfs home directories. he’s working hard to make sure this will be working very nicely soon. i talked to emmanuele about gproperty and we came up with a solution for how we could improve the performance to the point where it’s improving all use cases and regressing none while providing what i believe to be an easier API for implementors.

the idea for removing support for grabs from gtk was met with general support. implicit grabs will be supported of course (like the sort of grab that happens when you click the mouse down until the point that you release it). the only sort of explicit grab behaviour that we care to support is the ‘popup menu’ case where you want a window to be dismissed when you click away from it. this will be offered as a simple boolean property on GtkWindow and the implementation will be done in the best way for each platform. there was an idea that we could even have a window manager hint to help us do this on X without the use of grabs (or other evil hacks like fullscreen input-only windows). there is presently nobody to tackle this task, but i may be able to find some time.

we’re going to deprecate gdk threads next cycle. (yay!)

we’re going to deprecate gtk_dialog_run() next cycle. (yay!)

there’s the increasing idea that we want to move away from using atk for accessibility purposes. it would be more efficient if we spoke to dbus directly, using our own abstraction that was kept in-tree and suited to our widgets. this abstraction could then also be made to support gtk accessibility on windows and mac in the future (instead of having an abstraction around an abstraction as would be the case if we tried to include atk as another option there). the plan for now is that it would continue to speak the same dbus protocol (which puts us pretty close to what qt is doing in this area).

i also spent some time hanging out with the documentation team who was having a hackfest next door. there seems to be a big change in focus there from working on user help to working on developer documentation (which we desperately need more of). gnome is pretty easy to use already, but i can’t tell you how many times i’ve heard that it’s difficult to develop for due to the lack of good documentation for beginners. hopefully good things will be coming soon.

air canada +1, british air -1000

this is one of those “never do business with again” posts that i promised myself i’d write. so here we go:

avoid british air at all costs.

the story goes like this: i’m travelling to budapest from toronto with heathrow in the middle. air canada takes my two bags (a gym bag and a backpack) for the first leg. i checked both just because i didn’t want to have the extra bag on the plane and air canada allows for it. the bags were tagged straight through to budapest.

when i get to heathrow i have to pick up my boarding card. british air is my connecting flight and they inform me that their allowance is one bag and there will be an extra charge. i tell them that the bags are checked through and i expect them to be carried through. no? well, fine. just *give* me one of my bags and i’ll take it as a carryon, then (either one would have been small enough). i’m informed that the only way that i can access my bags in heathrow is if i cancel my ticket (no refund, of course), cancelling my return flight home as a side-effect.

i talked to various managers and made sure i got very clear answers. one guy in particular (‘chris’) seemed to be taking extreme pleasure in the fact that ‘he was in charge here’ and was ruining my day. everyone seemed to agree on the fact that i had exactly two choices in this situation:

  • pay 35£ for the privilege of boarding a flight that i’d already paid for
  • be stranded in heathrow with all my flights cancelled

at this point i was quite enraged. i wanted nothing to do with those fuckers anymore. i went to the air canada desk and asked if they could book me on a connecting flight with one of their partners so that i could avoid ever having to look at british airways ever again in my life. i was willing to pay for this.

air canada informed me that what british airways is doing is certainly immoral, probably illegal, and that they are the only airline that does it. i was certainly shocked as hell to learn that it was possible to be in an “insert more money to continue” situation in the middle of a connection. air canada also told me to not bother wasting my money on a rebooking — they’d cover the extra charge that british airways was extracting from me. i got an incident number from an agent at the air canada desk and was told to email my details to air canada’s refund services along with the number.

i paid ba and told them that this would be the last of my money they’d ever see. i thanked the girl at the desk (who had an honest “i’m sorry, but i just work here” look) and told her to tell her manager to go fuck himself.

i emailed air canada, they wrote back once asking for a scan of the receipt from ba and three weeks later i had a cheque in the mail (for the exact amount of the charge as it appeared on my visa bill, in canadian dollars).

so, for those keeping score:

air canada +1.

british airways -1000.

gearing up to montréal

the first montréal summit is just around the corner and things are shaping up very nicely. the weather for the weekend is forecast to be absolutely flawless.

we also scored a rather nice set of sponsors for the event:

collabora is sponsoring a party on Sunday evening.

codethink is providing the coffee and snacks for the summit.

finally, google has provided monetary support that is being used to cover travel costs.

one last nag: if you haven’t already, please add yourself to the list of participants: https://live.gnome.org/Montreal2011/Participants.

i hope everyone has a safe trip. see you on saturday!

travel sponsorship: two notes

i’d like to take a moment to remind everyone that the most effective way in which the GNOME foundation can spend its money is by bringing relevant people to the events that it organises. i’d also like to mention that we’ve extended the deadline for travel sponsorships for Montréal Summit until this friday at 23:59 EDT.

https://live.gnome.org/Travel

coming to montréal? tell us!

Montreal2011/Participants