a lowercase manifesto

It’s 2013. The GNOME summit is coming back to Montréal!

The summit will be on the usual Canadian Thanksgiving long weekend (Columbus day long weekend for the US): October 12, 13, 14.

The summit is being hosted by Savoir-faire Linux. The venue will most likely be their offices in Montréal, but we may move to a local university if the projected list of attendees gets too large. Founded in 1999, Savoir-faire Linux has an outstanding team of 80 Free Software consultants based in Montreal, Quebec City and Ottawa, and is a leading provider of training, consulting, development and support services on open source technologies.

A wiki page has been put online here: https://wiki.gnome.org/Montreal2013. In particular this year, please make sure to add yourself to the confirmed or tentative lists so that we know if we will need a larger venue.

Savoir-faire is already offering a venue and will be sponsoring a party during one of the evenings, but we are still looking for additional sponsors. If you’d like to help out with catering, a dinner, or to offset the costs of travel and hotel bookings, please get in touch!

More information about travel sponsorship, hotels, and a definitive venue confirmation will be coming soon. Stay tuned.

Have radio items or checkboxes in your GMenu? Probably they’re effectively proxies for a key in GSettings somewhere.

Check out g_settings_create_action().

I write about this because someone suggested to me the other day that this may be a neat feature. I responded that I had thought about it before but never got around to implementing it. Clearly I had implemented it and just forgot. If I was able to forget that this exists, probably few other people know about it. Indeed, a quick ‘grep’ of my jhbuild checkoutdir shows very few users.

i recently bought an intel 520 series drive which has advertised support for hardware disk encryption based on the ata security feature set. thinkpads have long had support for issuing ata security feature set commands during boot in order to lock and unlock these drives. this means that you end up with transparent full disk encryption based on a password entered at the bios, which is pretty great. i’ve been doing this for a while with an intel 320 series drive.

turns out that there is some weird incompatibility between intel’s 520 series drives and lenovo’s bios, however. the bios is unable to set the drive password. lenovo blames intel, intel blames lenovo. typical mess. that’s not what this post is about.

the obvious solution to the issue of the bios being unable to set the drive password is to use hdparm to do this (from a livecd or so).

that idea isn’t working out so well.

it turns out that thinkpads use some proprietary hashing algorithm for passwords that you enter into the bios that makes them incompatible with just about everything else. after quite a lot of googling, i find that this incompatibility exists between thinkpads and hdparm, thinkpads and other laptops, between different models of thinkpads (t60 was a strange one, apparently), between thinkpads and themselves if you change certain bios options (something about “use passphrase”), and (as reported by one poor soul) a thinkpad and itself if you upgrade the firmware.

take note: if you are using an ata password with a thinkpad bios and you value your data then you should not assume the data on your drive will be accessible if your machine dies (unless you have an exact duplicate of the machine). backups.

one other thing that i notice is that the passwords entered into the bios of (at least my) thinkpad t420 are case insensitive. this suggests to me that at least *some* form of mangling is in use. i tried the obvious idea of using an all-lowered or all-capsed password to hdparm; it didn’t work.

my question for the lazyweb: does anyone know what algorithm/hash a t420 would be using to turn the password i type into its bios into an ata command? the ata security feature set doesn’t say much about the exact format of passwords other than that they are a 16 word (32 byte) field. for reference, hdparm appears to be using a direct memcpy() and padding with nul bytes up to 32 characters.

I’ve often lamented the fact that jhbuild is dumb about the way that it does download, build, download, build, download, build. It totally misses the chance to download ahead of time so that it never has to stop building.

I just discovered “jhbuild update”, which will download everything ahead of time.

Of course, you can start “jhbuild update” and let it grab the first few packages and then “jhbuild build” in another terminal. As long as they don’t step on each others’ feet, that should speed things up a lot…

TIL.

I just wrote an email to the gtk-devel-list with an update on the potentially-dangerous work that has gone into GLib since the Boston Summit (where we planned quite a lot of changes).

One item in particular needs to be mentioned.

A few moments ago I just landed a relatively small patch that refuses attempts to add an interface to a class that has already been initialised (which usually occurs at first instantiation), emitting a g_warning() instead.

This functionality was used in a dirty way by pygobject until earlier today when Martin landed a rewritten version of the code. libsoup also made theoretical use of this feature, although it’s unclear if the code in question was ever executed in the real world (and is certainly never used by any module in jhbuild).

The new warning looks something like this:

(process:30660): GLib-GObject-WARNING **: attempting to add an interface (TestIface1) to class (BaseObject) after class_init

The main upshot of this is that if you are building glib in jhbuild and you fail to also build the new pygobject then you may encounter problems. If you are making packages for a distribution then you should also take care to make sure that you manage the upgrades to glib and pygobject in lock-step.

The patch was written in a way that is intentionally minimal and easy to back out. At this point we’re playing wait-and-see to find out if other modules are broken by this change. If you spot any other instances of this warning in the wild (apart from those mentioned above), please let me know.

dconf changes landing. watch for falling rocks.

I’ve been putting a lot of work into dconf lately. I just reached a landmark today by landing that work on master. There have been over 100 commits touching 108 files with 8109 insertions and 3023 removals. That may seem like a rather large increase in code size, but nearly half of the additions come in the form of tests (which there were almost none of before) and the rest is inflated by a relatively liberal commenting of the newly-introduced code.

The story behind that is that the first pass at writing dconf aimed at efficiency and speed at the cost of… almost everything else. There has been a lot of renewed interested in adding new features to dconf lately:

• an API for writing to system-level databases
• an API for managing lockdown
• an API for managing lists of items (aka GSettingsList)
• removing some cruft from the way GSettings dealt with “delayed” writes
• proper NFS support
• some stuff I forget

The code was in too brittle of a state to get away with adding any of these features in a clean way (and not for lack of trying by various people on both the GSettingsList and NFS fronts).

One of the reasons that I’m blogging is because with such a massive refactoring of code (almost a rewrite, I’d say) there are bound to be some issues. The new code will be going out with GNOME 3.5.4 next week, so keep your eyes peeled and let me know if you notice anything.

some thoughts on (unit) testing

The other reason I’m blogging is to talk about the major focus of the rewrite: testing and testability.

dconf was the nightmare situation for testability. It’s a relatively complicated system that requires complex interactions between client programs and the service via the filesystem, shared memory and D-Bus. This is not your typical dbus-test-runner situation. To make matters worse, it’s a component that tries to read and write files in /etc and the user’s home directory. People tend to get upset when they end up having their personal data written to by a testcase…

I decided to focus on the client library for testing. The service is a relatively simple component and it operates in relative isolation, safely on the other side of D-Bus, receiving simple requests. The client library is used in a far more interesting set of situations.

The first step was to identify the interactions that dconf on the client side has with the world around it:

1. environment variables like DCONF_PROFILE and XDG_CONFIG_HOME
2. reading profile files from /etc/dconf/profile/ via fopen()
3. reading database files using gvdb
4. communicating with the service via the shared memory region (shm)
5. communicating with the service via D-Bus

The first item is easy to deal with: testcases can easily set environment variables. The second item is solved in a bit of a dirty way using the classic trick of ELF symbol interposition to replace the implementation of fopen().

The remaining three items are solved by turning each of them into a separate module, compiled as an in-tree static library that is used as an intermediate step to making the final library. In addition to the standard niceness you feel about “good software engineering” from using modules, each static library can now be used in separate unit tests — and dconf now has a unit test for each of gvdb, shm and its various D-Bus backends.

The most powerful part of doing this is the ability to compile the core dconf code (called “engine”) as a static library too, excluding these modules that it depends on. The dconf testsuite now features “mock” versions of each of the underlying components components that can be linked with the dconf engine library for unit-testing of the engine code. We don’t have to interact with the real filesystem or talk on the real D-Bus — it can all be mocked.

Of course, there is a lot to be said for proper integration testing. Again, this is a hard problem, but I have some creative ideas about how I could address those situations…

test coverage reporting is easy

One tool that I’ve found to be absolutely invaluable during this process is lcov. Coverage reports make testing seem a whole lot more purposeful (it’s fun to try to get to 100%!) and they point out areas that you may have missed testing. Sometimes you failed to test particular cases because they’re actually impossible (in which case they help you remove dead code).

Getting coverage reporting setup was a lot easier than I imagined it would be. There are basically only two requirements to doing that. First you need to build the project with the proper CFLAGS and LDFLAGS. These couple of lines in configure.ac will do (linked to instead of inlined due to WordPress’ inability to do most things properly):

http://git.gnome.org/browse/dconf/tree/configure.ac#n59

The next step is to have some way of actually generating the report. Here’s a fragment from my toplevel Makefile.am that I stole from glib’s gtester Makefile and heavily modified:

http://git.gnome.org/browse/dconf/tree/Makefile.am#n14

One more thing to mention: lcov often generates rather silly coverage results: marking lines like ‘g_assert_not_reached();’ as untested, or marking lines like ‘g_assert();’ as only having tested one of two possible branches. There is a small script that I wrote (now living in dconf’s git) that attempts to deal with some of those situations:

http://git.gnome.org/browse/dconf/tree/trim-lcov.py

Rick Spencer pulled me over today to help with an IRC client that he’s working on. He’s using python irclib to talk to IRC and Gtk/Webkit for the UI. The trouble with that combination is that the two are not using the same mainloop.

We sat down for a while and finally figured out that the ‘IRC’ object in irclib has three hooks for user-provided functions to help with this:

1. add fd watch
2. remove fd watch
3. add timeout

The fd watch functions are passed a python socket object to add or remove a watch for. The intent is that you will watch for the socket becoming readable. As far as I can tell, irclib always performs blocking writes on the assumption that it won’t be a problem.

GLib lacks functionality for easily hooking up watches for fds (although we have some proposals for that in bug #658020 which I will be looking at more closely soon). You can use GIOChannel but that’s always somewhat annoying and as far as I can tell cannot be used from Python with a unix fd (possibly due to a binding issue?). The remaining solution is to implement a GSource from python, which is tricky. Rick made me promise I’d blog about the code that I came up with to help with that. Here it is:

class SocketSource(GLib.Source):
    def __init__(self, callback):
        GLib.Source.__init__(self)
        self.callback = callback
        self.pollfds = []

    def prepare(self):
        return False

    def check(self):
        for pollfd in self.pollfds:
            if pollfd.revents:
                return True

        return False

    def dispatch(self, callback, args):
        self.callback()
        return True

    def add_socket(self, socket):
        pollfd = GLib.PollFD(socket.fileno(), GLib.IO_IN)
        self.pollfds.append(pollfd)
        self.add_poll(pollfd)

    def rm_socket(self, socket):
        fd = socket.fileno()
        for pollfd in self.pollfds:
            if pollfd.fd == fd:
                self.remove_poll(pollfd)
                self.pollfds.remove(pollfd)

The callback function provided to the constructor is called when one of the sockets becomes ready for reading. That maps nicely for irclib’s process_once function. The add_socket and rm_socket fit nicely with irclib’s fn_to_add_socket and fn_to_remove_socket. Using the code looks something like so:

simple = irclib.SimpleIRCClient()
source = Socketsource(simple.ircobj.process_once)
simple.ircobj.fn_to_add_socket = source.add_socket
simple.ircobj.fn_to_remove_socket = source.rm_socket
source.attach()

Timeouts are left as an exercise to the reader.

I went downtown today to grab a tea with Behdad and pick up some electronics for a fun weekend project: a Mini Arduino Pro and a Bluetooth Mate. It turns out that although these two components are designed to work together, they don’t work very *well* together. I’m writing this so that anyone else who tries to plug these two devices into each other can save some time.

[[ Some background for those unfamiliar with the way the Arduino is programmed: The Arduino contains a bootloader that runs for the first moment after the device is powered on or reset. It’s possible to communicate with this bootloader using a serial protocol during this time. After the timeout, the software that you loaded onto the board takes over. The tricky part is the timing of sending your code while the bootloader is running. To help with this, the reset pin on the Arduino is wired to a pullup and a capacitor that connects to the DTR input. This means that pulling the DTR active (low) is enough to briefly pulse the reset line (until the pullup recharges the capacitor). This causes the Arduino to reset when a serial connection is made which helps ensure that the computer sending the code over will find a bootloader on the other side. ]]

The Bluetooth Mate (made by Sparkfun) attempts to come close to the pinout of a standard FTDI cable (for compatibility with various Arduino boards that pair with this cable). Unfortunately, it falls short in a major way: where the DTR pin is supposed to be there is instead a RTS output. This wouldn’t be such a big deal (since both are often turned on at the same time anyway) except that the RTS pin is wired through to the local UART for the RN-42 rather than the signal from the host computer.

This means two things. The first is that if your Arduino is sending data to the serial port faster than it can be transmitted out over Bluetooth, the RN-42 will attempt hardware flow control by dropping the RTS line, causing the reset signal to pulse. Your board will reboot in the middle of whatever it was doing.

Possibly worse is that the host is prevented from signalling the start of the programming sequence which pretty much makes programming over Bluetooth impossible unless your timing at manually pressing the reset switch is very good.

Fortunately, the hardware flow control signals are available as output pins on the RN-42 itself and the problem is merely that they aren’t wired through to the pin header. The RN-42 datasheet lists pins 33 and 34 as “PIO10” and “PIO11”. It also incorrectly labels them as “Input to RN42 with weak pulldown”, so it’s best to ignore that part if you’re reading the datasheet.

The RN-42 AT Command Set reference linked from the Sparkfun product page mentions in Sections 6.1 and 6.2 that PIO10 and PIO11 can be used to output the host DTR and RTS if the device is in DUN-DCE mode. It also mentions that the DTR signal is active high, which is a problem for us, because the DTR pin needs to be pulled low in order to cause the Arduino to reset.

Fortunately the RTS pin is active low and is active at the same time as the DTR when programming so using RTS will work okay. It’s not a perfect solution because it’s possible that the host computer could attempt hardware flow control and accidentally reset the board instead but that’s unlikely as long as your software can keep up.

A more correct solution would probably involve using a transistor to flip the DTR signal. There’s a FET on the board (Q2 in the schematic) that’s used to raise the RTS signal from 3.3V to 5V. Since it’s quite likely that you don’t want to use that line anymore, cutting some traces, removing the input pullup resistor and soldering on some new connections (DTR to gate, ground to source) would be enough to repurpose it, and as a bonus it’s even already connected to the correct output pin. I didn’t do this because I didn’t want to make too many modifications to my board.

One last note: the Arduino I used is a 16MHz atmega328. After some trial and error I was able to determine that it likes to be programmed at 57600 instead of the more usual 115200 baud.

All that said, here’s the way to get a Mini Arduino Pro working with a Bluetooth Mate:

1. despite both boards having a pin header with 6 positions, solder only a 5-pin connector on the Bluetooth Mate (skipping the RTS pin)
2. solder a lead on to pin 33 of the RN-42 module (see picture) and plug that into the DTR input on the Arduino board instead
3. enable the RTS pinout by setting the RN-42 to DUN-DCE mode: enter command mode (‘$$$‘ within 60 seconds of power on) and send ‘S~,1‘
4. change the baud rate of the RN-42 to 57600: ‘SU57.6‘ in command mode

After all of that it should now be possible to program the board over Bluetooth and the board should stop spontaneously resetting itself when sending large amounts of data.

On the software side of things, the “official” Arduino IDE seems to have some pretty serious problems when dealing with Bluetooth devices. It seems to enjoy opening all the serial ports on your computer every time you click on the menubar(!) which causes the entire application to freeze as the kernel attempts to establish the Bluetooth rfcomm connection. Programming is also problematic since the IDE claims that the device is already in use (and when you check which process is using it, is the IDE itself). Using avrdude directly is a good substitute. In order to do that, pair with the device (pin is 1234) and then:

1. rfcomm bind 0 aa:bb:cc:dd:ee:ff
2. avrdude -v -patmega328p -carduino -P/dev/rfcomm0 -b57600 (programming commands here)

If you happen across this page and find that I’ve made a mistake somewhere, please comment below so that I can fix it.

A GObject is a GTypeInstance.

GTypeInstance has its own system of constructing and initialising instances. It’s very simple: you call g_type_create_instance(). GType allocates the necessary memory for the instance and then calls each _init() function, from the base type to the most derived type. There are no properties here.

But we want to create a GObject. That’s always done with g_object_new(). The first thing that g_object_new() does is to sort the properties that you pass to it into construct and non-construct properties. The list of construct properties is in the order that you passed them. Added to this list (at the end) are any construct properties that you did not pass to g_object_new(), with their default values.

g_object_new() then calls the constructor() virtual function for your class with the list of construct properties. This is usually the internal function g_object_construct, but you can override it (but you must chain up). This is the first place that you can possibly intercept construction of a GObject.

g_object_construct() calls g_type_create_instance() which results in all the _init() functions being called (as described above). It then sets the construct properties in the order that they were given to g_object_new() (with the default values at the end). This results in you getting a lot of _set_property() calls.

Then it returns. Your constructor() override gets another chance to do things here if you want.

Back in g_object_new(), the constructed() virtual function is called to let you know that we’re (almost) done. If you implement this then you need to chain up.

Finally, the non-construct properties that you passed to g_object_new() are set in the order passed. More _set_property() calls for you.

Then g_object_new() returns. A lot of people take this chance to do some extra things in their C API constructor function (but binding authors and those who want to subclass you may get angry if you do that).

The long and the short of it is that there are a bunch of different parts of your code that you can hook into to do various things during object construction (in order of when they are called):

• constructor() override (before chainup) of each class from most derived to base
• GType _init() function of each class from base to most derived
• _set_property() function for each construct property given to g_object_new(), in order
• _set_property() function for each construct property not given to g_object_new()
• constructor() override (after chainup) of each class from base to most derived
• constructed() override (before chainup) of each class from most derived to base
• constructed() override (after chainup) from each class from base to most derived
• _set_property() for the remaining properties passed to g_object_new()
• whatever code you do in your _new() function after g_object_new()

There are a few conclusion to draw from all of this that a lot of people first using GObject are often surprised about:

• you do not have access to values of properties passed to g_object_new() (even construct properties) from your GType _init() function. You should think of _init() as doing the work necessary to make it safe for _set_property() calls to begin happening to your object.
• _init() is not a great place to set values for properties on your parent class. If those properties are construct properties then they will later be reset to their default values by the later call to _set_property() (even if the property was not passed to g_object_new()). Consider using constructed() for this instead.
• none-the-less, as a potential parent class, you should expect that your subclasses may attempt to set non-construct properties from their GType _init() functions which means that you may end up seeing _set_property() for non-construct properties before you see it for construct properties (delivered by GObject in the usual way).

And a couple more notes and general advice:

• since _init() doesn’t have access to any properties, you should limit it to simple always-the-same initialisation tasks such as setting up your ->priv pointer or allocating internal arrays and hashtables or so on.
• overriding constructor() is difficult and ugly. Attempting to manipulate the property array that gets passed through is even more complicated. You almost certainly don’t want to do this unless you are doing something exceptionally evil.
• constructed() is easy to override and at this point all your construct properties have been set. This is a good place to do the ‘heavy lifting’ of initialising your object that is dependent on construct properties. This is where GSettings opens its schemas, for example.
• constructed() is somewhat less useful than it would have been if it was called after all properties (not just construct-only ones) were set, but we probably can’t change it at this point.

i returned from the gtk+ hackfest in brno a few days ago.

the biggest thing we dealt with is touch. you can probably read about that on various other blogs. we’re taking a somewhat minimalist approach this cycle with big plans for the future: event controllers. those are bits of code that treat sequences of events as higher level ideas. for example, a press-release mouse button sequence without substantial pointer movement is a “click”. this is somewhat different from our current model where it’s left to individual widgets to determine what constitutes this sort of interaction based on separate delivery of the individual events. naturally, this is almost the same idea as the philosophy of utouch: that most interesting touch interactions are gestures. this should make future integration of gtk and utouch rather natural.

i had a chance to meet with marek about dconf working better with nfs home directories. he’s working hard to make sure this will be working very nicely soon. i talked to emmanuele about gproperty and we came up with a solution for how we could improve the performance to the point where it’s improving all use cases and regressing none while providing what i believe to be an easier API for implementors.

the idea for removing support for grabs from gtk was met with general support. implicit grabs will be supported of course (like the sort of grab that happens when you click the mouse down until the point that you release it). the only sort of explicit grab behaviour that we care to support is the ‘popup menu’ case where you want a window to be dismissed when you click away from it. this will be offered as a simple boolean property on GtkWindow and the implementation will be done in the best way for each platform. there was an idea that we could even have a window manager hint to help us do this on X without the use of grabs (or other evil hacks like fullscreen input-only windows). there is presently nobody to tackle this task, but i may be able to find some time.

we’re going to deprecate gdk threads next cycle. (yay!)

we’re going to deprecate gtk_dialog_run() next cycle. (yay!)

there’s the increasing idea that we want to move away from using atk for accessibility purposes. it would be more efficient if we spoke to dbus directly, using our own abstraction that was kept in-tree and suited to our widgets. this abstraction could then also be made to support gtk accessibility on windows and mac in the future (instead of having an abstraction around an abstraction as would be the case if we tried to include atk as another option there). the plan for now is that it would continue to speak the same dbus protocol (which puts us pretty close to what qt is doing in this area).

i also spent some time hanging out with the documentation team who was having a hackfest next door. there seems to be a big change in focus there from working on user help to working on developer documentation (which we desperately need more of). gnome is pretty easy to use already, but i can’t tell you how many times i’ve heard that it’s difficult to develop for due to the lack of good documentation for beginners. hopefully good things will be coming soon.