Category Archives: General

(allow-none) is dead. long live (nullable)!

This is a public service announcement.

Writers of introspected libraries have long been familiar with (allow-none) as an annotation on the arguments of functions.

This annotation had a few shortcomings:

  • it had a strange name with an implied direction
  • it was not supported for return values
  • for (out) parameters it does not mean “null may be returned” but rather “you may pass NULL to the C API if you wish to ignore this optional output”; there was no way to say that an (out) parameter may return null
  • many people were confused about the last point

As such, (allow-none) is deprecated. We now have two new annotations: (nullable) and (optional).

(nullable) always means “the range of this value includes the possibility of the null value”. This applies for in parameters, out parameters, inout parameters and return values (with plans to expand it to properties and struct fields). For Vala, this translates more or less directly to the ? found in type names.

(optional) always means “it is possible to ignore this optional output by passing NULL“. It is only meaningful for (out) parameters.

GNOME 3.12 and FreeBSD (and a virtual machine)

As a result of the work that has been going into increasing the portability of GNOME this cycle, I’m happy to announce the availablility (on “day 0″) of a virtual machine image of GNOME 3.12.0 running on FreeBSD.

You can download it here if you want to try it out: FreeBSD GNOME 3.12 VM image (471 MB).

Loading it into gnome-boxes is unlikely to work. You’ll want to use virt-manager’s “Import existing disk image” option with the OS type given as “FreeBSD 10.x (or later)” — you’ll have to choose “Show all OS options” in order to see that (then it will appear under “UNIX”). You’ll want to give it at least 4096MB of RAM.

Rawhide has a known-bad version of seabios which will cause the image to kernel panic on startup. A fix is hopefully on the way soon. Other systems may be affected, but Fedora 20 and Ubuntu Trusty are both known-good. You can work around the issue by changing your VM’s video card to “Cirrus” but this will trigger some pretty awful video corruption, so I’d recommend against that.

There have been reports that some CPU models might not work quite right. If you’re having issues, “core2duo” is probably a good one to try.

Some notes:

  • the passwords for the root and user account are both “beastie”
  • ssh is installed and running by default: don’t expose it to a public network unless you change both passwords first
  • a lot of “non-essential” files have been removed to keep the image small — headers, static libraries, translations, etc.

Some known issues with the image:

  • all applications based on WebKit2 are broken (bug is here: help wanted)
  • the keyring is not unlocked at login time
  • probably many other things are broken — download it and give it a try, and file bugs

This VM image would not exist if not for the break-neck speed of packaging the 3.12.0 release into the FreeBSD “experimental” ports collection by FreeBSD hackers Gustau Perez and Koop Mast. Many thanks to both of them.

Thanks as well to everyone who was involved in making GNOME 3.12.

On portability

After nagging me for several days to write a post on the topic, Matthias stole my thunder and wrote his own. I agree with almost everything he wrote there (and indeed, I wrote both of the documents that he links to as our policies on portability), but I’d like to add a bit more.

Some interesting things have been going on lately. Debian has decided on systemd. Ubuntu surprised many and quickly followed. Most people in the GNOME community (and even many Canonical employees) are very happy about this — myself included. A big part of that is because the discussion is finally over.

I also feel a little bit worried.

systemd provides us with a great set of APIs to get done what we want to get done. In ‘earlier times’, we had to do these things for ourselves. Not so long ago, gnome-settings-daemon shipped some system-level services that were a poorly-written mess of #ifdef in order to do basic things like set the time and enable/disable NTP just so that we could provide a UI for this in gnome-control-center. We had to handle the cases for each operating system, for ourselves, often without having access to those systems for testing. These are components that run as root.

Those days are gone forever, and I am very happy for that.

At the same time, I share the concerns of many on the pro-Upstart side of the discussion leading up to the decision of the Debian technical committee. Since Upstart itself is taken out of the picture, I think it would be more fair to call these people “pro-diversity”, “pro-choice” or “pro-flexibility”. They’re right to be concerned.

During the gigantic mess that was the battle over ballot wording and tight vs. loose coupling, a very interesting debate emerged: the question of if “software outside of an init system’s implementation” would be allowed to depend on a particular init system. We’re talking about “GNOME” here, of course. To me, this seems a bit obvious: GNOME should not be allowed to depend on systemd.

For a long time, the release team has had a clear policy on the topic: No hard compile time dep on systemd for “basic functionality”. My worry is that now that Debian and Ubuntu (the last big hold outs on GNU/Linux) are on a path to systemd adoption, we might finally slip towards allowing hard dependencies on systemd. I believe this would be a mistake.

The Debian discussion went directly to the core of this issue: what is a “dependency”? In one post, Colin Watson outlines three possibilities:

  1. direct dependency on init system, no attempt to make allowances
  2. dependency on reasonably-understood/specified interface that happens to have only one implementation right now
  3. dependency on interface with multiple implementations

I take “resonably-understood/specified” in this case to mean “it would be easy to write an independent implementation without resorting to hacks”.

We’ve always had a big problem with portability in the GLib and Gtk+ stack. These components, more than any others in GNOME, must be portable to a wider range of operating systems. People are using Gtk+ on Mac OS and Windows. People are also using it on IRIX, HP-UX, AIX, Solaris and the Hurd. We often get patches to add support for obscure compilers or quirks in operating systems that make us say “I didn’t even know that still existed”.

On the other hand, we’ve often removed “old hacks”, or added new features and done a release. Six months later we’d get a bug filed telling us that our (now-six-months-old) release doesn’t even build on someone’s platform. It’s clear that many of our users are only using our software very far after the fact. FreeBSD is only on GNOME 3.6, as an example.

This has resulted in a sort of paralysis for us. Particularly in GLib, we’re often faced with some gnarly hack in our code and the question of “do we still need this?”. We’ve tried various approaches over the course of many years to get on top of this issue with things like lists of required features, or supported platforms, but we’ve never gotten very far. Even when we decide to remove features, we’re never quite sure if it was the right thing to do, or if we will be hearing about it a few months later…

Our policy has always been more or less “we try to support everything — please send patches”, but it hasn’t been working. To that end, I thought that it might be helpful if we tried to support a specific set of systems, and if we could regularly test these systems.

A few months ago I wrote a mail on this topic to the gtk-devel-list. I also started reaching out to people in some non-Linux operating system communities. Having previously had positive collaboration with Antoine Jacoutot of OpenBSD, I tried setting up an OpenBSD VM. I also installed FreeBSD, having run it myself on servers, some 10 years ago. I soon discovered a team of excited and very friendly FreeBSD hackers who were interested in GNOME. The goal was originally to get a server setup where we could do regular jhbuilds of GLib and Gtk+ on FreeBSD for testing purposes. We set up a wiki page and got working on getting the required work upstream to allow an “out of the box” jhbuild to work on FreeBSD, without additional patches.

We got a bit carried away — a couple of months later, with well over 100 issues reported and fixed upstream, we’re in a state that all of jhbuild GNOME, fresh out of git master, is running on FreeBSD. There are still about half a dozen particularly thorny issues that need to be addressed, but we have patches for all of them, and they’re all filed upstream.

We have semi-regular tinderbox builds going on already — daily or better. You can read about it at the page that we’ve been using for collaboration and status tracking. A huge thanks goes to FreeBSD hacker Koop Mast and former GNOME Summer of Code student Ting-Wei Lan for their tireless efforts here.

These efforts have not gone unnoticed. Antoine is now in the process of getting a daily jhbuilder going on OpenBSD. I’ve also talked with people in the NetBSD and OpenIndiana communities who are interested in doing the same.

Last night, I wrote a page describing our new policy on supported platforms in GLib.

The (perhaps unfriendly) TL;DR, stated bluntly: if you want special support for your platform in GLib, please talk to us about setting up a daily builder.

A big part of this comes down to the most hated portability feature in all existence: the #ifdef. This is GLib — we’re in the portability business, so it can’t really be avoided. What we can do, though, is have a policy that no #ifdef section goes untested. Stated that way, it just seems like common sense.

This policy is simultaneous a friendlier and a more hardened stance compared to our previous approach. We’re getting serious about portability and making sure we do a good job of it, but the days of accepting random patches introducing #ifdef are over.

One thing is worth mentioning: the mere act of actively targeting and testing two completely independent systems already gets us 90% of the way for all of the other systems. FreeBSD alone lets us find out about all of the unintentional Linuxisms and GNUisms in our code and build system. Even if a system is not on our “officially supported” list, our general conformance to POSIX should be greatly improved by the simple existence of multiple diverse systems regularly building GLib.

Some components like GLib (mentioned above), Gtk (for display system backends), or even accountsservice (for platform-specific user management and categorisation tweaks) and NetworkManager (although it’s not there yet) can be seen at least partially as portability frameworks. Having platform-specific code in these modules is part of what makes these modules useful to their users.

Most components of GNOME are not in the portability business. In general, people hate to see #ifdef in their code and are (rightly) hostile towards the idea of patches that introduce it.

People just want one API that gives them what they need.

The #ifdef is not the only solution to portability. There is another solution that is very good: a reasonably-understood/specified interface (ideally) with multiple implementations.

We largely have two kinds of interfaces in common use between modules:

  • public D-Bus interfaces: make a call to a specific well-defined interface
  • pkg-config files and headers: add the pkg-config to your build and #include the header, using the interfaces specified therein

Everyone thinks about the D-Bus interfaces when talking about public APIs, but in fact, these two cases are not very much different from each other. If anything, the second case is more common: POSIX is exactly this (albeit with no pkg-config file). Include a header and use a function with a given name.

Some interfaces provided by systemd are awesome: they are perfect for getting the job done, they are well documented, and they are completely capable of being independently implemented. Some examples of these are the excellent interfaces for timedated, localed and hostnamed.

My opinion is that if presented with an interface like this, we should always use them, even if they don’t yet have multiple implementations. Because it’s D-Bus, the worst thing that can happen is that the call will fail (which is a case that the application should already be prepared to deal with gracefully). In my discussions with FreeBSD hackers, one thing is exceedingly clear: they do not mind providing implementations of the interfaces that we expect, assuming that those interfaces are well-documented and stable.

Some interfaces provided by systemd are less awesome. Even at the D-Bus level, the interface for PID 1 or logind are so complicated and implementation-specific that they could never be reasonably independently implemented. These interfaces often mix multiple functionality sets into one: for the logind case, for example, only a small subset of this is ever required by a desktop environment running as a normal user. Many other calls on the same interface are only called by other operating system components.

Another example is udev. It exposes largely-undocumented and only-vaguely-stable sysfs attributes through its interface. It also couples unrelated functionalities into a single API. I recently tried to do some “pkg-config and header” style independent implementation of the (relatively self-contained and sane) hwdb parts of udev, but was rejected upstream.

The udev issue may be going away soon anyway, at least for GNOME: Lennart has stated that he is no longer interested in maintaining the GLib bindings as part of systemd, and that it would make sense for similar functionality to live inside of GIO. I agree with this and it’s something that I hope to have time to work on in the future. This also gives us a good point at which to support multiple platforms while not interfering with the desire of application authors to have “just one API”.

Another example is the (pkg-config and header) logind library API. Although this library is perfectly scoped (“purely passive access and monitoring of seats, sessions and users”), I don’t consider this interface to be sufficiently “neutral” enough to allow reasonable independent implementation. At the same time, I don’t like seeing #ifdef in application code. I think we need one API that provides this functionality that we can use everywhere, always. If necessary, I think we should provide a stub version of this library for use by people who are not yet able to provide their own version. I prefer this to seeing conditional dependencies scattered around various bits of code.

Portability is not an issue that will go away. GNOME is not “Linux-only” and it never should be. People are running GNOME on FreeBSD, OpenBSD and very many other non-GNU/Linux systems. People are running GNOME on GNU/Linux systems that don’t have systemd. This will continue to be the case. We need to continue supporting these people.

At the same time, I think our current approach to portability has been a little too soft, and we suffer for it. We should be bold about depending on functionality that we need, and we should do so unconditionally. We should, however, only depend on interfaces — not specific system components, and we should only do so when the interfaces in question are well-documented and capable of being independently implemented.

GNOME in Montréal

It’s 2013. The GNOME summit is coming back to Montréal!

The summit will be on the usual Canadian Thanksgiving long weekend (Columbus day long weekend for the US): October 12, 13, 14.

The summit is being hosted by Savoir-faire Linux. The venue will most likely be their offices in Montréal, but we may move to a local university if the projected list of attendees gets too large. Founded in 1999, Savoir-faire Linux has an outstanding team of 80 Free Software consultants based in Montreal, Quebec City and Ottawa, and is a leading provider of training, consulting, development and support services on open source technologies.

A wiki page has been put online here: https://wiki.gnome.org/Montreal2013. In particular this year, please make sure to add yourself to the confirmed or tentative lists so that we know if we will need a larger venue.

Savoir-faire is already offering a venue and will be sponsoring a party during one of the evenings, but we are still looking for additional sponsors. If you’d like to help out with catering, a dinner, or to offset the costs of travel and hotel bookings, please get in touch!

More information about travel sponsorship, hotels, and a definitive venue confirmation will be coming soon. Stay tuned.

PSA: g_settings_create_action()

Have radio items or checkboxes in your GMenu? Probably they’re effectively proxies for a key in GSettings somewhere.

Check out g_settings_create_action().

I write about this because someone suggested to me the other day that this may be a neat feature. I responded that I had thought about it before but never got around to implementing it. Clearly I had implemented it and just forgot. If I was able to forget that this exists, probably few other people know about it. Indeed, a quick ‘grep’ of my jhbuild checkoutdir shows very few users.

dear lazyweb: thinkpads and ata passwords

i recently bought an intel 520 series drive which has advertised support for hardware disk encryption based on the ata security feature set. thinkpads have long had support for issuing ata security feature set commands during boot in order to lock and unlock these drives. this means that you end up with transparent full disk encryption based on a password entered at the bios, which is pretty great. i’ve been doing this for a while with an intel 320 series drive.

turns out that there is some weird incompatibility between intel’s 520 series drives and lenovo’s bios, however. the bios is unable to set the drive password. lenovo blames intel, intel blames lenovo. typical mess. that’s not what this post is about.

the obvious solution to the issue of the bios being unable to set the drive password is to use hdparm to do this (from a livecd or so).

that idea isn’t working out so well.

it turns out that thinkpads use some proprietary hashing algorithm for passwords that you enter into the bios that makes them incompatible with just about everything else. after quite a lot of googling, i find that this incompatibility exists between thinkpads and hdparm, thinkpads and other laptops, between different models of thinkpads (t60 was a strange one, apparently), between thinkpads and themselves if you change certain bios options (something about “use passphrase”), and (as reported by one poor soul) a thinkpad and itself if you upgrade the firmware.

take note: if you are using an ata password with a thinkpad bios and you value your data then you should not assume the data on your drive will be accessible if your machine dies (unless you have an exact duplicate of the machine). backups.

one other thing that i notice is that the passwords entered into the bios of (at least my) thinkpad t420 are case insensitive. this suggests to me that at least *some* form of mangling is in use. i tried the obvious idea of using an all-lowered or all-capsed password to hdparm; it didn’t work.

my question for the lazyweb: does anyone know what algorithm/hash a t420 would be using to turn the password i type into its bios into an ata command? the ata security feature set doesn’t say much about the exact format of passwords other than that they are a 16 word (32 byte) field. for reference, hdparm appears to be using a direct memcpy() and padding with nul bytes up to 32 characters.

TIL: jhbuild update

I’ve often lamented the fact that jhbuild is dumb about the way that it does download, build, download, build, download, build. It totally misses the chance to download ahead of time so that it never has to stop building.

I just discovered “jhbuild update”, which will download everything ahead of time.

Of course, you can start “jhbuild update” and let it grab the first few packages and then “jhbuild build” in another terminal. As long as they don’t step on each others’ feet, that should speed things up a lot…

TIL.

a warning about glib

I just wrote an email to the gtk-devel-list with an update on the potentially-dangerous work that has gone into GLib since the Boston Summit (where we planned quite a lot of changes).

One item in particular needs to be mentioned.

A few moments ago I just landed a relatively small patch that refuses attempts to add an interface to a class that has already been initialised (which usually occurs at first instantiation), emitting a g_warning() instead.

This functionality was used in a dirty way by pygobject until earlier today when Martin landed a rewritten version of the code. libsoup also made theoretical use of this feature, although it’s unclear if the code in question was ever executed in the real world (and is certainly never used by any module in jhbuild).

The new warning looks something like this:

(process:30660): GLib-GObject-WARNING **: attempting to add an interface (TestIface1) to class (BaseObject) after class_init

The main upshot of this is that if you are building glib in jhbuild and you fail to also build the new pygobject then you may encounter problems. If you are making packages for a distribution then you should also take care to make sure that you manage the upgrades to glib and pygobject in lock-step.

The patch was written in a way that is intentionally minimal and easy to back out. At this point we’re playing wait-and-see to find out if other modules are broken by this change. If you spot any other instances of this warning in the wild (apart from those mentioned above), please let me know.

“dconf update”

dconf changes landing. watch for falling rocks.

I’ve been putting a lot of work into dconf lately. I just reached a landmark today by landing that work on master. There have been over 100 commits touching 108 files with 8109 insertions and 3023 removals. That may seem like a rather large increase in code size, but nearly half of the additions come in the form of tests (which there were almost none of before) and the rest is inflated by a relatively liberal commenting of the newly-introduced code.

The story behind that is that the first pass at writing dconf aimed at efficiency and speed at the cost of… almost everything else. There has been a lot of renewed interested in adding new features to dconf lately:

  • an API for writing to system-level databases
  • an API for managing lockdown
  • an API for managing lists of items (aka GSettingsList)
  • removing some cruft from the way GSettings dealt with “delayed” writes
  • proper NFS support
  • some stuff I forget

The code was in too brittle of a state to get away with adding any of these features in a clean way (and not for lack of trying by various people on both the GSettingsList and NFS fronts).

One of the reasons that I’m blogging is because with such a massive refactoring of code (almost a rewrite, I’d say) there are bound to be some issues. The new code will be going out with GNOME 3.5.4 next week, so keep your eyes peeled and let me know if you notice anything.

some thoughts on (unit) testing

The other reason I’m blogging is to talk about the major focus of the rewrite: testing and testability.

dconf was the nightmare situation for testability. It’s a relatively complicated system that requires complex interactions between client programs and the service via the filesystem, shared memory and D-Bus. This is not your typical dbus-test-runner situation. To make matters worse, it’s a component that tries to read and write files in /etc and the user’s home directory. People tend to get upset when they end up having their personal data written to by a testcase…

I decided to focus on the client library for testing. The service is a relatively simple component and it operates in relative isolation, safely on the other side of D-Bus, receiving simple requests. The client library is used in a far more interesting set of situations.

The first step was to identify the interactions that dconf on the client side has with the world around it:

  1. environment variables like DCONF_PROFILE and XDG_CONFIG_HOME
  2. reading profile files from /etc/dconf/profile/ via fopen()
  3. reading database files using gvdb
  4. communicating with the service via the shared memory region (shm)
  5. communicating with the service via D-Bus

The first item is easy to deal with: testcases can easily set environment variables. The second item is solved in a bit of a dirty way using the classic trick of ELF symbol interposition to replace the implementation of fopen().

The remaining three items are solved by turning each of them into a separate module, compiled as an in-tree static library that is used as an intermediate step to making the final library. In addition to the standard niceness you feel about “good software engineering” from using modules, each static library can now be used in separate unit tests — and dconf now has a unit test for each of gvdb, shm and its various D-Bus backends.

The most powerful part of doing this is the ability to compile the core dconf code (called “engine”) as a static library too, excluding these modules that it depends on. The dconf testsuite now features “mock” versions of each of the underlying components components that can be linked with the dconf engine library for unit-testing of the engine code. We don’t have to interact with the real filesystem or talk on the real D-Bus — it can all be mocked.

Of course, there is a lot to be said for proper integration testing. Again, this is a hard problem, but I have some creative ideas about how I could address those situations…

test coverage reporting is easy

One tool that I’ve found to be absolutely invaluable during this process is lcov. Coverage reports make testing seem a whole lot more purposeful (it’s fun to try to get to 100%!) and they point out areas that you may have missed testing. Sometimes you failed to test particular cases because they’re actually impossible (in which case they help you remove dead code).

Getting coverage reporting setup was a lot easier than I imagined it would be. There are basically only two requirements to doing that. First you need to build the project with the proper CFLAGS and LDFLAGS. These couple of lines in configure.ac will do (linked to instead of inlined due to WordPress’ inability to do most things properly):

http://git.gnome.org/browse/dconf/tree/configure.ac#n59

The next step is to have some way of actually generating the report. Here’s a fragment from my toplevel Makefile.am that I stole from glib’s gtester Makefile and heavily modified:

http://git.gnome.org/browse/dconf/tree/Makefile.am#n14

One more thing to mention: lcov often generates rather silly coverage results: marking lines like ‘g_assert_not_reached();’ as untested, or marking lines like ‘g_assert();’ as only having tested one of two possible branches. There is a small script that I wrote (now living in dconf’s git) that attempts to deal with some of those situations:

http://git.gnome.org/browse/dconf/tree/trim-lcov.py

glib mainloop sources in python (e.g. for irclib)

Rick Spencer pulled me over today to help with an IRC client that he’s working on. He’s using python irclib to talk to IRC and Gtk/Webkit for the UI. The trouble with that combination is that the two are not using the same mainloop.

We sat down for a while and finally figured out that the ‘IRC’ object in irclib has three hooks for user-provided functions to help with this:

  1. add fd watch
  2. remove fd watch
  3. add timeout

The fd watch functions are passed a python socket object to add or remove a watch for. The intent is that you will watch for the socket becoming readable. As far as I can tell, irclib always performs blocking writes on the assumption that it won’t be a problem.

GLib lacks functionality for easily hooking up watches for fds (although we have some proposals for that in bug #658020 which I will be looking at more closely soon). You can use GIOChannel but that’s always somewhat annoying and as far as I can tell cannot be used from Python with a unix fd (possibly due to a binding issue?). The remaining solution is to implement a GSource from python, which is tricky. Rick made me promise I’d blog about the code that I came up with to help with that. Here it is:

class SocketSource(GLib.Source):
    def __init__(self, callback):
        GLib.Source.__init__(self)
        self.callback = callback
        self.pollfds = []

    def prepare(self):
        return False

    def check(self):
        for pollfd in self.pollfds:
            if pollfd.revents:
                return True

        return False

    def dispatch(self, callback, args):
        self.callback()
        return True

    def add_socket(self, socket):
        pollfd = GLib.PollFD(socket.fileno(), GLib.IO_IN)
        self.pollfds.append(pollfd)
        self.add_poll(pollfd)

    def rm_socket(self, socket):
        fd = socket.fileno()
        for pollfd in self.pollfds:
            if pollfd.fd == fd:
                self.remove_poll(pollfd)
                self.pollfds.remove(pollfd)

The callback function provided to the constructor is called when one of the sockets becomes ready for reading. That maps nicely for irclib’s process_once function. The add_socket and rm_socket fit nicely with irclib’s fn_to_add_socket and fn_to_remove_socket. Using the code looks something like so:

simple = irclib.SimpleIRCClient()
source = Socketsource(simple.ircobj.process_once)
simple.ircobj.fn_to_add_socket = source.add_socket
simple.ircobj.fn_to_remove_socket = source.rm_socket
source.attach()

Timeouts are left as an exercise to the reader.