a lowercase manifesto

Here’s a quick hack that I thought I’d share.

After accidentally pushing some extra commits that were on my local ‘master’ branch, Lars told me that he always tries to do a ‘git push –dry-run’ before pushing, followed by a ‘git log’ in order to check if the reported range is what he was actually intending to push.

After using –dry-run a couple of times and finding that I didn’t particularly care for the manual approach, I wrote a script called “git-wwpd”.

Here it is, in hopes that others will find it to be useful:

#!/usr/bin/env python2
# git-wwpd   (what would push do?)

import subprocess
import sys

dryrun = subprocess.Popen(['git', 'push', '--dry-run', '--porcelain'] + sys.argv[1:], stdout=subprocess.PIPE)

for line in dryrun.stdout:
    line = line.rstrip()
    parts = line.split('\t')

    if len(parts) == 3 and '..' in parts[2]:
        print('=== Would update {ref} with range {revs}'.format(ref=parts[1], revs=parts[2]))
        subprocess.check_call(['git', 'log', '--oneline', parts[2]])

    else:
        print(line)

It ends up looking like this:

desrt@humber:~/code/glib$ git wwpd origin master
To ssh://git.gnome.org/git/glib
=== Would update refs/heads/master:refs/heads/master with range 7417198..7d1d073
7d1d073 Doc: gio: enable gtkdoc-check
7242b7c Doc: glib: enable gtkdoc-check
2ef8ca5 Doc: gobject: enable gtkdoc-check
8446a00 configure.ac: Add ENABLE_GTK_DOC_CHECK conditional
30af941 Doc: gio: Fix all undocumented/unused/undeclared symbols
ed34c13 Doc: gobject: Fix all undocumented/unused/undeclared symbols
c6e0feb Doc: Fix GListModel/GListStore
b14f342 Win32: Move g_win32_check_windows_version() to the correct place in header
Done
desrt@humber:~/code/glib$ 

(by request of Lars): “This is a public service announcement.”

For some time, GCC has had support for __attribute__((cleanup)) which is a really nice way for automatically cleaning up variables when they go out of scope.

After a few attempts at people suggesting it for GLib, Alex finally convinced me to give it another thought. We originally resisted calls to implement it on the basis of portability, but having it in libgsystem (which has provided wrappers for a while) has shown it to be extremely useful and very popular. I came up with a pretty nice basic API concept and we iterated on it during the Developer Experience Hackfest that Collabora just hosted in Cambridge.

The changes just landed in GLib.

These macros only work with GCC and clang, which means that you should not use them on programs that you want to be buildable by MSVC (or other compilers).

The new API is best explained with an example:

{
  g_autoptr(GObject) object;
  g_autoptr(gchar) tmp;
  g_auto(GQueue) queue;

  g_queue_init (&queue);
  object = g_object_new (...);
  tmp = g_strdup_printf (...);

  // no free required
}

In order to support this for your types, you need to make use of the G_DEFINE_AUTOPTR_CLEANUP_FUNC and related macros. This will happen automatically if you make use of the new G_DECLARE_DERIVABLE_TYPE or G_DECLARE_FINAL_TYPE macros.

Please use responsibly.

I’ve struggled for some time with long-form tutorial style documentation for various bits of things in our platform. It feels out of place in the reference documentation (since it’s not reference documentation) and often it doesn’t fit neatly into one module or another.

In 2013 the GNOME foundation sponsored my attendance at OpenHelp and the documentation hackfest in Cincinnati. We talked about this problem for a while and I laid out a few simple criteria that I had at the time for making it less painful to write docs:

• must be a non-xml markdown style language
• must not involve using version control tools (git, etc.)
• must not involve getting patches reviewed
• needs to go online instantly (and not after the next tarball is released)

The best possible solution that we could think up at the time was to make use of the wiki to launch a new experiment called “HowDoI”. A HowDoI is a wiki page that describes how to make use of a specific GNOME technology or platform feature. The target audience is generally developers who know their way around but are not yet familiar with a particular new feature, or for those looking for the latest “best practice”.

There is a How Do I HowDoI? page that you can read for more information.

A couple of years on, this has been a moderate success. We have HowDoI pages on a reasonable range of important topics and they have been very popular with the people who have used them.

In general, it is my opinion that we should be aiming to write these pages for new technologies as they appear in GNOME. I just wrote one that makes use of the new type declaration macros, for example.

If you didn’t know about these, check them out — they contain some helpful hints. If you did know about these, and you are writing new GNOME technologies, please write one!

… 7 years later.

This is a public service announcement.

Please stop writing this:

#define G_DESKTOP_APP_INFO(o) (G_TYPE_CHECK_INSTANCE_CAST ((o), G_TYPE_DESKTOP_APP_INFO, GDesktopAppInfo))
#define G_DESKTOP_APP_INFO_CLASS(k) (G_TYPE_CHECK_CLASS_CAST((k), G_TYPE_DESKTOP_APP_INFO, GDesktopAppInfoClass))
#define G_IS_DESKTOP_APP_INFO(o) (G_TYPE_CHECK_INSTANCE_TYPE ((o), G_TYPE_DESKTOP_APP_INFO))
#define G_IS_DESKTOP_APP_INFO_CLASS(k) (G_TYPE_CHECK_CLASS_TYPE ((k), G_TYPE_DESKTOP_APP_INFO))
#define G_DESKTOP_APP_INFO_GET_CLASS(o) (G_TYPE_INSTANCE_GET_CLASS ((o), G_TYPE_DESKTOP_APP_INFO, GDesktopAppInfoClass))

typedef struct _GDesktopAppInfo GDesktopAppInfo;
typedef struct _GDesktopAppInfoClass GDesktopAppInfoClass;
 			
struct _GDesktopAppInfoClass
{
  GObjectClass parent_class;
};

GType g_desktop_app_info_get_type (void) G_GNUC_CONST;

and start writing this:

G_DECLARE_FINAL_TYPE(GDesktopAppInfo, g_desktop_app_info, G, DESKTOP_APP_INFO, GObject)

Thank you for your attention.

This is a public service announcement.

Writers of introspected libraries have long been familiar with (allow-none) as an annotation on the arguments of functions.

This annotation had a few shortcomings:

• it had a strange name with an implied direction
• it was not supported for return values
• for (out) parameters it does not mean “null may be returned” but rather “you may pass NULL to the C API if you wish to ignore this optional output”; there was no way to say that an (out) parameter may return null
• many people were confused about the last point

As such, (allow-none) is deprecated. We now have two new annotations: (nullable) and (optional).

(nullable) always means “the range of this value includes the possibility of the null value”. This applies for in parameters, out parameters, inout parameters and return values (with plans to expand it to properties and struct fields). For Vala, this translates more or less directly to the ? found in type names.

(optional) always means “it is possible to ignore this optional output by passing NULL“. It is only meaningful for (out) parameters.

As a result of the work that has been going into increasing the portability of GNOME this cycle, I’m happy to announce the availablility (on “day 0”) of a virtual machine image of GNOME 3.12.0 running on FreeBSD.

You can download it here if you want to try it out: FreeBSD GNOME 3.12 VM image (471 MB).

Loading it into gnome-boxes is unlikely to work. You’ll want to use virt-manager’s “Import existing disk image” option with the OS type given as “FreeBSD 10.x (or later)” — you’ll have to choose “Show all OS options” in order to see that (then it will appear under “UNIX”). You’ll want to give it at least 4096MB of RAM.

Rawhide has a known-bad version of seabios which will cause the image to kernel panic on startup. A fix is hopefully on the way soon. Other systems may be affected, but Fedora 20 and Ubuntu Trusty are both known-good. You can work around the issue by changing your VM’s video card to “Cirrus” but this will trigger some pretty awful video corruption, so I’d recommend against that.

There have been reports that some CPU models might not work quite right. If you’re having issues, “core2duo” is probably a good one to try.

Some notes:

• the passwords for the root and user account are both “beastie”
• ssh is installed and running by default: don’t expose it to a public network unless you change both passwords first
• a lot of “non-essential” files have been removed to keep the image small — headers, static libraries, translations, etc.

Some known issues with the image:

• all applications based on WebKit2 are broken (bug is here: help wanted)
• the keyring is not unlocked at login time
• probably many other things are broken — download it and give it a try, and file bugs

This VM image would not exist if not for the break-neck speed of packaging the 3.12.0 release into the FreeBSD “experimental” ports collection by FreeBSD hackers Gustau Perez and Koop Mast. Many thanks to both of them.

Thanks as well to everyone who was involved in making GNOME 3.12.

After nagging me for several days to write a post on the topic, Matthias stole my thunder and wrote his own. I agree with almost everything he wrote there (and indeed, I wrote both of the documents that he links to as our policies on portability), but I’d like to add a bit more.

Some interesting things have been going on lately. Debian has decided on systemd. Ubuntu surprised many and quickly followed. Most people in the GNOME community (and even many Canonical employees) are very happy about this — myself included. A big part of that is because the discussion is finally over.

I also feel a little bit worried.

systemd provides us with a great set of APIs to get done what we want to get done. In ‘earlier times’, we had to do these things for ourselves. Not so long ago, gnome-settings-daemon shipped some system-level services that were a poorly-written mess of #ifdef in order to do basic things like set the time and enable/disable NTP just so that we could provide a UI for this in gnome-control-center. We had to handle the cases for each operating system, for ourselves, often without having access to those systems for testing. These are components that run as root.

Those days are gone forever, and I am very happy for that.

At the same time, I share the concerns of many on the pro-Upstart side of the discussion leading up to the decision of the Debian technical committee. Since Upstart itself is taken out of the picture, I think it would be more fair to call these people “pro-diversity”, “pro-choice” or “pro-flexibility”. They’re right to be concerned.

During the gigantic mess that was the battle over ballot wording and tight vs. loose coupling, a very interesting debate emerged: the question of if “software outside of an init system’s implementation” would be allowed to depend on a particular init system. We’re talking about “GNOME” here, of course. To me, this seems a bit obvious: GNOME should not be allowed to depend on systemd.

For a long time, the release team has had a clear policy on the topic: No hard compile time dep on systemd for “basic functionality”. My worry is that now that Debian and Ubuntu (the last big hold outs on GNU/Linux) are on a path to systemd adoption, we might finally slip towards allowing hard dependencies on systemd. I believe this would be a mistake.

The Debian discussion went directly to the core of this issue: what is a “dependency”? In one post, Colin Watson outlines three possibilities:

1. direct dependency on init system, no attempt to make allowances
2. dependency on reasonably-understood/specified interface that happens to have only one implementation right now
3. dependency on interface with multiple implementations

I take “resonably-understood/specified” in this case to mean “it would be easy to write an independent implementation without resorting to hacks”.

…

We’ve always had a big problem with portability in the GLib and Gtk+ stack. These components, more than any others in GNOME, must be portable to a wider range of operating systems. People are using Gtk+ on Mac OS and Windows. People are also using it on IRIX, HP-UX, AIX, Solaris and the Hurd. We often get patches to add support for obscure compilers or quirks in operating systems that make us say “I didn’t even know that still existed”.

On the other hand, we’ve often removed “old hacks”, or added new features and done a release. Six months later we’d get a bug filed telling us that our (now-six-months-old) release doesn’t even build on someone’s platform. It’s clear that many of our users are only using our software very far after the fact. FreeBSD is only on GNOME 3.6, as an example.

This has resulted in a sort of paralysis for us. Particularly in GLib, we’re often faced with some gnarly hack in our code and the question of “do we still need this?”. We’ve tried various approaches over the course of many years to get on top of this issue with things like lists of required features, or supported platforms, but we’ve never gotten very far. Even when we decide to remove features, we’re never quite sure if it was the right thing to do, or if we will be hearing about it a few months later…

Our policy has always been more or less “we try to support everything — please send patches”, but it hasn’t been working. To that end, I thought that it might be helpful if we tried to support a specific set of systems, and if we could regularly test these systems.

A few months ago I wrote a mail on this topic to the gtk-devel-list. I also started reaching out to people in some non-Linux operating system communities. Having previously had positive collaboration with Antoine Jacoutot of OpenBSD, I tried setting up an OpenBSD VM. I also installed FreeBSD, having run it myself on servers, some 10 years ago. I soon discovered a team of excited and very friendly FreeBSD hackers who were interested in GNOME. The goal was originally to get a server setup where we could do regular jhbuilds of GLib and Gtk+ on FreeBSD for testing purposes. We set up a wiki page and got working on getting the required work upstream to allow an “out of the box” jhbuild to work on FreeBSD, without additional patches.

We got a bit carried away — a couple of months later, with well over 100 issues reported and fixed upstream, we’re in a state that all of jhbuild GNOME, fresh out of git master, is running on FreeBSD. There are still about half a dozen particularly thorny issues that need to be addressed, but we have patches for all of them, and they’re all filed upstream.

We have semi-regular tinderbox builds going on already — daily or better. You can read about it at the page that we’ve been using for collaboration and status tracking. A huge thanks goes to FreeBSD hacker Koop Mast and former GNOME Summer of Code student Ting-Wei Lan for their tireless efforts here.

These efforts have not gone unnoticed. Antoine is now in the process of getting a daily jhbuilder going on OpenBSD. I’ve also talked with people in the NetBSD and OpenIndiana communities who are interested in doing the same.

Last night, I wrote a page describing our new policy on supported platforms in GLib.

The (perhaps unfriendly) TL;DR, stated bluntly: if you want special support for your platform in GLib, please talk to us about setting up a daily builder.

A big part of this comes down to the most hated portability feature in all existence: the #ifdef. This is GLib — we’re in the portability business, so it can’t really be avoided. What we can do, though, is have a policy that no #ifdef section goes untested. Stated that way, it just seems like common sense.

This policy is simultaneous a friendlier and a more hardened stance compared to our previous approach. We’re getting serious about portability and making sure we do a good job of it, but the days of accepting random patches introducing #ifdef are over.

One thing is worth mentioning: the mere act of actively targeting and testing two completely independent systems already gets us 90% of the way for all of the other systems. FreeBSD alone lets us find out about all of the unintentional Linuxisms and GNUisms in our code and build system. Even if a system is not on our “officially supported” list, our general conformance to POSIX should be greatly improved by the simple existence of multiple diverse systems regularly building GLib.

…

Some components like GLib (mentioned above), Gtk (for display system backends), or even accountsservice (for platform-specific user management and categorisation tweaks) and NetworkManager (although it’s not there yet) can be seen at least partially as portability frameworks. Having platform-specific code in these modules is part of what makes these modules useful to their users.

Most components of GNOME are not in the portability business. In general, people hate to see #ifdef in their code and are (rightly) hostile towards the idea of patches that introduce it.

People just want one API that gives them what they need.

The #ifdef is not the only solution to portability. There is another solution that is very good: a reasonably-understood/specified interface (ideally) with multiple implementations.

We largely have two kinds of interfaces in common use between modules:

• public D-Bus interfaces: make a call to a specific well-defined interface
• pkg-config files and headers: add the pkg-config to your build and #include the header, using the interfaces specified therein

Everyone thinks about the D-Bus interfaces when talking about public APIs, but in fact, these two cases are not very much different from each other. If anything, the second case is more common: POSIX is exactly this (albeit with no pkg-config file). Include a header and use a function with a given name.

Some interfaces provided by systemd are awesome: they are perfect for getting the job done, they are well documented, and they are completely capable of being independently implemented. Some examples of these are the excellent interfaces for timedated, localed and hostnamed.

My opinion is that if presented with an interface like this, we should always use them, even if they don’t yet have multiple implementations. Because it’s D-Bus, the worst thing that can happen is that the call will fail (which is a case that the application should already be prepared to deal with gracefully). In my discussions with FreeBSD hackers, one thing is exceedingly clear: they do not mind providing implementations of the interfaces that we expect, assuming that those interfaces are well-documented and stable.

Some interfaces provided by systemd are less awesome. Even at the D-Bus level, the interface for PID 1 or logind are so complicated and implementation-specific that they could never be reasonably independently implemented. These interfaces often mix multiple functionality sets into one: for the logind case, for example, only a small subset of this is ever required by a desktop environment running as a normal user. Many other calls on the same interface are only called by other operating system components.

Another example is udev. It exposes largely-undocumented and only-vaguely-stable sysfs attributes through its interface. It also couples unrelated functionalities into a single API. I recently tried to do some “pkg-config and header” style independent implementation of the (relatively self-contained and sane) hwdb parts of udev, but was rejected upstream.

The udev issue may be going away soon anyway, at least for GNOME: Lennart has stated that he is no longer interested in maintaining the GLib bindings as part of systemd, and that it would make sense for similar functionality to live inside of GIO. I agree with this and it’s something that I hope to have time to work on in the future. This also gives us a good point at which to support multiple platforms while not interfering with the desire of application authors to have “just one API”.

Another example is the (pkg-config and header) logind library API. Although this library is perfectly scoped (“purely passive access and monitoring of seats, sessions and users”), I don’t consider this interface to be sufficiently “neutral” enough to allow reasonable independent implementation. At the same time, I don’t like seeing #ifdef in application code. I think we need one API that provides this functionality that we can use everywhere, always. If necessary, I think we should provide a stub version of this library for use by people who are not yet able to provide their own version. I prefer this to seeing conditional dependencies scattered around various bits of code.

Portability is not an issue that will go away. GNOME is not “Linux-only” and it never should be. People are running GNOME on FreeBSD, OpenBSD and very many other non-GNU/Linux systems. People are running GNOME on GNU/Linux systems that don’t have systemd. This will continue to be the case. We need to continue supporting these people.

At the same time, I think our current approach to portability has been a little too soft, and we suffer for it. We should be bold about depending on functionality that we need, and we should do so unconditionally. We should, however, only depend on interfaces — not specific system components, and we should only do so when the interfaces in question are well-documented and capable of being independently implemented.