On portability

After nagging me for several days to write a post on the topic, Matthias stole my thunder and wrote his own. I agree with almost everything he wrote there (and indeed, I wrote both of the documents that he links to as our policies on portability), but I’d like to add a bit more.

Some interesting things have been going on lately. Debian has decided on systemd. Ubuntu surprised many and quickly followed. Most people in the GNOME community (and even many Canonical employees) are very happy about this — myself included. A big part of that is because the discussion is finally over.

I also feel a little bit worried.

systemd provides us with a great set of APIs to get done what we want to get done. In ‘earlier times’, we had to do these things for ourselves. Not so long ago, gnome-settings-daemon shipped some system-level services that were a poorly-written mess of #ifdef in order to do basic things like set the time and enable/disable NTP just so that we could provide a UI for this in gnome-control-center. We had to handle the cases for each operating system, for ourselves, often without having access to those systems for testing. These are components that run as root.

Those days are gone forever, and I am very happy for that.

At the same time, I share the concerns of many on the pro-Upstart side of the discussion leading up to the decision of the Debian technical committee. Since Upstart itself is taken out of the picture, I think it would be more fair to call these people “pro-diversity”, “pro-choice” or “pro-flexibility”. They’re right to be concerned.

During the gigantic mess that was the battle over ballot wording and tight vs. loose coupling, a very interesting debate emerged: the question of if “software outside of an init system’s implementation” would be allowed to depend on a particular init system. We’re talking about “GNOME” here, of course. To me, this seems a bit obvious: GNOME should not be allowed to depend on systemd.

For a long time, the release team has had a clear policy on the topic: No hard compile time dep on systemd for “basic functionality”. My worry is that now that Debian and Ubuntu (the last big hold outs on GNU/Linux) are on a path to systemd adoption, we might finally slip towards allowing hard dependencies on systemd. I believe this would be a mistake.

The Debian discussion went directly to the core of this issue: what is a “dependency”? In one post, Colin Watson outlines three possibilities:

  1. direct dependency on init system, no attempt to make allowances
  2. dependency on reasonably-understood/specified interface that happens to have only one implementation right now
  3. dependency on interface with multiple implementations

I take “resonably-understood/specified” in this case to mean “it would be easy to write an independent implementation without resorting to hacks”.

We’ve always had a big problem with portability in the GLib and Gtk+ stack. These components, more than any others in GNOME, must be portable to a wider range of operating systems. People are using Gtk+ on Mac OS and Windows. People are also using it on IRIX, HP-UX, AIX, Solaris and the Hurd. We often get patches to add support for obscure compilers or quirks in operating systems that make us say “I didn’t even know that still existed”.

On the other hand, we’ve often removed “old hacks”, or added new features and done a release. Six months later we’d get a bug filed telling us that our (now-six-months-old) release doesn’t even build on someone’s platform. It’s clear that many of our users are only using our software very far after the fact. FreeBSD is only on GNOME 3.6, as an example.

This has resulted in a sort of paralysis for us. Particularly in GLib, we’re often faced with some gnarly hack in our code and the question of “do we still need this?”. We’ve tried various approaches over the course of many years to get on top of this issue with things like lists of required features, or supported platforms, but we’ve never gotten very far. Even when we decide to remove features, we’re never quite sure if it was the right thing to do, or if we will be hearing about it a few months later…

Our policy has always been more or less “we try to support everything — please send patches”, but it hasn’t been working. To that end, I thought that it might be helpful if we tried to support a specific set of systems, and if we could regularly test these systems.

A few months ago I wrote a mail on this topic to the gtk-devel-list. I also started reaching out to people in some non-Linux operating system communities. Having previously had positive collaboration with Antoine Jacoutot of OpenBSD, I tried setting up an OpenBSD VM. I also installed FreeBSD, having run it myself on servers, some 10 years ago. I soon discovered a team of excited and very friendly FreeBSD hackers who were interested in GNOME. The goal was originally to get a server setup where we could do regular jhbuilds of GLib and Gtk+ on FreeBSD for testing purposes. We set up a wiki page and got working on getting the required work upstream to allow an “out of the box” jhbuild to work on FreeBSD, without additional patches.

We got a bit carried away — a couple of months later, with well over 100 issues reported and fixed upstream, we’re in a state that all of jhbuild GNOME, fresh out of git master, is running on FreeBSD. There are still about half a dozen particularly thorny issues that need to be addressed, but we have patches for all of them, and they’re all filed upstream.

We have semi-regular tinderbox builds going on already — daily or better. You can read about it at the page that we’ve been using for collaboration and status tracking. A huge thanks goes to FreeBSD hacker Koop Mast and former GNOME Summer of Code student Ting-Wei Lan for their tireless efforts here.

These efforts have not gone unnoticed. Antoine is now in the process of getting a daily jhbuilder going on OpenBSD. I’ve also talked with people in the NetBSD and OpenIndiana communities who are interested in doing the same.

Last night, I wrote a page describing our new policy on supported platforms in GLib.

The (perhaps unfriendly) TL;DR, stated bluntly: if you want special support for your platform in GLib, please talk to us about setting up a daily builder.

A big part of this comes down to the most hated portability feature in all existence: the #ifdef. This is GLib — we’re in the portability business, so it can’t really be avoided. What we can do, though, is have a policy that no #ifdef section goes untested. Stated that way, it just seems like common sense.

This policy is simultaneous a friendlier and a more hardened stance compared to our previous approach. We’re getting serious about portability and making sure we do a good job of it, but the days of accepting random patches introducing #ifdef are over.

One thing is worth mentioning: the mere act of actively targeting and testing two completely independent systems already gets us 90% of the way for all of the other systems. FreeBSD alone lets us find out about all of the unintentional Linuxisms and GNUisms in our code and build system. Even if a system is not on our “officially supported” list, our general conformance to POSIX should be greatly improved by the simple existence of multiple diverse systems regularly building GLib.

Some components like GLib (mentioned above), Gtk (for display system backends), or even accountsservice (for platform-specific user management and categorisation tweaks) and NetworkManager (although it’s not there yet) can be seen at least partially as portability frameworks. Having platform-specific code in these modules is part of what makes these modules useful to their users.

Most components of GNOME are not in the portability business. In general, people hate to see #ifdef in their code and are (rightly) hostile towards the idea of patches that introduce it.

People just want one API that gives them what they need.

The #ifdef is not the only solution to portability. There is another solution that is very good: a reasonably-understood/specified interface (ideally) with multiple implementations.

We largely have two kinds of interfaces in common use between modules:

  • public D-Bus interfaces: make a call to a specific well-defined interface
  • pkg-config files and headers: add the pkg-config to your build and #include the header, using the interfaces specified therein

Everyone thinks about the D-Bus interfaces when talking about public APIs, but in fact, these two cases are not very much different from each other. If anything, the second case is more common: POSIX is exactly this (albeit with no pkg-config file). Include a header and use a function with a given name.

Some interfaces provided by systemd are awesome: they are perfect for getting the job done, they are well documented, and they are completely capable of being independently implemented. Some examples of these are the excellent interfaces for timedated, localed and hostnamed.

My opinion is that if presented with an interface like this, we should always use them, even if they don’t yet have multiple implementations. Because it’s D-Bus, the worst thing that can happen is that the call will fail (which is a case that the application should already be prepared to deal with gracefully). In my discussions with FreeBSD hackers, one thing is exceedingly clear: they do not mind providing implementations of the interfaces that we expect, assuming that those interfaces are well-documented and stable.

Some interfaces provided by systemd are less awesome. Even at the D-Bus level, the interface for PID 1 or logind are so complicated and implementation-specific that they could never be reasonably independently implemented. These interfaces often mix multiple functionality sets into one: for the logind case, for example, only a small subset of this is ever required by a desktop environment running as a normal user. Many other calls on the same interface are only called by other operating system components.

Another example is udev. It exposes largely-undocumented and only-vaguely-stable sysfs attributes through its interface. It also couples unrelated functionalities into a single API. I recently tried to do some “pkg-config and header” style independent implementation of the (relatively self-contained and sane) hwdb parts of udev, but was rejected upstream.

The udev issue may be going away soon anyway, at least for GNOME: Lennart has stated that he is no longer interested in maintaining the GLib bindings as part of systemd, and that it would make sense for similar functionality to live inside of GIO. I agree with this and it’s something that I hope to have time to work on in the future. This also gives us a good point at which to support multiple platforms while not interfering with the desire of application authors to have “just one API”.

Another example is the (pkg-config and header) logind library API. Although this library is perfectly scoped (“purely passive access and monitoring of seats, sessions and users”), I don’t consider this interface to be sufficiently “neutral” enough to allow reasonable independent implementation. At the same time, I don’t like seeing #ifdef in application code. I think we need one API that provides this functionality that we can use everywhere, always. If necessary, I think we should provide a stub version of this library for use by people who are not yet able to provide their own version. I prefer this to seeing conditional dependencies scattered around various bits of code.

Portability is not an issue that will go away. GNOME is not “Linux-only” and it never should be. People are running GNOME on FreeBSD, OpenBSD and very many other non-GNU/Linux systems. People are running GNOME on GNU/Linux systems that don’t have systemd. This will continue to be the case. We need to continue supporting these people.

At the same time, I think our current approach to portability has been a little too soft, and we suffer for it. We should be bold about depending on functionality that we need, and we should do so unconditionally. We should, however, only depend on interfaces — not specific system components, and we should only do so when the interfaces in question are well-documented and capable of being independently implemented.

22 thoughts on “On portability”

  1. Matthias Clasen wrote a great article about portability and GNOME:
    http://blogs.gnome.org/mclasen/2014/02/19/on-portability/

    It did a great job of helping the community unwind, since everyone has been tense about systemd, and in particular since the Debian mailing list threads. I applaud Mr. Shuttleworth in choosing systemd for Ubuntu: this will do a great job of unifying the different distro communities so that we can work together to solve interesting problems instead of having to make sure every program works with every weird init system out there.

    Sadly, Ryan Lortie wrote a follow up article:
    http://blogs.gnome.org/desrt/2014/02/19/on-portability/

    Here are my thoughts on this article:

    * Mr. Lortie goes out of his was to incite the issue, which will most likely cause upset, more conflicts and wasted time.

    * It’s clear that what the GNU/GNOME community needs is to focus on development, and providing a stable, functional working environment, backed by the awesome features that systemd can provide. GNOME still doesn’t have proper session management. Let systemd be one of the tools to help in this respect.

    * Since GNOME already states that they’ve been having to remove features all over the place due to the maintenance burden, why don’t they do the same and forget about *BSD until enough people want to step up in those communities and maintain things, and potentially also write a systemd compatible api for their init system. *BSD’s are essentially deprecated for desktop environments anyways. Power management anyone?

    * The FreeBSD community is a very small user base of Free Software, and in particular of GNOME. If you look at the debian statistics for example, Debian/kFreeBSD is a small, small minority. The Debian devs point this out.

    * The FreeBSD community, thinks that GPL’ed software is a problem: https://wiki.freebsd.org/GPLinBase as a result, I say, leave them behind. Make GNOME awesome, and if they want to support the port, let them do it themselves.

    * A lot of users and developers in the GNU/Linux/GNOME communities find Mr. Lortie very difficult to deal with. Just ask anyone who’s been to GUADEC, FOSDEM, or any of the other hackfests that he attends. Unfortunately he doesn’t seem to have management which knows, or which can corral him.

    * If Mr. Lortie is good at hacking, and writing patch, then please Mr. Lortie, do that, but don’t write non-technical blog posts in an attempt to point out divisive community issues, where what we need is to work together and get at least _ONE_ working operating system. Let the communities focus on that. Clearly that is GNU/Linux.

    Sorry if I come off annoyed, someone was wrong ( https://xkcd.com/386/ ) on the internet.

    If you don’t agree, that’s okay, please tell me why. I decided to post this here, even though the chance of having a neutral discussion on Mr. Lortie’s blog comments isn’t likely.

    Cheers,
    - Anonymous GNOME hacker

    PS: I’ve kept this anonymous, because I don’t want to cause personal tensions between myself and Mr. Lortie. I hope you understand.

  2. So I guess what you propose is ensuring that the APIs we rely on are re-implementable right? E.g. we could rely on logind dbus API, but then just the bits which someone else could reimplement? This is what I fully agree with. I partly like systemd everywhere, partly dislike it.

    I’m kind of saddened to read we should’ve just reached out to *BSD. We could’ve done that ages ago. Thanks for doing that.

    I don’t quite understand the seeming contradiction in your blogpost. E.g. “We should be bold about depending on functionality that we need, and we should do so unconditionally.”: You’ll make Debian and Gentoo packagers very angry :-P

  3. When I say that we should depend on things unconditionally, I’m appealing in part to the statement I made a couple of paragraphs up: “If necessary, I think we should provide a stub version of this library for use by people who are not yet able to provide their own version.”

    I think that this solution of having stubbed-out versions of the functionality in one place is far preferable to having individual modules having configure checks and #ifdef scattered around their code.

  4. Regarding the Debian debate, I don’t think it’s accurate to portray that pro-Upstart folks as the pro-flexibility side, as that suggests that others were anti-flexibility. From what I saw, *everyone* on the committee was in favour of flexibility – the fighting over tight or loose coupling came down to people’s views on how strong a statement the committee should be making on the subject – and over just how much work they expected others to do to achieve the flexibility.

    My views on the subject basically come down to the fact that the people doing the work get to make the decisions. If almost all Gnome development is being done by people running Linux, Linux is what will be supported – not because the developers are against portability, but because only a tiny number of users are either interested or capable of actually implementing that portability. We’ve been through all this before with HAL and all the other projects that spun off it over the years – upstream was happy to accept patches to make it work on FreeBSD, but with only a single BSD developer working part-time on it, it was never all that well supported.

    Basically, if you care about portability, you need to be prepared to do the work – and not just a one-off patch, but committing to being a long-term maintainer for that work. And very few non-Linux communities have the resources to make that commitment…

  5. I have no idea what anonymouse is talking about. I thought Ryan’s post presented actual issues with actual proposals for how to deal with them. Nothing in here sounded inflammatory or divisive. How is “I reached out to BSD hackers and we got stuff done together” even remotely approach being divisive?

    I’ve been to GUADEC and to numerous hackfests with Ryan, and I don’t know anybody who finds him difficult to deal with. Anybody. This is literally the first time I’ve ever heard anybody say that.

    As for anonymouse’s last point, Ryan is in fact good at hacking. Phenomenal. One of the best we have. And he should blog more.

  6. People in the FreeBSD community are interesting in using Gnome 3 and are interested in working on it. A few years ago before I started I started a forum post on how to install Gnome 3 in PCBSD and it has since received over 25,000 views.

    http://forums.pcbsd.org/forumdisplay.php?f=28

    As it has been if not the highest one of the highest viewed topics on the PCBSD forums for some time I really do think there is some interest. As a result of improved packaging tools I started a 3rd party packagesite and eventually assisted in committing Gnome3 directly in to PCBSD. So now PCBSD 10 now includes Gnome 3 as an option.

    I appreciate the efforts that Ryan Lortie, Koop Mast, and others have put in to getting portability issues with Gnome 3 fixed upstream. I was looking forward to assisting with Gnome 3.11 development on FreeBSD myself so that Gnome 3.12 can be installable for PCBSD users as well. I do hope that Gnome 3 will continue to be portable into the future and that portability support will continue to improve.

  7. Regarding logind (and possibly others), it may not be feasible or even a good idea to reimplement the whole thing, but would making the (for example) GNOME-facing api cross-platform and the rest implementation-specific be possible?

  8. Thanks for writing this informative post, that complements Matthias Clasen’ one about the topic, and for helping to improve the support of GNOME3 for the BSD family. So far, I only hear about the excelent work of Antoine Jacoutot in OpenBSD.

    I think promoting testing in those platforms which support GNOME is a step forward to improve the support and can help developers of other platforms to join forces and focus on the gaps where the GNU/Linux support is, obviously, ahead.

    I’ve been to GUADEC several times and Ryan is one of the members of the GNOME community who is easiest to reach out and ask for help in any situation. So I think he’s a valuable member of our community, and I know I am not the only one who thinks so :-)

  9. You acknowledge that logind is probably not “neutral” enough for independent implementations outside systemd. Do you have any plans, or know of any within the community, which would rectify this situation inside GNOME (and preferably also at fdo level for upcoming KDE and other DE’s plans) in terms of implementability of the API used by GNOME, or at least strictly defining the subset of logind needed by desktop environments and moving forward from there?
    Right now it feels like CK was thrown out without any reasonable alternatives due to corner-case bugs that may indeed been too hard to fix inside CK, but not many actually hit in practice. So we only has a rather systemd dependent alternative for basics like suspend capability – I’m on record on disagreeing with the gnome releng what “basic functionality” entails these days, as it seems they believe laptop support is not basic these days or something.

  10. About a udev abstraction in GLib, I think it’s a terrible idea, it’s the wrong level. udev is mostly about enumerating devices & monitoring them, which certainly could be abstracted, but then the actual APIs to access the devices are platform specific. Being able to enumerate cameras on Mac or Windows would be nice, but if the application only speaks v4l2, it’s useless. That’s why I think such abstractions belong at the next level. For multimedia related things, we’re working on a GstDevice to abstract enumeration in the various API-specific plugins.

    I was also very happy about Canonical switching to systemd, because having systemd in every major distribution means that we now have a richer platform to build on, just like having Xorg (and not some antique X server) means that gnome-shell can have nicer features, having systemd means that we can have great multi-seat support. That said, I’m pretty sure that building a single-seat logind stub would be pretty easy.

    I’m also really looking forward to having the desktop make a better use of the modern features of Linux, for example, having each application in its own cgroup so as to better track them and to prevent firefox from using all of my CPUs. Being able to rely on kdbus also comes to mind. I’m planning a prototype of having a GStreamer pipeline across two separate processes connected using kdbus, this kind of things just can’t be done outside of Linux.

    That said, making sure the lower levels of our stack are portable is super important, especially to Windows/Mac/Android/iOS, as they as the platforms that more people use.

  11. @ocrete: After talking to a few people today, I’m also starting to second guess the udev-in-GLib idea, but not necessarily for the reasons that you state. Certain things like network interfaces, USB devices and even cameras (FreeBSD has v4l) can be presented in reasonably consistent ways, so supporting those could indeed be useful.

    I was asked today, though “why an API for enumerating PCI devices in GLib and not one for enumerating the current set of logged-in users?” and I didn’t have a good answer.

    It’s going to take a while to work out these sorts of details…

  12. Nice try, anoymous, but the only time Ryan is difficult to deal with at a conference is when you have to interrupt is under-the-table nap break to ask him a question.

    You can try to pick on one of the nicest and most competent dudes in Free software, but it won’t work very well when everyone else knows that he’s awesome.

  13. Fantastic article Ryan, and I fully agree with the way forward that you’re guiding us. At Tracker we’ve had our share of portability issues too. For example file monitoring APIs are a troublesome area. In a way is the Tracker project about providing a clear and well defined (reimplementable) API for metadata, too. I hope we’ll get to become that or define it and it was among the reasons for choosing the Nepomuk ontology and the SPARQL and SPARQL INSERT query language.

    I disagree with Anonymous GNOME hacker that you are difficult to work with. Some background for people who don’t know me: I have been to many GUADECs in the past, I’ve been to each and every FOSDEM including the first OSDEM that have happened and I’ve been to some of the first hackfests that you attended. You’re maybe sometimes a bit special when having to work with, but aren’t we all? When we met I was probably the guy who was more difficult to work with anyway ;-). Anyway, so I disagree and I also think that how difficult people are to work with isn’t very relevant to how portable GNOME should or shouldn’t be.

  14. anonymouse, the only thing I can conclude from your comment is that you’ve mistaken someone else for Ryan Lortie. “Difficult to deal with” and “divisive” are not only inaccurate, they’re the opposite of the friendly, helpful, inclusive Ryan you would encounter at any hackfest, GUADEC or FOSDEM if you took the trouble to walk up to him and say hello.

  15. @desrt: So I guess we basically agree, enumerating devices isn’t really a single problem, it needs different APIs for different kind of devices. GLib could certainly have an API to enumerate network devices for example (see interfaces.c in libnice, where we implemented that kind of abstraction). Enumerating USB or PCI devices isn’t much useful in an application unless you have the specific APIs to use it.

  16. @desrt: I should also mention that for example, for the v4l2 case, I understand that FreeBSD doesn’t have an API for dynamic device monitoring, so one can just use the pre-udev v4l2 specific APIs to enumerate them.

  17. I think you’re overstating Gnome’s current portability.
    Gtk+ on Windows is a highly problematic beast where,
    for example, printing hasn’t worked at all on Gtk+3.x and print preview probably hasn’t worked since Gtk+1.x.

    Patches languish in bugzilla with no action.

  18. @leio: I had a discussion with Lennart about this and he is not interested in making the logind APIs more “neutral”. I also found this chart, meanwhile: http://www.freedesktop.org/wiki/Software/systemd/InterfacePortabilityAndStabilityChart/

    I think I’ve concluded that I want to define an interface that will provide the functionality that people get from logind today, and with a similar style of API: non-blocking but synchronous. I don’t know if this would look like a very slim wrapper around logind (pkg-config, plus a header with a bunch of defines) or if it would be a GObject-based library. There is certainly room for some GObject niceness here — the fd-based approach with the monitor is particularly inconvenient, as an example.

    Doing a GObject approach would cut me off from many potential users (most things outside of GNOME and its offshoots like MATE). On the other hand, my intention with this library is specifically to limit its interface to the things that GNOME actually needs (which happens to avoid all of the systemd-specific bits of the logind API anyway), so this as well may already limit the usefulness.

    I’m not sure which way I will proceed here.

    There will likely be a ‘stub’ backend, in any case. Other operating systems may also completely reimplement the API. We may also see the possibility of having a CK compatibility backend, but this would be a “bad implementation” in the sense that it is extremely difficult to imagine how it could possibly keep the non-blocking synchronous interface without substantial modifications to CK — so it would probably just block, despite interface promises not to do so.

Comments are closed.