Safe as Milk

The Lifecycle of a Patch (or: Working Upstream)

January 14, 2011 community, freesoftware, gimp, gnome, inkscape, maemo, meego 5 Comments

Yesterday I looked into what it means to be a maintainer of a package. Today, I’m going to examine how to affect change in a distribution like MeeGo, and what it means to work upstream. To do so, we’re going to look at how code gets from a developer’s brain into the hands of a user.

So – how can you make a change in a Linux-based distribution? Here’s what happens when everything works as it should:

You open a bug report for the feature against your distribution
You identify the module or modules you need to change to implement the new feature
You open bug reports for each of the modules concerned, detailing the feature and the changes needed in that module for the feature
You write a patch to implement the feature, and propose it (appropriately cut up for ease of review) to the maintainers of those modules
Once the code has gone through the appropriate review process, it will be committed to the source control of the module(s)
Some time later, the maintainer of each module will include that code in a stable release of the module
Some time after that, the new stable versions will be packaged and uploaded to MeeGo
Your code will be included in the next release of the distribution following the upload.

When people talk about “working upstream” in MeeGo or Linaro, this is what they mean.

To simplify matters for our analysis, let’s consider that the feature we want to implement is self-contained in one module (or related modules which release together). There are two different scenarios we’ll consider:

The module is maintained by people not associated with your distribution (for example, a GNU or GNOME project)
The module is maintained by people closely related to your distribution (for example, Unity in Ubuntu, or oFono in MeeGo)

We will also look at a third situation, where you find and fix a bug in the software you are using – that is, a released version of a distribution (the proverbial “scratching an itch”).

For each case, I will try to pick a representative feature/patch and follow it from developer through to distribution to Real Users.

What if your code changes different projects?

If your code touches several modules (for example, if you are proposing some new API in GTK+ which you want to use in the GIMP) then things can get complicated – you will need a stable version of GTK+ to be released before you can ship a stable release of the GIMP which depends on it.

This issue of staggered releases is one that Andrew Cowie pointed out a few years ago for language bindings. To avoid making bindings on shifting sands, he preferred to package new APIs once they had been included in a stable GNOME release. In turn, Java GNOME developers rarely depend on development release bindings, and they would wait for the new API to be included in a stable bindings release. For example, the gtk_orientable_get_orientation, added to GTK+ at the end of September 2008, was released in GTK+ 2.16, in March 2009. The first version of Java-GNOME which depended on GTK+ 2.16 was version 4.0.13, released in August 2009. That was packaged in distributions in Autumn 2009, and so most users would not have access to the newer bindings for a few months after that – perhaps early 2010 – at which point, the API was written 18 months beforehand.

And that is when you have a regular release schedule you can rely on! Pity the developer who wants to release a GIMP plug-in which depends on some API included in GIMP 2.8 – the last stable GIMP release, 2.6, came out in October 2008, and over two years later, 2.8 still has not released. And when you combine unreliable release schedules for distributions and applications, the results are cumulative: users of the stable Debian distribution are still using GIMP 2.4 releases. The GIMP 2.4 released in October 2007. Features added to the GIMP in late 2007 are still not in the hands of users of stable Debian distributions.

Getting features to users

It is difficult to generalise when users upgrade their Linux distributions, or even to say what proportion of Linux users are new users at any given time. It would be over-simplifying to say that developers use bleeding-edge distributions, power users upgrade early to the latest and greatest, new users install the latest distributions available, but will only upgrade every 18 months or so afterwards, and conservative users stick with “Long term service” or stable distributions. Most developers I know use their computer for work (and thus want a stable distribution) and only install the latest versions of various dependencies they need to work on their project. But let’s generalise and say that this is roughly the case. So (guesstimating) about 10% of your users will be upgrading to the latest distribution very quickly after its release, a further 20% in the months after when the bugs are shaken out, and the rest will follow along in their own time, perhaps 12 or 18 months later.

To make this concrete, let’s follow the life of a single patch. This is complete anecdata, but in my defence, the patch has been chosen by random, from a project which I know has good community processes and release management in place. The patch we’re going to follow adds an extension to Inkscape to render objects along triangular paths.

Bug #226001 opened on 2008-05-03 by inductiveload, with a description of the feature to be added, and proposed code to implement it. The code, as an extension, may have a lower bar for acceptance than code which is core to a project.
Patch submission reviewed on 2008-05-03, minor comments, but patch is accepted (note: This was not the authors first submission to Inkscape)
Patch corrected to respond to comments and committed on 2008-05-03 (did I mention these guys had good community processes!?!)
Inkscape 0.47-pre0, containing the Triangle extension, released on 2009-07-02
Inkscape 0.47-pre4 included in Ubuntu 9.10

So for a feature developed in mid 2008, most Inkscape users will still not have the feature by the end of 2009, 18 months later. This is both a typical and atypical example: in many projects, patch proposals lay unreviewed for days, weeks, sometimes months, but the 0.47 release cycle was a particularly long one for Inkscape. However, I think the lag from code written to presence on user’s hard drives of ~12 to 18 months is about correct.

Does it have to be this hard?

If this were the only way to get features into a distribution, trying to improve MeeGo by contributing upstream would be a very frustrating experience. Happily, there are ways to accelerate the process. Taking the MeeGo kernel as an example, where Greg Kroah-Hartman recently threw in the towel on persuading people to propose patches upstream; the process is supposed to work like this:

Propose a patch for inclusion upstream. This patch will then ship in a future stable kernel release (let’s say 2.6.38).
After peer review, when the code has been accepted for inclusion in the kernel upstream, propose a backport for inclusion in the MeeGo kernel. The back-ported patch will be maintained across the next MeeGo release, and will be dropped when the kernel version included in the MeeGo project catches up with 2.6.38.

The overhead here is reduced basically to the peer review process of the upstream project, and the cumulative cost of merging a patch over the course of 6 months.

As a distributor (or a developer working on a specific distribution), this allows you to get code to everyone, eventually, and have that code included in your distribution as soon as you are sure that it is up to the standard expected by the community. Currently in MeeGo, the trend seems to be more towards submitting patches concurrently upstream and to MeeGo kernel maintainers (or even submitting them upstream once they have been accepted into the MeeGo kernel). In the case that a patch requires substantial modifications, or is rejected outright, upstream, the kernel maintainers are then left carrying a patch indefinitely in the distribution. For one patch, this might not be a big deal, but for thousands of patches, the maintenance and integration burden of these patches adds up.

It is also not unusual for kernel developers to maintain their own git branches for a long time. Three examples that come to mind are inotify, which Robert Love maintained for over a year for both Novell and in the kernel before it was accepted into the mainline, ReiserFS, which was maintained for several years out-of-tree before being shipped with the Linux kernel in 2001, and the fast desktop patchset which Con Kolivas maintained for almost five years on the -ck kernel branch. Distributions will occasionally ship a substantial diff to upstream if there is a maintainer committed to getting the code upstream eventually. Allocating someone to work over a long period to make everyone happy and comfortable with your code may enable you to ship a big patch to upstream, but this will not be sustainable long term.

To summarise: when working upstream, as a distribution, you should only ship with patches which have been accepted in a development version of upstream already, if you can help it.

Meetings in telephone boxes

Sometimes, however, when upstream and downstream coincide, you can simplify things considerably, while also adding a small measure of risk.

In MeeGo, to continue with that example, the distribution architects have a pretty good idea when they can expect emergency telephony to be ready for oFono and the MeeGo telephony stack, because they’re writing it. By co-ordinating the upstream release management with downstream packaging, you can make promises as a distribution which you can’t with community-developed software.

When upstream and downstream are co-ordinating each other, we cut out the middleman. The workflow becomes:

Report a bug/feature request against a component of the distribution
Develop a patch which implements the feature, and submit it directly to the distribution bug tracker
Once it has been reviewed and accepted, you know that your patch will be included in the next version of the distribution.

This gives a distribution much more control, both over what gets done, and when, and explains both the Ayatana and MeeGo UX development projects. However, being able to plan around the release is no guarantee that the release will happen on time: GNOME has in the past been stung by planning during the 2.6 development cycle to depend on a new version of GTK+, only to find that the release was delayed. In the end, the GTK+ release shipped in time for the 2.6 release at the end of March.

Scratch scratch

The other patch lifecycle I’d like to mention, because it is so relevant to distributions, was pointed out to me by Federico Mena Quintero yesterday. What happens to a patch that someone makes and submits to a distribution when they find a bug in stable released software? This is one of the key advantages of free software – if you find a bug in the software you use, and you have the wherewithall, you can fix the bug and share that fix with everyone else.

However, as we have seen, there is typically a lag of several months from the time that software is released and the time it is being used by large numbers of users through distributions. With releases of Red Hat Enterprise Linux, Novell Suse Linux Desktop and Ubuntu LTS being supported for up to 5 years, it is possible that important bugs will be fixed in these stable versions for years after the original developers have moved on, and are no longer maintaining older stable versions.

Let’s say I find and fix a bug in Rhythmbox 0.12.5, which ships with Ubuntu 9.10. I open a bug report on Launchpad, attach a fix to the source .deb there, and I update my local copy. As a user, I’m happy – I have fixed my problem and shared the solution with others. If I’m particularly conscientious, I might open a bug on gnome.org against Rhythmbox and attach my patch there, but since the development version is now 0.13.2, the best you can hope for is that the patch applies cleanly to the master branch, and will be included in the next release. It is very unlikely that the upstream maintainers will release another update to the 0.12 series at this point.

Now imagine that you are a maintainer for Suse, and someone reports the same bug against a long-term service release.In practice, there are several different versions being maintained by different distributions, and no good way to know if the same bug has been reported and fixed by someone else. You end up searching for a fix in upstream bug trackers, and in the bug trackers of each of the other main distributions. According to Federico at the time:

Patches for old versions are traded in the black market. You have friends in another distro? You ask them first, “did you guys already fix this?” Those patches don’t ever manage to reach CVS, where everyone would be able to get them.

Ideally, you could collaborate ahead of time with other distributions to ensure that you are all using the same branch of upstream modules, and are committing patches upstream. The Linux kernel is moving to this model, and there are also discussions underway in GNOME to co-ordinate this type of activity. Mark Shuttleworth has also pushed for something similar by encouraging projects in the core Linux platform to have a regular cadence of releases, so that everyone can synchronise their longer term service offerings every couple of years.

But at the moment, the best you can hope for is that your patch will be included in an upcoming release for your distribution, and which point other users of the distro can avail of it, and that upstream will patch their development version and latest stable versions, and get your patch to everyone in a few months.

Working upstream

The goal of this article is to explain what working upstream actually means, and how to make that more palatable for a distribution that wants to get features written and included in their next release. Hopefully, by pointing out some of the shortcomings of the way patches circulate from developers to users, some of these issues can be addressed.

In any case, one thing is clear – if you are carrying a patch as a distribution without ever submitting it upstream, you are making a costly mistake. You will be carrying code that others won’t, and bearing all of the merge and maintenance burden for that code for years to come. The path to maximum happiness is to co-ordinate with other distributions and with upstream to ensure that everyone is working in the same place, and sharing work as much as possible.

Code:Free – a reminder that our software is for doing stuff

October 15, 2009 community, freesoftware, gimp, inkscape, scribus 5 Comments

I recently came across Code:Free, a webzine (made with Scribus) which showcases some great examples of art created with free software tools, and tutorials on how to achieve some nice effects – it’s kind of a compilation of the best of Deviantart made with Free tools. After seeing Ton Roosendaal keynote the Maemo Summit last weekend, it was a reminder that the goal behind creating software is to have your users take it & do cool stuff with it.

The webzine itself is gorgeously laid out and the art in it is very good indeed. Congratulations to Chrisdesign (of gimpforums.de fame) in this great initiative, long may it continue!

Will we pass $5000 today?

April 15, 2009 gimp, inkscape, libre graphics meeting, scribus No Comments

The Libre Graphics Meeting fundraiser has been inching higher in recent weeks, and we are very close to the symbolic level of $5,000 raised, with less than a week left in the fundraising drive.

I am sure we will manage to get past $5,000, but I wonder if we will do it today? To help us put on this great conference, and help get some passionate and deserving free software hackers together, you still have time to give to the campaign. (Update 2019-03-29: Pledgie is no more – for an explanation why, read this).

Thanks very much to the great Free Art and Free Culture community out there for your support!

Update: Less than 2 hours after posting, Mark Wielaard pushed us over the edge with a donation bringing us to exactly $5000. Thanks Mark! Next stop: $6000.

Libre Graphics Meeting fundraiser update

March 18, 2009 community, freesoftware, gimp, gnome, inkscape, libre graphics meeting, maemo, scribus No Comments

With little fanfare, this year’s Libre Graphics Meeting fundraiser has been progressing nicely.

In the three weeks since the announcement of the launch of the campaign, we have raised almost $3,000 in community donation – mostly smaller than $50 – from 71 individual donors. Much of the credit for the campaign this year has to go to Jon Phillips of Creative Commons, Inkscape and OpenClipart fame.

The campaign has started earlier this year than last year, when we were really caught unawares by our difficulties in getting sponsors, and has lacked some of the frenzy of the last campaign, but Jon has been doing stellar work keeping the fire burning, and ensuring a regular stream of donations from supporters of projects related to Libre graphics.

It is hard to overstate the importance this conference has to the communities working on projects like Inkscape, GIMP and Scribus, among others, and to overstate the progress we have made because of these conferences in the past few years in the realm of graphics applications on Linux.

It’s useful to point out that in the Linux Foundation desktop linux surveys, the most popular applications which companies and individuals want for Linux are graphics applications – Adobe Photoshop, Illustrator, Premier, Autodesk AutoCAD, Adobe Dreamweaver and Microsoft Visio are the top 6 applications which people are missing on Linux. This conference is all about encouraging the development of applications destined to fulfil those needs. Also worth noting, when asked whether they wanted the applications above ported to Linux, or they wanted to use equivalent Linux applications where possible, a large majority want to use native equivalents, rather than ported commercial applications.

For any of you looking for a good cause which will go directly to supporting high quality applications that you use, I’d encourage you to contribute to the Libre Graphics Meeting. The conference is only as worthwhile as the people attending it, let’s ensure that we get a critical mass once again and provide energy and momentum to all of the participating projects for the coming year.

Governance best practices

February 20, 2009 community, freesoftware, inkscape, maemo 2 Comments

Jono asked on the AOC blog for successful governance stories, and while I’m happy to comment on the blog, now that I’ve taken the time to write some down, I thought I might as well share them 🙂

Governance comes in many shapes & sizes of course. My favourite governance stories are about federating individuals, who manage to channel community efforts, maintain a meritocracy where code talks, and yet don’t come across as authoritarians.

Outside of Linus (who’s a good example), Ton Roosendaal of Blender has this kind of presence. Talking to Ton, it is easy to see that he cares about Blender and about the Blender Community. The care and attention that he brings to projects like “Elephants Dream” and “Big Buck Bunny”, or to the supporting documentation and conferences he organises for the community, illustrate the esteem in which he holds his users and his developer community. Even the way the Blender Foundation came into being was amazing.

One of my favourite communities is Inkscape. When they broke from Sodipodi, there was this acrimonious flame war, and something of a bitter taste in people’s mouth. So what Bryce Harrington, Nathan Hurst, MenTaLguY and Ted Gould did when they split was decide to throw open the doors, and accept code from all comers. They set a direction and some ambitious goals, but they were very clear from the start – come right in, you’re welcome. And this gave the project some great results, especially early on when it was still establishing itself. Bryce describes one of them in this article.

The success of the Inkscape project’s governance model is borne out by its ability to escape founder’s syndrome – Bryce, Nathan and Ted have now backed away from the project to some extent, they’re still there as wise heads, but they have passed off the direction of the project to other capable people.

I think the way that Drizzle was born bears some resemblance to this, and I really like the way they have consciously broken down the walls which were necessarily up around MySQL. Brian Aker’s been something of an inspiration on this. His mission statement at the announcement of the project was astounding.

Subversion‘s governance model is an exemplar of best practices too. Set a clear project scope (“Subversion is a compelling alternative to CVS”), clear goals, establish transparent and fair community processes, and open up the gates. Anything within the scope of the project is fair game. And once again, code talks. This story, from Karl Fogel’s “Producing OSS” illustrated the robustness of their governance, and the confidence the project’s leaders had in their ability to influence the project.

The Maemo Community Council has the potential to be a very good governance structure, I think. The idea of a governing body of the community, by the community, for the community, whose goal is to canalise the efforts of a disparate group into something coherent, and to provide a legitimate point of contact for technical decision-makers in Nokia, is a novel one, and hasn’t been tried, as far as I can tell, by other companies.

Counter-examples of good governance are all around, I won’t name any in particular to protect the guilty. But many of them stem from a misguided belief in absolute free speech, to the detriment of the quality of discourse and code in the project (“we are all created equal”) which results in very chatty, but unproductive, individuals taking senior positions in the community, or a sort of shyness of the founder or leader, who doesn’t believe that it’s his place to set a direction and tone.In company-run projects, excessive control or influence is an equally toxic characteristic. Companies who retain a veto on community decisions are companies who do not trust their communities.

Correction (for the record)

April 16, 2008 freesoftware, gimp, inkscape, libre graphics meeting 2 Comments

I just listened to LUG Radio 5×15, including my interview about LGM this year. And I have to make a correction. I have tried, but I cannot find any way to drag & drop or copy & paste a curve from Inkscape into the GIMP. I can go via the intermediary of an SVG, since importing gradients and curves from an SVG drop or load is supported in the GIMP, but I can’t figure out how to drag & drop elements from the Inkscape canvas into any other application – when I hit the edge of the window, it just starts scrolling. And cut/copy & paste isn’t any more successful.

Any Inkscape people out there able to set me right?

One interesting drag & drop thing I love showing to people is dragging a chart created in Gnumeric into Inkscape – the drop is a proper SVG, and you can ungroup & manipulate individual elements from the chart in your favourite vector graphics application. Which is nice.

Libre Graphics Meeting stories: colour management

April 8, 2008 freesoftware, gimp, inkscape, libre graphics meeting, scribus 1 Comment

Continuing the series of articles I started last week (part 1, part 2), the next fall-out which has come from past Libre Graphics Meetings is the movement towards colour management everywhere over the past few years.

Let’s look back to where we were 3 years ago. Outside of Scribus and CinePaint, there was essentially no colour management in free software graphics apps, in spite of the existence of a high quality color management library, little cms.

In 2005, that story started to change a bit – the GIMP started conserving ICC profiles in JPEG files and allowing the user to see the list of ICC profiles with the 2.3.2 release, in July 2005. Scribus added support for soft proofing in version 1.2.3 in September 2005. Krita released version 1.5 with support for color profiles in December 2005.

In the first Libre Graphics Meeting, one of the most popular presentations was by Marti Maria of little cms, who gave an overview of what color management is, how ICC color profiles fit into the picture, and finally what applications need to do to integrate color management support. One of the outstanding memories I have from the conference was Carl Worth of Cairo being very excited about the conference, and in particular about meeting Marti.

Since 2005, things have changed significantly. Color management support has been completed for the GIMP in 2.4.0. Inkscape added support for ICC profiles in 0.44, in June 2006, soon after the first Libre Graphics Meeting in Lyon, and this support has been further improved in the recent 0.46 release.

And since, color management has become almost ubiquitous – via the “ICC profiles in X” spec, all applications who support the spec (including, at last count, the GIMP, Eye of GNOME, Krita, UFRaw and Inkscape) get soft proofing for your screen when X contains the ICC profile atom.

I’m not so presumptuous as to attribute the advent of color management to the Libre Graphics Meeting, but at least in the case of Inkscape, the work started at the conference. And for other developments, the bridges built and conversations started during LGM and other similar conferences has played a significant part in improving the state of affairs.

Libre Graphics Meeting stories: Shared resources

April 5, 2008 freesoftware, gimp, inkscape, libre graphics meeting, scribus No Comments

Graphics applications have lots of associated files that we call resources or assets, which help the artist achieve what he’s looking for.

Some of the resources which are used by an application like the GIMP are paintbrushes, patterns, gradients, fonts, colour profiles and palettes. On top of this, you can add things like clip-art and probably others that I’ve forgotten which are important to other types of application.

Three years ago, pretty much none of this data was shared between applications. The GIMP had long been using very simple formats for its data, designed to be easy to write and easy to parse. Some other applications, for example Krita, adopted the same file formats for some of the formats, and thus created some de facto standards for things like dynamic bitmap brushes and patterns, but there wasn’t much sharing going on.

Through the Create project and the shared resources spec current practices and formats were documented and attention brought to what we could share. The OpenICC spec proposed a way to share ICC device profiles throughout the system for colour management. In the first and second Libre Graphics Meeting, progress has been made on improving the situation of shared resources. Today, you can share patterns between the GIMP, Krita, and Cinepaint. Palettes (or swatches) can be shared between Inkscape, Scribus, the GIMP, Krita and Cinepaint. Gradients can be shared between Krita, the GIMP and Inkscape among others.

Some areas where work would be useful would be in defining a shared access point for clipart to be used by all applications, and have the various applications (including OpenOffice.org) ship with a clipart browser which allowed applications to easily take advantage of the work of the Open Clipart library, and finally splitting out all of the resources which can be shared into a separate package, which would be installed in one place and used by everyone. But already, we have come a long way in being able to share all of the resources you expect among lots of different applications, in large part because of the collaboration that has happened at the Libre Graphics Meeting.

Almost $2000 in 2 days campaigning

April 4, 2008 freesoftware, gimp, inkscape, libre graphics meeting, scribus No Comments

I woke up this morning to find out that we are fast approaching the $2000 level in our fundraising campaign! As I write this, we have now raised $1780 in the two days since the campaign opened on the 2nd of April, bringing us to 9% of our goal amount, with 14 days left in the campaign.

As of tomorrow, the community will be listed on the Libre Graphics Meeting website as a silver sponsor, the same level as Google, Intel and the Free Software Foundation, and as we continue to pass the sponsorship levels, we will move the community to Gold, and then to Cornerstone sponsor levels.

I am deeply in awe of the generosity of the people who are donating, and deeply impressed by the passion of the user community of these applications which we’re helping improve by hosting this conference.

Thank you all very much!

Update: One hour after posting this, Sebastian Bober pushed us over the $2000 mark – we’re now at over 10% of our objective!