Safe as Milk

What community?

October 6, 2011 community, General, gnome, maemo, work 5 Comments

With the announcement of Tizen (pronounced, I learned, tie-zen, not tea-zen or tizz-en) recently, I headed over to the website to find out who the project was aimed at. I read this on the “Community” page:

The Tizen community is made up of all of the people who collectively work on or with Tizen:

• Product contributors: kernel/distribution developers, release managers, quality assurance, localization, etc.
• Application developers: people who write applications to run on top of Tizen
• Users: people who run Tizen on their device and provide feedback
• Vendors: companies who create products based on Tizen
• Other contributors: promotion, documentation, and much more

Anyone can contribute by:

• Submitting patches
• Filing bugs
• Developing applications
• Helping with wiki documentation
• Participating in other community efforts and programs

Wow! That’s a diverse target audience, and a very wide ranging list of ways you can help out. But is it really helpful to scope the project so wide, and try to cater to such a wide range of use-cases from the start? And is the project at a stage where it even makes sense to advertise itself to some of these different types of users?

I have talked about the different meanings of “maintainer” before, depending on whether you’re maintaining a code project or are a package maintainer for a distribution. I have also talked about the different types of community that build up around a project, and how each of them needs their own identity – particularly in the context of the MeeGo trademark. I particularly like Simon Phipps’s analysis of the four community types as a way to clarify what you’re talking about.

For Tizen, I see between three and five different types of community, each with different needs, and each of which can form at different stages in the life-cycle of the project. Trying to “sell” the project to one type of community before the project is ready for them will result in disappointment and frustration all round – managing the expectations of people approaching Tizen will be vital to its long-term success, even if it opens you up to short-term criticism. Unless each of these communities is targeted individually and separately, and at the right time, I am sceptical about the results.

“Upstream” software developers

The first and most identifiably “Open Source” family of communities will be the software developers working on components and applications which will end up in the core of Tizen. For the most part, these communities exist already, and Samsung and Intel engineers are working with them. These are the projects we commonly call “upstreams” – projects you don’t control, but from whom code flows into your product.

In other cases, code will originate from Intel and/or Samsung. In the same way that Buteo, oFono and the various applications which were developed for the MeeGo Netbook UX were very closely associated with MeeGo, there will be similar projects (sometimes the same projects) which will have a close association with Tizen. Each of these projects will have their own personality, their own maintainers, roadmaps, specs – and each of them should have their own identity, and space to collaborate and communicate.

Communities form around programming projects not because of the code, but because of a shared vision and values. Each project will attract different people – the people who are interested in metadata and search are not the same as the people who will be passionate about system-wide contact integration. Each project needs its own web space, maintainers, bug tracker, mailing list, and wiki space. Of course, many projects can share the same infrastructure, and a lot of the same community processes (for things like code governance), and for projects closely related to Tizen, we can provide common space to help create a Tizen developer community in the same way there’s a GNOME developer community. But each community around each component will have its own personality and will need its own space.

At the level of Tizen, we could start with an architecture diagram, perhaps – and for each component on the architecture diagram, link to the project’s home page – many of the links will point to places like kernel.org, gnome.org, freedesktop.org and so on. For Tizen-specific projects, there could be a link to the project home page, with a list of stuff that needs to be done before the component is “ready”.

Core platform packagers, testers, integrators

Once we have a set of components which are working well together, we get to the heart of what I think will be Tizen’s early activity – bringing those components together into a cohesive whole. Tizen will be, basically, a set of distributions aimed at different form factors. And the deliverable in a distribution is not code or a Git tag, it’s a complete, integrated stack.

The engineering skills, resources and processes required to integrate a distribution are different to those of a code project. Making a great integrated Linux platform is obviously difficult – otherwise Red Hat would not be making money, and Ubuntu would not have had the opportunity to capture so much mind-share. Both Red Hat and Canonical do something right which others failed at before them.

Distributions attract a different type of contributor than code projects, and need a different set of tools and infrastructure to allow people to collaborate.At the distribution level, it is more likely you will be debating whether or not to integrate a particular package or its competitor than it is to debate whether to implement a feature in a specific package. Of course, it is possible to influence upstream projects to get specific features implemented, not least by providing developer resources, and there will be a need for some ambassadors to bridge the gap to upstream projects. And it is possible for a distribution to carry patches to upstream packages if that community disagrees. But in general, not much code gets written in distributions.

What the distro community needs and expects is infrastructure for continuous integration, bug tracking software, a way to submit and build software packages, good release engineering, an easy way to find out what packages need a maintainer (see Debian’s WNPP list  or Ubuntu’s “need-packaging” list for examples) and a way to influence what packages or features are included in future releases (see Fedora or Ubuntu for examples). They also want tools to allow packaging, testing and  deploying the integrated distribution – for an embedded distro, that might mean an emulator and an image creator, perhaps.

Vendors and carriers

Communities of companies are worth a special mention. Companies have very different ways of working together and agreeing on things than communities of individuals. I was tempted to just roll vendors into the “Platform integrators” community type, but they are sufficiently different to be considered another type of community. Vendors have different constraints and motivations than individual contributors to the platform, and we should be aware of those.

Vendors like to have a business relationship – some written agreement that shows where everyone stands. They have a direct relationship with people who buy their hardware, and have an interest (potentially in conflict with other communities) in owning the user relationship – through branded application stores, UI and support forums, for example. And since vendors are typically working on hardware development in parallel with software development, they care a lot about a reliable release schedule and quality level from the stack. Something that companies care about which individuals usually don’t are legal concerns around working with the process – do they have patent rights to the code they ship? Are they giving up any of their own potential patent claims?

3rd party application developers

Application developers don’t care, in general, whether the platform is open source or closed, or developed collaboratively or by one party (witness the popularity of Android and iOS with application developers). What they do care about are developer tools, documentation, and the ability to share their work with device users and other application developers. Some application developers will want to develop their applications as free software, and it is possible to enable that, but I think the most important thing for application developers is that it’s easy to do things with your platform, that there are good tools for developing, testing and deploying your application, that your platforms APIs are enabling the developer to do what he wants, and that you are providing a channel for those developers to get their apps to users of your platform.

An application developer doesn’t want to have to ship his software to 5 different app stores on every release – in contrast to vendors, he would like a single channel to his market. Other things he cares about are being able to form a relationship with his users – so app stores need to be social, allow user ratings and comments, and allow the author to interact with his users. Clear terms of engagement are vital here too – especially for commercial application developers. And application developers are also another type of community – they will want to share tips and tricks, code, and their thoughts on the project leaders in some kind of app developer knowledge base.

Device users

There is another potential community which I should mention, and that is users of your platform – typically, these will be users of devices running your platform. It should be possible for engaged users to share information, opinions, tips & tricks, and interesting hacks among each other. It should also be possible to rate and recommend applications easily – this is in the interests of both your user community and your application developer ecosystem.

OK, so what?

Each of these community types is different, and they don’t mix well. They mature at different rates. There is no point in trying to build a user platform until there are devices running your platform on the market, for example

So each type of community needs a separate space to work. There is no point in catering to a 3rd party application developer until you have developer tools and a platform for him to develop against. Vendors will commit to products when they see a viable integrated platform. And so on.

What is vital is to be very clear, for each type of community, what the rules of engagement are. As an example, one company can control the integration of a platform and the development of many of its components (as is the case for Android) and everyone is relatively happy, because they know where they stand and what they’re getting into. But if you advertise as an open and transparent project, and a small group of people announce the decisions of what components are included or excluded from the stack (as was the case in MeeGo), then in spite of being vastly more open, people who have engaged with the project will end up unhappy, because of a mismatch between the message and the practice in the project.

So what about Tizen? I think it is a mistake to announce the projects as a place to “submit patches, report bugs and develop applications” when there is no identifiable code base, no platform to try, and no published SDK to develop against. By announcing that Tizen is an Open Source platform, Intel and Samsung have set an expectation for people – and these are people who have gone through the move to MeeGo under two years ago, and who have seen Nokia drop the project earlier this year. If they are disappointed by the project’s beginnings because the expectations around the project have been set wrong from the offset, it could take a long time to recover.

Personally, I would start low-key by announcing an architecture diagram and concentrating on code and features that need writing, then ramp up the integrator community with some alpha images and tools to allow people to roll their own; finally, when the platform stabilises roll out the developer SDK and app store and start building up an application developer community. But by aiming too big with the messaging, Tizen runs the risk of scaring some people away early. Time will tell.

 

The Real Life “Lord of the Flies” experiment that went wrong

September 1, 2011 community, gnome 8 Comments

While reading “The Illusion of Asymmetric Insight”, it occurred to me that the real cost (and tragedy) of Unity and GNOME Shell, or KDE and GNOME, is that it turns us into “us” and “them” – and at that point it is really easy to fall into the trap of reducing all criticism to “haters gonna hate”.

 

GNOME News

June 24, 2011 gnome, marketing 7 Comments

Does everyone know about GNOME News? Since we moved gnome.org to WordPress, we have a news channel, syndicated “below the fold” on the front page of gnome.org, and updated regularly with news, announcements and articles from around the GNOME world.

In the past few weeks, we have published interviews with Desktop Summit keynotes Thomas Thwaites and Claire Rowland, some announcements related to the Desktop Summit and GNOME Asia Summit, and of course the great news that Karen Sandler recently became the GNOME Foundation’s new Executivee Director.

Since I only found out about it recently, I’m guessing that maybe others aren’t aware of it yet either. We’re always on the look-out for compelling GNOME related stories to keep it interesting too – if you have something you think we should share, contact marketing-list@gnome.org

Something to add to your feed readers.

 

Effective mentoring programs

May 31, 2011 community, freesoftware, General, gimp, gnome 18 Comments

I’ve been thinking a lot recently about mentoring programs, what works, what doesn’t, and what the minimum amount of effort needed to bootstrap a program might be.

With the advent of Google Summer of Code and Google Code-In, more and more projects are formalising mentoring and thinking about what newcomers to the project might be able to do to learn the ropes and integrate themselves into the community. These programs led to other organised programs like GNOME’s Women Summer Outreach Program. Of course, these initiatives weren’t the first to encourage good mentoring, but they have helped make the idea of mentors taking new developers under their wing much more widespread.

In addition to these scheduled and time-constrained programs, many projects have more informal “always-on” mentoring programs – Drupal Dojo and GNOME Love come to mind. Others save smaller easier tasks for newcomers to cut their teeth on, like the “easy hacks” in LibreOffice. Esther Schindler wrote a fantastic article a few years ago documenting best and worst practices in mentoring among different projects.

Most mentoring programs I have seen and participated in don’t have a very good success rate, though. In this article, I look at why that is the case, and what can be put in place to increase the success rate when recruiting new developers.

Why most mentoring fails

Graham Percival, a GNU/LilyPond developer, decided in 2008 to run an experiment. At one point, Graham decided that he would quit the project, but felt guilty about doing so in one go. So he  started the “Great Documentation Project” to recruit a replacement documentation team to follow on after his departure. He then spent 12 months doing nothing but mentoring newcomers to get them involved in the project, and documented his results. Over the course of a year, he estimates that he spent around 800 hours mentoring newcomers to the project.

His conclusions? The net result for the project was somewhere between 600-900 hours of productivity, and at the end of the year, 0 new documentation team members. In other words, Graham would have been better off doing everything himself.

Graham found that “Only 1 in 4 developers was a net gain for the project” – that is, for every 4 apprentices that Graham spent time mentoring, only 1 hung around long enough for the project to recoup the time investment he put into mentoring. A further 1 in 4 were neither gain or loss – their contribution roughly equalled the mentor time that they took up. And the remainder were a net loss for the project, with much more time spent mentoring than the results could justify.

The GNOME Women’s Summer Outreach Program in 2006 had 6 participants. In 2009, the GNOME Journal ran a “Where are they now?” follow-up article. Of the 6 original participants, only one is still involved in free software, and none are involved in GNOME. Murray Stokely did a follow-up in 2008 to track the 57 alumni of Summer of Code who had worked on FreeBSD. Of these, 10 students went on to get full commit access, and a further 4 students were still contributing to FreeBSD or OpenBSD after the project. Obey Arthur Liu also did a review of Debian participants in 2008. Of 11 students from 2008 who had no previous Debian developer experience, he found that 4 remained active in the project one year later.

From my own experience as a replacement mentor and administrator in the Summer of Code for the GIMP in 2006, we had 6 projects, most of which were considered a success by the end of the summer, yet of the participating students, none have made any meaningful contribution to the GIMP since.

I feel safe in saying that the majority of mentoring projects fail – and Graham’s 1 in 4 sounds about right as an overall average success/failure rate. This begs the question: why?

Most mentored projects take too long

What might take a mentor a couple of hours working on his own could well take an apprentice several days or weeks. All of the experience that allows you to hit the ground running isn’t there. The most important part of the mentoring experience is getting the student to the point where he can start working on the problem. To help address this point, many projects now require Summer of Code applicants to compile the project and propose a trivial patch before they are accepted for the program, but understanding the architecture of a project and reading enough code to get a handle on coding conventions will take time. It will also take mentor time. It takes longer to teach a newcomer to your project than to do the work yourself, as anyone who has ever had a Summer intern will attest.

When you set a trainee task which you estimate to be about 4 hours work, that will end up costing a few weeks of volunteer effort for your apprentice, and 8 to 10 hours mentoring time for you during that time. Obviously, this is a big investment on both sides, and can lead to the apprentice giving up, or the mentor running out of patience. I remember in the first year of Summer of Code, projects were taking features off their wishlists that had not been touched for years, and expecting students new to the project to come in and work full time implementing them perfectly over the course of 12 weeks. The reality that first year was that most of the time was spent getting a working environment set up, and getting started on their task.

Mentoring demands a lot of mentors

As a free software developer, you might not have a lot of time to work on your project. Josh Berkus, quoted in Schindler’s article, says “being a good mentor requires a lot of time, frequently more time than it would take you to do the development yourself”.  According to the Google Summer of Code FAQ, “5 hours a week is a reasonable estimate” for the amount of time you would need to dedicate to mentoring. Federico Mena Quintero suggests that you will need to set aside “between 30 and 60 minutes a day“.

When you only have 10 hours a week to contribute to a project, giving up half of it to help someone else is a lot. It is easy to see how working on code can get a higher priority than checking in with your apprentice to make sure everything is on track.

Communication issues

More mentoring projects fail for lack of communication than for any other reason.

Apprentices may expect their mentors to check in on them, while mentors expect apprentices to ask questions if they have any. Perhaps newcomers to the project are not used to working on mailing lists, or are afraid of asking stupid questions, preferring to read lots of documentation or search Google for answers. In the absence of clear guidelines on when and how parties will talk to each other, communication will tend towards “not enough” rather than “too much”.

No follow through

Many mentoring programs stop when your first task is complete. The relationship between the mentor and the apprentice lasts until the end of the task, and then either the apprentice goes off and starts a new task, with a new mentor, or that is the end of their relationship with the project. I would be really interested to hear how many Summer of Code mentors maintained a relationship with their students after the end of the Summer, and helped them out with further projects. I suspect that many mentors invest a lot of time during the program, and then spend most of their time catching up with what they wanted to do.

Project culture

In her OSCON keynote in 2009, Skud talked about the creation of a welcoming and diverse community as a prerequisite for recruiting new developers. Sometimes, your project culture just doesn’t match newcomers to the project. If this happens regularly, then perhaps the project’s leaders need to work on changing the culture, but this is easier said than done. As Chris di Bona says in this video, “the brutality of open source is such that people will learn to work with others, or they will fail”. While many think that this kind of trial-by-fire is fine, the will not be the environment for everyone. It is really up to each project and its leaders to decide how “brutal” or forgiving they want to be. There is a trade-off: investing time in apprentices who will contribute little is a waste of time, but being too dismissive of a potential new developer could cost your project in the long run.

Mentoring best practices

Is all the effort worth it? If mentoring programs are so much hassle, why go to the bother?

Mentoring programs are needed to ensure that your project is long-term sustainable. As Graham says in his presentation: “Core developers do most of the work. Losing core developers is bad. Projects will lose core developers.” Do you need any other reason to start actively recruiting new blood?

There are a few simple things that you can put in place to give your mentoring program a better chance of success.

Small tasks

Mentored tasks should be small, bite-sized, and allow the apprentice to succeed or fail fast. This has a number of advantages: The apprentice who won’t stick around, or who will accomplish nothing, has not wasted a lot of your mentor’s time. The apprentice who will stay around gets a quick win, gets his name in the ChangeLog, and gains assurance in his ability to contribute. And the quick feedback loop is incredibly rewarding for the mentor, who sees his apprentice attack new tasks and increase his productivity in short order. Graham implies that a 10 minute task is the right size, with the expectation that the apprentice might take 1 hour to accomplish the task.

A ten minute task might even take longer to identify and list than it would to do. You can consider this cost the boot-strapping cost of the mentoring program. Some tasks that are well suited to this might include:

• Write user documentation for 1 feature
• Get the source code, compile it, remove a compiler warning, and submit a patch
• Critique 1 unreviewed patch in Bugzilla
• Fix a trivial bug (a one line/local change)

Of course, the types of tasks on your list will change from one project to the next.

Mentoring is management

Just as not everyone is suited to being a manager, not everyone is suited to being a mentor. The skills needed to be a good mentor are also those of a good manager – keeping track of someone else’s activity while making them feel responsible for what they’re working on, communicating well and frequently, and reading between the lines to diagnose problems before it’s too late to do anything about them.

When you think of it in this way, there is an opportunity for developers who would like to gain management experience to do so as a mentor in a free software project. Continually recruiting mentors is just as important as recruiting developers. Since mentoring takes a lot of time, it’s important that mentors get time off and new mentors are coming in in their place.

Pair apprentices with mentors, not tasks

An apprentice should have the same mentor from the day he enters the mentoring program until he no longer needs or wants the help. The relationship will ideally continue until the apprentice has himself become a mentor. Free software communities are built on relationships, and the key point to a mentoring program is to help the creation of a new relationship. Mentoring relationships can be limited in time also, 6 months or a year seem like good time limits. The time needed to mentor will, hopefully, go down over this period.

Regular meeting times

Mentors and apprentices should ensure that there is a time on their calendar for a “one on one” at regular times. How regularly will depend on the tasks, and the amount of time you can spend on it. Weekly, fortnightly or monthly are all reasonable in different situations. This meeting should be independent of any other communication you have with the person – it is too easy for the general business of a project to swallow up a newbie and prevent their voice from being heard. Rands said it well when he said “this chatter will bury the individual voice unless someone pays attention.”

Convert apprentices into mentors

Never do you understand the pain of the initial learning curve better than when you have just gone through it. The people best suited to helping out newcomers to the project are those who have just come through the mentoring program themselves.

This is a phenomenon that I have seen in the Summer of Code. Those students who succeed and stay with the project are often eager to become mentors the following year. And they will, in general, be among the best mentors in the project.

Keep track

For all involved, it’s useful to have some idea of the issues newcomers have – ensure that documenting solutions is part of what you ask. It’s also useful to know how successful your mentoring program is. Can you do better than the 1 in 4 success rate of LilyPond? Keeping track of successes and failures encourages new mentors, and gives you data to address any problems you run into.

Manage the mentors

All of this work has overhead. In a small project with 1 or 2 core developers, it’s easy enough to have each core developer take an apprentice under their wing, and co-ordinate on the mailing list. In bigger projects, keeping track of who is a mentor, and who is mentoring who, and inviting new mentors, and ensuring that no-one falls through the cracks when a mentor gets too busy, is a job of itself. If your mentoring program goes beyond more than ~5 mentors or so, you might want to consider nominating someone to lead the program (or see who steps up to do the job). This is the idea behind the Summer of Code administrator, and it’s a good one.

Go forth and multiply

Developer attrition is a problem in open source, and recruitment and training of new developers is the only solution. Any project which is not bringing new developers up to positions where they can take over maintainership is doomed to failure. A good mentoring program, however, with a retention rate around 25%, organised continuously, should ensure that your project continues to grow and attract new developers.

Replenishing your stock of mentor tasks and recruiting new mentors will take effort, and continual maintenance of someone putting in a few hours a week. If you execute well, then you will have helped contribute to the long term diversity and health of your project.

Desktop Summit t-shirt design contest

May 12, 2011 freesoftware, gnome, guadec 1 Comment

The closing date for the Desktop Summit t-shirt design contest is coming up fast!

I have not heard much about it, so it seems like we need to get the word out about it. You can submit a t-shirt design by email to ds-team@desktopsummit.orgin SVG format, and the winner will get printed on the official conference t-shirt this year. It’s a great honour and ego boost to see your design up there on the conference t-shirt, and as this year is a joint conference with KDE targeting the free software desktop as a whole, there will be added significance.

If you know of any artists & designers who might be interested in submitting a design, please pass this news along – they have only a few days left to make a submission.

Getting people together

May 9, 2011 community, freesoftware, gimp, gnome, guadec, libre graphics meeting, maemo, openwengo 3 Comments

One of the most important things you can do in a free software project, besides writing code, is to get your key contributors together as often as possible.

I’ve been fortunate to be able to organise a number of events in the past 10 years, and also to observe others and learn from them over that time. Here are some of the lessons I’ve learned over the years from that experience:

Venue

The starting point for most meetings or conferences is the venue. If you’re getting a small group (under 10 people) together, then it is usually OK just to pick a city, and ask a friend who runs a business or is a college professor to book a room for you. Or use a co-working space. Or hang out in someone’s house, and camp in the garden. Once you get bigger, you may need to go through a more formal process.

If you’re not careful, the venue will be a huge expense, and you’ll have to find that money somewhere. But if you are smart, you can manage a free venue quite easily.

Here are a few strategies you might want to try:

• Piggy-back on another event – the Linux Foundation Collaboration Summit, OSCON, LinuxTag, GUADEC and many other conferences are happy to host workshops or meet-ups for smaller groups. The GIMP Developers Conference in 2004 was the first meet-up that I organised, and to avoid the hassle of dealing with a venue, finding a time that suited everyone, and so on, I asked the GNOME Foundation if they wouldn’t mind setting aside some space for us at GUADEC – and they said yes.Take advantage of the bigger conference’s organisation, and you get the added benefit of attending the bigger conference at the same time!
• Ask local universities for free rooms – This won’t work once you go over a certain size, but especially for universities which have academics who are members of the local LUG, they can talk their department head into booking a lecture theatre & a few classrooms for a weekend. Many universities will ask to do a press release and get credit on the conference web-site, and this is a completely fair deal.The first Libre Graphics Meeting was hosted free in CPE Lyon, and the GNOME Boston Summit has been hosted free for a number of years in MIT.

• If the venue can’t be free, see if you can get someone else to pay for it – Once your conference is bigger than about 200 people, most venues will require payment. Hosting a conference will cost them a lot, and it’s a big part of the business model of universities to host conferences when the students are gone. But just because the university or conference center won’t host you for free doesn’t mean that you have to be the one paying.

  Local regional governments like to be involved with big events in their region. GUADEC in Stuttgart, the Gran Canaria Desktop Summit, and this year’s Desktop Summit in Berlin have all had the cost of the venue covered by the host region. An additional benefit of partnering with the region is that they will often have links to local industry and press – resources you can use to get publicity and perhaps even sponsorship for your conference.

• Run a bidding process – by encouraging groups wishing to host the conference to put in bids, you are also encouraging them to source a venue and talk to local partners before you decide where to go. You are also putting cities in competition with each other, and like olympic bids, cities don’t like to lose competitions they’re in!

Budget

Conferences cost money. Major costs for a small meet-up might be
covering the travel costs of attendees. For a larger conference, the
major costs will be equipment, staff and venue.

Every time I have been raising the budget for a conference, my rule of
thumb has been simple:

1. Decide how much money you need to put on the event
2. Fundraise until you reach that amount
3. Stop fundraising, and move on to other things.

Raising money is a tricky thing to do. You can literally spend all of
your time doing it. At the end of the day, you have a conference to put
on, and the amount of money in the budget is not the major concern of
your attendees.

Remember, your primary goal is to get project participants together to
advance the project. So getting the word out to prospective attendees,
organising accommodation, venue, talks, food and drinks, social
activities and everything else people expect at an event is more
important than raising money.

Of course, you need money to be able to do all the rest of that stuff,
so finding sponsors, fixing sponsorship levels, and selling your
conference is a necessary evil. But once you have reached the amount of
money you need for the conference, you really do have better things to
do with your time.

There are a few potential sources of funds to put on a conference – I
recommend a mix of all of these as the best way to raise your budget.

• Attendees – While this is a controversial topic among many communities, I think it is completely valid to ask attendees to contribute something to the costs of the conference. Attendees benefit from the facilities, the social events, and gain value from the conference.Some communities consider attendance at their annual event as a kind of reward for services rendered, or an incitement to do good work in the coming year, but I don’t think that’s a healthy way to look at it.

  There are a few ways for conference attendees to fund the running of the conference:

  1. Registration fees – This is the most common way to get money from conference attendees. Most community conferences ask for a token amount of fees. I’ve seen conferences ask for an entrance fee of €20 to €50, and most people have not had a problem paying this.

     A pre-paid fee also has an additional benefit of massively reducing no-shows among locals. People place more value on attending an event that costs them €10 than one where they can get in for free, even if the content is the same.

  2. Donations – very successfully employed by FOSDEM. Attendees are offered an array of goodies, provided by sponsors (books, magazine subscriptions, t-shirts) in return for a donation. But those who want can attend for free.
  3. Selling merchandising – Perhaps your community would be happier hosting a free conference, and selling plush toys, t-shirts, hoodies, mugs and other merchandising to make some money. Beware: in my experience you can expect less from profits from merchandising sales than you would get giving a free t-shirt to each attendee with a registration fee.

• Sponsors – Media publications will typically agree to “press sponsorship” – providing free ads for your conference in their print magazine or website. If your conference is a registered non-profit which can accept tax-deductible donations, offer press sponsors the chance to invoice you for the services and then make a separate sponsorship grant to cover the bill. The end result for you is identical, but it will allow the publication to write off the space they donate to you for tax.

  What you really want, though, are cash sponsorships. As the number of free software projects and conferences has multiplied, the competition for sponsorship dollars has really heated up in recent years. To maximise your chances of making your budget target, there are a few things you can do.

  1. Conference brochure – Think of your conference as a product you’re selling. What does it stand for, how much attention does it get, how important is it to you, to your members, to the industry and beyond? What is the value proposition for the sponsor?

     You can sell a sponsorship package on three or four different grounds: perhaps conference attendees are a high-value target audience for the sponsor, perhaps (especially for smaller conferences) the attendees aren’t what’s important, it’s the attention that the conference will get in the international press, or perhaps you are pitching to the company that the conference is improving a piece of software that they depend on.

     Depending on the positioning of the conference, you can then make a list of potential sponsors. You should have a sponsorship brochure that you can send them, which will contain a description of the conference, a sales pitch explaining why it’s interesting for the company to sponsor it, potentially press clippings or quotes from past attendees saying how great the conference is, and finally the amount of money you’re looking for.

  2. Sponsorship levels – These should be fixed based on the amount of money you want to raise. You should figure on your biggest sponsor providing somewhere between 30% and 40% of your total conference budget for a smaller conference. If you’re lucky, and your conference gets a lot of sponsors, that might be as low as 20%. Figure on a third as a ball-park figure. That means if you’ve decided that you need €60,000 then you should set your cornerstone sponsor level at €20,000, and all the other levels in consequence (say, €12,000 for the second level and €6,000 for third level).

     For smaller conferences and meet-ups, the fundraising process might be slightly more informal, but you should still think of the entire process as a sales pitch.

  3. Calendar – Most companies have either a yearly or half-yearly budget cycle. If you get your submission into the right person at the right time, then you could potentially have a much easier conversation. The best time to submit proposals for sponsorship of a conference in the Summer is around October or November of the year before, when companies are finalising their annual budget.

     If you miss this window, all is not lost, but any sponsorship you get will be coming out of discretionary budgets, which tend to get spread quite thin, and are guarded preciously by their owners. Alternatively, you might get a commitment to sponsor your July conference in May, at the end of the first half budget process – which is quite late in the day.

  4. Approaching the right people – I’m not going to teach anyone sales, but my personal secret to dealing with big organisations is to make friends with people inside the organisations, and try to get a feel for where the budget might come from for my event. Your friend will probably not be the person controlling the budget, but getting him or her on board is your opportunity to have an advocate inside the organisation, working to put your proposal in front of the eyes of the person who owns the budget.

     Big organisations can be a hard nut to crack, but free software projects often have friends in high places. If you have seen the CTO or CEO of a Fortune 500 company talk about your project in a news article, don’t hesitate to drop him a line mentioning that, and when the time comes to fund that conference, a personal note asking who the best person to talk to will work wonders. Remember, your goal is not to sell to your personal contact, it is to turn her into an advocate to your cause inside the organisation, and create the opportunity to sell the conference to the budget owner later.

  Also, remember when you’re selling sponsorship packages that everything which costs you money could potentially be part of a sponsorship package. Some companies will offer lanyards for attendees, or offer to pay for a coffee break, or ice-cream in the afternoon, or a social event. These are potentially valuable sponsorship opportunities and you should be clear in your brochure about everything that’s happening, and spec out a provisional budget for each of these events when you’re drafting your budget.

Content

Conference content is the most important thing about a conference. Different events handle content differently – some events invite a large proportion of their speakers, while others like GUADEC and OSCON invite proposals and choose talks to fill the spots.

The strategy you choose will depend largely on the nature of the event. If it’s an event in its 10th year with an ever increasing number of attendees, then a call for papers is great. If you’re in your first year, and people really don’t know what to make of the event, then setting the tone by inviting a number of speakers will do a great job of helping people know what you’re aiming for.

For Ignite Lyon last year, I invited about 40% of the speakers for the first night (and often had to hassle them to put in a submission, and the remaining 60% came through a submission form. For the first Libre Graphics Meeting, apart from lightning talks, I think that I contacted every speaker except 2 first. Now that the event is in its 6th year, there is a call for proposals process which works quite well.

Schedule

Avoiding putting talks in parallel which will appeal to the same people is hard. Every single conference, you hear from people who wanted to attend talks which were on at the same time on similar topics.

My solution to conference scheduling is very low-tech, but works for me. Coloured post-its, with a different colour for each theme, and an empty talks grid, do the job fine. Write the talk titles one per post-it, add any constraints you have for the speaker, and then fill in the grid.

Taking scheduling off the computer and into real life makes it really easy to see when you have clashes, to swap talks as often as you like, and then to commit it to a web page when you’re happy with it.

I used this technique successfully for GUADEC 2006, and Ross Burton re-used it in 2007.

Parties

Parties are a trade-off. You want everyone to have fun, and hanging out is a huge part of attending a conference. But morning attendance suffers after a party. Pity the poor community member who has to drag himself out of bed after 3 hours sleep to go and talk to 4 people at 9am after the party.

Some conferences have too many parties. It’s great to have the opportunity to get drunk with friends every night. But it’s not great to actually get drunk with friends every night. Remember the goal of the conference: you want to encourage the advancement of your project.

I encourage one biggish party, and one other smallish party, over the course of the week. Outside of that, people will still get together, and have a good time, but it’ll be on their dime, and that will keep everyone reasonable.

With a little imagination, you can come up with events that don’t involved loud music and alcohol. Other types of social event can work just as well, and be even more fun.

At GUADEC we have had a football tournament for the last number of years. During the OpenWengo Summit in 2007, we brought people on a boat ride on the Seine and we went on a classic 19th century merry-go-round afterwards. Getting people eating together is another great way to create closer ties – I have very fond memories of group dinners at a number of conferences. At the annual KDE conference Akademy, there is typically a Big Day Out, where people get together for a picnic, some light outdoors activity, a boat ride, some sightseeing or something similar.

Extra costs

Watch out for those unforeseen costs! One conference I was involved in, where the venue was “100% sponsored” left us with a €20,000 bill for labour and equipment costs. Yes, the venue had been sponsored, but setting up tables and chairs, and equipment rental of whiteboards, overhead projectors and so on, had not. At the end of the day, I estimate that we used about 60% of the equipment we paid for.

Conference venues are hugely expensive for everything they provide. Coffee breaks can cost up to $10 per person for a coffee & a few biscuits, bottled water for speakers costs $5 per bottle, and so on.  Rental of an overhead projector and mics for one room for one day can cost €300 or more, depending on whether the venue insists that equipment be operated by their a/v guy or not.

When you’re dealing with a commercial venue, be clear up-front about what you’re paying for.

On-site details

I like conferences that take care of the little details. As a speaker, I like it when someone contacts me before the conference and says they’ll be presenting me, what would I like them to say? It’s reassuring to know that when I arrive there will be a hands-free mic and someone who can help fit it.

Taking care of all of these details needs a gaggle of volunteers, and it needs someone organising them beforehand and during the event. Spend a lot of time talking to the local staff, especially the audio/visual engineers.

In one conference, the a/v guy would switch manually to a screen-saver at the end of a presentation. We had a comical situation during a lightning talk session where after the first speaker, I switched presentations, and while the next presentation showed up on my laptop, we still had the screensaver on the big screen. No-one had talked to the A/V engineer to explain to him the format of the presentation!

So we ended up with 4 Linux engineers looking at the laptop, checking connections and running various Xrandr incantations, trying to get the overhead projector working again! We eventually changed laptops, and the a/v engineer realised what the session was, and all went well after that – most of the people involved ended up blaming my laptop.

Have fun!

Running a conference, or even a smaller meet-up, is time consuming, and consists of a lot of detail work, much of which will never be noticed by attendees. I haven’t even dealt with things like banners and posters, graphic design, dealing with the press, or any of the other joys that come from organising a conference.

The end result is massively rewarding, though. A study I did last year of the GNOME project showed that there is a massive project-wide boost in productivity just after our annual conference, and many of our community members cite the conference as the high point of their year.

Lessons learned

March 11, 2011 community, freesoftware, gnome 119 Comments

After my rather controversial question a few days ago and multiple reactions from around the KDE & Canonical world, a lot of reading and digging into archives, and a lot of conversations with people across the spectrum, I have some preliminary findings and lessons which I hope can serve us going forward to help improve things. Nothing in here is controversial, I think, but each of these is a contributing factor to the current mess we find ourselves in.

tl;dr

For those without the patience to read this article (which is much longer than I intended it to be when I started!), here are the headline points:

1. FreeDesktop.org is broken as a standards body
2. Mark Shuttleworth doesn’t understand how GNOME works
3. GNOME is not easy to understand
4. Deep mistrust has developed between Canonical, GNOME & KDE
5. Difficult people are prominent in each of these projects
6. Behind closed doors conversations are poison
7. For people to work together, they need to be in the same place

In summary, there are a number of things we can do to move forward from where we are now: improve processes & structure for freedesktop.org (this will require buy-in from key GNOME & KDE people), make the operation of GNOME (and the operation of individual modules) more transparent from outside the project, cut out a lot of the back-channel conversations that have been happening over the phone, in person & on IRC, in favour of documented & archived discussions and agreements on mailing lists & wikis, and work to ensure that people working on similar problem areas are talking to each other.

The major challenge we have is how to move beyond the deep mistrust which has evolved between members of our communities, who are all to eager to assume the worst of others, and how we can improve the tone of discourse when some of the most prominent members of our communities are also hard to work with.

Now, to elaborate:

FreeDesktop.org is broken as a standards body

This is not surprising when you consider that it’s written right there on the front page: “freedesktop.org is not a formal standards organization”.

In the case of the StatusNotifier spec, the brokenness shines through. Work started in April 2009 by Aaron Seigo, using the Galago spec as a starting point. Once KDE had begun working on an implementation, Marco Martin started on an initial draft of a spec. The first round draft there was mostly done on September 17 and proposed as the KNotificationItem spec. Then Aurélien Gâteau and Ted Gould made some (offline) suggestions, resulting in a rename, and some revisions, in late October. The spec was proposed as the StatusNotifier spec in December 2009.

At the point that GNOME developers Dan Winship & Mathias Clasen, and Citrix developer Giles Atkinson, reviewed the spec and made comments on it, too much had been invested in it to make major revisions. At that point, it is disingenuous to call StatusNotifier a cross-desktop standard. Hosting a document on the freedesktop.org wiki does not a cross-desktop standard make.

It’s interesting and ironic to see Aaron mention the nascent DConf specification from 2005 in these terms:

instead, the idea was, “If we propose it on fd.o, then people have to accept it because otherwise they won’t be cooperating with fd.o.” this is completely different from trying to work with others and having those efforts ignored.

In fact, that is exactly how StatusNotifiers were perceived (and exactly how Mark & Aaron are messaging GNOME’s non-adoption of the spec).

There is no freedesktop.org process for proposing standards, identifying those which are proposals and those which are de facto implemented, and perhaps more importantly, there is no process for building consensus around a specification, and signalling that consensus.

If I were in charge, I would require every spec to start with a problem definition. Only by agreeing on the problem can we hope to arrive at a solution which will be acceptable to all. The problem statement is the guiding light of a spec. Then I would make sure that the people with an interest in solving the problem were committed to the project. Only then do you start working on a spec and implementations. Without agreement on a problem, and without the right people at the table from the start, the effort is doomed. Some guidance on the process for the creation of a spec would be a start.

In this case, there was no founding problem statement. The spec proposed by Marco Martin listed this as the problems which it was solving:

The new protocol is based upon D-Bus, and separates the presentation of the items from the logic, in our case the painting is completely controlled by Plasma and the applications registers via D-bus (with a small client library shared across KDE) to a central server, while there can be zero or more instances of the systemtray. if either the serve or no instances of systemtrays that supports this protocol are registered the system will fall back using the old freedesktop.org systray specification.

This is not a compelling problem statement. No user ever had a problem because notifications didn’t use D-Bus.

It’s clear when reading Dan Winship’s follow-up comments that there was disagreement on the problem to solve, as well as disagreement on how to solve it. Dan felt that a spec should include policy, and document expected behaviour, while Aaron and Marco were at that point committed to the separation of “the visualisation” and the API. With a better problem statement, this would have been a minor implementation point, without one, two people ended up arguing over positions, and not interests.

If, instead, people had agreed on the problem of the panel or the issues they wanted to address before starting to work on a solution, things might have gone more smoothly. Note that the wiki page linked above was created at the end of December 2009, and the mailing list post was from February 2010 – to communicate what had already been written, not to concentrate people on a common problem.

Mark Shuttleworth really doesn’t know how GNOME works

This one really surprised me, but I think it’s indisputable. Mark wants GNOME to have “strong, mature technical leadership”. He talks about a GNOME cabal, and GNOME’s strategy being “whatever Jon McCann wants to do with the panel”. Mark and others don’t understand why libappindicator was rejected as an external dependency, misunderstanding that external dependencies are, by definition, dependencies of GNOME modules. He admits himself that he has failed to have Canonical developments considered as “internal to Gnome”, and clearly does not understand the position that the GNOME community as a whole has taken with respect to copyright assignment, or the history behind that position.

My understanding of GNOME is this: GNOME does not have technical leadership – it hasn’t had clear technical leadership since, as I understand it, the creation of the GNOME Foundation (at which point, by design, the board was given a mandate to build and define GNOME, and then soon afterwards removed that mandate from itself). The foundation does not now dictate any vision or direction for GNOME.

It can be argued that this is something which should be changed. That change will be effected by people involved in the foundation and the project. It is not enough for Mark to tell the project that “you need leadership”, or Jono Bacon telling foundation members (as he told me in 2007) that they should step up to the plate. Decisions are made by those who turn up – and I consider Mark, Jono, Ted Gould and others as members of the GNOME project, with as much mandate to change GNOME as I do. If Mark wants GNOME to have strong leadership, then he needs to help make that happen.

Given that this is not (yet) how GNOME works, to get things done in GNOME, you need to talk to the right people. That means, defining your problem, and identifying the stakeholders who are also interested in that problem, and working out a solution with them (am I repeating myself?). Mark seems to want GNOME to behave like a company, so that he can get “his people” to talk to “our people” and make it happen. I think that this misunderstanding of how to wield influence within the GNOME project is a key problem.

But then again, over the years I have heard similar feedback from GNOME Mobile participants, and people in Nokia – so it’s not all Mark’s fault. As Jono says here: GNOME does have a reputation of being hard to work with for companies – no point in denying it (then again, so does the kernel, and they seem to get along fine).

GNOME is not easy to understand

When I evaluated GNOME’s governance for Simon Phipps recently, I scored the project 0 (on a scale of -1 to 1) for the criteria of oligarchic governance. The notes from the evaluation were:

Newcomers to GNOME often have trouble figuring out who’s in charge. The Release Team is responsible primarily for the release process and has not traditionally set any strategic direction for GNOME, and individual module governance rules are varied. The foundation board is responsible primarily for maintaining the infrastructure of the foundation, and dealing with sponsors and benefactors, and does not set any technical direction.

Score: Governance is open, membership of the release team oligarchy is meritocratic – scoring zero for oligarchy because much of the governance is devolved to maintainers, making it hard to figure out how to accomplish project-wide change.

Finding the right person inside the GNOME project to help work on a given problem is not straightforward. If you want to make a change to one module, then it’s as simple as working with the maintainers. If, on the other hand, you want to propose a system-wide change, it is a much harder job – you need to work with module maintainers to get them to adopt your proposal, then work with the release team & the wider GNOME community through the module proposal period to get your library included in one of the module sets. Libraries I can think of in recent times that have not gained sufficient traction include Beagle, Geoclue, Soylent, or LeafTag. Other projects like Pyro, GNOME Online Desktop, or Zeitgeist have had baptisms of fire. Even libraries like GStreamer and Telepathy have taken a long time to get traction in core GNOME applications.

Even once you’re in the right place, having work reviewed can take time & effort. I have been told stories of dropped or unreviewed patch-sets by developers I’ve known across a number of projects for many years – one that is mentioned frequently is GNOME Control Center. Maybe persistence was all that would be required, maybe the patches were submitted in a way other than the usual method, or maybe the maintainer was just stuck for time & forgot – in any case, patches were lost, or their integration delayed, and contributors ended up disenfranchised.

But then you can say the same thing about the Linux kernel – contributing to the project is so confusing that Jon Corbet wrote a book about how to contribute, or even KDE – Stuart Jarvis wrote a timely article yesterday explaining that “KDE is not like [a company]. We don’t have leaders. We have prominent community members, but they tend to operate within their own areas of expertise.” Sounds familiar.

The bottom line is that GNOME can improve, but it is not going to change its nature, and working with GNOME needs to be done on the terms of the community you’re working with, not on your terms.

Deep mistrust has developed between Canonical, GNOME and KDE

Regardless of the causes & the history, it’s been made very clear over the past two days that people in our communities are prepared to believe the worst about their fellow free software developers.

Aaron Seigo, for example, clearly has no confidence in GNOME developers as a whole. He writes in a comment that:

@Tom: “do you think the Gnome my way or the high way attitude is connected to company agenda?”

i don’t think so. […] it really seems to be something common to the culture of the project rather than the culture of the companies they work for.

and later:

it’s not a belief that GNOME has decided to not collaborate on this (and other) initiatives for no good reason: it’s a fact. there is a demonstrated “if isn’t invented here, it isn’t used here” pattern of behavior.

Mark clearly believes that GNOME Shell is a Red Hat project. He feels short-changed, feels he and his team have made a good faith effort to engage which has been rejected, offer suggestions which have been ignored. On the other hand, Jon McCann does not see things the same way. And a lot of GNOME people see the move to Unity as a deliberate effort to undermine GNOME Shell, one more in a series of initiatives designed to give Ubuntu differentiation over their competitors without feeding the results into the upstream ecosystem.

Looking at some of the tweets & comments on the various posts, I see an employee of Intel, a developer from the Junta de Andalucia, a number of ex Canonical employees, a Novell employee, an unaffiliated volunteer, and others.Mark’s article blaming GNOME for the problems in the relationship was literally met with “WTF”s and laughter.

Ill will toward Canonical as a company is not limited to GNOME – Greg Kroah Hartman’s infamous presentation at the Linux Plumber’s Conference in 2008 comes to mind. Clearly frustration has been building across the community for a number of years, and it’s far too easy to dismiss it as jealousy because Ubuntu has so many users.

Difficult people are prominent in each of these projects

At this point, the participants in what has become a menage à trois each have a world view which is so different and a prioris so ingrained about the motivations & attitude of everyone associated with another project that undoing the damage will be very difficult.

It’s made even more difficult because a number of key contributors in the projects in question have a reputation of being hard to work with. In GNOME we have our share of people who, to use a phrase Jon Corbet uses to describe kernel hackers, are “not always concerned with showing a high degree of politeness”. I could come up with 10 names of hackers who might themselves be surprised to be on the list, who would be considered by people who have worked with them to be “prickly” to say the least. These people can be found at all levels of the project – prominent on the GNOME Foundation mailing list, maintainers of modules, employees of all of the major companies working with GNOME, even on the release team.

On the KDE side, Aaron also has this reputation – one KDE contributor I spoke to recently said that if Aaron had been a little more open to the feedback he received rather than adopting his “habitual air of superiority” that things might have gone better. And he’s not alone.

Reading the thread on freedesktop.org where the StatusNotifier spec was being discussed, it’s clear that Dan and Matthias considered that Aaron was being dismissive of their concerns – and I can certainly see why. Aaron, on the other hand, in his blog post, considers that “there was a lot of communication about Status Notifiers on the freedesktop.org xdg list where good feedback was offered and the specification improved significantly as a result. So communication really can’t be to blame here, at least not communication by those outside of GNOME” – there is a clear disconnect between how the thread was perceived by Aaron and by other participants.

Mark himself is no angel – I’m sure he will recognise that he is not one to avoid polarising positionn, or to change his mind easily once it’s made up. On the subject of copyright assignment, his mind is made up, for example. There is no revenue motive  behind the decision (I am convinced of this), but on principle, Mark has come to believe that controlling the copyrights to code is a best practice, and nothing will change his mind about that. Similarly, he has made up his mind on GNOME Shell it is “McCann’s” plaything, design suggestions made early in the process were ignored, and even though he now admits that the result is better than the early mock-ups, it is clear to me that there is no chance that Ubuntu will ever voluntarily adopt GNOME Shell.

We have two problems: first, that key figures in our communities can rub people up the wrong way. Second, it’s easy to ascribe to entire groups the characteristics of the people we come in contact with.

To solve the second problem, we need to start using names instead of project names. It’s too easy to ascribe ill-will to an anonymous faceless project, it’s another thing to do so to an individual with a name & a face. GNOME didn’t reject the StatusNotifier spec – two GNOME contributors on the xdg@freedesktop.org mailing list who read the spec (and who were in a position to do something about it) felt that their concerns were getting short-changed, and disengaged. I’d wager that 90% of GNOME project members didn’t even know about xdg before this week. Let’s call each other by our names (and be nice while we’re at it).

Behind closed doors conversations are poison

Another major issue we’ve had is a distinct lack of tracability. It isn’t helped by many Canonical developers using infrastructure which the appropriate GNOME developers don’t – but in fact I don’t care what public forum you use to develop & talk about your software. What is harmful is what I’ve dubbed the Water Cooler – when too many conversations happen in private email, at conferences over drinks, or on IRC, there is no tracability. One example: A key event in the timeline of shell/notifiers/Unity was the UX hackfest in 2008. Different people say different things were said. And there is no written trace of agreements or proposals concerning notifications. I was told that there were some conversations after the conference on IRC, but nothing was sent to a mailing list or recorded in a wiki page.

A member of the GNOME Shell team recently, in response to some questions about design decisions, said that a lot of the discussions & reasons behind the design would have happened on IRC. So there is no trace.

I have personally been involved in dozens of OTR discussions with Canonical people. I recall one where Jono urged me to “step up to the plate” and provide technical leadership for GNOME – my response was that that wasn’t the role of the board, that the distributors who depended on GNOME for their products had to set the lead. It would have been nice to have the discussion in public.

I know that writing stuff down after the fact is a pain. But it is required to allow for a traceability of community decisions and agreements – and also to highlight misunderstandings. If I had a dollar for every time I’ve had a conversation with a client about a technical spec, and he understood something different than me, I’d have enough money for a nice meal. By writing down understandings in clear and unambiguous terms, you are also giving the other party to the discussion the opportunity to correct misunderstandings early.

I also understand that there is an interest in putting on a good face, and not airing your dirty laundry in public (ironic, eh?) – for the past few years, the party line in Canonical has been “We love GNOME, we’re a GNOME shop” while behind the scenes there have been heartfelt conversations about the various problems which exist in GNOME & how to address them. The problem is, because these discussioons happen behind the scenes, they stay there. We never get beyond discussions, agreeing there is a problem, but never working together on a solution.

The party line in GNOME & KDE has been “we’re all pulling in the same direction, we like each other” – and for the most part it’s true. But if this week has shown nothing else, it’s shown that senior members of the KDE & GNOME projects actively mistrust each others motives & don’t believe we have the same interests at heart. At least, this is clear from Aaron Seigo’s comments on his blog post.

For people to work together, they need to be in the same place

I have seen a number of people say that Canonical worked on libappindicator and Unity “internally, not in the open” or that “a lot of design in free software are (sic) developed in secret”. Yet the code is open, the entire history is in Bazaar… how is this consistent?

First, design is not code. Design documents can behave like code, however, with peer review and an iterative process, and can be wedded to the process of developing code, evolving as technical constraints and schedule pressure get in the way of the original design. Good designers work with coders – and this is how it happens in Canonical too.

However, Canonical has occasionally opted to create new projects, housed in Launchpad, rather than engage existing projects to evolve them. libappindicator is an example – several people suggested that it should/could be part of GTK+ – what changed between January 2010 when Ted Gould said ” I’d like to think that the code in libappindicator would useful, and maybe even migrate into a replacement for GtkStatusIcon in GTK+” and February 2010 when he wrote ” Q: Shouldn’t this be in GTK+?
A: Apparently not”?

Canonical has a policy that Canonical development is done in Launchpad, using Bazaar. Sometimes that’s fine – if you’re originating a project, then you get to choose the infrastructure. Bazaar & Launchpad are working just fine for a plethora of projects. But when you are working with other projects, you need to be where they are.

Cody Russell, long-time GNOME contributor, former Canonical developer, and the developer of client side decorations for GTK+ among other things, wrote in a comment to Aaron’s blog:

CSD is really not a good example of how stuff development between Canonical and GNOME should work. I’m the person at Canonical who started CSD, but never finished it.

It started as just an experimental hack, and somehow got picked up as a “Canonical project”. Once that happened my immediate manager told me to stop committing code to GNOME git and do any further work on it privately in bzr.

For me this made developing it further much more difficult, because it was an extremely large and intrusive change into GTK+ source code and my manager didn’t want upstream developers to help me with at least peer code review.

Apparently there was originally some desire to have libappindicator developed as part of GTK+. I don’t know why this did not happen, but perhaps the quote above can give some insight into why the project was developed as an independent module.

Similarly, having a discussion on a freedesktop.org list does not ensure that you are getting appropriate cross-platform buy-in for your ideas. There is no guarantee that you are talking to the right people.

Most free software developers I know are on lots of mailing lists, and for all but a small number directly related to their day-to-day work, they glance over them. I certainly fall into this camp – there are about half a dozen lists I’m on where I will open maybe only 1 in 10 emails, with a subject that looks like it might concern me directly. If you want to directly get my attention outside of that, a personal email, IRC ping or IM asking me to comment on something is a good way to get my attention.

In the early days of freedesktop.org, this is how things worked. There were well defined problems that needed solving, and the people concerned made a conscious effort to get the right people into a central desktop-agnostic mailing list. As time goes on, maintainerships evolve, people change jobs, new people arrive – there is no longer any guarantee that the people on the freedesktop.org mailing lists are the best people to be talking to.

Moving on

So where do we go from here? Well, first GNOME 3. We have a release coming up, and so does Ubuntu, and we’re both going to have a bumpy ride for the next few months, so that is presumably going to be the priority for everyone.

After that, the Desktop Summit will be an opportunity to start building bridges. We’ve made an effort this year to avoid tribalism in the conference, by framing the call for papers according to problem area (multimedia, mobile, platform, etc.) rather than by desktop. We will be continuing this, I’m sure, through paper selection and drafting the agenda. That said, you can bring a horse to water, but you can’t make him drink.

Looking through the list of headings here, a number of them are easily fixable, a number of them are much more troubling, and a result of letting discontent fester for months or years.

We can certainly improve the operation of freedesktop.org – currently there is no freedesktop.org as such, it’s a wiki & a mailing list server. To improve it, there needs to be a process whereby things are agreed, and a way to ensure that all concerned parties are engaged in that process. There were discussions about this in Gran Canaria, including members of both the GNOME and KDE projects. But to evolve freedesktop.org, buy-in of a number of key GNOME developers is essential – I can’t imagine any long-term changes happening without Owen Taylor’s agreement, for example.

We can increase the transparency of the operation of individual GNOME modules. This is one of the things I hoped to achieve last year with the GNOME census. By identifying the key contributors for each module, and the processes under which each module operates, we can help reduce the friction when people try to figure out how to work with GNOME. Ideally, something like Jon’s guide to the kernel will help reduce the number of dropped or unreviewed patches, make it easier for people to see what kinds of contributions would actually be welcomed by module maintainer teams, and help people figure out how to gain influence in a specific module and eventually become a maintainer themselves.

We must reduce the amount of back-channel discussions between the various project participants. Any important decisions or agreements that happen off-line must get written down & agreed to after the fact. IRC usage has become predominant in some teams, resulting in a lack of transparency of operation – GNOME should adopt the Apache policy of “if it didn’t happen on the mailing list, it didn’t happen” and encourage companies who want to encourage change in the project to do the same. I would appreciate all participants committing to a general policy of releasing design specs and code early to peer review – and in the case of Canonical, working upstream before working in their own distribution.

I think there is a potential for a GNOME Design group, for example, with qualified designers in a publicly archived, but invitation-only, mailing list, to allow design collaboration without a high level of poor quality amateur participation which has typified public usability or design lists.

Finally, smaller, focused teams, started on a case by case basis, will serve us better than long-living “collaboration” mailing lists like desktop-architects or xdg. To ensure that the right people are at the table, they need to be invited, and their presence needs to be documented, on a project-by-project basis. Of course discussions on these lists should be publicly archived, but they should only be useful as long as the specific problem area is being addressed, and should die a natural death afterwards.

That’s the easy stuff.

The more difficult issue is that we have allowed relationships to degrade so far. It feels like GNOME & Canonical are in a bad relationship – we used to love each other, and now every time we talk it seems like we’re speaking a different language. For a while, it seemed like GNOME & KDE contributors were working productively together & overcoming some of the historical issues between the projects, but over the past 3 years, it’s been clear that the progress we had achieved was illusory, and that deep-seated ill-feeling among a small number of project leaders have ensured that any early progress has been squandered.

In addition, all of GNOME, KDE & Canonical have allowed personality issues to build up. One need only follow the discussions within the GNOME foundation concerning the Code of Conduct to see that the GNOME community has allowed some loud & confrontational characters to gain positions of authority in the project, and KDE is also no stranger to such personality issues among prominent developers.

Solving this problem is much more difficult, if it’s solvable at all. Change inside the GNOME project can only come from the grass roots, and the same goes for KDE. Adopting a code of conduct is less important than actually being nice. Too many people confusing being rude and abrasive as being terse and efficient. And getting a critical mass of community leaders in the same place at the same time to concentrate on common issues and approaches to solving them is difficult when there is so much pent up frustration and ill-will involved.

The Desktop Summit will be an important meeting point this year, where hopefully some of these issues can be resolved. In the meantime, I hope that we can start some small conversations soon to get people talking and trusting each other’s motives again. It will be a long and arduous process, and will require everyone to accept part of the blame for the situation we find ourselves in, and to accept the better days are possible.

As I said to Jono Bacon yesterday when he suggested we should all just get along & stop digging up the past: “Those who ignore the past are doomed to repeat it. And we have been failing for some time to understand the issues which people have working in freedesktop.org, or with GNOME, KDE or Canonical”. Unlike some people commenting on the various blog posts this week, I think that getting some of the dirty laundry out into the open will be beneficial to the general working environment. Sunlight is a great disinfectant, they say, and a number of issues have been kept under wraps too long by people who want to put a brave face on & pretend for public benefit that everything is rosy.

Well, everything isn’t rosy, and now even a fool could see that. But I don’t think there’s anything broken that we can’t fix. Let’s concentrate on getting GNOME 3.0 and Ubuntu 11.04 out the door, and then get to work mending relationships.

Has GNOME rejected Canonical help?

March 7, 2011 community, freesoftware, gnome 74 Comments

Through the fall-out from the Unity decision, and now the fall-out from the packaging of Banshee on Natty, I have repeatedly read Canonical & Ubuntu people say “We offered our help to GNOME, and they didn’t want it”.

Exhibit #1:

For starters, some people in the GNOME community moan about how Ubuntu doesn’t pull its weight upstream.They then make it difficult for Ubuntu-y folks to contribute things upstream.

Exhibit #2:

For the app indicators we also had a lot of community involvement, it was based on a Freedesktop.org spec, worked on with consultancy from KDE, we invited GNOME developers to participate in the Freedesktop discussion and proposed them to the GNOME community for inclusion, but it’s not up to us, if they take it or not

Exhibit #3:

Where tensions between Canonical and GNOME have occurred, according to Bacon, is in Canonical’s desktop innovations for improved usability, such as the Ayatana indicators for sound and social media, and the new Unity desktop, all of which were submitted to GNOME and rejected, leaving Canonical to develop them outside the GNOME project. […] Asked whether Canonical could have developed its usability modifications within GNOME, he replies, “To be honest with you, I don’t think it could have been done. The fact that nothing’s been accepted is a pretty reasonable indicator that the two projects have widely different directions.”

Exhibit #4:

We committed to build Unity […] because we had ample reason to believe that the trajectory of the alternatives was going to fail. And it did fail – Gnome 3 looks much more like the vision we painted with Unity than the original vision […] I am sorry that a few Gnome leaders have blocked Gnome’s adoption of Unity API’s, and the stress that will cause, but I feel proud that we had the guts, and the capacity, to design and deliver something wonderful.

I have seen and heard this mentioned by others too, but cannot find any others right now – additional pointers in comments would be welcome!

So – given that GNOME is a project which scores very highly as being Open By Rule (disclosure: I put together the evaluation of GNOME for Simon), I thought I would go back through the archives and see how true this was.

Looking at what was actually proposed for inclusion in GNOME from Ayatana work, libappindicator was rejected because (quoting directly from the release team’s decision):

• it doesn’t integrate with gnome-shell
• probably depends on GtkApplication, and would need integration in GTK+ itself
• we wished there was some constructive discussion around it, pushed by the libappindicator developers; but it didn’t happen
• there’s nothing in GNOME needing it

I went back to see where the discussion happened for the libappindicator proposal. There was a discussion, some over & back, Ted was (as usual) forthcoming & helpful, and things appeared to be moving approximately in the right direction. There were some issues over copyright assignment, and the discussion petered out. No feedback I could see from the GNOME Shell team – positive or negative – to depending on the library.

Now, this is hardly ideal. I would love to see debate on why there wasn’t a more in-depth debate on using libappindicator in GNOME Shell. Was this ever proposed? If so, where? I can’t find any reference. Was there any reaction other than “we don’t think it’s an issue” to the copyright assignment issue? Perhaps there was a lot more discussion in another forum that I haven’t linked to – on the release-team list, on IRC, or elsewhere? Comments, please!

I would love to point to other instances of work which has been proposed upstream from Canonical and which has been rejected, but my (admittedly, brief) search has not turned up much useful stuff. I can’t find any online reference to displeasure with the GNOME Shell vision, or proposals of alternatives, nor can I find situations of “Paper Cut” patches being rejected because they were from Canonical or Ubuntu. In fact, the one reference I found to the UX hackfest in 2008 from Mark seemed quite positive about the whole thing.

There are apocryphal stories about patches submitted twice by different people before they were accepted, other stories about people being “impossible to work with”, design feedback being ignored, and more – I would love to see some evidence of this, or some documented criticism from 2008 of some of the GNOME Shell design documents. I hear often that some of the design decisions were unacceptable, but ask which ones, where the discussion took place, or how much effort was spent trying to get things changed, and hand-wavy “lots of stuff” type answers is what you get back.

I would really like to shed some sunlight on this – if we do not have publicly archived references to places where these disagreements have happened, then there are a couple of possible conclusions we can draw: either insufficient effort was made to collaborate, or the effort was made, and GNOME Shell is not sufficiently transparent for the developers and designers to be accountable.

So please – pile in on the comments. I want to know of instances when GNOME has (allegedly) refused contributions or help from Canonical, with links to Bugzilla, mailing lists, even IRC logs or wiki pages. Let’s get to the bottom of this & see if we can’t solve the problem.

Updated to clarify that the reasons for rejecting libappindicator were not mine, but were copied from the release team decisions, after reading Aaron Seigo’s response

Drawing up a roadmap

February 7, 2011 community, freesoftware, General, gimp, gnome, maemo, openwengo, work 6 Comments

One of the most important documents a project can have is some kind of elaboration of what the maintainers want to see happen in the future. This is the concrete expression of the project vision – it allows people to adhere to the vision, and gives them the opportunity to contribute to its realisation. This is the document I’ll be calling a roadmap.

Sometimes the word “roadmap” is used to talk about other things, like branching strategies and release schedules. To me, a release schedule and a roadmap are related, but different documents. Releasing is about ensuring users get to use what you make. The roadmap is your guiding light, the beacon at the end of the road that lets you know what you’re making, and why.

Too many projects fall into the trap of having occasional roadmap planning processes, and then posting a mighty document which stays, unchanged, until the next time the planning process gets done. Roadmaps like these end up being historical documents – a shining example of how aspirations get lost along the way of product development.

Other projects are under-ambitious. Either there is no roadmap at all, in which case the business as usual of making software takes over – developers are interrupt-driven, fixing bugs, taking care of user requests, and never taking a step back to look at the bigger picture. Or your roadmap is something you use to track tasks which are already underway, a list of the features which developers are working on right now. It’s like walking in a forest at night with a head-light – you are always looking at your feet avoiding tree-roots, yet you have no idea where you’re going.

When we drew up the roadmap for the GIMP for versions 2.0 and 2.2 in 2003, we committed some of these mistakes. By observing some projects like Inkscape (which has a history of excellent roadmapping) and learning from our mistakes, I came up with a different method which we applied to the WengoPhone from OpenWengo in 2006, and which served us well (until the project became QuteCom, at least). Here are some of the techniques I learned, which I hope will be useful to others.

Time or features?

One question with roadmaps is whether hitting a date for release should be included as an objective. Even though I’ve said that release plans and roadmaps are different documents, I think it is important to set realistic target dates on way-points. Having a calendar in front of you allows you to keep people focussed on the path, and avoid falling into the trap of implementing one small feature that isn’t part of your release criteria. Pure time-based releases, with no features associated, don’t quite work either. The end result is often quite tepid, a product of the release process rather than any design by a core team.

I like Joel’s scheduling technique: “If you have a bunch of wood blocks, and you can’t fit them into a box, you have two choices: get a bigger box, or remove some blocks.” That is, you can mix a time-based and feature-based schedule. You plan features, giving each one a priority. You start at the top and work your way down the list. At the feature freeze date, you run a project review. If a feature is finished, or will be finished (at a sufficient quality level) in time for release, it’s in. If it won’t realistically be finished in time for the release date, it’s bumped. That way, you stick to your schedule (mostly), and there is a motivation to start working on the biggest wood blocks (the most important features) first.

A recent article on lessons learned over years of Bugzilla development by Max Kanat-Alexander made an interesting suggestion which makes a lot of sense to me – at the point you decide to feature freeze and bump features, it may be better to create a release branch for stabilisation work, and allow the trunk to continue in active development. The potential cost of this is a duplication of work merging unfinished features and bug fixes into both branches, the advantage is it allows someone to continue working on a bumped feature while the team as a whole works towards the stable release.

Near term, mid term, long term

The Inkscape roadmap from 2005 is a thing of beauty. The roadmap mixes beautifully long-term goals with short-term planning. Each release has a by-line, a set of one or two things which are the main focus of the release. Some releases are purely focussed on quality. Others include important features. The whole thing feels planned. There is a vision.

But as you come closer and closer to the current work, the plans get broken down, itemised further. The BHAGs of a release in 2 years gets turned into a list of sub-features when it’s one year away, and each of those features gets broken down further as a developer starts planning and working on it.

The fractal geometer in me identifies this as a scaling phenomenon – coding software is like zooming in to a coastline and measuring its length. The value you get when measuring with a 1km long ruler is not the same as with a 1m ruler. And as you get closer and closer to writing code, you also need to break down bigger tasks into smaller tasks, and smaller tasks into object design, then coding the actual objects and methods. Giving your roadmap this sense of scope allows you to look up and see in the distance every now and again.

Keep it accurate

A roadmap is a living document. The best reason to go into no detail at all for  future releases beyond specifying a theme is that you have no idea yet how long things will take to do when you get there. If you load up the next version with features, you’re probably aiming for a long death-march in the project team.

The inaccurate roadmap is an object of ridicule, and a motivation killer. If it becomes clear that you’re not going to make a date, change the date (and all the other dates in consequence). That might also be a sign that the team has over-committed for the release, and an opportunity to bump some features.

Leave some empty seats

In community projects, new contributors often arrive who would like to work on features, but they don’t know where to start. There is an in-place core team who are claiming features for the next release left & right, and the new guy doesn’t know what to do. “Fix some bugs” or “do some documentation” are common answers for many projects including GNOME (with the gnome-love keyword in Bugzilla) and LibreOffice (with the easy hacks list). Indeed, these do allow you to get to know the project.

But, as has often been said, developers like to develop features, and sometimes it can be really hard what features are important to the core team. This is especially true with commercial software developers. The roadmap can help.

In any given release, you can include some high priority features – stuff that you would love to see happen – and explicitly marked as “Not taken by the core team”. It should be clear that patches of a sufficiently high standard implementing the feature would be gratefully accepted. This won’t automatically change a new developer into a coding ninja, nor will it prevent an ambitious hacker from biting off more than he can chew, but it will give experienced developers an easy way to prove themselves and earn their place in the core team, and it will also provide some great opportunities for mentoring programs like the Google Summer of Code.

The Subversion roadmap, recently updated by the core team, is another example of best practice in this area. In addition to a mixed features & time based release cycle, they maintain a roadmap which has key goals for a release, but also includes a separate list of high priority features.

The end result: Visibility

The end result of a good roadmap process is that your users know where they stand, more or less, at any given time. Your developers know where you want to take the project, and can see opportunities to contribute. Your core team knows what the release criteria for the next release are, and you have agreed together mid-term and long-term goals for the project that express your common vision. As maintainer, you have a powerful tool to explain your decisions and align your community around your ideas. A good roadmap is the fertile soil on which your developer community will grow.

The Lifecycle of a Patch (or: Working Upstream)

January 14, 2011 community, freesoftware, gimp, gnome, inkscape, maemo, meego 5 Comments

Reposted from Neary Consulting

Yesterday I looked into what it means to be a maintainer of a package. Today, I’m going to examine how to affect change in a distribution like MeeGo, and what it means to work upstream. To do so, we’re going to look at how code gets from a developer’s brain into the hands of a user.

So – how can you make a change in a Linux-based distribution? Here’s what happens when everything works as it should:

1. You open a bug report for the feature against your distribution
2. You identify the module or modules you need to change to implement the new feature
3. You open bug reports for each of the modules concerned, detailing the feature and the changes needed in that module for the feature
4. You write a patch to implement the feature, and propose it (appropriately cut up for ease of review) to the maintainers of those modules
5. Once the code has gone through the appropriate review process, it will be committed to the source control of the module(s)
6. Some time later, the maintainer of each module will include that code in a stable release of the module
7. Some time after that, the new stable versions will be packaged and uploaded to MeeGo
8. Your code will be included in the next release of the distribution following the upload.

When people talk about “working upstream” in MeeGo or Linaro, this is what they mean.

To simplify matters for our analysis, let’s consider that the feature we want to implement is self-contained in one module (or related modules which release together). There are two different scenarios we’ll consider:

1. The module is maintained by people not associated with your distribution (for example, a GNU or GNOME project)
2. The module is maintained by people closely related to your distribution (for example, Unity in Ubuntu, or oFono in MeeGo)

We will also look at a third situation, where you find and fix a bug in the software you are using – that is, a released version of a distribution (the proverbial “scratching an itch”).

For each case, I will try to pick a representative feature/patch and follow it from developer through to distribution to Real Users.

What if your code changes different projects?

If your code touches several modules (for example, if you are proposing some new API in GTK+ which you want to use in the GIMP) then things can get complicated – you will need a stable version of GTK+ to be released before you can ship a stable release of the GIMP which depends on it.

This issue of staggered releases is one that Andrew Cowie pointed out a few years ago for language bindings. To avoid making bindings on shifting sands, he preferred to package new APIs once they had been included in a stable GNOME release. In turn, Java GNOME developers rarely depend on development release bindings, and they would wait for the new API to be included in a stable bindings release. For example, the gtk_orientable_get_orientation, added to GTK+ at the end of September 2008, was released in GTK+ 2.16, in March 2009. The first version of Java-GNOME which depended on GTK+ 2.16 was version 4.0.13, released in August 2009. That was packaged in distributions in Autumn 2009, and so most users would not have access to the newer bindings for a few months after that – perhaps early 2010 – at which point, the API was written 18 months beforehand.

And that is when you have a regular release schedule you can rely on! Pity the developer who wants to release a GIMP plug-in which depends on some API included in GIMP 2.8 – the last stable GIMP release, 2.6, came out in October 2008, and over two years later, 2.8 still has not released. And when you combine unreliable release schedules for distributions and applications, the results are cumulative: users of the stable Debian distribution are still using GIMP 2.4 releases. The GIMP 2.4 released in October 2007. Features added to the GIMP in late 2007 are still not in the hands of users of stable Debian distributions.

Getting features to users

It is difficult to generalise when users upgrade their Linux distributions, or even to say what proportion of Linux users are new users at any given time. It would be over-simplifying to say that developers use bleeding-edge distributions, power users upgrade early to the latest and greatest, new users install the latest distributions available, but will only upgrade every 18 months or so afterwards, and conservative users stick with “Long term service” or stable distributions. Most developers I know use their computer for work (and thus want a stable distribution) and only install the latest versions of various dependencies they need to work on their project. But let’s generalise and say that this is roughly the case. So (guesstimating) about 10% of your users will be upgrading to the latest distribution very quickly after its release, a further 20% in the months after when the bugs are shaken out, and the rest will follow along in their own time, perhaps 12 or 18 months later.

To make this concrete, let’s follow the life of a single patch. This is complete anecdata, but in my defence, the patch has been chosen by random, from a project which I know has good community processes and release management in place. The patch we’re going to follow adds an extension to Inkscape to render objects along triangular paths.

1. Bug #226001 opened on 2008-05-03 by inductiveload, with a description of the feature to be added, and proposed code to implement it. The code, as an extension, may have a lower bar for acceptance than code which is core to a project.
2. Patch submission reviewed on 2008-05-03, minor comments, but patch is accepted (note: This was not the authors first submission to Inkscape)
3. Patch corrected to respond to comments and committed on 2008-05-03 (did I mention these guys had good community processes!?!)
4. Inkscape 0.47-pre0, containing the Triangle extension, released on 2009-07-02
5. Inkscape 0.47-pre4 included in Ubuntu 9.10

So for a feature developed in mid 2008, most Inkscape users will still not have the feature by the end of 2009, 18 months later. This is both a typical and atypical example: in many projects, patch proposals lay unreviewed for days, weeks, sometimes months, but the 0.47 release cycle was a particularly long one for Inkscape. However, I think the lag from code written to presence on user’s hard drives of ~12 to 18 months is about correct.

Does it have to be this hard?

If this were the only way to get features into a distribution, trying to improve MeeGo by contributing upstream would be a very frustrating experience. Happily, there are ways to accelerate the process. Taking the MeeGo kernel as an example, where Greg Kroah-Hartman recently threw in the towel on persuading people to propose patches upstream; the process is supposed to work like this:

1. Propose a patch for inclusion upstream. This patch will then ship in a future stable kernel release (let’s say 2.6.38).
2. After peer review, when the code has been accepted for inclusion in the kernel upstream, propose a backport for inclusion in the MeeGo kernel. The back-ported patch will be maintained across the next MeeGo release, and will be dropped when the kernel version included in the MeeGo project catches up with 2.6.38.

The overhead here is reduced basically to the peer review process of the upstream project, and the cumulative cost of merging a patch over the course of 6 months.

As a distributor (or a developer working on a specific distribution), this allows you to get code to everyone, eventually, and have that code included in your distribution as soon as you are sure that it is up to the standard expected by the community. Currently in MeeGo, the trend seems to be more towards submitting patches concurrently upstream and to MeeGo kernel maintainers (or even submitting them upstream once they have been accepted into the MeeGo kernel). In the case that a patch requires substantial modifications, or is rejected outright, upstream, the kernel maintainers are then left carrying a patch indefinitely in the distribution. For one patch, this might not be a big deal, but for thousands of patches, the maintenance and integration burden of these patches adds up.

It is also not unusual for kernel developers to maintain their own git branches for a long time. Three examples that come to mind are inotify, which Robert Love maintained for over a year for both Novell and in the kernel before it was accepted into the mainline, ReiserFS, which was maintained for several years out-of-tree before being shipped with the Linux kernel in 2001, and the fast  desktop patchset which Con Kolivas maintained for almost five years on the -ck kernel branch. Distributions will occasionally ship a substantial diff to upstream if there is a maintainer committed to getting the code upstream eventually. Allocating someone to work over a long period to make everyone happy and comfortable with your code may enable you to ship a big patch to upstream, but this will not be sustainable long term.

To summarise: when working upstream, as a distribution, you should only ship with patches which have been accepted in a development version of upstream already, if you can help it.

Meetings in telephone boxes

Sometimes, however, when upstream and downstream coincide, you can simplify things considerably, while also adding a small measure of risk.

In MeeGo, to continue with that example, the distribution architects have a pretty good idea when they can expect emergency telephony to be ready for oFono and the MeeGo telephony stack, because they’re writing it. By co-ordinating the upstream release management with downstream packaging, you can make promises as a distribution which you can’t with community-developed software.

When upstream and downstream are co-ordinating each other, we cut out the middleman. The workflow becomes:

1. Report a bug/feature request against a component of the distribution
2. Develop a patch which implements the feature, and submit it directly to the distribution bug tracker
3. Once it has been reviewed and accepted, you know that your patch will be included in the next version of the distribution.

This gives a distribution much more control, both over what gets done, and when, and explains both the Ayatana and MeeGo UX development projects. However, being able to plan around the release is no guarantee that the release will happen on time: GNOME has in the past been stung by planning during the 2.6 development cycle to depend on a new version of GTK+, only to find that the release was delayed. In the end, the GTK+ release shipped in time for the 2.6 release at the end of March.

Scratch scratch

The other patch lifecycle I’d like to mention, because it is so relevant to distributions, was pointed out to me by Federico Mena Quintero yesterday. What happens to a patch that someone makes and submits to a distribution when they find a bug in stable released software? This is one of the key advantages of free software – if you find a bug in the software you use, and you have the wherewithall, you can fix the bug and share that fix with everyone else.

However, as we have seen, there is typically a lag of several months from the time that software is released and the time it is being used by large numbers of users through distributions. With releases of Red Hat Enterprise Linux, Novell Suse Linux Desktop and Ubuntu LTS being supported for up to 5 years, it is possible that important bugs will be fixed in these stable versions for years after the original developers have moved on, and are no longer maintaining older stable versions.

Let’s say I find and fix a bug in Rhythmbox 0.12.5, which ships with Ubuntu 9.10. I open a bug report on Launchpad, attach a fix to the source .deb there, and I update my local copy. As a user, I’m happy – I have fixed my problem and shared the solution with others. If I’m particularly conscientious, I might open a bug on gnome.org against Rhythmbox and attach my patch there, but since the development version is now 0.13.2, the best you can hope for is that the patch applies cleanly to the master branch, and will be included in the next release. It is very unlikely that the upstream maintainers will release another update to the 0.12 series at this point.

Now imagine that you are a maintainer for Suse, and someone reports the same bug against a long-term service release.In practice, there are several different versions being maintained by different distributions, and no good way to know if the same bug has been reported and fixed by someone else. You end up searching for a fix in upstream bug trackers, and in the bug trackers of each of the other main distributions. According to Federico at the time:

Patches for old versions are traded in the black market. You have friends in another distro? You ask them first, “did you guys already fix this?” Those patches don’t ever manage to reach CVS, where everyone would be able to get them.

Ideally, you could collaborate ahead of time with other distributions to ensure that you are all using the same branch of upstream modules, and are committing patches upstream. The Linux kernel is moving to this model, and there are also discussions underway in GNOME to co-ordinate this type of activity. Mark Shuttleworth has also pushed for something similar by encouraging projects in the core Linux platform to have a regular cadence of releases, so that everyone can synchronise their longer term service offerings every couple of years.

But at the moment, the best you can hope for is that your patch will be included in an upcoming release for your distribution, and which point other users of the distro can avail of it, and that upstream will patch their development version and latest stable versions, and get your patch to everyone in a few months.

Working upstream

The goal of this article is to explain what working upstream actually means, and how to make that more palatable for a distribution that wants to get features written and included in their next release. Hopefully, by pointing out some of the shortcomings of the way patches circulate from developers to users, some of these issues can be addressed.

In any case, one thing is clear – if you are carrying a patch as a distribution without ever submitting it upstream, you are making a costly mistake. You will be carrying code that others won’t, and bearing all of the merge and maintenance burden for that code for years to come. The path to maximum happiness is to co-ordinate with other distributions and with upstream to ensure that everyone is working in the same place, and sharing work as much as possible.