Safe as Milk

Increasing Ecosystem Co-operation

December 20, 2008 10:46 pm Dave Neary community, freesoftware, General, gnome, marketing, work

Reposting from Neary Consulting: This is an article accompanying the presentation I gave at MAPOS 08 in London on December 9th 2008.

Moving the Mobile industry from purchasing to co-development in free software communities

Recently, Matt Aslett wrote an article about the way that attitudes to free software evolve over time within a company, using a graphic he got from the Eclipse Foundation, based on some Nortel funded research. Software sneaks in on the ground floor, going from simple use of components to a real understanding of community-driven development, resulting, long-term, in building free software projects and strategies.

Matt sees an evolution in attitudes as the software and its value is discovered at different levels of the organisation, before finally the business development side of the company picks up the ball and drives free software into the heart of the company’s product strategy.

I have also seen this learning process in action, but I would express it differently. People discover the value of the freedoms granted by free software one by one, more or less independently of their level in an organisation – exploring each freedom before discovering its limitations, and thus discovering the value of the next freedom, and qualifying for the next level.

The core freedoms  in the Free Software Definition which are granted to the user of free software are:

1. Freedom to use
2. Freedom to modify
3. Freedom to share, freedom to redistribute
4. Freedom to participate

As companies start to integrate free software components into their products, they discover the value of these freedoms one by one.

Use

The first thing that people see about free software is FREE! As in zero cost. The days when companies reject a product out of hand because they don’t have to pay for it are gone – Linux, OpenOffice.org, Apache, Red Hat and a plethora of other “free” products have proven themselves in the marketplace, and companies are now prepared to allow free software components into their solutions, after appropriate consideration of the licences involved.

To quote one attendee at MAPOS 08, “why would I want to write a compression library, when I can download the best one in the world from zlib.org?” In the area of specialised components for secure communications, compression/decompression, a commodity kernel, and a bunch of other situations, it is appropriate to use free software components off-the-shelf. We expect them to work, and we don’t expect to ever need to talk to the maintainer.

Free software components are in use like this in thousands of systems solutions and commercial products, often without their authors even being aware of it. The main advantage of this for a systems or product company is a saving of time and money, through having a fully functional component without having to go through a purchasing process, and a reduced software bill of materials. An additional advantage is the simplification of your licensing due diligence, thanks to the relatively well-understood consequences of the various popular free software licences.

The difficulty arises when the software doesn’t meet your needs. In many cases, libraries are written by an individual to scratch an itch – it works for him, but is not quite up to your requirements. As one friend of mine put it: “Open Source: 80% as good as the last guy needed it to be”.

Perhaps it’s software that works on 32 bit platforms, but has never been tested for 64 bit. Perhaps it has not been ported to ARM or MIPS. Or perhaps the author simply never imagined that anyone would want the feature which you find indispensable.

In this situation, you can always ask the software author to write the feature or fix the bug for you – but since there is no client/supplier relationship between you, it is entirely reasonable for a volunteer to put your request on the long finger, or reject it outright.

At this point, you realise the value of having the source code – you can modify the software to meet your needs, or pay someone else to do it for you.

Modify

Being able to modify software that doesn’t quite meet your needs is amazing. This is the way things used to work by default, but the shrink-wrapped software revolution of the 1980s got everyone used to the idea that software was a valuable asset to be protected from public view at all costs. When I worked for Informix in the late ’90s, we used to refer to the source code of our leading product as “the crown jewels”.

With the widespread acceptance of free software as an alternative, developers are no longer surprised when they may see how a program works, and change its behaviour. This ability brings two important and immediate benefits – you have control of the behaviour of the software, and you can adapt it to suit exactly your needs. The old choice of build vs buy has become: build vs buy vs extend.

This situation is common in software services companies which provide vertically integrated “solutions” to corporate clients. You take components where you can find them to speed up initial development, stick everything together with duct-tape, hack whatever you need in whatever libraries you’re using to make everything pass the client’s integration tests, and then publish a set of .tar.gz files somewhere on the website of the company to fulfil any licensing requirements.

This control and ability to tailor a solution comes at a price, however. Over and above the cost of making the changes, your team is lumbered with a maintenance problem. Let’s say that implementing the features you need on top of a component the first time round takes a month. Fixing bugs in the features when it has been rolled out can take another few weeks. A few months later, the upstream product you’re based on goes and releases a shiny new version, with lots of compelling new features that you really want.

The cost of integrating your features into the newer version, and doing extensive regression testing before rolling out the new version, might take you another 6 weeks. It is not unusual for time spent integrating your work into later versions to quickly outweigh initial development time and investment. Inconveniently, this is typically effort which is not budgeted for beforehand.

After a company has run into this problem a couple of times, over the course of a year or two, someone will usually suggest that you propose that the features you have developed be sent upstream to the projects you work with – if the feature is accepted, you have solved your maintenance problem, it will be in all future releases of the project, and all of that tricky integration work and regression testing work will get done upstream, as part of normal maintenance.

Redistribute

And so you tell your star hacker Jack that he has two weeks to get your 5,000 line patch down to manageable size by getting your work integrated upstream. (when I said this at MAPOS, no-one laughed – so maybe this does not sound as ridiculous as I thought it did).

He diligently goes to work, cleaning up his code, getting rid of all the warnings, spliting up the big diff into small manageable chunks, creating accounts in 10 different bug trackers, signing up to a dozen mailing lists, creating 47 bugs with terse descriptions, attaching proposed bug fixes, and for major features he sends email telling people that the feature is there and asking for review.

By the end of a frantic month, two weeks more than he was given, he reckons that if everything he’s submitted is accepted, your 5,000 patch will be down to a more manageable 2,000 line patch.

What happens next is… underwhelming.

Major features and bug fixes lie unreviewed for weeks or months. Those that are reviewed need changes which take time and effort. Some patches are rejected outright because they’re too big and the feature is difficult to review.

A post mortem analysis of the project of “giving back to the community” might identify some of the following conclusions:

• Not enough time and resources were devoted to advocating your changes upstream
• Personal relationships between Jack and the project maintainers led to a much higher acceptance rate for patches and feature requests
• The projects were initially evaluated on technical grounds, no thought was given to the developer community underpinning it
• In some cases, maintainers priorities were ill-understood

There are two common conclusions that people make from this kind of analysis;

1. It’s not worth it. They don’t want our work, and the time we’re spending is costing us more than maintaining out-of-tree patches
2. Perhaps if you had engaged with the projects before modifying them heavily, or had been regularly sending contributions, that the maintainers would have been more encouraging, and might have been more prepared to consider your work. If someone from your company was a maintainer or committer already, you would have had a valuable short-cut to getting your agenda implemented in the upstream project.

If you choose door number 1, you will go no further in your quest to really understanding free software processes. This is a reasonable thing to do, but the costs involved are often miscalculated. In addition, the benefits of influencing upstream projects are often vastly underestimated.

If you choose door number 2, you have concluded, in short, that it is madness to include a component in one of your products and exert no influence with upstream projects.

Participate

To have influence, you must understand how the community around a project works. Someone within the team must become an active, trusted member of the community. Once they have gained the trust of the community through their contributions, there may be some procedure to follow for them to become a maintainer of the project, or to gain commit privileges.

These considerations are not technical, for the most part. Friendship and trust are fuzzy human concepts. And this more than anything else brings me to my final point.

Community is hard

For a start, every community is different. They all have different people, different behavioural norms, different dynamics, different forums for communication.

Taking GNOME Mobile as an example, there are 18 projects in the GNOME Mobile platform, with another 10 or so in incubation. Within that, we have a large number of projects housed on gnome.org, and governed by our rules, procedures and conventions. And yet each project has its own set of maintainers – GTK+ is maintained by a committee of around 10 people, EDS is maintained principally by Novell employees, gtkmm has one core maintainer, and so on.

On top of this are a number of freedesktop.org projects, and a couple more which are not under either of these umbrellas. To be an effective influencer of GNOME Mobile, you need to learn the culture of over 20 projects, of wildly varying sizes and baggage.

There are a number of issues to bear in mind when you approach a free software community for the first time. The main one is that while the vast majority of projects think that they are welcoming people with open arms and are very welcoming, if you are a stranger to their land, it is very likely that you will be getting exactly the opposite message.

In some cases, the extent of the welcome is “go and read wiki page telling people how to contribute to the project”. In other cases, no wiki page exists. Occasionally, you will be told that you’re asking your question on the wrong mailing list, or in the wrong way, or that you should read the relevant documentation first. It is not unusual for people to answer questions with a very terse answer – perhaps a link to a mailing list discussion or web-page where the answer can be found.

In general, all of these things are intended to fulfil a simple goal – get you the information you want as quickly as possible, in a way that wastes the time of people already in the project as little as possible. An admirable goal indeed, but as a newcomer, this is not how people are used to being welcomed. Eric Raymond wrote extensively about this in his essay “How to ask questions the smart way”.

Indeed, one of the hardest things to do as an outsider looking in is to evaluate when a community is healthy and viable, and when it has problems which will prevent you from working effectively in partnership. Few resources which talk about healthy free software community projects exist – “Producing Open Source Software”, by Karl Fogel, is something of a bible on the subject, and should be required reading for anyone considering investing in free software. I have also found some presentations, including Simon Phipps’s 2006 OSCon keynote “The Zen of Free” and “How Open Source Projects Survive Poisonous People” by Ben Collins-Sussman and Brian Fitzpatrick, to be excellent resources in helping identify traits of what makes up a healthy community. Two other useful papers which include metrics on measuring the openness of a community, including its governance model, are Pia Waugh’s “The Foundations of Openness” and François Druel’s Ph.D. Thesis (in French) “Évaluation de la valeur à l’ère du Web ” (PDF – rough translation: “Measuring value in the era of the Web”).

Some of the considerations when evaluating a community are whether there is clear leadership, whether that leadership is an individual, a group, or a company, how the leaders are chosen (if they are chosen), what technological and social barriers to participating in the project exist, whether the community processes are documented and transparent, what recourse one has if one feels badly treated, what the behavioural norms of the community are (and whether they are documented) – the list goes on. Pia’s paper in particular gives a great overview in the section “Open Governance”.

Call to arms

And so I close with a call to arms to both free software communities, and companies planning on developing an “open source strategy”.

First, developers, document your communities. Think of yourselves as guides, explaining the cultural quirks of your country to a newly arrived immigrant. Be explicit. In addition to explaining where and how your community works, document how one gains trust and responsibility. Ensure that a newcomer can learn quickly what he needs to do to become a citizen and from there a project maintainer. I am not saying that it should be easy for someone to become a maintainer. What I am suggesting is that it should be easy to see how one becomes a maintainer before doing it

Next, project managers, software developers, company leaders: please, please, please – save yourself time and money and, when you reach the point where you will be building products which depend on good free software components, let the second thing that you do, right after a technical evaluation, be to evaluate the health of the community. A community where you can earn influence and guide the project to better meet your needs is a better long-term investment than betting on a slightly technically superior solution with an unhealthy governance model.

You are building products that you will be selling, supporting, and hopefully profiting from. In this situation, does it really make sense not to have the developer’s ear?