Safe as Milk

The value of engagement

(Reposted from Neary Consulting)

Mal Minhas of the LiMo Foundation announced and presented a white paper at OSiM World called “Mobile Open Source Economic Analysis” (PDF link). Mal argues that by forking off a version of a free software component to adjust it to your needs, run intensive QA, and ship it in a device (a process which can take up to 2 years), you are leaving money on the table, by way of what he calls “unleveraged potential” – you don’t benefit from all of the features and bug fixes which have gone into the software since you forked off it.

While this is true, it is also not the whole story. Trying to build a rock-solid software platform on shifting sands is not easy. Many projects do not commit to regular stable releases of their software. In the not too distant past, the FFMpeg project, universally shipped in Linux distributions, had never had a stable or unstable release. The GIMP went from version 1.2.0 in December 1999 to 2.0.0 in March 2004 in unstable mode, with only bug-fix releases on the 1.2 series.

In these circumstances, getting both the stability your customers need, and the latest & greatest features, is not easy. Time-based releases, pioneered by the GNOME project in 2001, and now almost universally followed by major free software projects, mitigate this. They give you periodic sync points where you can get software which meets a certain standard of feature stability and robustness. But no software release is bug-free, and this is true for both free and proprietary software. In the Mythical Man-Month, Fred Brooks described the difficulties of system integration, and estimated that 25% of the time in a project would be spent integrating and testing relationships between components which had already been planned, written and debugged. Building a system or a Linux distribution, then, takes a lot longer than just throwing the latest stable version of every project together and hoping it all works.

By participating actively in the QA process of the project leading up to the release, and by maintaining automated test suites and continuous integration, you can mitigate the effects of both the shifting sands of unstable development versions and reduce the integration overhead once you have a stable release. At some stage, you must draw a line in the sand, and start preparing for a release. In the GNOME project, we have a progressive freezing of modules, progressively freezing the API & ABI of the platform, the features to be included in existing modules, new module proposals, strings and user interface changes, before finally we have a complete code freeze pre-release. Similarly, distributors decide early what versions of components they will include on their platforms, and while occasional slippages may be tolerated, moving to a new major version of a major component of the platform would cause integration testing to return more or less to zero – the overhead is enormous.

The difficulty, then, is what to do once this line is drawn. Serious bugs will be fixed in the stable branch, and they can be merged into your platform easily. But what about features you develop to solve problems specific to your device? Typically, free software projects expect new features to be built and tested on the unstable branch, but you are building your platform on the stable version. You have three choices at this point, none pleasant – never merge, merge later, or merge now:

• Develop the feature you want on your copy of the stable branch, resulting in a delta which will be unique to your code-base, which you will have to maintain separately forever. In addition, if you want to benefit from the features and bug fixes added to later versions of the component, you will incur the cost of merging your changes into the latest version, a non-negigible amount of time.
• Once you have released your product and your team has more time, propose the features you have worked on piecemeal to the upstream project, for inclusion in the next stable version. This solution has many issues:
  • If the period is long enough, your feature additions will be long removed from the codebase as it has evolved, and merging your changes into the latest unstable tree will be a major task
  • You may be redundantly solving problems that the community has already addressed, in a different or incompatible way.
  • Feature requests may need substantial re-writing to meet community standards. This problem is doubly so if you have not consulted the community before developing the feature, to see how it might best be integrated.
  • In the worst case, you may have built a lot of software on an API which is only present in your copy of the component’s source tree, and if your features are rejected, you are stuck maintaining the component, or re-writing substantial amounts of code to work with upstream.
• Develop your feature on the unstable branch of the project, submit it for inclusion (with the overhead that implies), and back-port the feature to your stable branch once included. This guarantees a smaller delta from the next stable version to your branch, and ensures you work gets upstream as soon as possible, but adds a time & labour overhead to the creation of your software platform

In all of these situations there is a cost. The time & effort of developing software within the community and back-porting, the maintenance cost (and related unleveraged potential) to maintaining your own branch of a major component, and the huge cost of integrating a large delta back to the community-maintained version many months after the code has been written.

Intuitively, it feels like the long-term cheapest solution is to develop, where possible, features in the community-maintained unstable branch, and back-port them to your stable tree when you are finished. While this might be nice in an ideal world, feature proposals have taken literally years to get to the point where they have been accepted into the Linux kernel, and you have a product to ship – sometimes the only choice you have is to maintain the feature yourself out-of-tree, as Robert Love did for over a year with inotify.

While addressing the raw value of the code produced by the community in the interim, Mal does not quantify the costs associated with these options. Indeed, it is difficult to do so. In some cases, there is not only a cost in terms of time & effort, but also in terms of goodwill and standing of your engineers within the community – this is the type of cost which it is very hard to put a dollar value on. I would like to see a way to do so, though, and I think that it would be possible to quantify, for example, the community overhead (as a mean) by looking at the average time for patch acceptance and/or number of lines modified from intial proposal to final mainline merge.

Anyone have any other thoughts on ways you could measure the cost of maintaining a big diff, or the cost of merging a lot of code?

Frustration

I wonder if it was a mistake to adopt the “evaluate as they come in” method for Maemo Summit presentations. As we received proposals, for each proposal on its merits we said yes, no or maybe. If you were a yes, you were added to the schedule. A no got a nice email. A maybe stayed in the queue.

We set a deadline for submissions of September 13th, but this was a deadline for us to finish the schedule, not for people who wanted to give presentations to submit. I said as much in the call for content: “The final deadline for submissions will be September 13th but the sooner you submit your proposal the better chances you will have to get a slot”.

After Nokia World, a bunch of people came out of the woodwork to propose quality presentations, and after reviewing pending proposals last week, we now have an agenda which is almost full – there are 5 open slots and about 8 open lightning talk slots, about half of which are potentially taken already.

So it’s slightly frustrating to see 16 new submissions come in over the past 2 days as people saw the deadline arriving and the schedule filling up. If they were all there before, our choices might have been different, but now we will unfortunately be obliged to reject otherwise great presentations, simply because the proposers waited too long to ask for a slot.

It’s a tough problem to solve, though – if we had set an earlier deadline, we would not have received many of those presentations, or they would have been vague proposals like “can’t say much yet, but this’ll be a cool presentation about something related to Fremantle”. Approving presentations early allowed the council to have better information for travel subsidies and allowed people to book travel earlier and thus cheaper. But we’re going to miss out on some presentations I think would be pretty good. Pity.

Ton Roosendaal to keynote the Summit community days

Ton Roosendaal (on the left)

I’m very pleased to share that the opening keynote for the community days during the Maemo Summit will be Ton Roosendaal of the Blender Foundation. It’s been my pleasure to know Ton, mostly from afar, for the past few years, and he is one of the most amazing people I know in the free software world.

Ton is one of those people who has a sense of doing things big, and doing them right. Over the past few years, Ton has raised money to hire artists and developers to work on commercial quality films and games, resulting in Project Orange, Project Apricot, Project Peach and now Project Durian is in pre-production (you can buy your copy now and get your name on the credits!), with the goal of showing off what Blender can do and making the program better by working closely with artists to see what needs work. The results are truly impressive, and the amount of foresight and hard work which went into getting each of these projects off the ground and completed is amazing.

The Blender community is also amazing. Ton has continually given passionate users a reason to stay around, and ways to help the project, and that has been rewarded by a diverse community of artists and developers working together. The BlenderNation fan/news site is a testament to the creativity and passion of the community.

I’m really looking forward to hearing Ton speak.

Maemo Community Council elections Q3 09: Nominations open

The nomination period for candidatures for the Q3 2009 Maemo community council election is now open.

Candidates eligible for election according to the rules of the council can be nominated by anyone in the community. If a maemo.org community member nominates someone other than themselves, the nomination must be accepted by the nominee before it is official.

Nominations may be made before 23:59 UTC, September 20th, at which time
a voting period of one week will open, by sending an email to the
maemo-community mailing list with the subject “Council Nomination:”
followed by the name of the nominee. Nominations can be confirmed by the
nominee replying to this email.

I encourage anyone who would like to be on the council to nominate
themselves early, and I would encourage all community members to be
forthcoming with questions for the candidates.

Important election dates:

Sept 7

Nominations open for Maemo Community Council elections

Sept 21

Nominations close, voting opens

Sept 28

Voting closes, provisional results declared

Oct 5

If no challenges are upheld, results for elections are final

Good luck to all!

GCDS round-up 5: Mobile Day

Nearing the end of the series on the Gran Canaria Desktop Summit.

On Wednesday morning (after SMASHED), we had to get to the new location for the conference. I missed the bus window of 8am to 9am, so I took a taxi, without knowing the address of where we were going, other than knowing that it was the “Gran Canaria university, informatics building”. Turns out that’s not enough information for a taxi driver 🙂 Anyway, got there eventually, late for the opening session, and a little more expensive than expected. I also lost some change down the back of the bucket seat, so he even got a tip.

Anyway, the rest of the day went pretty well, and we had some great mobile related presentations (to compliment all of the other mobile related content in the conference):

• Multimedia in your pocket, by Stefan Kost: Nice presentation on using MAFW to build complex multimedia applications
• Designing Moblin-Netbook. A free desktop on a 7-10″ Screen, by Nick Richards: Great overview of the Moblin platform, and the design principles guiding it – from design requirements, personas, and dealing with constraints.
• Hildon desktop in Maemo 5 by Kimmo Hämäläinen: An overview of the Hildon desktop on a whiteboard by Kimmo.
• MAFW: the Media Application Framework for Maemo by Iago Toral: Drilling down into the details of MAFW.
• Why its easier to re-invent rather than participate on the mobile? by Shreyas Srinivasan: My favourite presentation of the day. Shreyas laid out what he had expectied from GNOME Mobile, the problems he encountered, his understanding of the issues, and some proposed solutions to those problems. All in 15 minutes. I really appreciate people who don’t pad out the content that they have to present and instead focus on making a high-impact presentation.
• GNOME Mobile BOF, led by myself: We talked about how far we’ve come, the original goals of the initiative, and identified a bunch of things that we can improve short-term and medium-term.

Had a great dinner again on Wednesday, in a tapas bar with some Red Hatters and Michael Meeks, and then on to the party. Wednesday night was the golf club party, sponsored by Collabora, with a free bar until 1 (of which I mostly did not avail – I was being good), and I was in bed by 2. It was a great party, and I picked up another couple of cyclists for the outing I had been planning for Friday, before they wimped out on me.

GCDS round-up 4: Days 2 – 4

Sunday, Monday and Tuesday were the “core” days of the Gran Canaria Desktop Summit, with cross-desktop and KDE & GNOME specific presentations throughout. I caught a number of presentations, but mostly I was chatting in the hallway track, or doing work on the schedule, or actually working.

For me, the story of the 3 days was “parties”. I missed the early sessions on Sunday and Monday to get breakfast at 10am, after the parties hosted by Nokia (Sunday night) and Igalia (Monday night) – I was relieved that there was no party planned on Tuesday night, my 35 year old body couldn’t stand the pace! Great parties, not marred by excessive boozing mostly, and some great chats, notably with jrb, and Adam Dingle and Jim Nelson from Yorba, makers of Shotwell, a Vala photo manager with some really nice features and plans. And some great discussions with Michael Meeks and Matthew Garrett on the fouton during the Igalia party, with Federico Mena Quintero on architecture design patterns, and Jorge Castro on dinosaurs. I also got to meet Joaquim from Igalia, the Macacque band were great, but I’m sure that a hoarse Lefty regretted sweet home chicago and smoke on the water the day after.

I did get to some presentations though (here with a one line summary):

• Power management by Matthew Garrett: “Power management isn’t doing the same amount of work, slower. Do less work, or you’re killing polar bears.”
• ConnMan by Not Marcel Holtmann (Joshua Lock from Intel gave the talk in the end – thanks Emmanuele!): “ConnMan solves some problems for Moblin that NetworkManager wasn’t designed to solve.” (I think).
• Bluetooth on Linux by Bastien Nocera: “It mostly works now”
• Introduction to GNOME Shell, by Owen Taylor: “It’s pretty cool stuff already”
• GNOME Zeitgeist, by Thorsten, Seif and Federico: “We record what you’re doing”
• Communicating design in development, by Celeste Lyn Paul: “Keep it simple until they get the design principle, excessive realism too early just makes the discussion about the details”. Unfortunately, I don’t see a video available, highly recommended viewing if there was one.
• GNOME 3.0: A live circus^Wstatus update, by Vincent Untz et al: “It’s not just GTK+, Zeitgeist and GNOME Shell”
• GNOME 1,2,3, by Fernando Herrera and Xan Lopez: “A history of GNOME with thanks to YouTube” (my favourite presentation of the conference)
• Personal Passion lightning talks, by Aaron Bockover: “We’re not just Free Software hackers!” This was absolutely my second favourite session of the conference. We got a 10 minute overview of the burnout cycle from Jono Bacon, underlining how important it is to have a life outside of software, and heard from people whose passion was running (complete with a soundtrack of me finishing a marathon), airplanes, motorcycling, cooking, bacon, dinosaurs, Aikido, buddhism and calligraphy, trekking in Argentina, and also a couple of geeky ones on icon design and scheme (which was very enjoyable indeed, thanks Andy!)

Update: Memory playing tricks with me – for of course, Tuesday evening was the highly anticipated meeting of SMASHED. We finally met at the Mare Baja again, where the opening night party was held, and enjoyed a bunch of tapas courtesy of CodeThink, before scoffing down some great whisk[e]y, including (from memory) a 21yo Highland Park, a nice 16yo Longmorn, a very lod bottle of Oude Ginever from Lefty, an old standard Connemara single malt, and a Yamazaki 10yo I brought.

SMASHED 2009 in Gran Canaria

Festivities carried on until after 1pm, when I left with Andrew Savory and someone else (whose name I don’t recall), and Behdad got in an unprovoked fight with the footpath on the way back to the hotel – it came right up and hit him in the face. Some nice KDE people took him to the hospital to get sewn up – luckily the group photos had been taken earlier in the day.

Got back to the hotel around 2, and tried to catch up on some of that beauty sleep before Mobile Day on Wednesday in the new conference location in the university.

Gran Canaria wrap-up 2: Day 0

Note: I actually wrote something like this already in GNOME Blog, and a combination of the Intel graphics freezes in Jaunty and GNOME Blog not creating a local copy of in-progress entries cost me the lot. Funny that WordPress, a web-app, offers better transparent data retention across unexpected events than a local client. I have resolved to use Tomboy for drafting blog entries off-line now, and to figure out how to patch GNOME Blog to save drafts.

My Gran Canaria adventure started in a funny ha ha way when I got the airport and I was told I wasn’t on the plane which I had a ticket for. I checked my email to ensure I hadn’t received any schedule change emails, and found the last mail I received from Expedia, indicating I was booked on the 15h flight from Lyon to Madrid. But the friendly & helpful people at the Air France desk eventually figured it out, the airline had bumped me to an 8am flight, with a transfer to Gran Canaria arriving in the early afternoon. The travel agent wasn’t aware of it (I checked later when I got some internet access). So the Air France people asked if I minded flying through Bilbao, I said no (imagining they meant that I’d be flying from Lyon to Bilbao), and they checked in my bags, and gave me a boarding pass. For the plane to Madrid.

“I don’t understand”, I said. “We can’t issue you a boarding pass for the Madrid-Bilbao or Bilbao-Las Palmas legs now”, they explained. Ah. When I looked at the transfer times, and realised that (if we were on time) I would have 30 minutes to transfer in Bilbao, I was told that I would probably be able to get a boarding pass for the Bilbao-Las Palmas in Madrid.

When I got to Madrid, I queued behind some Swedes who were on their way to some holiday destination and had just been told that their flights were over-booked, and that they’d be staying in Madrid for the night. Happy happy joy joy. I also surprisingly ran into Alex Larsson, who was looking for a boarding card for his flight to Gran Canaria, which was delayed. I debated asking to get on the flight with him for a second, but figured that my bags wouldn’t make it even if I did, so I decided to play it safe.

The transfer desk in Madrid couldn’t issue me a boarding card for Las Palmas, so with 35 minutes transfer time, I would have to find the transfer desk, get a boarding pass, and hope that both my bags and I made it to the plane on time. I was not optimistic. After checking in, I bought a nice bottle for the SMASHED meeting, a Yamazaki 10yo.

Landed in Bilbao (the approach looks beautiful, I really want to visit the Basque country now), and found that there was no transfer desk. I had to go past security, with my newly purchased bottle of Yamazaki, check in, go back through security, and have my bags and I both make the plane. I have learned over time that the quality of hustle is important in airports. Relax when things are beyond your control, and when you can do something about it, run. So I ran. Headless chicken style.

An airport attendant who took pity on my cause very kindly brought me out through the security check-point, and I left my whiskey with the security guard. Ran to the first check-in counter I found to ask where I could check in for my flight. And by complete coincidence, the girl who was supposed to be manning the check-in desk had stepped away, since the flight was almost closed, to chat with her friend, who was minding the check-in desk I ran to.

Checked in, registered baggage tags to get the bags on the plane, back through security, got my whiskey back, ran to the plane, and (with take-of delayed a few minutes) felt much more confident about making it to the islands that night, with baggage in tow. Be thankful for the kindness of strangers. And it’s better to be born lucky than rich. All in all, a day made much better by the desire of everyone I met to be nice & helpful, in spite of the bureaucracy they work under.

Landed, picked up my bags, got a taxi to the hotel, and dropped them off. Said hello to someone with a laptop in the lobby (Hi mpt!), and ran to the welcome party to see if anyone was left, as it was now almost midnight local time.

I forgot this was Spain.

I met lots of people on the way. Lots of people (but no free beer) were still at the party. Talked briefly with Stormy, Lefty and family, Quim, Oskari, Henri, Sebas, Richard Dale, Rob Taylor and many more over a couple of nice beers. Thanks Canonical for the t-shirt and for the party, a great time was had. Home & to bed by 3. So endeth day 0.

Why I disagree with RMS concerning Mono

The GNOME press contact alias got a mail last weekend from Sam Varghese asking about the possibility of new Mono applications being added to GNOME 3.0, and I answered it. I didn’t think much about it at the time, but I see now that the reason Sam was asking was because of Richard Stallman’s recent warnings about Mono – Sam’s article has since appeared with the ominous looking title “GNOME 3.0 may have more Mono apps“. And indeed it may. It may also have more alien technology, we’re not sure yet. We’re still working on an agreement with the DoD to get access to the alien craft in Fort Knox.

Anyway – that aside, Richard’s position is that it’s dangerous to include Mono to the point where removing it is difficult, should that become necessary to legally distribute your software. On the surface, I agree. But he goes a little further, saying that since it is dangerous to depend on Mono, we should actively discourage its use. And on this point, we disagree.

I’m not arguing that we should encourage its use either, but I fundamentally disagree with discouraging someone from pursuing a technology choice because of the threat of patents. In this particular case, the law is an ass. The patent system in the United States is out of control and dysfunctional, and it is bringing the rest of the world down with it. The time has come to take a stand and say “We don’t care about patents. We’re just not going to think about them. Sue us if you want.”

The healthy thing to do now would be to provoke a test case of the US patent system. Take advantage of one of the many cease & desist letters that get sent out for vacuous patented technology to make a case against the US PTO’s policy pertaining to software and business process patents. Run an “implement your favourite stupid patent as free software” competition.

In all of the projects that I have been involved in over the years, patent fears have had a negative affect on developer productivity and morale. In the GIMP, we struggled with patent issues related to compression algorithms for GIF and TIFF, colour management, and for some plug-ins. In GNOME, it’s been Mono mostly, but also MP3, and related (and unrelated) issues have handicapped basic functionality like playing DVDs for years. In Openwengo, the area of audio and video codecs is mined with patent restrictions, including the popular codecs G729 and H264 among others.

What could we have achieved if standards bodies had a patent pledge as part of their standardisation process, and released reference implementations under an artistic licence? How much further along would we be if cryptography, filesystems, codecs and data compression weren’t so heavily handicapped by patents? Or if we’d just ignored the patents and created clean-room implementations of these patented technologies?

That’s what I believe we need to do. Ignore the patent system completely. I believe strongly in respecting licencing requirements related to third party products and developer packs. I think it’s reasonable to respect people’s trademarks and trade secrets. But having respect for patents, and the patent system, is ridiculous. Let a thousand flowers bloom, and let the chips fall where they may.

So if you want to write a killer app in Mono, then don’t let anyone tell you otherwise. If you build it, they will come.

Maemo Summit – help make it great

This year, I’ve been asked to help with the content selection for the Maemo Summit, which will be held in October, in Amsterdam. We’re aiming for a very cool conference with lots of tips, tricks, hacks and general hardware coolness over 3 days.

Nokia is organising the first day, and the second and third days are entirely organised by the community. After a round of discussion, myself, Valerio Valerio and Jamie Bennett will be choosing content for the summit from among presentations proposed by the community. We’re aiming for presentations which will target three main audiences: tablet users, application developers and platform developers.

You can read more about the call for content or how to submit a presentation on the Maemo wiki. We’ve agreed on a fairly novel way of filling the schedule – we are starting from an empty grid, with three tracks, a couple of plenary sessions, and some lightning talks. As great talks come in, we will add them directly to the grid. If we don’t think that talks are up to scratch, they will be rejected, the submission will move to the Talk page for the Submissions wiki page, and if we are hesitant, the proposals will stay in the Submissions queue.

This has some great benefits over the usual call for papers/deadline/selection/publish the entire schedule scheme of things. Most proposers will know straight away whether their talk has been accepted, rejected, or converted into a lightning talk. Attendees will see the schedule building up and be able to propose sessions to account for topics that are not yet accounted for. And we will be able to keep some small number of slots until quite late in the organisation cycle for “late breaking news” – those great presentations that arrive too late for your deadline, but which you would really love to see get onto the schedule. And it is a kind of auction system – you have a great interest in getting your presentation proposal in early, rather than waiting for the last minute.

Anyway – let’s see how it works. You can follow the progress of the schedule on the wiki as well.

Good luck to all!

Too many platforms?

Fabrizio Capobianco of Funambol wondered recently if there are too many mobile Linux platforms.

The context was the recent announcement of oFono by Intel and Nokia, and some confusion and misunderstanding about what oFono represents. Apparently, several people in the media thought that oFono would be Yet Another Complete Stack, and Fabrizio took the bait too.

As far as I can tell, oFono is a component of a mobile stack, supplying the kind of high-level API for telephony functions which Gstreamer does for multimedia applications. If you look at it like this, it is a natural compliment to Moblin and Maemo and potentially a candidate technology for inclusion in the GNOME Mobile module set.

Which brings me to my main point. Fabrizio mentions five platforms besides oFono in his article: Android, LiMo, Symbian, Maemo and Moblin. First, Symbian is not Linux. Of the other four, LiMo, Maemo and Moblin share a bunch of technology in their platforms. Common components across the three are: The Linux kernel (duh), DBus, Xorg, GTK+, GConf, Gstreamer, BlueZ, SQLite… For the most part, they use the same build tools. The differences are in the middleware and application layers of the platform, but the APIs that developers are mostly building against are the same across all three.

Maemo and Moblin share even more technology, as well as having very solid community roots. Nokia have invested heavily in getting their developers working upstream, as has Intel. They are both leveraging community projects right through the stack, and focusing on differentiation at the top, in the user experience. The same goes for Ubuntu Netbook Edition (the nearest thing that Moblin has to a direct competitor at the moment).

So where is the massive diversity in mobile platforms? Right now, there is Android in smartphones, LiMo targeting smartphones, Maemo in personal internet tablets and Moblin on netbooks. And except for Android, they are all leveraging the work being done by projects like GNOME, rather than re-inventing the wheel. This is not fragmentation, it is adaptability. It is the basic system being tailored to very specific use-cases by groups who decide to use an existing code base rather than starting from scratch. It is, in a word, what rocks about Linux and free software in general.