I Want To Teach The World To Sing, In Perfect Harmony

May 3, 2004 1:51 pm General

On looking back over the past couple of months, it’s pretty clear that JDS has become a hugely successful product. We’ve broken previous records of mass deployments in the deal with the Chinese Government, we’ve started selling cheap PC’s with JDS pre-installed at Walmart, and everyone in Sun wants to have some part of our project. JDS is the new thing within Sun, and I have been hugely excited and motivated being a part of it.

However, despite the huge strides we’ve been making, both from a technical level, marketing and planning level, things have not run easy. One of the issues we’re facing is how we might effectively track a moving community product like GNOME. With continuing lack of resources [but who doesn’t, right?], this is one of the key things we need to learn, if only by continuing to make mistakes.

What should we feel the need to track? Arguably we could just take the GNOME sources, apply various levels of branding, bug fixing and feature development and ship it as our product. We’ve done this for CDE in the past, but without the feature development and innovation that’s needed on the desktop, that we can both leverage from the community and drive ourselves, people will soon start to turn their eyes to other more exciting projects.

GNOME is a rapidly evolving project, perhaps a little more rapidly evolving than JDS, and keeping pace with it is fraught with difficulties. With a defined, tested and proven 6 monthly release schedule we’re in a better situation than most open source projects, in that we, Sun, can plan and expect releases of the core desktop and platform.

The problem statement was the following –

” We maintain separate versions of community code based on a snapshot of a stable release. This means excessive forward porting to different community and internal branches ”

We can break this down into a number of potential development areas, some of which almost certainly overlap.

Bug Fixes

Bug fixes are generally, but not always, straight forward patches that usually go upstream as soon as they are created.

Despite the relative build up of bug fixes, most are likely to go upstream. So long as there is a mentality of pushing patches upstream as soon as possible these should be less of an issue. Red Hat by comparison only fix high priority bugs in their local stable branch. With very different QA rules to Sun, their developers have the opportunity to mark bugs fixed as ‘UPSTREAM’, meaning that the fix will go in the latest community sources and, only then, propagate down into their future releases. The priority, depending on the bug in question, is on fixing it in the community. This allows them to spend their precious resources on feature development. There are no religious reasons for doing this – it is purely practical, and it seems to work well. Could Sun be quite so practical – perhaps not, based on our current customer expectations, which may well be a good thing.

Branding

Branding of our product seems to include making trivial changes eg. menu re-ordering, default panel setup, theming, …

While these changes are relatively simplistic, with increase numbers we end up maintaining many patches that may never go upstream. If we’re not careful, these patches can create more patches. For example, moving away from the community default menu has resulted in moving away from the community documentation causing us not only shipping different versions of online user help but also having to provide patches to fix up the various help links around the desktop.

Even a simple string change, has huge consequences. GNOME has translations for almost 70 different languages, each of variable levels of support within the project. With the latest GNOME 2.6 release, there are 36 fully supported languages – coverage of 80%+. While Sun officially has support for it’s ’10 Big Language Rule’, changing just 1 string is going to break 70 – 10 languages. That’s a big hit, and we need to think carefully before making such changes.

Value Add / Features

While these could be arguably considered as ‘branding’, these are generally non-trivial changes with a large level of impact eg. Launch button, menu stripe, Nautilus Mount/Sharing.

While we certainly shouldn’t prevent innovating and adding new features to the desktop, we should carefully plan them, so that they get the right amount of project time. Too often have we created new features without any real thought into project planning and whether we have enough time in our schedule to properly finish off the implementation. As a result, we are stuck with many patches that are poorly written/planned [as a direct result of schedule], unpolished and far from ready to be taken upstream. As these numbers of patches increase we start to develop a fork of our product, and updating to new versions of the community product becomes a time consuming, cruel lesson in merging and rewriting. We have neither time nor resources for this, and if the level of patches of these type increase, we will be stuck in maintainer land rather than developer and innovation land.

Module Replacement

Module replacement is replacing a community shipped module and replacing it with another module of equal or better functionality eg. Glow and Evolution, Yelp and Java Help, GNOME Dictionary and jDictionary

This opens up another can of worms, and it’s sometimes hard to balance the argument between desktop integration and better functionality. Obviously we should strive to ship the best set of applications available to us, and that the user will benefit from. Ultimately we have to do what’s best for Sun but we such a decision should be taken very reluctantly. Not only does such a move result in a ‘fork’ but also results in more fragmentation in the Linux desktop. Is fragmentation of the desktop such an important issue? In my mind, yes, crucially. While it might be argued that as long as we keep conforming to open standards, every divergent move by Sun potentially results in a less cooperative and supportive community, with positive influence being harder to achieve. Having a community that trust you, is key to Sun’s continuing involvement.

What’s best for Sun often isn’t what is best for our users/customers and the support that we can leverage from the open source community. Achieving that balance is always going to be hard. Being stuck with some code that you are going to have to maintain, localize and document, let alone innovate, on your own is harder.

So, what are the conclusions and solutions can be drawn other than to be more careful with our planning, development and project management?

Ideally, the sensible approach is to work on community code and avoid diverging JDS as much as possible from the core desktop and platform. It is clear that we have a huge dependancy on the success of the GNOME project – without which we would not have a product [for which we owe many thanks for]. While we can continue to invest developer time into innovative, future projects like Looking Glass, we can not hope to bolster the resources for a complete, featureful and stable desktop environment like a typical open source community. ‘Innovation happens elsewhere’ has been a typical quote within Sun walls for many years. The belief is returning that Sun can indeed lead with innovative ideas, but as we start to fragment our applications and, to a lesser extent, our platform we lose out on many remarkable advances within the wider open source community. That’s a slightly harder pill to swallow, and can certainly appreciate the reluctance of such an approach.

Can we afford to see history repeat itself? We’re already seeing vendors all push for their own combinations of solutions, yet desperately wish they could all standardize to some extent. We’ve all seen and suffered greatly from the UNIX wars of the past. Have we forgotten what lessons we have learnt from this? Moreover, how often should we expect to see similar quotes such as this –

” And as long as Sun thinks they are going to beat Microsoft by replaying the Unix wars and fragmenting Linux, they are going to continue losing. Funny that all they worry about with Java is fragmentation, when GPL communities do a much better job of not fragmenting than they do. ” – Luis Villa

and more importantly, be able to come up with a honest reply to them?

How might we achieve this? With a predictable schedule for GNOME, it should easily be possible to time our development, and synchronize with the community. Like it, or lump it, the community unstable development phase is our security and our insurance for future JDS work. Unless we ensure its survival and take part in its direction, it will be hard for JDS, and GNOME, to be considered as a viable desktop alternative to Windows and, harder still, to prove that we’re innovating on the desktop. With a predictable schedule already in place, we need to synchronize the schedules at our end. Many resources are wasted by continuously changing plans mid way through a developmental cycle. We need to stick to our original plans where possible, develop the exact same fundamentals of time based releases, and pull together our product releases where it makes sense.

Why does this make sense? For a start, developing the features we want and working with community unstable development means that we have a positive influence and recognition within the community. We are developing both for our own benefit, and the benefit of the GNOME project. Both sides win, period.

With a change in development plan, we would not only be guaranteed that our features get upstream to community code, since by their nature they are developed there, but also that the maintenance of our feature patches becomes minimal. We could invest some degree of effort back porting some of these features with the knowledge that no porting will be required in the future. Not only do we align ourselves better with community development, and on fundamentally important projects like ‘Desktop Lockdown’, we benefit from exposure within the many sub groups of GNOME – everything from documentation and usability, to translation and QA. Any features that we develop in an unstable stream also benefit from ‘soak’ time, in which they are well tested, well thought out and implemented according to community coding standards.

We’re not the only ones with this issue. In fact many groups within Sun are facing the same problems on a daily basis. And not just Sun.

” I think the main reason much of XD2 didn’t make it upstream was that it was done behind closed doors without buy-in and much of it was done in a pretty hacky way (for understandable reasons). If we instead take the approach of first making the change in 2.8 in an upstream-acceptable way and then backporting to 2.6, then we know stuff won’t be lost. ” – Havoc Pennington

But is a development plan enough? It is not only about work practices. It is also about mindset. To work effectively and to get the most out of open source, you need to consider yourself a part of the community. Buy into the strategy, and if you don’t agree with that, work hard to change it. With proper rationale, there is no reason that any community member won’t feel the same way as you.

Fortunately there are people within Sun who *do* care about these types of problems and are willing to spend time to find solutions and create a new mindset, however hard that challenge might be. These aren’t exactly new and they’re not particularly earth shattering but they are the challenges that drive me and make me want to stay at Sun. Is Sun all doom and gloom right now? Hell no, far from it. We’re changing, and it is wonderful to see it.

Thanks to the bunch of people I bounced this off for review – it’s hugely been appreciated.

Comments are closed.