JHBuild Updates

The progress on JHBuild has continued (although I haven’t done much in the last week or so). Frederic Peters of JhAutobuild fame now has a CVS account to maintain the client portion of that project in tree.

Perl Modules (#342638)

One of the other things that Frederic has been working on is support for building Perl modules (which use a Makefile.PL instead of a configure script). His initial patchworked fine for tarballs, but by switching over to the new generic version control code in jhbuild it was possible to support Perl modules maintained in any of the supported version control systems without extra effort.

Speed Up Builds (#313997)

One of the other suggestions for jhbuild that came up a while ago was to make it “eleventy billion times faster”. In essence, adding a mode where it would only rebuild modules that had changed. While the idea has merrit, the proposed implementation had some problems (it used the output of “cvs update” to decide whether things had changed).

I’d like to get something like this implemented, preferably with three possible behaviours:

  1. Build everything (the current behaviour).
  2. Build only modules that have changed.
  3. Build only modules that have changed, or have dependencies that have changed.

The second option is obviously the fastest, and is a useful option for collections of modules that should be API stable. The third option is essentially an optimisation of the first option. For both the second and third option, it is necessary to be able to tell if the code in a module has been updated. The easiest way to do this is to record an identifier for the tree state, and the identifier is different after an update.

The identifier should be cheap to calculate too, so will probably be dependent on the underlying version control system:

  • CVS – a hash of the names and versions of all files in the tree. Something like this, maybe (can be constructed by reading the CVS/Root, CVS/Repository and CVS/Entries files in the tree).
  • Subversion – a combination of (a) the repository UUID, (b) the path of the tree inside the repository and (c) the youngest revision for this subtree in the repository.
  • Arch – the output of “baz tree-id“.
  • Bzr – the working tree’s revision ID.
  • Darcs – a hash of the sequence of patches representing the tree, maybe?
  • Tarballs – the version number for the tarball.

On a successful build, the ID for the tree would be recorded. On subsequent builds, the ID gets recalculated after updating the tree. The new and old IDs are then used to decide on whether to build the module or not, according to the chosen policy.

JHBuild Improvements

I’ve been doing most JHBuild development in my bzr branch recently. If you have bzr 0.8rc1 installed, you can grab it here:

bzr branch http://www.gnome.org/~jamesh/bzr/jhbuild/jhbuild.dev

I’ve been keeping a regular CVS import going at http://www.gnome.org/~jamesh/bzr/jhbuild/jhbuild.cvs using Tailor, so changes people make to module sets in CVS make there way into the bzr branch. I’ve used a small hack so that merges back into CVS get recorded correctly in the jhbuild.cvs branch:

  1. Apply the diff between jhbuild.cvs and jhbuild.dev to my CVS checkout and commit.
  2. Modify tailor to pause before committing the to jhbuild.cvs.
  3. While tailor is paused, run bzr revert followed by a merge of the same changes from jhbuild.dev.
  4. Let tailor complete the commit.

It’s a bit of a hack, but it allows me to do repeated merges from the CVS import to my development branch (and back again). It also means that any file moves I do in my bzr branch are reflected in the CVS import when merged.

So now when filing bug reports on jhbuild, you can submit fixes in the form of bzr branches as well as patches.

So, on to the improvements:

Generic Version Control Interface

Previously, to add support for a new version control system the following additions were needed:

  • Some code to invoke the version control utility to make checkouts and update working trees.
  • Code to implement the build state machine for modules using the version control system (these classes would generally derive from AutogenModule which implemented most of the build logic).
  • Code to create instances of the above module type when parsing .modules files.

This was quite a bit of work, and in the end would only help if the code in question was set up to build the same way as most Gnome modules (i.e. with a autogen.sh script and autotools). If you wanted to build a module using Python distutils out of Subversion, a new module type would be needed. If you wanted to build a distutils module from a tarball, that would be another module type again.

With the new system, the different version control support modules provide a common interface. This means that a single module type is capable of implementing the build state machine for any version control system. Similarly, it should now be possible to implement distutils module support such that it will work with any supported version control system.

This work is not yet finished though. A bit more work needs to be done to parse version control system agnostic module definitions from .modules files. When this is done, a fair bit of the current syntax can be deprecated and eventually removed. When this is done, adding support for a new version control system shouldn’t take more than 100-200 lines.

Module Type Simplifications

As well as reducing the number of module types that need to be maintained in JHBuild, I’ve been working on simplifying the code in these module types. Previously, each stage of a module build was represented by a method call on the module type. The return value of the method was used to say (a) whether the stage succeeded or not, (b) what the next state would be and (c) if an error occurred some alternative next states to go to (e.g. offer to rerun autogen.sh).

With the new system, the next state and error states are declared as attributes on the method object. The method can indicate a failure by raising a particular exception. This greatly simplifies the cases where a build stage involves a number of separate actions that could each fail individually, since the exception cuts processing short without the error checks getting in the way of the code.

There are still a few module build stages not converted to the new system since their next state depends on various config settings (e.g. if running “make check” has been enabled or not). Since these generally involve skipping a stage based on some criteria, the plan is to move the logic to the stage being skipped, which should simplify things further.

Using Tailor to Convert a Gnome CVS Module

In my previous post, I mentioned using Tailor to import jhbuild into a Bazaar-NG branch. In case anyone else is interested in doing the same, here are the steps I used:

1. Install the tools

First create a working directory to perform the import, and set up tailor. I currently use the nightly snapshots of bzr, which did not work with Tailor, so I also grabbed bzr-0.7:

$ wget http://darcs.arstecnica.it/tailor-0.9.20.tar.gz
$ wget http://www.bazaar-ng.org/pkg/bzr-0.7.tar.gz
$ tar xzf tailor-0.9.20.tar.gz
$ tar xzf bzr-0.7.tar.gz
$ ln -s ../bzr-0.7/bzrlib tailor-0.9.20/bzrlib

2. Prepare a local CVS Repository to import from

The import will run a lot faster with a local CVS repository. If you have a shell account on window.gnome.org, this is trivial to set up:

$ mkdir cvsroot
$ cvs -d `pwd`/cvsroot init
$ rsync -azP window.gnome.org:/cvs/gnome/jhbuild/ cvsroot/jhbuild/

3. Check for history inconsistency

As I discovered, Tailor will bomb if time goes backwards at some point in your CVS history, and will probably bomb out part way through. The quick fix for this is to directly edit the RCS ,v files to correct the dates. Since you are working with a copy of the repository, there isn’t any danger of screwing things up.

I wrote a small program to check an RCS file for such discontinuities:

http://www.gnome.org/~jamesh/code/backward-time.py

When editing the dates in the RCS files, make sure that you change the dates in the different files in a consistent way. You want to make sure that revisions in different files that are part of the same changeset still have the same date after the edits.

4. Create a Tailor config file

Here is the Tailor config file I used to import jhbuild:

#!
"""
[DEFAULT]
verbose = True
projects = jhbuild
encoding = utf-8

[jhbuild]
target = bzr:target
start-revision = INITIAL
root-directory = basedir/jhbuild.cvs
state-file = tailor.state
source = cvs:source
subdir = .
before-commit = remap_author
patch-name-format =

[bzr:target]
encoding = utf-8

[cvs:source]
module = jhbuild
repository = basedir/cvsroot
encoding = utf-8
"""

def remap_author(context, changeset):
    if '@' not in changeset.author:
        changeset.author = '%s <%s@cvs.gnome.org>' % (changeset.author,
                                                      changeset.author)
    return True

The remap_author function at the bottom maps the CVS user names to something closer to what bzr normally uses.

5. Perform the conversion

Now it is possible to run the conversion:

$ python tailor-0.9.20/tailor -vv --configfile jhbuild.tailor

When the conversion is complete, you should be left with a bzr branch containing the history of the HEAD branch from CVS. Now is a good time to check that the converted bzr looks sane.

6. Use the new branch

Rather than using the converted branch directly, it is a good idea to branch off it and do the development there:

$ bzr branch jhbuild.cvs jhbuild.dev

The advantage of doing this is that you have the option of rsyncing in new changes to the CVS repository and running tailor again to incrementally import them. You can then merge those changes to your development branch.

Revision Control Migration and History Corruption

As most people probably know, the Gnome project is planning a migration to Subversion. In contrast, I’ve decided to move development of jhbuild over to bzr. This decision is a bit easier for me than for other Gnome modules because:

  • No need to coordinate with GDP or GTP, since I maintain the docs and there is no translations.
  • Outside of the moduleset definitions, the large majority of development and commits are done by me.
  • There aren’t really any interesting branches other than the mainline.

I plan to leave the Gnome module set definitions in CVS/Subversion though, since many people help in keeping them up to date, so leaving them there has some value.

I performed a test conversion using Tailor 0.9.20. My first attempt at performing the conversion failed part way through. Looking at what had been imported, it was apparent that the first few changesets created weren’t the first changesets I’d created in CVS. What was weirder still was the dates on those changesets: they were dated 1997, while I hadn’t started jhbuild til 2001.

It turns out that it was caused by clock skew on the CVS server back in September 2003, so the revision dates for a few files are not monotonic. I did the quick fix of directly editing the RCS files (I was working off a local copy of the repo), which allowed the conversion to run through to completion. The problem has been reported as bug #37 in Tailor’s bug tracker.

This made me a bit worried about whether the CVS to Subversion conversion script being used for the rest of the Gnome modules was also vulnerable to this sort of clock skew problem. Sure enough it was, and the first real changeset of jhbuild had been imported as revision 323.

I did a bit more checking of the CVS repository, and found that there were 98 other modules exhibiting clock skew in their revision history, spread over 1245 files (some files with multiple points of skew). I’ve only checked the SVN test conversions of some of these modules, but all the ones I checked exhibited the same type of corruption.

It is going to be a fair bit of work cleaning it all up before the final conversion.

GraphViz

On the gtk-doc-list mailing list, Matthias mentioned that the GraphViz license has been changed to the CPL (the same license as used for Eclipse), which is considered Free by both the FSF and OSI (although still GPL incompatible). This should remove the barriers that prevented it getting packaged by Linux distributions.

Due to the previous licensing, RMS urged developers of GNU software to not even produce output in the form that the GraphViz tools use as input. Maybe that can change now. While the license is GPL incompatible, the GraphViz tools can easily be invoked from the command line, passing a .dot file in, and getting output in PNG, PS, SVG, etc format (or even another .dot file with the layout information added), which is enough for pretty much all uses of the tools.

One of the features I added to JHBuild fairly early on was the ability to dump the dependency tree for a set of modules in the .dot format. So to visualise the dependencies for Gnome, you could run a command like this:

jhbuild dot meta-gnome-desktop | dot -Tps > gnome-2.10.eps

(of course, given the number of modules that are needed to build the entire Gnome desktop, you might get a better picture by picking a smaller number of modules).

8 December 2004

Mataró

I’ve been in Mataró (about an hour from Barcelona) now since Sunday, and it’s quite a nice place. It is a bit cooler than Perth due to it being the middle of Winter here, but the way most of the locals are rugged up you’d think it was a lot colder. It’s great to catch up with everyone, and a number of pygtk developers will be turning up over the next few days for the BOF on the weekend.

Gnome Foundation Elections

Congratulations to the new board members. It is a little disappointing that only about 56% of members voted though. Once the membership committee has the anonymous voting stuff set up, it might be worth doing the preferential voting referrendum.

jhbuild

I’ve been working on some preliminary documentation for JHBuild, which is available here. It should be useful for new users and people looking at writing new module sets for it. It has a fairly complete command reference and config file reference, so it is probably useful for current users too. It would be good to add some information about setting up a tinderbox like the one Luis set up for Gnome.

20 October 2004

Even More Icon Theme Stuff

To make it a bit easier to correctly display themed icons, I added support to GtkImage, so that it is as easy as calling gtk_image_new_from_icon_name() or gtk_image_set_from_icon_name(). The patch is attached to bug #155688.

This code takes care of theme changes so the application developer doesn’t need to. Once this is in, it should be trivial to add themed icon support to various other widgets that use GtkImage (such as GtkAbout and GtkToolItem).

JHBuild

I started work on some extended documentation for JHBuild. At the moment, this just includes some information on setting it up and basic use. I’ll extend it to hold a reference to all JHBuild commands, some documentation on the module set file format and some frequently asked questions.

It would be good to include some information on setting up JHBuild’s tinderbox mode (like Luis has). Getting a few more tinderboxes running for Gnome on other platforms such Solaris/SPARC would be really useful — Luis’s build logs have already helped track down a few build failures, so having build information for a few more platforms would be very useful.

New Computer

I just got the last components for my new computer (an Athlon 64 system). It is a fair bit faster than the laptop I’ve been doing most of the development on, so should be quite nice once it is all set up.

It is amazing how much hardware has improved and gone down in price. The motherboard alone is packed with features I wouldn’t have expected for something costing AU$220:

  • Gigabit ethernet
  • An 802.11g wireless card (PCI)
  • An extra Promise SATA chip, bringing the number of SATA connectors up to 4, and the IDE connectors to 3.
  • Firewire
  • SPDIF output (both electrical and optical).
  • 6 headphone-style jacks on the back, so you can get 6 channel audio output without losing your line in and microphone jacks.

I also got a Raptor hard drive for the system. These drives seem to have up to twice the performance of most 7200rpm desktop drives, and make a big difference to the overall performance of the system.

It should be a nice system once I finish building it. Also, since it is an x86-64 system, it effectively provides two architectures to test stuff on.

4 October 2004

Icon Theme APIs (continued)

Of course, after recommending that people use gtk_icon_theme_load_icon() to perform the icon load and scale the icon for you, Ross manages to find a bug in that function.

If the icon is not found in the icon theme, but instead in the legacy $prefix/share/pixmaps directory, then gtk_icon_theme_load_icon() will not scale the image down (it will scale them up if necessary though).

jhbuild

Jhbuild now includes a notification icon when running in the default terminal mode. The code is loosely based on Davyd’s patch, but instead uses Zenity’s notification icon support. If you have the HEAD branch of Zenity installed, it should display without any further configuration. Some of the icons are a little difficult to tell apart at notification icon sizes, so it would be good to update some of them.

DVDs

The Double the Fist DVD is great. I hope they do another season, and release the second half of the first season on DVD. It is a satire on extreme sports and reality TV shows among other things, and is worth watching. Apparently it was originally shown on ABC digital, so not many people saw it during its original screening (digital television is fairly new in Australia, and equipment is still fairly expensive).

6 September 2004

linux.conf.au

The LCA2004 team have put together the conference CD and DVD. Apparently they will arrive in the mail in about a week.

They put the CD contents on the web first, and I was a bit disappointed that the recording of my talk was missing (it does include my slides though). However, when they put the DVD contents up I saw that it included a video recording of the talk, which is pretty cool.

There are links to the CD and DVD contents on the wiki. The video recording can be found by following one of the “Explore DVD” links, and looking at the entry second from the bottom. There is also a video of Havoc’s keynote in there.

jhbuild

It sounds like Fluendo are looking at using the Subversion support I added to jhbuild. There were a few bugs in the code that jdahlin fixed, but it seems to be working pretty well. I still need to fix up the Arch support so that you don’t need tla unless you actually build a module managed by Arch.

I’ve also dropped one of the old versions of Automake (1.6) from the bootstrap moduleset and sanity checks. Maybe after Gnome 2.8 is out we can clean up the last few modules still requiring Automake 1.4, which should drop the number of Automake versions I need to deal with even further.

Elections

Today is the last day people can enrol to vote in the federal election. Last week we had John Howard defending one of his part members, Trish Worth, for comparing refugees to animals at a forum organised by the group Justice for Refugees.

There is also a Liberal (Peter King) who lost preselection, but is still running as an independent. He has been accused of splitting the conservative vote, which is a bit strange. I’d assume that conservatives who vote for him would give their second preference to the Liberal candidate, and vice versa. What might happen is that Labor voters might pick the independent candidate over the Liberal candidate (this happened in my electorate when a similar thing happened a few years back).

Meanwhile, the National party leader is going round telling people that the Greens are really communists: “They are watermelons. many of them – green on the outside and very, very, very red on the inside.”

One other weird thing was the postal vote applications sent out by the current MP. The weird thing was that they came with reply paid envelopes to send the application back to the MP instead of the AEC. She explained why in response to a letter in the local paper, but it still seems a bit weird for the applications to pass through the office of the currently elected member.

20 May 2004

Mail Viruses

The barrage of mail viruses and their side effects is getting quite annoying. In the past week, I’ve had a gnome.org mailing list subscriptions disabled twice. After looking at the mailing list archive, it was pretty obvious why.

The mail server that serves my account is set up to reject windows executables a few other viruses at SMTP delivery time (so it isn’t responsible for generating bounces). Unfortunately, a number of viruses got through to the mailing lists and were subsequently rejected before reaching my account. After a certain number of bounces of this type, mailman helpfully disables delivery.

It’d be nice if mail.gnome.org was set up to reject these sort of messages too (in the case of gnome.org it’d probably be safe to block zip files as well, which would cut out virtually all the viruses).

It also seems that the email viruses don’t pick the sender and recipient completely at random. Apparently a number of infected machines keep on mailing the XML mailing list with my address as the sender. It got so bad that the list admin put me in the “always moderate” list. Of course, this meant that I ended up receiving many messages telling me my message awaits moderation (which are pretty easy to filter). Luckily the new version of Mailman limits itself to 10 of these messages a day.

jhbuild

I’ve merged in some of Thomas Fitzsimmons’ jhbuild patches. It isn’t yet at a stage where you can build GCJ using an unmodified jhbuild, but we’ve got some of the basics in there. A big part of the changes involve adding support for srcdir != builddir builds, which is apparently the preferred way of compiling GCJ. This is accomplished by setting the buildroot config key to the directory where you want builds to be performed. Things aren’t fully working yet, but at least some modules build in this mode. We’ll probably need to add support for marking some modules as not supporting srcdir != builddir builds, since some modules will most likely never support it.

gnome-common

I’ve been doing some work to simplify the gnome-common autogen script. A lot of the infrastructure dates back to the early 2.0 days where it was important to make sure developers could hack on 1.x apps and 2.0 stuff at the same time, which involved complicated infrastructure to make sure 2.0 packages didn’t see the Gnome 1.x autoconf macros and vice versa.

Since then things have changed. Developing Gnome 1.x apps isn’t really a priority any more (and no one was using the stuff installed by gnome-common for 1.x work anyway). We also have far fewer autoconf macros in gnome-common, and they aren’t particularly Gnome 2 specific. This is partly because I killed a lot of them last year, and deprecated most of the rest. While looking through the macros this time, it turned out I could remove another one, and get rid of the deprecated macros altogether. This just leaves some macros for setting compiler warning flags, one for adding a --enable-debug configure option.

The patch moves the remaining autoconf macros to the normal $(datadir)/aclocal directory so that aclocal can find it easily, and install the common autogen script as $(bindir)/bin/gnome-autogen.sh (which was previously a small script that would choose which set of macros and autogen script to call based on an environment variable).

Hopefully these simplifications will make it easier to debug autotool failures in the various Gnome packages. Many people seem to find autoconf hard enough to understand as is without us making things more complicated and adding extra ways that things could fail.