Random stuff

JHBuild Improvements

I’ve been doing most JHBuild development in my bzr branch recently. If you have bzr 0.8rc1 installed, you can grab it here:

bzr branch http://www.gnome.org/~jamesh/bzr/jhbuild/jhbuild.dev

I’ve been keeping a regular CVS import going at http://www.gnome.org/~jamesh/bzr/jhbuild/jhbuild.cvs using Tailor, so changes people make to module sets in CVS make there way into the bzr branch. I’ve used a small hack so that merges back into CVS get recorded correctly in the jhbuild.cvs branch:

  1. Apply the diff between jhbuild.cvs and jhbuild.dev to my CVS checkout and commit.
  2. Modify tailor to pause before committing the to jhbuild.cvs.
  3. While tailor is paused, run bzr revert followed by a merge of the same changes from jhbuild.dev.
  4. Let tailor complete the commit.

It’s a bit of a hack, but it allows me to do repeated merges from the CVS import to my development branch (and back again). It also means that any file moves I do in my bzr branch are reflected in the CVS import when merged.

So now when filing bug reports on jhbuild, you can submit fixes in the form of bzr branches as well as patches.

So, on to the improvements:

Generic Version Control Interface

Previously, to add support for a new version control system the following additions were needed:

  • Some code to invoke the version control utility to make checkouts and update working trees.
  • Code to implement the build state machine for modules using the version control system (these classes would generally derive from AutogenModule which implemented most of the build logic).
  • Code to create instances of the above module type when parsing .modules files.

This was quite a bit of work, and in the end would only help if the code in question was set up to build the same way as most Gnome modules (i.e. with a autogen.sh script and autotools). If you wanted to build a module using Python distutils out of Subversion, a new module type would be needed. If you wanted to build a distutils module from a tarball, that would be another module type again.

With the new system, the different version control support modules provide a common interface. This means that a single module type is capable of implementing the build state machine for any version control system. Similarly, it should now be possible to implement distutils module support such that it will work with any supported version control system.

This work is not yet finished though. A bit more work needs to be done to parse version control system agnostic module definitions from .modules files. When this is done, a fair bit of the current syntax can be deprecated and eventually removed. When this is done, adding support for a new version control system shouldn’t take more than 100-200 lines.

Module Type Simplifications

As well as reducing the number of module types that need to be maintained in JHBuild, I’ve been working on simplifying the code in these module types. Previously, each stage of a module build was represented by a method call on the module type. The return value of the method was used to say (a) whether the stage succeeded or not, (b) what the next state would be and (c) if an error occurred some alternative next states to go to (e.g. offer to rerun autogen.sh).

With the new system, the next state and error states are declared as attributes on the method object. The method can indicate a failure by raising a particular exception. This greatly simplifies the cases where a build stage involves a number of separate actions that could each fail individually, since the exception cuts processing short without the error checks getting in the way of the code.

There are still a few module build stages not converted to the new system since their next state depends on various config settings (e.g. if running “make check” has been enabled or not). Since these generally involve skipping a stage based on some criteria, the plan is to move the logic to the stage being skipped, which should simplify things further.

New Default Branch Format in Bzr

One of the new features in the soon to be released bzr 0.8 is the new “knit” storage format.

When comparing the size of the repository data for jhbuild with “knit” and “metadir” formats (metadir is just the old storage format with repository, branch and checkout bookkeeping separated), I see the following:

metadir knit
Size 9.9MB 5.5MB
Number of files 1267 307

The reason for the smaller number of files is that information about all revisions in the repository is now stored together rather than in separate files. So the file count comes out at a constant plus 2 times the number of tracked files (a knit index file plus the knit data file). For comparison, the CVS repository I imported this from was 4.4MB, and comprised 143 files.

As well as reducing storage requirements, the new knit repository format is designed to reduce network traffic. With the current weave repository format, the weave file for each file touched by a commit gets rewritten to include the contents of the new revision. In contrast to this, the information about the new revision can simply be appended to the knit data file and the knit index file updated to match. This means publishing a branch to a server via sftp mainly involves append operations, resulting in a nice speed up.

Similarly when pulling new changes from a published branch, bzr only needs to download a knit index to find out which sections of the knit data are missing locally. It can then ask for just the changed sections (by an HTTP range request or a partial read with sftp), rather than downloading the entire contents of the changed weaves.

Overall, this should make bzr 0.8 a lot more usable than 0.7 for various network operations.

Repositories in Bzr

One of the new features comming up in the next release of bzr is support for shared repositories. This provides a way to reduce disk space needed to store multiple related branches. To understand how repositories work, it helps to know a bit about how branches are stored by bzr.

[bzr repository diagram]

There are three concepts that make up a bzr branch:

  1. A checkout or working tree. This is the source files you are working with. It represents the state of the source code at some recorded revision plus any local changes you’ve made. In the diagram on the right, it is represented as the red node.
  2. The branch, consisting of a linear sequence of revisions. This is represented by the blue nodes in the diagram. Note that there may be multiple paths from the first revision to the current revision due to branching and merging. The branch revision history indicates the path that was taken by this particular branch.
  3. The repository, being a store of the text of all the revisions in the ancestry of the branch, plus metadata about those revisions. This essentially stores information about every node and edge in the diagram.

In previous versions of bzr, this information was not clearly separated. However with the new default branch format in bzr 0.8 they are separated, and a particular directory need not contain all three parts, which is what makes the space savings and performance improvements possible.

One of the biggest space savings is achieved from sharing the repository data between branches. If a particular branch does not contain any repository information, bzr will recursively check the parent directory til it finds a repository. If a collection branches share some of their history, then the single shared repository will be significantly smaller than the space used if each branch had its own repository data.

Another way to reduce disk usage is to create branches without checkouts. This is useful when publishing a branch, since people pulling or merging from that branch don’t use the checkout files.

Finally, it is possible to create a checkout which does not contain branch or repository data, instead containing a pointer to where that data is located. This is quite useful when combined with a central shared repository.

So how big is this space saving? When I converted JHBuild to bzr, the repository data totals to 10MB, the branch data totals 100KB and a checkout is 1.4MB.

So to publish a second branch without the use of shared repositories means another 10MB of storage (a bit more if I include a checkout at the published location). If I use shared repositories, the cost of the second branch is 100KB plus an amount proportional to the size of the changes I make on that branch. So for many projects, the cost of publishing another branch is lost in the noise.

intltool and po/LINGUAS

  • Post author:
  • Post category:Uncategorized

Rodney: my suggestions for intltool were not intended as an attack. I just don’t really see much benefit in intltool providing its own po/Makefile.in.in file.

The primary difference between the intltool po/Makefile.in.in and the version provided by gettext or glib is that it calls intltool-update rather than xgettext to update the PO template, so that strings get correctly extracted from files types like desktop entries, Bonobo component registration files, or various other XML files.

The current method intltool uses to get intltool-update called (providing its own po/Makefile.in.in) is a lot better than the previous method (maintaining patches for the po/Makefile.in.in files from various versions of gettext and then deciding which one to apply), however it can make it difficult to take advantage of new gettext features (the po/LINGUAS file being the most recent example). If it was possible for intltool-update to be called without any modification to the po/Makefile.in.in file that gettext installs then this sort of problem wouldn’t occur.

The standard po/Makefile.in.in uses the makefile variable $(XGETTEXT) as the program to extract translations for the PO template. If intltool had a program (or a mode for one of the existing programs) that was command line argument compatible with xgettext, then all that would be necessary would be to redefine $(XGETTEXT) to the appropriate value. Since $(XGETTEXT) is set through a simple autoconf substitution, this should be very easy to do from intltool’s M4 autoconf macro.

po/LINGUAS

One issue that was meantioned as a Gnome Goal was to switch packages to use a po/LINGUAS file.

The idea makes sense — translators only need to edit a simple text file to add a new translation to an application, rather than having to modify the configure.in/configure.ac file without breaking things. Unfortunately, the suggested way of supporting this is a pretty big hack. A better long term solution would be to use the upstream gettext macros and po/Makefile.in.in infrastructure.

For a Gnome module that doesn’t use intltool, the following steps should work.

  1. Make sure the module is being built with Automake 1.8 or 1.9. If it isn’t, upgrade to 1.9.
  2. Create an m4 subdirectory in your project if it doesn’t exist, add it in CVS and then create and add a m4/.cvsignore file (there are a number of files that will get created here by gettext that you don’t want to check into CVS).
  3. Mark the m4 subdirectory as the macro dir in the configure.ac file:
    AC_CONFIG_MACRO_DIR([m4])

    And make sure that the macro dir gets checked if the makefile reruns aclocal:

    AC_SUBST([ACLOCAL_AMFLAGS], ["-I $ac_macro_dir \${ACLOCAL_FLAGS}"])
  4. If you aren’t using the gnome-common autogen.sh script, you will also need to make sure that aclocal is called with “-I m4“. If you are using the gnome-common script, then this will happen automatically.
  5. Remove the AM_GLIB_GNU_GETTEXT call from configure.ac and replace it with:
    AM_GNU_GETTEXT([external])
AM_GNU_GETTEXT_VERSION([0.14.1])
  6. If you aren’t using the gnome-common autogen.sh script, change the call to glib-gettextize to autopoint, and make sure it gets run before aclocal (again, unneeded if you are using the gnome-common script).
  7. Now rerun autogen.sh so that autopoint gets run. This should result in a number of files getting created under m4, and some new files under po.
  8. Copy po/Makevars.template to po/Makevars and customise the variables. You might want to set DOMAIN to $(GETTEXT_PACKAGE) rather than $(PACKAGE). Add this new file in CVS.
  9. Update po/LINGUAS from the ALL_LINGUAS variable in configure.ac, and then remove the ALL_LINGUAS definition. Add po/LINGUAS to CVS.
  10. Finally update m4/.cvsignore and po/.cvsignore to ignore the new generated files.

As I said at the start, this change is only appropriate for apps not using intltool, since intltool overwrites the po/Makefile.in.in file with an incomaptible version.

To get things working with intltool, I believe it would make most sense to modify intltool as follows:

  • Make intltool provide some commands that are command line argument compatible with xgettext and msgmerge.
  • Make IT_PROG_INTLTOOL alter XGETTEXT and MSGMERGE with the appropriate intltool functions.
  • Don’t overwrite po/Makefile.in.in.
  • If additional makefile rules are needed in the po subdirectory, install a po/Rules-intltool file containing them. The gettext M4 macros will include them into the resulting Makefile.