Release notes on GNOME Library

In a combined effort Frederic Peters and me moved the GNOME 2.12+ release notes to the GNOME Library. There have been various changes over the years in how these release notes were generated (only discussing GNOME 2.x). In the beginning, the release notes were stored in the releng module. Someone had to fetch the docbook sources, generate HTML from these files and commit the HTML to the www.gnome.org module (gnomeweb-wml). Starting with 2.10 the release notes were translated into various languages. Someone still had to commit the HTML files to gnomeweb-wml. That changed as of 2.12. The release-notes were put into gnomeweb-wml and its build was changed to automatically generate the HTML (in various languages). The release-notes build process within gnomeweb-wml wasn’t smart. It would generate HTML files no matter what. Even if you just changed a file in www.gnome.org/projects/ it would still generate the 2.12 (and newer) release notes in various languages. This severely slowed down the update of www.gnome.org. From 2.18 onwards the build was changed to not recreate release notes for older GNOME versions (as no change would be expected); although ideally it should just rebuild when needed.

As of now, you can see the release notes on the GNOME Library (I’ll wait with the URL until 2.22). This has the following benefits:

  • General GNOME Library goodness. Meaning: users should automatically get the release notes in their language (when available)
  • No duplication of docbook intelligence
  • Less maintenance
    Initially I changed the gnomeweb-wml release notes build system to change the layout to match
  • Only builds when needed

However, it took some effort to get these release notes on the GNOME Library. Most importantly, the GNOME Library builds from tarballs, not SVN. This meant somehow giving a new tarball to the GNOME Library after a commit. Secondly, GNOME Library assumes tarballs do not change their contents (e.g. you cannot just give it an updated release-notes-2.22.tar.gz ). And the last important issue was that the old release notes had some invalid docbook.

Creating a new tarball involved the moving of the 2.12+ release-notes from gnomeweb-wml into a new repository. This to make the tarball generating scripts easier, make damned-lies integration easier and it is a lot cleaner as well. Moving these files with history meant a painful combination of svndumpfilter, vim (to change the locations), and trial and error until the repository import worked again (due to usage of svn cp between release note versions). Basically the stuff changed from /trunk/www.gnome.org/start/$VERSION(/notes)?/ to /branches/gnome-$VERSION/help. The ‘help’ directory is there to ease damned-lies (has some defaults); same for the ‘branches’.
After the new repository was created, a tarball had to be made. Although actually a tarball per branch. This was a problem as although there was enough infrastructure to build something after a commit (the ‘gnomeweb’ scripts), these had no notion of branches. Everything works using trunk and the post-commit script assumes the commit was done in trunk. First task was making a post-commit script that would figure out which branch(es) a commit was made to. For this I started with a copy of the damned-lies post-commit script, changed it enough to introduce a horrible bug and spent loads of time figuring out why it wasn’t working in some cases. Oh well.
Next up was adding branch intelligence to the ‘gnomeweb’ scripts. There are basically two scripts. One notices that a commit was made. That script is started after receiving a certain email. The other script starts via cron and is smart enough to avoid updating e.g. GNOME Library if there is another update running atm. However, it will notice if a new commit was made while a build was running. In that case, it’ll rebuild again (and again). This is important as the GNOME Library doesn’t support 2 updated running at the same time (although it usually works fine). After the branch support was added, it was time to make ‘hook’ script (the ‘gnomeweb’ script optionally runs a different hook script depending on the svn repos name).

The hook script itself is pretty simple (not perfect, but works and it is pretty clear what is going on):

#!/bin/sh

DIR="$(basename "$(pwd)")"
TGZ="/usr/local/www/gnomeweb/libgo-extra-tarballs/$DIR.tar.gz"

cd help
for LANG in $(find -maxdepth 1 -type d)
do
    PO_FILE="$LANG/$LANG.po"
    if [ -e "$PO_FILE" ]
    then
        for XML_FILE in C/*.xml
        do
            DEST="$LANG/$(basename $XML_FILE)"
            BUILD=0
            test -e "$DEST" || BUILD=1
            test "$PO_FILE" -nt "$DEST" && BUILD=1
            test "$XML_FILE" -nt "$DEST" && BUILD=1
            if [ "$BUILD" = "1" ]
            then
                xml2po -e -p "$PO_FILE" "$XML_FILE" > "$DEST"
            fi
        done
    fi
done
cd ..

cd ..
rm -f "$TGZ"
tar czf "$TGZ" --exclude=".svn" "$DIR"

# Trigger update of GNOME Library
echo "$DIR forced" | /home/admin/bin/gnomeweb/set_buildflag.py -m library-web

Comments on above hook script: The gnomeweb script automatically runs this hook script from the checkout directory. Further, I added some assumptions to that script on how to name the checkout directory for a branch. Basically: $REPOSNAME-$BRANCHNAME, but if the branchname looks like: gnome-2-20, name the checkout directory like: $REPOSNAME-2.20. This allows the hook script to easily tar that directory. The directory name is important as (AFAIK) GNOME Library wants a ‘release-notes-2.20.tar.gz’ tarball to have a ‘release-notes-2.20’ directory.

Above hook script triggers a rebuild of the GNOME Library. I’ve added some (customizable) ordering to the gnomeweb scripts to ensure ‘library-web’ is checked after ‘release-notes’.

Frederic worked on making GNOME Library (AKA libgo / library-web) show these release-notes somewhere. I’m not fully aware of the various things he had to do, but hopefully he’ll make a post if something is missing (or I’ll update this one). Initially he had added to libgo to always rebuild certain modules (meaning: release-notes). Later he changed that to look at the timestamp of the tarball. Of course he had to add workarounds due to the various ways in which the release-notes XML was wrong. Stuff like a no/wrong ‘id’ being set on the article element. Another issue was with getting a title/abstract from the release-notes. This is done automatically and works perfectly for every document aside from the release-notes. However, for the release-notes it only gets some random text. Forcing Frederic to add an option not to fetch a title/abstract for some modules (release-notes) and using some standard text instead.
‘Layout’-wise Frederic made various important changes. First of all the release notes appeared under ‘users’, while the target is everyone (users, developers, administrators, …). Frederic made a new category for the release-notes called ‘misc’. Another change was to render the release notes ‘flat’. Meaning: everything in one big page. This as we learned in older release notes that people 1) didn’t bother to click the link to the release notes 2) often stopped at the first page. Further, he added a list of languages to the side of the release notes. Before we had a link to these languages within the release notes themselves. Meaning: if there we’re 45 languages, the translators would translate those links again. Now GNOME Library handles that. Also avoids changes to the release notes docbook just because it was translated.
He also made some changes that aren’t really release-notes specific. First of all a translated document will have links to the specific translated image. Meaning it links to image.png.es instead of image.png. Before it had a link to image.png. This relied on the user having the correct browser setting being correct. That broke down whenever you e.g. we’re intentially looking at a language other than the one your browser was set to. Second change was how links within the same document we’re made. Before it always had links like: $FILENAME#someid. Now it only does that if it links to another file. Otherwise if you’re looking at release-notes/2.20/, your browser would load index.html.en#someid — but that is the same document!

Currently a commit to release-notes will start the following process:
commit on socket -> post-hook command -> nasty gawk script -> email to menubar -> fwd’ed to window -> procmail as gnomeweb user -> setting of buildflag -> cron -> update_sites.py -> svn checkout/update -> release-notes hook script -> tarball -> forced libgo update -> library-web svn update -> library-web hook script -> lots of magic (I’m not Frederic) -> updated document. This whole process takes 5-7 minutes (due to cron — that is on purpose so it sometimes combines multiple commits).

6 Replies to “Release notes on GNOME Library”

  1. Thanks you. It’s great to know that this has been made sane.

    There’s still hope that the new plone-based site could make translated web sites work this well without this intermediate steps, but I don’t recommend holding our breath for that.

  2. There is only one small todo item left. Currently the release-notes tarballs have been set specifically in the library-web config. It should just add/look for release-notes-*.tar.gz. This is easily possible (as you can use Python in the config file), just hasn’t been done yet. Then a new GNOME just means adding another branch (e.g. svn cp).

    The various scripts are mostly generic enough that we can repeat this for other documents as well.

    I also have some thoughts about having library-web show SVN version by linking it (sort of) to damned-lies (meaning: same server). But that is a long term idea/wish.

  3. Re: “Second change was how links within the same document we’re made. Before it always had links like: $FILENAME#someid. Now it only does that if it links to another file. Otherwise if you’re looking at release-notes/2.20/, your browser would load index.html.en#someid — but that is the same document!”

    But if I click “Looking Forward to GNOME 2.22”, I get sent to “index.html.en#rnlookingforward”. This happens with any link in the header nav block.

    Also, is there a plan to allow access to these from the l.g.o home? The only obvious link seems to be from the w.g.o home (“GNOME 2.20 Released!”). But that doesn’t allow easy access to prior versions.

    Oh, just now tried “Download” tab on w.g.o – that leads to the release notes with a trailing “//”. Maybe “…/2.20/#rninstallation” instead?

  4. John: Thanks for your comments.

    Regarding the #someid stuff: It will do that when the document is regenerated. I tested it with 2.22, it works there. I’ll have lgo rebuild the older release notes. Every once in a while lgo will rebuild every document (takes hours), after that is done the whole site will be fixed.

    Regarding the ‘Download’: I don’t agree with having a ‘Download’ link. IMO we should find another word for that.

    ATM it isn’t shown on main lgo because we don’t want people to look at the 2.22 release notes yet. Have to check the plans for the ‘misc’ section. It should be clear what that contains. I don’t want people to e.g. first look into ‘Users’, then in ‘Misc’ when they couldn’t find something in ‘Users’.

  5. Olav and Frederic: Cool, great to know about 2.22 page working and just needing a regen for previous versions. Sorry about the noise, I guess I jumped the gun. 🙂

    “Download”: didn’t mean so much to keep it, just thought the redirection behavior odd (though if the nav is changing anyway, no worries).

    For my point about accessing them, I didn’t expect you’d want people looking at 2.22 notes yet. I just wasn’t sure if/how they’d be organized to allow access in the future. They’re all nicely archived, but the only way to look at prior notes (as far a I can tell) is to edit the URL. Not sure if there’s other info like that, that’s only available via Google (what I’ve resorted to to find the current release notes before 😉 ).

    I agree about the misc/user split being undesirable (maybe “misc” links are available in the navigation for the other three sections? i.e. physical location != logical location).

    Regardless, great job on the ongoing revamp of the *.g.o family of sites!

Comments are closed.