Month: February 2021

A few days ago, I posted to desktop-devel-list asking how we can ensure releases happen, especially beta releases for the freeze. I was frustrated and my language was too abrasive, and I’m sorry for that. My intention was really to open a discussion on how we can improve our release process. Emmanuele replied with a thorough analysis of which bits are hard to automate, which I enjoyed reading.

Earlier today, I tweeted asking developers of other open source projects how they make releases, just to get a sense of what the rest of the world does. There have been a lot of responses, and it will take me a while to digest it all.

In the meantime, I wanted to share my process for rolling releases. I maintain five core GNOME modules, plus a handful of things in the wider open source world. My release process hasn’t fundamentally changed in the 18 years I’ve been a maintainer. A lot of other stuff has changed (merge requests, CI, freeze break approvals, etc), so I’m just trying to think of how any of this could be better. Anyway, here’s my process:

1. First, I run git status in my development checkout to do a sanity check for files I forgot to add to the repo. In at least one project, I have auto-generated docs files that I keep in git, because various tools rely on them being there.
2. Next, I always want to make sure I’m making releases from a clean checkout. So I will git clone a fresh checkout. Or sometimes, I already have a checkout I use just for releases, so I will git pull there.
3. Next, I actually roll a tarball before doing anything else, which I will promptly throw away. I’ve had a few times where I discovered a dist breakage after doing everything else, and I’ve had to start over.
4. Now it’s time to write a NEWS entry. I run git log --stat PREVTAG.. > changes, where PREVTAG is the tag name of the previous release. I edit changes to turn it into a NEWS entry, then I copy it to the top of the NEWS file.
5. I then bump the version number in either configure.ac or meson.build. I know a lot of people do pre-release version bumps. I don’t have a strong opinion on this, so I’ve never changed my habits.
6. Now it’s time to roll the tarball that I don’t throw away. The commands I run depend on the build system, of course. What matters is that I run these commands myself and have a tarball at the end.
7. Before I actually release that tarball, I run git commit and git push. If there have been any commits since I started, I either have to rebase and update the NEWS file, or do some branching. This is fortunately quite rare.
8. Also before releasing the tarballs, I tag the release with git tag -s and push the tag. Importantly, I only do this after pushing the commits, because otherwise if other commits have happened I have to do tag surgery. Nobody likes that.
9. Finally, I scp the tarball to a GNOME server, ssh into that server, and run a release script that the release team maintains.

The two things that take the most time are rolling a tarball (which I do at least twice), and creating the NEWS entry. Rolling tarballs is something that can happen in the background, so I usually have multiple terminal tabs, and I work on other releases while one release is building. So that part isn’t too bad, but sometimes I am just waiting on a build to finish with nothing else to do. I know some people auto-generate NEWS entries, or don’t write them at all, but I find hand-edited entries extremely valuable. (I do read them when writing help for apps, when I actually find time to do that, so a big thanks to app maintainers who write them.)

I’m tossing around in my head what a more GitLab-focused workflow would look like. I could imagine a workflow where I click a “Release” button, and I get prompted for a version number and a NEWS entry. I’m not even sure I care if the “news” is in a file in the tarball, as long as I know where to find it. (Distro packagers might feel differently. I don’t know what their processes look like.) I would still need to look at a commit log to write the news. And I guess I’d probably still want to do my own git status sanity check before going to GitLab, but I could probably catch a lot of that with either CI or local commit checks. Ensuring there are no dist breakages should almost certainly be done on every commit with CI.

I suppose another thing to consider is just maintainers remembering it’s time to make releases. I didn’t miss this one because I was eagerly awaiting playing with it and updating the help, but I’ve missed lots of releases in the past. Could we all get automatic issues added to our todo lists a few days in advance of expected releases? Would that be annoying? I don’t know.

I’ve just released the 40.alpha release of yelp-tools, the collection of tools for building and maintaining your documentation in GNOME. This is the first release using the new Meson build system. More importantly, it’s the first release since I ported the tools from shell scripts to Python.

Porting to Python is a pretty big deal, and it comes with more improvements than you might expect. It fixes a number of issues that are just difficult to do right in a shell script, and it’s significantly faster. For some commands, it can be as much as 20 times faster.

But that’s not all. You can now provide a config file with default values for all command-line arguments. This is useful, for example, with the --version option for yelp-check status. Previously, to ensure you weren’t getting stale status information, everybody had to remember to pass --version. Now, you can set the current version in your config file, and it will always do the right thing for everybody.

It gets better. The config file can specify custom checkers for yelp-check. I blogged about custom checkers a couple months ago. You can use XPath expressions to make assertions about a document. This is very similar to how Schematron works. But now you don’t have to figure out how to write a Schematron file, call xmllint with it, and parse the output.

Here’s an example of how to ensure that every page uses XInclude to include a boilerplate legal.xml file:

[namespaces]
mal = http://projectmallard.org/1.0/
xi = http://www.w3.org/2001/XInclude

[check:gnome-info-legal-xi]
select = /mal:page/mal:info
assert = xi:include[@href='legal.xml']
message = Must include legal.xml
xinclude = false

To run this check, you simply call:

yelp-check gnome-info-legal-xi

For more examples, check out the config file I already added to gnome-user-docs.

What I’d like to do now is figure out a good way to share custom checkers across modules, and then add CI pipelines to all GNOME modules with user docs to ensure consistency.