This is the first post in a series about tools used to develop GNOME and GNOME OS. Part two coming soon.
In the old age, developing the desktop was simple™️. You only had to install a handful of toolchains, development headers, and tools from the distribution packages, run make install, execute say_prayer.sh and if you had not eaten meat on Friday you had a 25% chance for your system to work after a reboot.
But how do you develop software in the brave new world of Image-Based systems and containerization?
Well, if you are an application developer Flatpak makes this very simple. Applications run against a containerized runtimes, with their own userspace. The only thing they need from the host system is a working Desktop Environment, Flatpak, and Portals. Builder and flatpak-builder along with all the integration we built into the Desktop make sure that it will be a breeze.
But what if you are developing a system component, like a daemon or perhaps GNOME Shell?
Till now the goto solution has been “Grab a fedora container and open enough sandbox holes until things work”. This is what Fedora Toolbox does, and it works great for a lot of things. However the sandbox makes things way more difficult than they need to be, and its rather limiting on what parts of the system you can test. But there is another way.
In GNOME OS we provide two images. The first one is how we envision GNOME to be, with all the applications and services we develop. The second one is complimentary, and it adds all the development tools, headers and debugging information we need to develop GNOME. Now the popular image based OSes don’t provide you with something like that, and you have to layer everything on your own. This makes it much harder to do things like running gdb against the host system. But in GNOME OS its easy, and by switching to the Development Edition/Image you get access to all the tools required to build any GNOME component and even GNOME OS itself.
Alright, I have a compiler and all the dependencies I need to build my project but /usr is still immutable. How do I install and run my modified build?
I am glad you asked! Enter systemd-sysext.
If you are familiar with ostree-based operating systems, you probably used ostree admin unlock at some point. systemd-sysext is another take on the same concept. You build your software as usual with a /usr (or /opt) prefix and install it in a special folder along with some metadata. Then upon running systemd-sysext merge, systemd creates an ovelayfs with the merged contents of /usr and your directory, and replaces the existing /usr mount atomically.
The format is very simple, there are 2 directories we care about for our usecase:
- /run/extensions
- /var/lib/extensions
/run/extensions is a temporary directory that’s wiped on reboot, so it’s excellent for experimental changes that might take down the system. After a power cycle you will boot back in a clean slate.
/var/lib/extensions is for persistent changes. Say you are experimenting with UI changes or want to thoroughly test a patch set for a couple days. Or you might simply want to have a local FFmpeg build with extra codecs, cause god lawyer forbid we have working video playback.
If you are installing a build in /run/extensions only thing you need to do is the following two commands:
sudo DESTDIR=/run/extensions/custom_install meson install -C _build
sudo systemd-sysext refresh --forceThis installs our tree into the custom_install directory and tells systext to refresh, which means that it will look at all extensions we might have, unmerge (unmount) them and then merge the updated contents again. Congrats, you can launch your new binaries now.
Normally sysext will check if the extension is compatible with the host operating system. This is done by using a metadata file that includes the ID= field you’d find in /etc/os-release (see more at the systemd-sysext manpage). The --force argument ignores this, as we build the project on host and we can be reasonably sure things will work.
If we want to have our build to be persistent and available across reboots, we have to install it in /var/lib/extensions/ and create the metadata file so systemd-sysext will mount it automatically upon boot. It’s rather simple to do but gets repetitive after a while and hunting in your console history is never fun. Here is a simple script that will take care of it.
#! /bin/bash
set -eu
EXTENSION_NAME="custom_install"
# DESTDIR="/var/lib/extensions/$EXTENSION_NAME"
DESTDIR="/run/extensions/$EXTENSION_NAME"
VERSION_FILE="$DESTDIR/usr/lib/extension-release.d/extension-release.$EXTENSION_NAME"
sudo mkdir -p $DESTDIR
# rm -rf _build
# meson setup _build --prefix=/usr
# meson compile -C _build
sudo meson install --destdir=$DESTDIR -C _build --no-rebuild
sudo mkdir -p $DESTDIR/usr/lib/extension-release.d/
# ID=_any to ignore it completly
echo ID=org.gnome.gnomeos | sudo tee $VERSION_FILE
sudo systemd-sysext refresh
sudo systemd-sysext listHere’s a demo, I used sysprof as the example since it’s more visible change than my gnome-session MR. You can also test gnome-shell the same way by installing, refreshing and then logging out and login in again.
Anothe example was from today, where I was bisecting gdm. Before systemd-sysext, I’d be building complete images with different commits of gdm in order to bisect. It was still fast, at ~25m per build for the whole OS, but that’s still 24 minutes more after it becomes annoying.
Now, I switched to the gdm checkout, started a bisect, compiled, installed and then run systemctl restart gdm.service. The login greeter would either come up and I’d continue the bisect, or it would be blank at which point I’d ssh in, switch to a tty or even hit the power button and continue knowing it’s a bad commit. Repeat. Bisect done in 10 minutes.
And the best is that we can keep updating the operating system image uninterrupted, and on next boot the trees will get merged again. Want to go back? Simply systemd-sysext unmerge or remove the extension directories!
One caveat when using systemd-sysext is that you might occasionally need to systemctl daemon-reload. Another one when using custom DESTDIRs, is that meson won’t run post-install/integration commands for you (nor would it work), so if you need to recompile glib schemas, you will have to first systemd-sysext refresh, compile the schemas, place the new binary in the extension or make a new extension, and systemd-sysext refresh again.
Another use case I plan on exploring in the near future, is generating systemd-sysext images in the CI for Merge Requests, same way we generate Flatpak Bundles for applications. This proved to be really useful for people wanting to tests apps in an easier way. Begone shall be the days where we had to teach designers how to setup JHBuild in order to test UI changes in the Shell. Just grab the disk image, drop it in GNOME OS, refresh and you are done!
And that’s not all, none of this is specific to GNOME OS, other than having bleeding edge versions of all the gnome components that is! You can use systemd-sysext the same way in Fedora Workstation, Arch, Elementary OS etc. The only requirement is having recent enough systemd and a merged /usr tree. Next time you are about to meson install on your host system, give systemd-sysext a try!
This whole post is basically a retelling of Lennart’s blogpost about systemd-sysext, It has more details and you should check it out. This is also how I initially found out about this awesome tool! I tried to get people hooked on it in the past but it didn’t bear fruit, so here’s one post specific to GNOME development!
Happy Hacking!