This is a follow-up to Carlos Soriano’s blog post about the new GNOME workflow that has emerged following the transition to gitlab.gnome.org. The post is pretty damn good and if you haven’t read it already you should. In this post I will walk through setting up the Flatpak build and test job that runs on the nautilus CI. The majority of the work on this was done by Carlos Soriano and Ernestas Kulik.
Let’s start by defining what we want to accomplish. First of all we want to ensure that every commit commit will be buildable in a clean environment and against a Flatpak runtime. Second to that, we want to ensure that the each project’s test suite will be run and pass. After these succeed we want to be able to export the resulting Flatpak to install and/or test it locally. Lastly we don’t want to waste precious time of the shared CI runners from other projects so we want to utilize Flatpak’s ostree artifacts for caching.
To summarize we want to achieve the following:
- Build the project
- Run the Test-Suite
- Create a Flatpak package/bundle and export it
- Use a caching mechanism to reduce build times
Building the project
If your Flatpak manifest targets gnome-3.26/3.28 or Freedesktop-1.6 runtime you can use these container images, provided by the Flatpak project directly. They are a good fit for your stable release branches too. Nautilus master branch though, as most GNOME projects, targets the GNOME Nightly runtime. I created a custom image using the same process the stable 3.26/3.28 images are built. It will be updated every day an hour after the new Nightly runtime is composed. You can use it by changing the image: key in the .gitlab-ci.yml to point to registry.gitlab.com/alatiera/gnome-nightly-oci/gnome-master:latest. Currently we invoke Flatpak builder manually to build the resulting Flatpak, the reason this happens is that the manifest is always pointing at Gnome/nautilus/master branch which would ignore the fork/branch we want it to build. That’s why we do the following in nautilus to sidestep that.
script: - flatpak-builder --stop-at=nautilus app build-aux/flatpak/org.gnome.Nautilus.json # Make sure to keep this in sync with the Flatpak manifest, all arguments # are passed except the config-args because we build it ourselves - flatpak build app meson --prefix=/app --libdir=/app/lib -Dprofile=development -Dtests=all _build - flatpak build app ninja -C _build install - flatpak-builder --finish-only --repo=repo app build-aux/flatpak/org.gnome.Nautilus.json
This builds all the modules up till nautilus. Then we take over and build nautilus ourselves with from the local checkout. Finally we call again the manifest to finish the build.
It works with Sdk-extensions too
There’s a small caveat here, I was only able to use sdk-extensions with flatpak-builder.
It’s probably possible to use flatpak build too, but my knowledge of Flatpak
is limited. If you know of a way to do it better please let me know!
I’ve created a Rust-sdk image for my own use. Here is an example of how to use its used.
If you need any other sdk-extension, like C#, open an issue in this repo or even better make an MR!
Running tests inside the Flatpak environment
In order to run the nautilus test suite we will add the following line:
flatpak build app ninja -C _build test
If your testsuite requires a display, you could use Xvfb. Since it’s quite common for GNOME apps I’ve included it in the gnome-nightly container image directly so you won’t have it to install it. Hopefully you can just prefix the above command with xvfb-run ... args cmd
xvfb-run -a -s "-screen 0 1024x768x24" flatpak build app ninja -C _build test
Thanks to Emmanuele Bassi for showing me how to get the display tests up and running withxvfb.
Retrieving a Flatpak package
To export a Flatpak bundle, named nautilus-dev.flatpak, we will add the following line:
flatpak build-bundle repo nautilus-dev.flatpak --runtime-repo=https://sdk.gnome.org/gnome-nightly.flatpakrepo org.gnome.NautilusDevel
Then we will add an artfact: block in order to extract our Flatpak package and be able to download it and extract it locally.
artifacts:
paths:
- nautilus-dev.flatpak
expire_in: 2 days
After that there should be a “Browse” and a “Download” button in the job’s logs from where you can download the Flatpak bundle. You can either get a zip with all the artifacts upon clicking “Download” or get the individual nautilus-dev.flatpak from “Browse”. After that to install it you can either open it with gnome-software (probably KDE discover too) or with the following command:
flatpak install --bundle nautilus-dev.flatpak
Caching in-between builds
In order to introduce caching in between CI runs we just need to add the following lines. In principle this should work, but there seem to be frequent cache misses that I am still investigating. If anyone is able to reduce the misses somehow please let me know.
cache:
paths:
- .flatpak-builder/cache
Complete config
Here is how the nautilus .gitlab-ci.yml config for the Flatpak job looks like right now. It might have slightly changed depending on when you read this.
flatpak:
image: registry.gitlab.com/alatiera/gnome-nightly-oci/gnome-master:latest
stage: test
script:
- flatpak-builder --stop-at=nautilus app build-aux/flatpak/org.gnome.Nautilus.json
# Make sure to keep this in sync with the Flatpak manifest, all arguments
# are passed except the config-args because we build it ourselves
- flatpak build app meson --prefix=/app --libdir=/app/lib -Dprofile=development -Dtests=all _build
- flatpak build app ninja -C _build install
- flatpak-builder --finish-only --repo=repo app build-aux/flatpak/org.gnome.Nautilus.json
# Make a Flatpak Nautilus bundle for people to test
- flatpak build-bundle repo nautilus-dev.flatpak --runtime-repo=https://sdk.gnome.org/gnome-nightly.flatpakrepo org.gnome.NautilusDevel
# Run automatic tests inside the Flatpak env
- xvfb-run -a -s "-screen 0 1024x768x24" flatpak build app ninja -C _build test
artifacts:
paths:
- nautilus-dev.flatpak
- _build/meson-logs/meson-log.txt
- _build/meson-logs/testlog.txt
expire_in: 2 days
cache:
paths:
- .flatpak-builder/cache
Going forward
The current config is not ideal yet. We have to keep it in sync with various parts of the Flatpak manifest and essentially replicate half of the functionality that’s specified in it. It would be nice if we could use the usual oneliner to build the Flatpak instead.
Also the cache misses issues mentioned above are driving me mad, when setting up gnome-builder‘s CI it would spend 11/12min rebuilding all the dependencies and half a minute building gnome-builder. What seems to happen is that it hits the ostree cache points for gnome-builder, but it is rejecting them for the modules/dependencies. Never figured out why sadly.
The Wikipedia page of Xvfb states that it has been replaced by xf86-video-dummy since X.org 7.8 and someone should probably look into using that for the tests instead.
But this setup should be good enough to get you started. If you are an app maintainer and would want to set this up but don’t have time or you are having trouble with something I want to hear from you! Feel free to ping me anytime at #gnome-hackers or send me an email.