Rust in Peace

Month: October 2025

  • Nightly Flatpak CI gets a cache

    Recently I got around tackling a long standing issue for good. There were multiple attempts in the past 6 years to cache flatpak-builder artifacts with Gitlab but none had worked so far.

    On the technical side of things, flatpak-builder relies heavily on extended attributes (xattrs) on files to do cache validation. Using gitlab’s built-in cache or artifacts mechanisms results in a plain zip archive which strips all the attributes from the files, causing the cache to always be invalid once restored. Additionally the hardlinks/symlinks in the cache break. One workaround for this is to always tar the directories and then manually extract them after they are restored.

    On the infrastructure of things we stumble once again into Gitlab. When a cache or artifact is created, it’s uploaded into the Gitlab’s instance storage so it can later be reused/redownloaded into any runner. While this is great, it also quickly ramps up the network egress bill we have to pay along with storage.
 And since its a public gitlab instance that anyone can make request against repositories, it gets out of hand fast.

    Couple weeks ago Bart pointed me out to Flathub’s workaround for this same problem. It comes down to making it someone else problem, and ideally one someone who is willing to fund FOSS infrastructure. We can use ORAS to wrap files and directories into an OCI wrapper and publish it to public registries. And it worked. Quite handy! OCI images are the new tarballs.

    Now when a pipeline run against your default branch (and assuming it’s protected) it will create a cache artifact and upload to the currently configured OCI registry. Afterwards, any build, including Merge Request pipelines, will download the image, extract the artifacts and check how much of it is still valid.

    From some quick tests and numbers, GNOME Builder went from a ~16 minute build to 6 minutes for our x86_64 runners. While on the AArch64 runner the impact was even bigger, going from 50 minutes to 16 minutes. Not bad. The more modules you are building in your manifest, the more noticeable it is.

    Unlike Buildstream, there is no Content Addressable Server and flatpak-builder itself isn’t aware of the artifacts we publish or can associate them with the cache keys. The OCI/ORAS cache artifacts are manual and a bit hacky of a solution but works well in practice and until we have better tooling. To optimize a bit better for less cache-misses consider building modules from pinned commits/tags/tarballs and building modules from moving branches as late as possible.

    If you are curious in the details, take a look at the related Merge Request in the templates repository and the follow up commits.

    Free Palestine ✊

  • The Flatpak Runtime drops the 32-bit compatibility extension

    Last month GNOME 49 was released, very smooth overall, especially given the amount of changes across the entire stack that we shipped.

    One thing that is missing and that flew under the radar however, is that 32 bit Compatibility extension (org.gnome.Platform.i386.Compat) of the GNOME Flatpak Runtime is now gone. We were planning on making an announcement earlier but life got in the way.

    That extension is a 32-bit version of the Runtime that applications could request to use. This is mostly helpful so Wine can use a 32 bit environment to run against. However your wine or legacy applications most likely don’t require a 32 bit build of GTK 4, libadwaita or WebkitGTK.

    We rebuild all of GNOME from the latest commits in git in each module, at least twice a day. This includes 2 builds of WebkitGTK, a build of mozjs and a couple of rust libraries and applications. Multiplied for each architecture we support. This is no small task for our CI machines to handle. There were also a couple of updates that were blocked on 32-bit specific build failures, as projects rarely test for that before merging the code. Suffice to say that supporting builds that almost nobody used or needed was a universal annoyance across developers and projects.

    When we lost our main pool of donated CI machines and builders, the first thing in the chopping block was the 32-bit build of the runtime. It affected no applications, as none are relying on the Nightly version of the extension but it would affect some applications on Flathub once released.

    In order to keep the applications working, and to avoid having to overload our runners again, we thought about another approach. In theory it would be possible to make the runtime compatible with the org.Freedesktop.i386.Compat extension point instead. We already use freedesktop-sdk as the base for the runtime so we did not expect many issues.

    There were exactly 4 applications that made use of the gnome specific extension, 2 in Flathub, 1 in Flathub Beta and 1 archived.

    Abderrahim and I worked on porting all the application to the GNOME 49 runtime and have Pull Requests open. The developers of Bottles were great help in our testing and the subsequent PR is almost ready to be merged. Lutris and Minigalaxy need some extra work to upgrade the runtime but its for unrelated reasons.

    Since everything was working we never re-published the i386 GNOME compatibility extension again in Nightly, and thus we also didn’t for GNOME 49. As a result, the GNOME Runtime is only available for x86_64 and AArch64.

    Couple years ago we dropped the normal armv7 and i386 build as of the Runtime. With the i386 compatibility extension also gone, it means that we no longer have any 32 bit targets we QA before releasing GNOME as a whole. Previously, all modules we released would be guaranteed to at least compile for i386/x86 but going forward that will not be the case.

    Some projects, for example glib, have their own CI specifically for 32 bit architectures. What was a project-wide guarantee before, is now a per-project opt-in. While many maintainers will no longer go out of their way to fix 32 bit specific issues anymore, they will most likely still review and merge any patches sent their way.

    If you are a distributor, relying on 32 bit builds of GNOME, you will now be expected to debug and fix issues on your own for the majority of the projects. Alternatively you could also get involved upstream and help avoid further bit rot of 32 bit builds.

    Free Palestine ✊