Today I woke up to a link of an interview from the current Fedora Project Leader, Matthew Miller. Brodie who conducted the interview mentioned that Miller was the one that reached out to him. The background of this video was the currently ongoing issue regarding OBS, Bottles and the Fedora project, which Niccolò made an excellent video explaining and summarizing the situation. You can also find the article over at thelibre.news. “Impressive” as this story is, it’s for another time.

What I want to talk in this post, is the outrageous, smearing and straight up slanderous statements about Flathub that the Fedora Project Leader made during the interview..

I am not directly involved with the Flathub project (A lot of my friends are), however I am a maintainer of the GNOME Flatpak Runtime, and a contributor to the Freedesktop-sdk and ElementaryOS Runtimes. I also maintain applications that get published on Flathub directly. So you can say I am someone invested in the project and that has put a lot of time into it. It was extremely frustrating to hear what would only qualify as reddit-level completely made up arguments with no base in reality coming directly from Matthew Miller.

Below is a transcript, slightly edited for brevity, of all the times Flathub and Flatpak was mentioned. You can refer to the original video as well as there were many more interesting things Miller talked about.

It starts off with an introduction and some history and around the 10-minute mark, the conversation starts to involve Flathub.

Miller: [..] long way of saying I think for something like OBS we’re not really providing anything by packaging that. Miller: I think there is an overall place for the Fedora Flatpaks, because Flathub part of the reason its so popular (there’s a double edged sword), (its) because the rules are fairly lax about what can go into Flathub and the idea is we want to make it as easy for developers to get their things to users, but there is not really much of a review

This is not the main reason why Flathub is popular, its a lot more involved and interesting in practice. I will go into this in a separate post hopefully soon.

Claiming that Flathub does not have any review process or inclusion policies is straight up wrong and incredibly damaging. It’s the kind of thing we’ve heard ad nauseam from Flathub haters, but never from a person in charge of one of the most popular distributions and that should have really really known better.

You can find the Requirements in the Flathub documentation if you spend 30 seconds to google for them, along with the submission guidelines for developers. If those documents qualify as a wild west and free for all, I can’t possibly take you seriously.

I haven’t maintained a linux distribution package myself so I won’t go to comparisons between Flathub and other distros, however you can find people, with red hats even, that do so and talked about it. Of course this is one off examples and social bias from my part. But it proves how laughable of a claim is that things are not reviewed. Additionally, the most popular story I hear from developers is how Flathub requirements are often stricter and sometimes cause annoyances.

Additionally, Flathub has been the driving force behind encouraging applications to update their metadata, completely reworking the User Experience and handling off permissions and made them prominent to the user. (To the point where even network access is marked as potentially-unsafe).

Miller: [..] the thing that says verified just says that it’s verified from the developer themselves.

No, verified does not mean that the developer signed off into it. Let’s take another 30 seconds to look into the Flathub documentation page about exactly this.

A verified app on Flathub is one whose developer has confirmed their ownership of the app ID […]. This usually also may mean that either the app is maintained directly by the developer or a party authorized or approved by them.

It still went through the review process and all the rest of requirements and policies apply. The verified program is basically a badge to tell users this is a supported application by the upstream developers, rather than the free for all that exists currently where you may or may not get an application released from years ago depending on how stable your distribution is.

Sidenote, did you know that 1483/3003 applications on Flathub are verified as of the writing of this post? As opposed to maybe a dozen of them at best in the distributions. You can check for yourself

Miller: .. and it doesn’t necessarily verify that it was build with good practices, maybe it was built in a coffee shop on some laptop or whatever which could be infected with malware or whatever could happen

Again if Miller had done the bare minimum effort, he would have come across the Requirements page which describes exactly how an Application in Flathub is built, instead of further spreading made up takes about the infrastructure. I can’t stress enough how damaging it has been throughout the years to claim that “Flathub may be potential Malware”. Why it’s malware? Because I don’t like its vibes and I just assume so..

I am sure If I did the same about Fedora in a very very public medium with thousand of listeners I would probably end up with a Layers letter from Redhat.

Now Applications in Flathub are all built without a network access, in Flathub’s build servers, using flatpak-builder and Flatpak Manifests which are a declarative format, which means all the sources required to build the application are known, validated/checksumed, the build is reproducible to the extend possible, you can easily inspect the resulting binaries and the manifest itself used to build the application ends up in /app/manifest.json which you can also inspect with the following command and use it to rebuild the application yourself exactly like how it’s done in Flathub.

$ flatpak run --command=cat org.gnome.TextEditor /app/manifest.json
{
  "id" : "org.gnome.TextEditor",
  "runtime" : "org.gnome.Platform",
  "runtime-version" : "47",
  "runtime-commit" : "d93ca42ee0c4ca3a84836e3ba7d34d8aba062cfaeb7d8488afbf7841c9d2646b",
  "sdk" : "org.gnome.Sdk",
  "sdk-commit" : "3d5777bdd18dfdb8ed171f5a845291b2c504d03443a5d019cad3a41c6c5d3acd",
  "command" : "gnome-text-editor",
  "modules" : [
    {
...

The exception to this, are proprietary applications naturally, and a handful of applications (under an OSI approved license) where Flathub developers helped the upstream projects integrate a direct publishing workflow into their Deployment pipelines. I am aware of Firefox and OBS as the main examples, both of which publish in Flathub through their Continues Deployment (CI/CD) pipeline the same way they generate their builds for other platforms they support and the code for how it happens is available on their repos.

If you have issues trusting Mozilla’s infrastructure, then how are you trusting Firefox in the first place and good luck auditing gecko to make sure it does not start to ship malware. Surely distribution packagers audit every single change that happens from release to release for each package they maintain and can verify no malicious code ever gets merged. The xz backdoor was very recent, and it was identified by pure chance, none of this prevented it.

Then Miller proceeds to describe the Fedora build infrastructure and afterward we get into the following:

Miller: I will give an example of something I installed in Flathub, I was trying to get some nice gui thing that would show me like my system Hardware stats […] one of them ones I picked seemed to do nothing, and turns out what it was actually doing, there was no graphical application it was just a script, it was running that script in the background and that script uploaded my system stats to a server somewhere.

Firstly we don’t really have many details to be able to identify which application it was, I would be very curious to know. Now speculating on my part, the most popular application matching that description it’s Hardware Probe and it absolutely has a GUI, no matter how minimal. It also asks you before uploading.

Maybe there is a org.upload.MySystem application that I don’t know about, and it ended up doing what was in the description, again I would love to know more and update the post if you could recall!

Miller: No one is checking for things like that and there’s no necessarily even agreement that that was was bad.

Second time! Again with the “There is no review and inclusion process in Flathub” narrative. There absolutely is, and these are the kinds of things that get brought up during it.

Miller: I am not trying to be down on Flathub because I think it is a great resource

Yes, I can see that, however in your ignorance you were something much worse than “Down”. This is pure slander and defamation, coming from the current “Fedora Project Leader”, the “Technically Voice of Fedora” (direct quote from a couple seconds later). All the statements made above are manufactured and inaccurate. Myths that you’d hear from people that never asked, looked or cared about any of these cause the moment you do you its obvious how laughable all these claims are.

Miller: And in a lot of ways Flathub is a competing distribution to Fedora’s packaging of all applications.

Precisely, he is spot on here, and I believe this is what kept Miller willfully ignorant and caused him to happily pick the first anit-flatpak/anti-flathub arguments he came across on reddit and repeat the verbatim without putting any thought into it. I do not believe Miller is malicious on purpose, I do truly believe he means well and does not know better.

However, we can’t ignore the conflict that arises from his current job position as an big influence to why incidents like this happened. Nor the influence and damage this causes when it comes from a person of Matthew Miller’s position.

Moving on:

Miller: One of the other things I wanted to talk about Flatpak, is the security and sandboxing around it. Miller: Like I said the stuff in the Flathub are not really reviewed in detail and it can do a lot of things:

Third time with the no review theme. I was fuming when I first heard this, and I am very very angry about still, If you can’t tell. Not only is this an incredibly damaging lie as covered above, it gets repeated over and over again.

With Flatpak basically the developer defines what the permissions are. So there is a sandbox, but the sandbox is what the person who put it there is, and one can imagine that if you were to put malware in there you might make your sandboxing pretty loose.

Brodie: One of the things you can say is “I want full file system access, and then you can do anything”

No, again it’s stated in the Flathub documentation, permissions are very carefully reviewed and updates get blocked when permissions change until another review has happened.

Miller: Android and Apple have pretty strong leverage against application developers to make applications work in their sandbox

Brodie: the model is the other way around where they request permissions and then the user grants them whereas Flatpak, they get the permission and then you could reject them later

This is partially correct, the first part about leverage will talk about in a bit, but here’s a primer on how permissions work in Flatpak and how it compares to the sandboxing technologies in iOS and Android.

In all of them we have a separation between Static and Dynamic permissions. Static are the ones the application always has access to, for example the network, or the ability to send you notifications. These are always there and are mentioned at install time usually. Dynamic permissions are the ones where the application has to ask the user before being able to access a resource. For example opening a file chooser dialog so the user can upload a file, the application the only gets access to the file the user consented or none. Another example is using the camera on the device and capturing photos/video from it.

Brodie here gets a bit confused and only mentions static permissions. If I had to guess it would be cause we usually refer to the dynamic permissions system in the Flatpak world as “Portals”.

Miller: it didn’t used to be that way and and in fact um Android had much weaker sandboxing like you could know read the whole file system from one app and things like that […] they slowly tightened it and then app developers had to adjust Miller: I think with the Linux ecosystem we don’t really have the way to tighten that kind of thing on app developers … Flatpak actually has that kind of functionality […] with portals […] but there’s no not really a strong incentive for developers to do that because, you know well, first of all of course my software is not going to be bad so why should I you know work on sandboxing it, it’s kind of extra work and I I don’t know I don’t know how to solve that. I would like to get to the utopian world where we have that same security for applications and it would be nice to be able to install things from completely untrusted places and know that they can’t do anything to harm your system and that’s not the case with it right now

As with any technology and adoption, we don’t get to perfection from day 1. Static permissions are necessary to provide a migration path for existing applications and until you have developed the appropriate and much more complex dynamic permissions mechanisms that are needed. For example up until iOS 18 it wasn’t possible to give applications access to a subset of your contacts list. Think of it like having to give access your entire filesystem instead of the specific files you want. Similarly partial-only access to your photos library arrived couple years ago in IOS and Android.

In an ideal world all permissions are dynamic, but this takes time and resources and adaptation for the needs of applications and the platform as development progresses.

Now about the leverage part.

I do agree that “the Linux ecosystem” as a whole does not have any leverage on applications developers. This is cause Miller is looking at the wrong place for it. There is no Linux ecosystem but rather Platforms developers target.

GNOME and KDE, as they distribute all their applications on Flathub absolutely have leverage. Similarly Flathub itself has leverage by changing the publishing requirements and inclusion guidelines. Which I kept being told they don’t exist.. Every other application that wants to publish also has to adhere by the rules on Flathub. ElementaryOS and their Appcenter has leverage on developers. Canonical does have the same pull as well with the Snapstore. Fedora on the other hand doesn’t have any leverage cause the Fedora Flatpak repository is irrelevant, broken and nobody wants to use it.

[..] The xz backdoor gets brought up when discussing dependencies and how software gets composed together.

Miller: we try to keep all of those things up to date and make sure everything is patched across the dist even when it’s even when it’s difficult. I think that really is one of the best ways to keep your system secure and because the sandboxing isn’t very strong that can really be a problem, you know like the XZ thing that happened before. If XZ is just one place it’s not that hard of an update but if you’ve got a 100 Flatpaks from different places […] and no consistency to it it’s pretty hard to manage that

I am not going to get in depth about this problem domain and the arguments over it. In fact I have been writing another blog post for a while. I hope to publish shortly. Till then I can not recommend high enough Emmanuele’s and Lennart’s blog posts, as well as one of the very early posts from Alex when Flatpak was in early design phase on the shortcomings of the current distribution model.

Now about bundled dependencies. The concept of Runtimes has served us well so far, and we have been doing a pretty decent job providing most of the things applications need but would not want to bundle themselves. This makes the Runtimes a single place for most of the high profile dependencies (curl, openssl, webkitgtk and so on) that you’d frequently update for security vulnerabilities and once it’s done they roll out to everyone without needing to do anything manual to update the applications or even rebuilt them.

Applications only need to bundle their direct dependencies,and as mentioned above, the flatpak manifest includes the exact definition of all of them. They are available to anyone to inspect and there’s tooling that can scan them and hopefully in the future alert us.

If the Docker/OCI model where you end bundling the entire toolchain, runtime, and now you have to maintain it and keep up with updates and rebuild your containers is good enough for all those enterprise distributions, then the Flatpak model which is much more efficient, streamlined and thought out and much much much less maintenance intensive, it is probably fine.

Miller: part of the idea of having a distro was to keep all those things consistent so that it’s easier for everyone, including the developers

As mentioned above, nothing that fundamentally differs from the leverage that Flathub and the Platform Developers have.

Brodie: took us 20 minutes to get to an explanation [..] but the tldr Fedora Flatpak is basically it is built off of the Fedora RPM build system and because that it is more well tested and sort of intended, even if not entirely for the Enterprise, designed in a way as if an Enterprise user was going to use it the idea is this is more well tested and more secure in a lot of cases not every case.
Miller: Yea that’s basically it

This is a question/conclusion that Brodie reaches with after the previous statements and by far the most enraging thing in this interview. This is also an excellent example of the damage Matthew Miller caused today and if I was a Flathub developer I would stop on nothing sort of a public apology from the Fedora project itself. Hell I want this just being an application developer that publishes on it. The interview has been basically shitting on both the Developers of Flathub and the people that choose to publish in it. And if that’s not enough there should be an apology just out of decency. Dear god..

Brodie: how should Fedora handle upstreams that don’t want to be packaged  like the OBS case here where they did not want there to be a package in Fedora Flatpak or another example is obviously bottles which has made a lot of noise about the packaging

Lastly I want to touch on this closing question in light of recent events.

Miller: I think we probably shouldn’t do it. We should respect people’s wishes there. At least when it is an open source project working in good faith there. There maybe some other cases where the software, say theoretically there’s somebody who has commercial interests in some thing and they only want to release it from their thing even though it’s open source. We might want to actually like, well it’s open source we can provide things, we in that case we might end up you having a different name or something but yeah I can imagine situations where it makes sense to have it packaged in Fedora still but in general especially and when it’s a you know friendly successful open source project we should be friendly yeah. The name thing is something people forget history like that’s happened before with Mozilla with Firefox and Debian.

This is an excellent idea! But it gets better:

Miller: so I understand why they strict about that but it was kind of frustrating um you know we in Fedora have basically the same rules if you want to take Fedora Linux and do something out of it, make your own thing out of it, put your own software on whatever, you can do that but we ask you not to call it Fedora if it’s a fedora remix brand you can use in some cases otherwise pick your own name it’s all open source but you know the name is ours. yeah and I the Upstream as well it make totally makes sense.

Brodie: yeah no the name is completely understandable especially if you do have a trademark to already even if you don’t like it’s it’s common courtesy to not name the thing the exact same thing

Miller: yeah I mean and depending on the legalities like you don’t necessarily have to register a trademark to have the trademark kind of protections under things so hopefully lawyers you can stay out of the whole thing because that always makes the situations a lot more complicated, and we can just get along talking like human beings who care about making good software and getting it to users.

And I completely agree with all of these, all of it. But let’s break it down a bit because no matter how nice the words and intentions it hasn’t been working out this way with the Fedora community so far.

First, Miller agrees the Fedora project should be respecting of application developer’s wishes to not have their application distributed by fedora but rather it be a renamed version if Fedora wishes to keep distributing it.

However, every single time a developer has asked for this, they have been ridiculed, laughed at and straight up bullied by Fedora packagers and the rest of the Fedora community. It has been a similar response from other distribution projects and companies as well, it’s not just Fedora. You can look at Bottle’s story for the most recent example. It is very nice to hear Miller’s intentions but means nothing in practice.

Then Miller proceeds to assure us why he understand that naming and branding is such a big deal to those projects (unlike the rest of the Fedora community again). He further informs us how Fedora has the exact same policies and asks from people that want to fork Fedora. Which makes the treatment that every single application developer has received when asking about the same exact thing ever more outrageous.

What I didn’t know is that in certain cases you don’t even need to have a trademark yet to be covered by some of the protections, depending on jurisdiction and all.

And last we come into lawyers. Neither Fedora nor application developers would want it to ever come to this, and it was stated multiple times by Bottles developers that they don’t want to have to file for a trademark so they can be taken seriously. Similarly, OBS developers said how resorting to legal action would be the last thing they would want to do and would rather have the issue resolved before that. But it took until OBS, a project of a high enough profile, with the resources required to acquire a trademark and to threaten legal action before the Fedora Leadership cared to treat application developers like human beings and get the Fedora packagers and community members to comply. (Something which they had stated multiple times they simply couldn’t do).

I hate all of this. Fedora and all the other distributions need to do better. They all claim to care about their users but happily keep shipping broken and miss configured software to them over the upstream version, just cause it’s what aligns with their current interests. In this case is the promotion of Fedora tooling and Fedora Flatpaks over the application in Flathub they have no control over. In previous incidents it was about branding applications like the rest of the system even though it was making them unusable. And I can find you and list you with a bunch of examples from other distributions just as easily.

They don’t care about their users, they care about their bottom line first and foremost. Any civil attempts at fixing issues get ignored and laughed at, up until there is a threat of a legal action or a big enough PR damage, drama and shitshow that they can’t ignore it anymore and have to backtrack on them.

This is my two angry cents. Overall I am not exactly sure how Matthew Miller managed in a rushed and desperate attempt at damage control for the OBS drama, to not only to make it worse, but to piss off the entire Flathub community at the same time. But what’s done is done, let’s see what we can do to address the issues that have festered and persisted for years now.

Christian was looking at PGO and BOLT recently I figured I’d write down my notes from the discussions we had on how we’d go about making things faster on our stack, since I don’t have time or the resource to pursue those plans myself atm.

First off let’s start with the basics, PGO (profile guided optimizations) and BOLT (Binary Optimization and Layout Tool) work in similar ways. You capture one or more “profiles” of a workload that’s representative of a usecase of your code and then the tools do their magic to make the common hot paths more efficient/cache-friendly/etc. Afterwards they produce a new binary that is hopefully faster than the old one and functionally identical so you can just replace it.

Now already we have two issues here that arise here:

First of all we don’t really have any benchmarks in our stack, let alone, ones that are rounded enough to account for the majority of usecases. Additionally we need better instrumentation to capture stats like frames, frame-times, and export them both for sysprof and so we can make the benchmark runners more useful.

Once we have the benchmarks we can use them to create the profiles for optimizations and to verify that any changes have the desired effect. We will need multiple profiles of all the different hardware/software configurations.

For example for GTK ideally we’d want to have a matrix of profiles for the different render backends (NGL/Vulkan) along with the mesa drivers they’d use depending on different hardware AMD/Intel and then also different architectures, so additional profile for Raspberrypi5 and Asahi stacks. We might also want to add a profile captured under qemu+virtio while we are it too.

Maintaining the benchmarks and profiles would be a lot of work and very tailored to each project so they would all have to live in their upstream repositories.

On the other hand, the optimization itself has to be done during the Tree/userland/OS composition and we’d have to aggregate all the profiles from all the projects to apply them. This is easily done when you are in control of the whole deployment as we can do for the GNOME Flatpak Runtime. It’s also easy to do if you are targeting an embedded deployment where most of the time you have custom images you are in full control off and know exactly the workload you will be running.

If we want distros to also apply these optimizations and for this to be done at scale, we’d have to make the whole process automatic and part of the usual compilation process so there would be no room for error during integration. The downside of this would be that we’d have a lot less opportunities for aggregating different usecases/profiles as projects would either have to own optimizations of the stack beneath them (ex: GTK being the one relinking pango) or only relink their own libraries.

To conclude, Post-linktime optimization would be a great avenue to explore as it seems to be one of the lower-hanging fruits when it comes to optimizing the whole stack. But it also would be quite the effort and require a decent amount of work to be committed to it. It would be worth it in the long run.

Hello everyone!

I am here to terrorize your calendar by dropping the dates for two back to back hackfests we are organizing in the beautiful city of Thessaloniki, Greece (who doesn’t like coming to Greece on work time, right?).

May 27-29th we will be hosting the annual GStreamer Spring Hackfest. If multimedia is your thing, you know the drill. Newcomers are also welcome ofc!

May 31st-June 5th we will be hosting another edition of the GNOME ♥️ Rust Hackfest. First in person Rust hackfest ever since the pandemic started. From what I heard, half of Berlin will be coming for this one so we might change its scope to an all around GNOME one, but we will see. You are all welcome!

See the pages of each hackfest for more details.

We are in the final steps of booking the venue but it will most likely be in the city center and it should be safe to book accommodation and traveling tickets.

Additionally the venue we are looking at can accommodate around 40 people, so please please add yourself to the organizing pad of each hackfest you are interested in, in addition to any dietary restrictions you might have.

See you all IRL!

This is the first post in a series about tools used to develop GNOME and GNOME OS. Part two coming soon.

In the old age, developing the desktop was simple™️. You only had to install a handful of toolchains, development headers, and tools from the distribution packages, run make install, execute say_prayer.sh and if you had not eaten meat on Friday you had a 25% chance for your system to work after a reboot.

But how do you develop software in the brave new world of Image-Based systems and containerization?

Well, if you are an application developer Flatpak makes this very simple. Applications run against a containerized runtimes, with their own userspace. The only thing they need from the host system is a working Desktop Environment, Flatpak, and Portals. Builder and flatpak-builder along with all the integration we built into the Desktop make sure that it will be a breeze.

But what if you are developing a system component, like a daemon or perhaps GNOME Shell?

Till now the goto solution has been “Grab a fedora container and open enough sandbox holes until things work”. This is what Fedora Toolbox does, and it works great for a lot of things. However the sandbox makes things way more difficult than they need to be, and its rather limiting on what parts of the system you can test. But there is another way.

In GNOME OS we provide two images. The first one is how we envision GNOME to be, with all the applications and services we develop. The second one is complimentary, and it adds all the development tools, headers and debugging information we need to develop GNOME. Now the popular image based OSes don’t provide you with something like that, and you have to layer everything on your own. This makes it much harder to do things like running gdb against the host system. But in GNOME OS its easy, and by switching to the Development Edition/Image you get access to all the tools required to build any GNOME component and even GNOME OS itself.

Alright, I have a compiler and all the dependencies I need to build my project but /usr is still immutable. How do I install and run my modified build?

I am glad you asked! Enter systemd-sysext.

If you are familiar with ostree-based operating systems, you probably used ostree admin unlock at some point. systemd-sysext is another take on the same concept. You build your software as usual with a /usr (or /opt) prefix and install it in a special folder along with some metadata. Then upon running systemd-sysext merge, systemd creates an ovelayfs with the merged contents of /usr and your directory, and replaces the existing /usr mount atomically.

The format is very simple, there are 2 directories we care about for our usecase:

• /run/extensions
• /var/lib/extensions

/run/extensions is a temporary directory that’s wiped on reboot, so it’s excellent for experimental changes that might take down the system. After a power cycle you will boot back in a clean slate.

/var/lib/extensions is for persistent changes. Say you are experimenting with UI changes or want to thoroughly test a patch set for a couple days. Or you might simply want to have a local FFmpeg build with extra codecs, cause god lawyer forbid we have working video playback.

If you are installing a build in /run/extensions only thing you need to do is the following two commands:

sudo DESTDIR=/run/extensions/custom_install meson install -C _build
sudo systemd-sysext refresh --force

This installs our tree into the custom_install directory and tells systext to refresh, which means that it will look at all extensions we might have, unmerge (unmount) them and then merge the updated contents again. Congrats, you can launch your new binaries now.

Normally sysext will check if the extension is compatible with the host operating system. This is done by using a metadata file that includes the ID= field you’d find in /etc/os-release (see more at the systemd-sysext manpage). The --force argument ignores this, as we build the project on host and we can be reasonably sure things will work.

If we want to have our build to be persistent and available across reboots, we have to install it in /var/lib/extensions/ and create the metadata file so systemd-sysext will mount it automatically upon boot. It’s rather simple to do but gets repetitive after a while and hunting in your console history is never fun. Here is a simple script that will take care of it.

#! /bin/bash

set -eu

EXTENSION_NAME="custom_install"
# DESTDIR="/var/lib/extensions/$EXTENSION_NAME"
DESTDIR="/run/extensions/$EXTENSION_NAME"
VERSION_FILE="$DESTDIR/usr/lib/extension-release.d/extension-release.$EXTENSION_NAME"

sudo mkdir -p $DESTDIR

# rm -rf _build 
# meson setup _build --prefix=/usr
# meson compile -C _build 
sudo meson install --destdir=$DESTDIR -C _build --no-rebuild

sudo mkdir -p $DESTDIR/usr/lib/extension-release.d/
# ID=_any to ignore it completly
echo ID=org.gnome.gnomeos | sudo tee $VERSION_FILE

sudo systemd-sysext refresh
sudo systemd-sysext list

Here’s a demo, I used sysprof as the example since it’s more visible change than my gnome-session MR. You can also test gnome-shell the same way by installing, refreshing and then logging out and login in again.

Anothe example was from today, where I was bisecting gdm. Before systemd-sysext, I’d be building complete images with different commits of gdm in order to bisect. It was still fast, at ~25m per build for the whole OS, but that’s still 24 minutes more after it becomes annoying.

Now, I switched to the gdm checkout, started a bisect, compiled, installed and then run systemctl restart gdm.service. The login greeter would either come up and I’d continue the bisect, or it would be blank at which point I’d ssh in, switch to a tty or even hit the power button and continue knowing it’s a bad commit.  Repeat. Bisect done in 10 minutes.

And the best is that we can keep updating the operating system image uninterrupted, and on next boot the trees will get merged again. Want to go back? Simply systemd-sysext unmerge or remove the extension directories!

One caveat when using systemd-sysext is that you might occasionally need to systemctl daemon-reload. Another one when using custom DESTDIRs, is that meson won’t run post-install/integration commands for you (nor would it work), so if you need to recompile glib schemas, you will have to first systemd-sysext refresh, compile the schemas, place the new binary in the extension or make a new extension, and systemd-sysext refresh again.

Another use case I plan on exploring in the near future, is generating systemd-sysext images in the CI for Merge Requests, same way we generate Flatpak Bundles for applications. This proved to be really useful for people wanting to tests apps in an easier way. Begone shall be the days where we had to teach designers how to setup JHBuild in order to test UI changes in the Shell. Just grab the disk image, drop it in GNOME OS, refresh and you are done!

And that’s not all, none of this is specific to GNOME OS, other than having bleeding edge versions of all the gnome components that is! You can use systemd-sysext the same way in Fedora Workstation, Arch, Elementary OS etc. The only requirement is having recent enough systemd and a merged /usr tree. Next time you are about to meson install on your host system, give systemd-sysext a try!

This whole post is basically a retelling of Lennart’s blogpost about systemd-sysext, It has more details and you should check it out. This is also how I initially found out about this awesome tool! I tried to get people hooked on it in the past but it didn’t bear fruit, so here’s one post specific to GNOME development!

Happy Hacking!

It’s that time of the month again, when some clueless guy tries to write a hit-piece about Flatpak and we all get dejavus.

One of my favorite past-time activities for a while now has been seeing people on the internet trying to rationalize concepts and decisions, and instead of asking why things ended up the way they did, what where the issues and goals of system A, and design B, and what were the compromise, they just pick the first idea that comes to their mind and go with it.

For example, a very common scenario is that someone picks a random Proprietary application and points out all the sandbox holes it needs to function and thus declares the sandbox as useless. At no point does one of them ever ask, “Hey why does Flatpak allow to punch such holes”, “What developments have been done to limit that”, “What tools are available to deal with that”, “Why am I a cherry-picking an evil proprietary application as my example that no distribution would be able to distribute anyway and I wouldn’t want any person to use” and “What went wrong with my life that I have to write hate posts to get attention and feel any kind of emotion”. These are just a few of the question that should have come up and given one pause, way before getting anywhere near the the publish button.

Now I can answer most of these questions, and you would be happy to know that even Chromium and Electron have been adopting more and more of the sandboxed Portal APIs as the years pass. But there isn’t any point in talking about it cause none of the delirium is about the technical decisions behind Flatpak or how it works. None.

Let me explain.

Flatpak itself is a piece of software. It provides major advantages to distributing and running applications such as: atomic updates, binary deltas, reproducible build and run environments, mandatory sandboxing for many resources, and so on. How the software is built and distributed however has nothing to do with Flatpak. If you think the distribution-model is what’s best for you, you can already use Fedora’s flatpaked applications, Canonical’s snaps or your fav distro version of this. Everything is still built from distribution packages, by your distribution vendor, vetted by the package maintainers, come with the same downstream patches you’d see in the normal rpm/deb/etc variations, and so on. And you would still get the advantages of sandboxing, atomicity, etc even though you don’t need them cause you love and trust your distro so much.

On the other hand what every single post really complains about is Flathub. You see, what Flatpak gave us was the ability to decouple the applications from the host system. Instead of taking the existing runtime from some distro, We (The platform and application developers) built our Runtimes from scratch, that we were in full control of, that we could update and mold at will, that was not bound to any existing distribution or corporation, that we could make sure our applications were fully functional with, without any downstream patches that made things orange or blue. And unlike the old distribution model, Flathub gave application developers the same autonomy. We no longer had to wait for dependencies to be packaged, or the worry about some distribution shipping an incompatible version. We didn’t have to wait until a new enough version of a library was included into an LTS release before making use of it. We could now ship our applications on our cadence, without gatekeepers, in the way we envisioned and intended.

This is what made applications truly work on any distribution. This is what was truly disruptive about Flatpak. This is what the haters are mad about.

Thanks to Flathub the social dynamic for distributing applications has changed. Now the people that create the Platforms (GNOME, KDE, Elementary, etc) and Applications are in charge of distributing them. The sysadmin-turned-distro-packager middleman trope from the 90s is gone and no developer, or user wants it back. This is why Flathub took over, this is why no application developer became a Fedora packager even when they could build Flatpaks from the rpms packaged. If we ever want “Desktop Linux” to succeed, we have to let go of the idea of Linux distributions and “Linux” as a monolith.

The old distribution model is still useful for very specific, enterprise environments where you depend on a single ISV for all your software, but unless your Surname is Mr. IBM or Mr. Canonical, you gain nothing by asking for this on your desktop.

If you want to read more on the subject I highly suggest these two blogposts, along with Richard Brown’s Fosdem 2023 talk.

Quick heads up, the GNOME Nightly Flatpak repository is currently undergoing maintenance, during which you may notice that some applications are currently missing from the repo.

For a couple of months now we have been plagued by a few bugs that have made maintenance of the repo very hard, and CI builds were constantly failing due to a lack of available space. In order to resolve this we had to wipe the majority of the refs/objects in the repository and start again with safeguards in place.

As such, we are currently re-populating the repository with fresh builds of all the applications, but it may take a while. If you want to help with this, make sure your Flatpak manifests are up to date and build-able, and that you have set up a daily or weekly scheduled CI build in your project. Your app may not be changing, but the runtime might, and it’s good to be on top of possible API/ABI breaks.

Go to your project, Settings -> CI/CD -> Schedules -> New schedule button -> Select the daily preset.

If you are a user and seeing warnings while updating, don’t worry – you won’t have to do anything and updates will start working again transparently once the applications are available in the repository.

$ flatpak update
Looking for updates…
F: Warning: Treating remote fetch error as non-fatal since runtime/org.gnome.Todo.Devel.Locale/x86_64/master is already installed: No such ref 'runtime/org.gnome.Todo.Devel.Locale/x86_64/master' in remote gnome-nightly
F: Warning: Treating remote fetch error as non-fatal since runtime/org.gnome.TextEditor.Devel.Locale/x86_64/master is already installed: No such ref 'runtime/org.gnome.TextEditor.Devel.Locale/x86_64/master' in remote gnome-nightly
F: Warning: Treating remote fetch error as non-fatal since runtime/org.gnome.TextEditor.Devel.Debug/x86_64/master is already installed: No such ref 'runtime/org.gnome.TextEditor.Devel.Debug/x86_64/master' in remote gnome-nightly
F: Warning: Treating remote fetch error as non-fatal since runtime/org.gnome.Photos.Locale/x86_64/master is already installed: No such ref 'runtime/org.gnome.Photos.Locale/x86_64/master' in remote gnome-nightly
F: Warning: Treating remote fetch error as non-fatal since runtime/org.gnome.Epiphany.Devel.Locale/x86_64/master is already installed: No such ref 'runtime/org.gnome.Epiphany.Devel.Locale/x86_64/master' in remote gnome-nightly
F: Warning: Treating remote fetch error as non-fatal since app/org.gnome.Todo.Devel/x86_64/master is already installed: No such ref 'app/org.gnome.Todo.Devel/x86_64/master' in remote gnome-nightly
F: Warning: Treating remote fetch error as non-fatal since app/org.gnome.TextEditor.Devel/x86_64/master is already installed: No such ref 'app/org.gnome.TextEditor.Devel/x86_64/master' in remote gnome-nightly
F: Warning: Treating remote fetch error as non-fatal since app/org.gnome.Photos/x86_64/master is already installed: No such ref 'app/org.gnome.Photos/x86_64/master' in remote gnome-nightly
F: Warning: Treating remote fetch error as non-fatal since app/org.gnome.Epiphany.Devel/x86_64/master is already installed: No such ref 'app/org.gnome.Epiphany.Devel/x86_64/master' in remote gnome-nightly

Sorry for the inconvenience and happy hacking.

Before we start, let’s get this out of the way because the week long delirium on social media has dragged enough.

Yes, libadwaita “hardcodes” Adwaita. Yes, applications, as is, will not be following a custom system theme. Yes this does improve the default behavior of application for GNOME when run on other platforms like Elementary OS. However, this is the result of a technical limitation, and not some evil plot as Twitter will keep telling you…

The reason is that in order for High Contrast (and the upcoming Dark Style) to work, libadwaita needs to override the theme name property so it doesn’t fallback to GTK’s “Default” High Contrast style. The “Default” style is an older version of Adwaita, not your system style.

Compared to GTK 3, there isn’t a new way to enforce the “hardcoded” style. The GTK_THEME Debug variable still works, as does ~/.config/gtk-4.0/gtk.css which you can use to set more permanent changes, and there are probably 3 other ways of doing this. The process to theme your system might be a bit different compared to GTK 3 but it will still work. Likewise, if you are developing a distribution, you have control of the end product and can do anything you want with the code. There is a plethora of options available. Apparently complaining on social media and bullying volunteers into submission was one such option…

And I guess this also needs to be stated: this change only affects apps that choose to use libadwaita and adopt the GNOME Design Guidelines, not “every” GTK 4 application.

As usual, the fact that the themes keep working doesn’t mean they are supported. The same issues about restyling applications when they don’t expect it apply and GNOME can not realistically support arbitrary stylesheets that none of the contributors develop against and test.

Now onto the actual blogpost.

There seems to be some confusion when it comes to libadwaita’s stylesheet and coloring APIs. It’s easy to mix them up when you haven’t heard of libadwaita before, so here is a short introduction on what they are and how they differ.

Keep in mind that the features discussed below are not guaranteed to land. After libadwaita 1.0 the stylesheet will be frozen and treated as an API. That means that if a feature doesn’t make it by 1.0 it will be a breaking change and will have to wait for libadwaita 2.0.

An Application Coloring API / Accent Colors

The idea here is that you can define “accent” colors to be applied for various parts of widgets. You can also recolor any part of a widget however you like. Take a look at Epiphany’s private mode header bar for an example. For this to be possible the whole stylesheet had to be reworked. Extra care was needed to ensure that the functionality wouldn’t conflict with the high contrast preference and wouldn’t need special handling. I hope Alexander will blog about this work in more detail, as it was truly fascinating. I am very excited to see what developers do with the coloring API.

For now the colors can be controlled with the GTK-specific @define-color, similar to CSS variables. Programmatic API will be added later on as the dust settles. The API will be based on AdwStyleManager which is getting introduced by the Dark style preference MR and hasn’t landed yet.

Here’s a quick example:

@define-color accent_color @yellow_5;
@define-color accent_bg_color @yellow_2;
@define-color accent_fg_color black;

.controls {
    color: white;
    background: linear-gradient(to right, shade(@blue_3, .8), @purple_2);
}

.slider > trough > highlight {
    background: linear-gradient(to left, shade(@red_1, .8), @yellow_4);
}

.controls textview text {
  background: none;
}

.controls entry,
.controls spinbutton,
.controls textview {
  background-color: alpha(black, .15);
  color: white;
}

.navigation-sidebar,
headerbar {
  background: alpha(white, .1);
  color: white;
}

For a more detailed example of what you can do check Federico’s recent blogpost.

System Accents

This is heavily inspired by system accent settings in elementary OS, and it’s similar in function. Think of it like a way to set the accent color system-wide, then applications can read it and decide to follow or override it. A case where you want to override would be if your app had a Sepia mode for example.

The coloring API mentioned above is designed in a way that makes this feature easy to implement. The interface and UI for this are not yet fleshed out completely, and it’s debatable if it’s going to be implemented/merged at all. There are a couple of design issues and concerns that need further research. It’s a possibility, but don’t bet on it.

Vendor Styling

The story behind this idea is extensive and best left for another post, so here’s the current status on this infamous topic.

There have been great accomplishments to reducing the possible fallout of restyling applications with brand colors. Nowadays vendors recognize that arbitrary restyling can be damaging to application developers and have taken some precautions.

Yaru reworked its style and rebased it on Adwaita, this helped reduced the changes to mostly the color palette and minor stylistic tweaks. This got rid of a lot of bugs surfacing in applications, as Yaru now at least has the same spacing, margins and padding as Adwaita. Pop!_OS followed suit shortly after, I believe it’s now based on Yaru.

However, both Ubuntu and Pop also introduced “Dark-modes”, Pop making it the default, which broke applications’ expectations. They did it despite being warned about it. As a result this ended up increasing the issues with theming by about an order of magnitude as now you would frequently end up with black on black, grey on grey and other fun coloring bugs. It should also be noted that neither Ubuntu nor System 76 approached any contributor I know of, about properly implementing a Dark Style preference upstream. Even though GNOME and Elementary contributors had been collaborating in public for the last 3 years.

Yaru developers did some research on the topic and there was a call for engagement by GNOME, but unfortunately ever since the last theming BoF in 2019, the conversation had dried up. The interested parties haven’t provided any details on what the scope of the API would need to be, how it would look like, or the detailed requirements. Nobody stepped up to help with the Adwaita changes that were required either, or with dark mode, or to work on the QA tooling, or to figure out the implementation details. Now we are sadly out of time for libadwaita 1.0 and there isn’t much hope for such a complex thing to be ready in the next 4 months.

Conclusion

For libadwaita 1.0 and GNOME 42 the work on recoloring widgets will likely be completed. A proper Dark Style setting will likely also be implement by then. System-wide accent colors are being discussed and looked at, but there are design related concerns about them, so it’s possible that they will never land. And there won’t be any “Theming API” for libadawaita 1.0. Maybe there will be renewed interest from the vendors that want it in the future, but given the story so far, I won’t hold my breath. I hope to be proven wrong.

Last week I needed a Debian system to test things, I had heard others were using toolbox with Debian images without much trouble so decided to give it a go instead of creating a VM.

Toolbox only requires a handful utilities to work with any given docker image. After a quick search I stumbled upon Philippe’s post which in turn linked into this PR about an Ubuntu based toolbox image. Looks like the last major issues where worked out recently in toolbox and there isn’t anything extra needed apart the image.

Until the upstream PR is merged, I’ve adapted the image for Debian Sid and inlined the deps in one Dockerfile so its easier to fetch it and use it. The Dockerfile is hosted in a gist here.

Here is how to use it:

curl -o debian.Dockerfile 'https://gitlab.gnome.org/-/snippets/1653/raw/master/debian.Dockerfile'
podman build -t debian-sid-toolbox -f debian.Dockerfile .
toolbox create -c sid --image debian-sid-toolbox
toolbox enter sid

Enjoy!