Christian was looking at PGO and BOLT recently I figured I’d write down my notes from the discussions we had on how we’d go about making things faster on our stack, since I don’t have time or the resource to pursue those plans myself atm.

First off let’s start with the basics, PGO (profile guided optimizations) and BOLT (Binary Optimization and Layout Tool) work in similar ways. You capture one or more “profiles” of a workload that’s representative of a usecase of your code and then the tools do their magic to make the common hot paths more efficient/cache-friendly/etc. Afterwards they produce a new binary that is hopefully faster than the old one and functionally identical so you can just replace it.

Now already we have two issues here that arise here:

First of all we don’t really have any benchmarks in our stack, let alone, ones that are rounded enough to account for the majority of usecases. Additionally we need better instrumentation to capture stats like frames, frame-times, and export them both for sysprof and so we can make the benchmark runners more useful.

Once we have the benchmarks we can use them to create the profiles for optimizations and to verify that any changes have the desired effect. We will need multiple profiles of all the different hardware/software configurations.

For example for GTK ideally we’d want to have a matrix of profiles for the different render backends (NGL/Vulkan) along with the mesa drivers they’d use depending on different hardware AMD/Intel and then also different architectures, so additional profile for Raspberrypi5 and Asahi stacks. We might also want to add a profile captured under qemu+virtio while we are it too.

Maintaining the benchmarks and profiles would be a lot of work and very tailored to each project so they would all have to live in their upstream repositories.

On the other hand, the optimization itself has to be done during the Tree/userland/OS composition and we’d have to aggregate all the profiles from all the projects to apply them. This is easily done when you are in control of the whole deployment as we can do for the GNOME Flatpak Runtime. It’s also easy to do if you are targeting an embedded deployment where most of the time you have custom images you are in full control off and know exactly the workload you will be running.

If we want distros to also apply these optimizations and for this to be done at scale, we’d have to make the whole process automatic and part of the usual compilation process so there would be no room for error during integration. The downside of this would be that we’d have a lot less opportunities for aggregating different usecases/profiles as projects would either have to own optimizations of the stack beneath them (ex: GTK being the one relinking pango) or only relink their own libraries.

To conclude, Post-linktime optimization would be a great avenue to explore as it seems to be one of the lower-hanging fruits when it comes to optimizing the whole stack. But it also would be quite the effort and require a decent amount of work to be committed to it. It would be worth it in the long run.

Hello everyone!

I am here to terrorize your calendar by dropping the dates for two back to back hackfests we are organizing in the beautiful city of Thessaloniki, Greece (who doesn’t like coming to Greece on work time, right?).

May 27-29th we will be hosting the annual GStreamer Spring Hackfest. If multimedia is your thing, you know the drill. Newcomers are also welcome ofc!

May 31st-June 5th we will be hosting another edition of the GNOME ♥️ Rust Hackfest. First in person Rust hackfest ever since the pandemic started. From what I heard, half of Berlin will be coming for this one so we might change its scope to an all around GNOME one, but we will see. You are all welcome!

See the pages of each hackfest for more details.

We are in the final steps of booking the venue but it will most likely be in the city center and it should be safe to book accommodation and traveling tickets.

Additionally the venue we are looking at can accommodate around 40 people, so please please add yourself to the organizing pad of each hackfest you are interested in, in addition to any dietary restrictions you might have.

See you all IRL!

This is the first post in a series about tools used to develop GNOME and GNOME OS. Part two coming soon.

In the old age, developing the desktop was simple™️. You only had to install a handful of toolchains, development headers, and tools from the distribution packages, run make install, execute say_prayer.sh and if you had not eaten meat on Friday you had a 25% chance for your system to work after a reboot.

But how do you develop software in the brave new world of Image-Based systems and containerization?

Well, if you are an application developer Flatpak makes this very simple. Applications run against a containerized runtimes, with their own userspace. The only thing they need from the host system is a working Desktop Environment, Flatpak, and Portals. Builder and flatpak-builder along with all the integration we built into the Desktop make sure that it will be a breeze.

But what if you are developing a system component, like a daemon or perhaps GNOME Shell?

Till now the goto solution has been “Grab a fedora container and open enough sandbox holes until things work”. This is what Fedora Toolbox does, and it works great for a lot of things. However the sandbox makes things way more difficult than they need to be, and its rather limiting on what parts of the system you can test. But there is another way.

In GNOME OS we provide two images. The first one is how we envision GNOME to be, with all the applications and services we develop. The second one is complimentary, and it adds all the development tools, headers and debugging information we need to develop GNOME. Now the popular image based OSes don’t provide you with something like that, and you have to layer everything on your own. This makes it much harder to do things like running gdb against the host system. But in GNOME OS its easy, and by switching to the Development Edition/Image you get access to all the tools required to build any GNOME component and even GNOME OS itself.

Alright, I have a compiler and all the dependencies I need to build my project but /usr is still immutable. How do I install and run my modified build?

I am glad you asked! Enter systemd-sysext.

If you are familiar with ostree-based operating systems, you probably used ostree admin unlock at some point. systemd-sysext is another take on the same concept. You build your software as usual with a /usr (or /opt) prefix and install it in a special folder along with some metadata. Then upon running systemd-sysext merge, systemd creates an ovelayfs with the merged contents of /usr and your directory, and replaces the existing /usr mount atomically.

The format is very simple, there are 2 directories we care about for our usecase:

• /run/extensions
• /var/lib/extensions

/run/extensions is a temporary directory that’s wiped on reboot, so it’s excellent for experimental changes that might take down the system. After a power cycle you will boot back in a clean slate.

/var/lib/extensions is for persistent changes. Say you are experimenting with UI changes or want to thoroughly test a patch set for a couple days. Or you might simply want to have a local FFmpeg build with extra codecs, cause god lawyer forbid we have working video playback.

If you are installing a build in /run/extensions only thing you need to do is the following two commands:

sudo DESTDIR=/run/extensions/custom_install meson install -C _build
sudo systemd-sysext refresh --force

This installs our tree into the custom_install directory and tells systext to refresh, which means that it will look at all extensions we might have, unmerge (unmount) them and then merge the updated contents again. Congrats, you can launch your new binaries now.

Normally sysext will check if the extension is compatible with the host operating system. This is done by using a metadata file that includes the ID= field you’d find in /etc/os-release (see more at the systemd-sysext manpage). The --force argument ignores this, as we build the project on host and we can be reasonably sure things will work.

If we want to have our build to be persistent and available across reboots, we have to install it in /var/lib/extensions/ and create the metadata file so systemd-sysext will mount it automatically upon boot. It’s rather simple to do but gets repetitive after a while and hunting in your console history is never fun. Here is a simple script that will take care of it.

#! /bin/bash

set -eu

EXTENSION_NAME="custom_install"
# DESTDIR="/var/lib/extensions/$EXTENSION_NAME"
DESTDIR="/run/extensions/$EXTENSION_NAME"
VERSION_FILE="$DESTDIR/usr/lib/extension-release.d/extension-release.$EXTENSION_NAME"

sudo mkdir -p $DESTDIR

# rm -rf _build 
# meson setup _build --prefix=/usr
# meson compile -C _build 
sudo meson install --destdir=$DESTDIR -C _build --no-rebuild

sudo mkdir -p $DESTDIR/usr/lib/extension-release.d/
# ID=_any to ignore it completly
echo ID=org.gnome.gnomeos | sudo tee $VERSION_FILE

sudo systemd-sysext refresh
sudo systemd-sysext list

Here’s a demo, I used sysprof as the example since it’s more visible change than my gnome-session MR. You can also test gnome-shell the same way by installing, refreshing and then logging out and login in again.

Anothe example was from today, where I was bisecting gdm. Before systemd-sysext, I’d be building complete images with different commits of gdm in order to bisect. It was still fast, at ~25m per build for the whole OS, but that’s still 24 minutes more after it becomes annoying.

Now, I switched to the gdm checkout, started a bisect, compiled, installed and then run systemctl restart gdm.service. The login greeter would either come up and I’d continue the bisect, or it would be blank at which point I’d ssh in, switch to a tty or even hit the power button and continue knowing it’s a bad commit.  Repeat. Bisect done in 10 minutes.

And the best is that we can keep updating the operating system image uninterrupted, and on next boot the trees will get merged again. Want to go back? Simply systemd-sysext unmerge or remove the extension directories!

One caveat when using systemd-sysext is that you might occasionally need to systemctl daemon-reload. Another one when using custom DESTDIRs, is that meson won’t run post-install/integration commands for you (nor would it work), so if you need to recompile glib schemas, you will have to first systemd-sysext refresh, compile the schemas, place the new binary in the extension or make a new extension, and systemd-sysext refresh again.

Another use case I plan on exploring in the near future, is generating systemd-sysext images in the CI for Merge Requests, same way we generate Flatpak Bundles for applications. This proved to be really useful for people wanting to tests apps in an easier way. Begone shall be the days where we had to teach designers how to setup JHBuild in order to test UI changes in the Shell. Just grab the disk image, drop it in GNOME OS, refresh and you are done!

And that’s not all, none of this is specific to GNOME OS, other than having bleeding edge versions of all the gnome components that is! You can use systemd-sysext the same way in Fedora Workstation, Arch, Elementary OS etc. The only requirement is having recent enough systemd and a merged /usr tree. Next time you are about to meson install on your host system, give systemd-sysext a try!

This whole post is basically a retelling of Lennart’s blogpost about systemd-sysext, It has more details and you should check it out. This is also how I initially found out about this awesome tool! I tried to get people hooked on it in the past but it didn’t bear fruit, so here’s one post specific to GNOME development!

Happy Hacking!

It’s that time of the month again, when some clueless guy tries to write a hit-piece about Flatpak and we all get dejavus.

One of my favorite past-time activities for a while now has been seeing people on the internet trying to rationalize concepts and decisions, and instead of asking why things ended up the way they did, what where the issues and goals of system A, and design B, and what were the compromise, they just pick the first idea that comes to their mind and go with it.

For example, a very common scenario is that someone picks a random Proprietary application and points out all the sandbox holes it needs to function and thus declares the sandbox as useless. At no point does one of them ever ask, “Hey why does Flatpak allow to punch such holes”, “What developments have been done to limit that”, “What tools are available to deal with that”, “Why am I a cherry-picking an evil proprietary application as my example that no distribution would be able to distribute anyway and I wouldn’t want any person to use” and “What went wrong with my life that I have to write hate posts to get attention and feel any kind of emotion”. These are just a few of the question that should have come up and given one pause, way before getting anywhere near the the publish button.

Now I can answer most of these questions, and you would be happy to know that even Chromium and Electron have been adopting more and more of the sandboxed Portal APIs as the years pass. But there isn’t any point in talking about it cause none of the delirium is about the technical decisions behind Flatpak or how it works. None.

Let me explain.

Flatpak itself is a piece of software. It provides major advantages to distributing and running applications such as: atomic updates, binary deltas, reproducible build and run environments, mandatory sandboxing for many resources, and so on. How the software is built and distributed however has nothing to do with Flatpak. If you think the distribution-model is what’s best for you, you can already use Fedora’s flatpaked applications, Canonical’s snaps or your fav distro version of this. Everything is still built from distribution packages, by your distribution vendor, vetted by the package maintainers, come with the same downstream patches you’d see in the normal rpm/deb/etc variations, and so on. And you would still get the advantages of sandboxing, atomicity, etc even though you don’t need them cause you love and trust your distro so much.

On the other hand what every single post really complains about is Flathub. You see, what Flatpak gave us was the ability to decouple the applications from the host system. Instead of taking the existing runtime from some distro, We (The platform and application developers) built our Runtimes from scratch, that we were in full control of, that we could update and mold at will, that was not bound to any existing distribution or corporation, that we could make sure our applications were fully functional with, without any downstream patches that made things orange or blue. And unlike the old distribution model, Flathub gave application developers the same autonomy. We no longer had to wait for dependencies to be packaged, or the worry about some distribution shipping an incompatible version. We didn’t have to wait until a new enough version of a library was included into an LTS release before making use of it. We could now ship our applications on our cadence, without gatekeepers, in the way we envisioned and intended.

This is what made applications truly work on any distribution. This is what was truly disruptive about Flatpak. This is what the haters are mad about.

Thanks to Flathub the social dynamic for distributing applications has changed. Now the people that create the Platforms (GNOME, KDE, Elementary, etc) and Applications are in charge of distributing them. The sysadmin-turned-distro-packager middleman trope from the 90s is gone and no developer, or user wants it back. This is why Flathub took over, this is why no application developer became a Fedora packager even when they could build Flatpaks from the rpms packaged. If we ever want “Desktop Linux” to succeed, we have to let go of the idea of Linux distributions and “Linux” as a monolith.

The old distribution model is still useful for very specific, enterprise environments where you depend on a single ISV for all your software, but unless your Surname is Mr. IBM or Mr. Canonical, you gain nothing by asking for this on your desktop.

If you want to read more on the subject I highly suggest these two blogposts, along with Richard Brown’s Fosdem 2023 talk.

Quick heads up, the GNOME Nightly Flatpak repository is currently undergoing maintenance, during which you may notice that some applications are currently missing from the repo.

For a couple of months now we have been plagued by a few bugs that have made maintenance of the repo very hard, and CI builds were constantly failing due to a lack of available space. In order to resolve this we had to wipe the majority of the refs/objects in the repository and start again with safeguards in place.

As such, we are currently re-populating the repository with fresh builds of all the applications, but it may take a while. If you want to help with this, make sure your Flatpak manifests are up to date and build-able, and that you have set up a daily or weekly scheduled CI build in your project. Your app may not be changing, but the runtime might, and it’s good to be on top of possible API/ABI breaks.

Go to your project, Settings -> CI/CD -> Schedules -> New schedule button -> Select the daily preset.

If you are a user and seeing warnings while updating, don’t worry – you won’t have to do anything and updates will start working again transparently once the applications are available in the repository.

$ flatpak update
Looking for updates…
F: Warning: Treating remote fetch error as non-fatal since runtime/org.gnome.Todo.Devel.Locale/x86_64/master is already installed: No such ref 'runtime/org.gnome.Todo.Devel.Locale/x86_64/master' in remote gnome-nightly
F: Warning: Treating remote fetch error as non-fatal since runtime/org.gnome.TextEditor.Devel.Locale/x86_64/master is already installed: No such ref 'runtime/org.gnome.TextEditor.Devel.Locale/x86_64/master' in remote gnome-nightly
F: Warning: Treating remote fetch error as non-fatal since runtime/org.gnome.TextEditor.Devel.Debug/x86_64/master is already installed: No such ref 'runtime/org.gnome.TextEditor.Devel.Debug/x86_64/master' in remote gnome-nightly
F: Warning: Treating remote fetch error as non-fatal since runtime/org.gnome.Photos.Locale/x86_64/master is already installed: No such ref 'runtime/org.gnome.Photos.Locale/x86_64/master' in remote gnome-nightly
F: Warning: Treating remote fetch error as non-fatal since runtime/org.gnome.Epiphany.Devel.Locale/x86_64/master is already installed: No such ref 'runtime/org.gnome.Epiphany.Devel.Locale/x86_64/master' in remote gnome-nightly
F: Warning: Treating remote fetch error as non-fatal since app/org.gnome.Todo.Devel/x86_64/master is already installed: No such ref 'app/org.gnome.Todo.Devel/x86_64/master' in remote gnome-nightly
F: Warning: Treating remote fetch error as non-fatal since app/org.gnome.TextEditor.Devel/x86_64/master is already installed: No such ref 'app/org.gnome.TextEditor.Devel/x86_64/master' in remote gnome-nightly
F: Warning: Treating remote fetch error as non-fatal since app/org.gnome.Photos/x86_64/master is already installed: No such ref 'app/org.gnome.Photos/x86_64/master' in remote gnome-nightly
F: Warning: Treating remote fetch error as non-fatal since app/org.gnome.Epiphany.Devel/x86_64/master is already installed: No such ref 'app/org.gnome.Epiphany.Devel/x86_64/master' in remote gnome-nightly

Sorry for the inconvenience and happy hacking.

Before we start, let’s get this out of the way because the week long delirium on social media has dragged enough.

Yes, libadwaita “hardcodes” Adwaita. Yes, applications, as is, will not be following a custom system theme. Yes this does improve the default behavior of application for GNOME when run on other platforms like Elementary OS. However, this is the result of a technical limitation, and not some evil plot as Twitter will keep telling you…

The reason is that in order for High Contrast (and the upcoming Dark Style) to work, libadwaita needs to override the theme name property so it doesn’t fallback to GTK’s “Default” High Contrast style. The “Default” style is an older version of Adwaita, not your system style.

Compared to GTK 3, there isn’t a new way to enforce the “hardcoded” style. The GTK_THEME Debug variable still works, as does ~/.config/gtk-4.0/gtk.css which you can use to set more permanent changes, and there are probably 3 other ways of doing this. The process to theme your system might be a bit different compared to GTK 3 but it will still work. Likewise, if you are developing a distribution, you have control of the end product and can do anything you want with the code. There is a plethora of options available. Apparently complaining on social media and bullying volunteers into submission was one such option…

And I guess this also needs to be stated: this change only affects apps that choose to use libadwaita and adopt the GNOME Design Guidelines, not “every” GTK 4 application.

As usual, the fact that the themes keep working doesn’t mean they are supported. The same issues about restyling applications when they don’t expect it apply and GNOME can not realistically support arbitrary stylesheets that none of the contributors develop against and test.

Now onto the actual blogpost.

There seems to be some confusion when it comes to libadwaita’s stylesheet and coloring APIs. It’s easy to mix them up when you haven’t heard of libadwaita before, so here is a short introduction on what they are and how they differ.

Keep in mind that the features discussed below are not guaranteed to land. After libadwaita 1.0 the stylesheet will be frozen and treated as an API. That means that if a feature doesn’t make it by 1.0 it will be a breaking change and will have to wait for libadwaita 2.0.

An Application Coloring API / Accent Colors

The idea here is that you can define “accent” colors to be applied for various parts of widgets. You can also recolor any part of a widget however you like. Take a look at Epiphany’s private mode header bar for an example. For this to be possible the whole stylesheet had to be reworked. Extra care was needed to ensure that the functionality wouldn’t conflict with the high contrast preference and wouldn’t need special handling. I hope Alexander will blog about this work in more detail, as it was truly fascinating. I am very excited to see what developers do with the coloring API.

For now the colors can be controlled with the GTK-specific @define-color, similar to CSS variables. Programmatic API will be added later on as the dust settles. The API will be based on AdwStyleManager which is getting introduced by the Dark style preference MR and hasn’t landed yet.

Here’s a quick example:

@define-color accent_color @yellow_5;
@define-color accent_bg_color @yellow_2;
@define-color accent_fg_color black;

.controls {
    color: white;
    background: linear-gradient(to right, shade(@blue_3, .8), @purple_2);
}

.slider > trough > highlight {
    background: linear-gradient(to left, shade(@red_1, .8), @yellow_4);
}

.controls textview text {
  background: none;
}

.controls entry,
.controls spinbutton,
.controls textview {
  background-color: alpha(black, .15);
  color: white;
}

.navigation-sidebar,
headerbar {
  background: alpha(white, .1);
  color: white;
}

For a more detailed example of what you can do check Federico’s recent blogpost.

System Accents

This is heavily inspired by system accent settings in elementary OS, and it’s similar in function. Think of it like a way to set the accent color system-wide, then applications can read it and decide to follow or override it. A case where you want to override would be if your app had a Sepia mode for example.

The coloring API mentioned above is designed in a way that makes this feature easy to implement. The interface and UI for this are not yet fleshed out completely, and it’s debatable if it’s going to be implemented/merged at all. There are a couple of design issues and concerns that need further research. It’s a possibility, but don’t bet on it.

Vendor Styling

The story behind this idea is extensive and best left for another post, so here’s the current status on this infamous topic.

There have been great accomplishments to reducing the possible fallout of restyling applications with brand colors. Nowadays vendors recognize that arbitrary restyling can be damaging to application developers and have taken some precautions.

Yaru reworked its style and rebased it on Adwaita, this helped reduced the changes to mostly the color palette and minor stylistic tweaks. This got rid of a lot of bugs surfacing in applications, as Yaru now at least has the same spacing, margins and padding as Adwaita. Pop!_OS followed suit shortly after, I believe it’s now based on Yaru.

However, both Ubuntu and Pop also introduced “Dark-modes”, Pop making it the default, which broke applications’ expectations. They did it despite being warned about it. As a result this ended up increasing the issues with theming by about an order of magnitude as now you would frequently end up with black on black, grey on grey and other fun coloring bugs. It should also be noted that neither Ubuntu nor System 76 approached any contributor I know of, about properly implementing a Dark Style preference upstream. Even though GNOME and Elementary contributors had been collaborating in public for the last 3 years.

Yaru developers did some research on the topic and there was a call for engagement by GNOME, but unfortunately ever since the last theming BoF in 2019, the conversation had dried up. The interested parties haven’t provided any details on what the scope of the API would need to be, how it would look like, or the detailed requirements. Nobody stepped up to help with the Adwaita changes that were required either, or with dark mode, or to work on the QA tooling, or to figure out the implementation details. Now we are sadly out of time for libadwaita 1.0 and there isn’t much hope for such a complex thing to be ready in the next 4 months.

Conclusion

For libadwaita 1.0 and GNOME 42 the work on recoloring widgets will likely be completed. A proper Dark Style setting will likely also be implement by then. System-wide accent colors are being discussed and looked at, but there are design related concerns about them, so it’s possible that they will never land. And there won’t be any “Theming API” for libadawaita 1.0. Maybe there will be renewed interest from the vendors that want it in the future, but given the story so far, I won’t hold my breath. I hope to be proven wrong.

Last week I needed a Debian system to test things, I had heard others were using toolbox with Debian images without much trouble so decided to give it a go instead of creating a VM.

Toolbox only requires a handful utilities to work with any given docker image. After a quick search I stumbled upon Philippe’s post which in turn linked into this PR about an Ubuntu based toolbox image. Looks like the last major issues where worked out recently in toolbox and there isn’t anything extra needed apart the image.

Until the upstream PR is merged, I’ve adapted the image for Debian Sid and inlined the deps in one Dockerfile so its easier to fetch it and use it. The Dockerfile is hosted in a gist here.

Here is how to use it:

curl -o debian.Dockerfile 'https://gitlab.gnome.org/-/snippets/1653/raw/master/debian.Dockerfile'
podman build -t debian-sid-toolbox -f debian.Dockerfile .
toolbox create -c sid --image debian-sid-toolbox
toolbox enter sid

Enjoy!