The 2023/2024 GNOME STF project is mostly wrapped up now, so it’s a good moment to look back at what was done as part of the project, and what’s next for the projects we worked on.
As a brief refresher, STF (Sovereign Tech Fund, recently renamed to Sovereign Tech Agency) is a program by the German Government to support critical public interest software infrastructure. Sonny Piers and I applied with a proposal to improve important, underfunded areas of GNOME and the free desktop and got an investment of 1 Million Euro for 2023/2024.
While we’ve reported individual parts of what we were able to achieve thanks to this investment elsewhere, it felt important to have a somewhat comprehensive post with all of it in one place. Everyone on the team contributed summaries of their work to help put this together, with final editing by Adrian Vovk and myself.
Table of contents:
Accessibility
Accessibility is an incredibly important part of the GNOME project, community, and platform, but unfourtunately it has historically been underfunded and undermaintained. This is why we chose to make accessibility one of the primary focus areas for the STF project.
Newton
The Assistive Technology Service Provider Interface (AT-SPI) is the current accessibility API for the Linux desktop. It was designed and developed in the early 2000s, under the leadership of Sun Microsystems. Twenty years later, we are feeling its limitations. It’s slow, requiring an IPC round trip for each query a screen reader may want to make about the contents of an app. It predates our modern desktop security technologies, like Wayland and Flatpak, so it’s unaware of and sometimes incompatible with sandboxing. In short: it’s a product of its time.
The STF project was a good opportunity to start work on a replacement, so we contracted Matt Campbell to make a prototype. The result was Newton, an experimental replacement for the Linux desktop accessibility stack. Newton uses a fundamentally different architecture from AT-SPI, where apps push their accessibility information to the screen reader. This makes Newton significantly more efficient than AT-SPI, and also makes it fully compatible with Wayland and the Flatpak sandbox.
The prototype required work all across the stack, including GTK, Mutter, Orca, and all the plumbing connecting these components. Apps use a new Wayland protocol to send accessibility info to Mutter, which ensures that the accessibility state an app reports is always synchronized with the app’s current visual state. Meanwhile, the prototype has Orca communicate with Mutter via a new D-Bus Protocol.
This D-Bus protocol also includes a solution for one of the major blockers for accessibility on Wayland. Due to Wayland’s anti-keylogging design, Orca is unable to intercept certain keys used to control the screen reader, like Insert or Caps Lock. The protocol gives this intercept functionality to screen readers on Wayland. Recently, RedHat’s Lukáš Tyrychtr has adapted this part of Matt’s work into a standalone patch, which landed in GNOME 48.
As part of this work, Matt added AccessKit support to GTK. This library acts as an abstraction layer over various OS-specific accessibility APIs, and Matt’s experimental fork included support for the Newton Wayland protocol. As a side effect, GTK accessibility now works on Windows and macOS! Matt’s original patch was rebased and merged by Matthias Clasen, and recently it was released in GTK 4.18.
Finally, to test and demonstrate this new accessibility stack, Matt integrated all his changes into a fork of GNOME OS and the GNOME Flatpak Platform.
For more details about Newton’s design and implementation, including a demo video of Newton in action, you can read Matt’s announcement blog post, and his subsequent update.
Orca
The STF project allowed Igalia’s Joanmarie Diggs to rewrite and modernize much of Orca, our screen reader. Between November 2023 and December 2024 there were over 800 commits, with 33711 insertions and 34806 deletions. The changes include significant refactoring to make Orca more reliable and performant as well as easier to maintain and debug. Orca is also used on other desktop environments, like KDE, so this work benefits accessibility on the Linux desktop as a whole.
Orca now no longer depends on the deprecated pyatspi library, and has switched to using AT-SPI directly via GObject Introspection. As part of this replacement, a layer of abstraction was added to centralize any low-level accessibility-API calls. This will make it significantly easier to port Orca to new platform accessibility APIs (like Newton) when the time comes.
Over the years, Orca has added many workarounds for bugs in apps or toolkits, to ensure that users are able to access the apps they need. However, enough of these workarounds accumulated to impact Orca’s performance and reliability. The STF project allowed the majority of these workarounds to be investigated and, where possible, removed. In cases where workarounds were still necessary, bugs were filed against the app or toolkit, and the workaround was documented in Orca’s code for eventual removal.
There is arguably no single “correct” order or number of accessibility events, but both order and number can impact Orca’s presentation and performance. Therefore, Orca’s event scheduling was reworked to ensure that events are recieved in a consistent order regardless of the source. Orca’s event-flood detection was also completely reworked, so that apps can no longer freeze Orca by flooding it with events.
A lot of work went into increasing Orca’s code quality. A couple of tightly-entangled systems were disentangled, making Orca a lot more modular. Some overly complicated systems were refactored to simplify them. Utility code that was unnecessarily grouped together got split up. Linter warnings were addressed and the code style was modernized. Overall, Orca’s sources are now a lot easier to read through and reason about, debug, analyze, and maintain.
Finally, building apps that are compatible with screen readers is occasionally challenging. Screen readers have complicated rules about what they present and when they present it, so sometimes app developers are unsure of what they need to do to make Orca present their app correctly. To improve the developer experience around building accessible apps, there’s now a new guide with tips and techniques to use. This guide is very much a work in progress, and additional content is planned.
WebKitGTK
WebKitGTK is GNOME’s default web rendering engine. GTK4 significantly reworked the accessibility API for GTK widgets, so when WebKitGTK was first ported to GTK4, a major missing feature was the accessibility of web pages. The screen reader was simply unable to see web content visible on screen. As part of the STF project, Igalia’s Georges Basile Stavracas Neto added support for GTK4’s new accessibility APIs to WebKitGTK. This landed in WebKitGTK 2.44, the first stable release with GTK4 support.
Around the same time, Joanmarie removed Orca’s custom WebKitGTK handling in favor of the generic “web” support, which aligns WebKitGTK’s user experience with Firefox and Chromium. This gives Orca users an additional choice when it comes to web browsing. Please note that there are still a couple of accessibility bugs that must be fixed before Orca users can enjoy the full benefits of this change.
The last hurdle to fully functional accessibility in WebKitGTK was Flatpak. Web browsers are generally hard to make work in Flatpak, due to the interaction between Flatpak’s sandbox and the browser’s own sandboxing features, which are usually either turned off, weakened, or replaced downstream. WebKitGTK, however, has strong support for sandboxing in Flatpak, and it actually uses Flatpak’s native subsandboxing support directly. Unfourtunately, the way the sandboxes interacted prevented WebKitGTK from exporting its accessibility information to the system. Georges takes a deep dive into the specifics in his GUADEC 2024 talk.
Since that talk, Georges added features to Flatpak (and a follow-up) that made WebKitGTK work with the screen reader. This makes GNOME Web the first web browser that is both fully accessible and fully Flatpak sandboxed!
Spiel
Text-to-speech (TTS) on Linux is currently handled by a service called SpeechDispatcher. SpeechDispatcher was primarily built for use in screen readers, like Orca. Thus, TTS on Linux has generally been limited to accessibility use cases. SpeechDispatcher is modular, and allows the user to replace the speech synthesizer (which defaults to the robotic-sounding eSpeak) with something that sounds more natural. However, this configuration is done via text files, and can thus be nontrivial to get right, especially if the user wishes to integrate a proprietary synthesizer they might have paid money for.
Eitan Isaacson ran up against these limitations when he was implementing the Web Speech API into Firefox. So, he created Spiel, a new TTS framework for the Linux desktop. Spiel is, at its core, a D-Bus protocol that apps and speech synthesizers can use to communicate. Spiel also has a client library that closely emulates the Web Speech API, which makes it easy for apps to make use of TTS. Finally, Spiel is a distribution system for voices, based on Flatpak. This part of Spiel is still in the early stages. You can learn more about Spiel via Eitan’s GUADEC 2024 Talk.
As part of the STF project, Andy Holmes and Eitan implemented an initial working implementation of Spiel in Orca, demonstrating its viability for screen readers. This helped stabalize Spiel, and encouraged engagement with the project. The Spiel client and server libraries were also hardened with sanitizer and static analysis testing.
Platform
The GNOME Platform consists of the libraries, system and session services, and standards provided by GNOME and Freedesktop. In short, this is the overarching API surface that we expose to app developers so that they can write GNOME apps. Clearly, that’s very important and so we focused much of the STF’s funding there. In no particular order, here’s some of the work that the STF made possible.
Libadwaita
Starting with GTK4, we’ve decoupled GTK from GNOME’s design guidelines. This means that GTK4 no longer includes GNOME’s style sheet, or GNOME-specific widgets. This has many benefits: first and foremost, this makes GTK4 a much more generic UI toolkit, and thus more suitible for use in other desktop environments. Second, it gives GNOME the flexibility to iterate on our design and UI without interfering with other projects, and on a faster timescale. This leads to “platform libraries”, which extend GTK4’s basic widgetry with desktop-specific functionality and styles. Of course GNOME has a platform library, but so do other platforms like elementary OS.
Adwaita is GNOME’s design language, and so GNOME’s platform library is called libadwaita. Libadwaita provides GNOME’s style sheet, as well as widgets that implement many parts of the GNOME Human Interface Guidelines, including the machinery necessary to build adaptive GNOME apps that can work on mobile phones.
The STF project allowed libadwaita’s maintainer, Alice Mikhaylenko, to close a few long-standing gaps as well as finish a number of stalled projects.
Bottom Sheets
Libadwaita now provides a new bottom sheet widget, which provides a sheet that slides up from below and can be swiped back down off screen. Optionally, bottom sheets can have a bottom bar that’s visible when the sheet is collapsed, and which morphs into the sheet whenever the user activates it. This pattern is common with many apps that wish to show some short status on a main page, but a detailed view if the user wants one. For example: music player apps tend to use this kind of pattern for their “now playing” screens.
This shipped in libadwaita 1.6, and apps like Shortwave (shown above) are already using it.
Adaptive Dialogs
Traditionally, in GNOME dialogs were just separate child windows of the app’s main window. This made it difficult, sometimes, to create dialogs and popups that behave correctly on small windows and on mobile devices. Now, libadwaita handles dialogs completely within the app’s main window, which lets them adapt between floating centered pop-ups on desktop, and bottom sheets on mobile. Libadwaita’s new dialogs also correctly manage the appearance of their own close button, so that users have the ability to exit out of dialogs even on mobile devices where windows don’t normally have a close button.
This shipped in libadwaita 1.5 and many apps across GNOME have already been updated to use the new dialogs.
Multi-Layout View
Libadwaita already provides a system of breakpoints, where widget properties are automatically updated depending on the size of the app’s window. However, it was non-trivial to use breakpoints to swap between different layouts, such as a sidebar on desktop and a bottom bar on mobile. The new multi-layout view allows you to define the different layouts an app can use, and control the active layout using breakpoints.
Work on multi-layout views started before the STF project, but it was stalled. The STF project allowed it to be completed, and the feature has shipped in libadwaita 1.6.
Wrap Box
Libadwaita now provides a new wrap box widget, which wraps its children similarly to how lines are wrapped in a text paragraph. This allows us to implement various layouts that we’ve wanted to, like the list of search filters in this mockup, or toolbars that wrap onto multiple lines when there’s not enough room.
Like the multi-layout view, this work was stalled until the STF project. The feature shipped in the recent libadwaita 1.7 release.
Toggle Groups
Libadwata also now provides a new toggle group widget, which is a group of buttons where only one at a time can be selected. This pattern is pretty common in GNOME apps, and usually this was implemented manually which was awkward and didn’t look great. The new widget is a big improvement.
Toggle groups were originally implemented by Maximiliano Sandoval, but the work was stalled. The STF project allowed Alice to bring this work over the finish line. The feature is part of libadwaita 1.7.
GTK CSS
GTK uses a custom version of CSS to style its widgets, with extensions for defining and transforming colors. These extensions were limited in various ways: for instance, the defined colors were global for the entire stylesheet, but it would be very convenient to have them per-widget instead. The color functions only worked in sRGB, which isn’t the optimal colorspace for some kinds of calculations.
Thanks to work by Alice Mikhaylenko and the rest of the GTK team, GTK now has support for the standard CSS variables, color mixing, and relative colors, with a variety of color spaces. The old extensions have been deprecated. This work has already shipped in GTK 4.16, and many apps and libraries (including libadwaita as of 1.6), are making extensive use of it.
This work gets us one step closer to our long-term goal of dropping SCSS in the future, which will simplify the development and distribution process for the GNOME stylesheet.
Notification API
Notifications are a critical component of any modern computing experience; they’re essential for keeping users informed and ensuring that they can react quickly to messages or events.
The original Freedesktop Notification Standard used on the Linux desktop saw almost no significant changes in the past decade, so it was missing many modern features that users have grown to know and expect from other platforms. There were thus various DE-specific extensions and workarounds, which made it difficult for app developers to expect a consistent feature set and behavior. Even within GNOME, there were technically three different supported notification APIs that apps could use, each of which had a different subset of features. Thanks to STF funding, Julian Sparber was able to spend the time necessary to finally untangle some of the difficult issues in this area.
After evaluating different directions, a path forward was identified. The two main criteria were to not break existing apps, and to reuse one of the existing APIs. We decided to extend Flatpak’s notification portal API. The new additions include some of the most essential and highly visible features we’re currently missing, like playing a notification sound, markup styling, and more granular control of notification visibility.
The visibility control is especially impactful because it allows apps to send less intrusive notifications and it improves user privacy. On the other hand, one mostly invisible feature to users was the inclusion of the XDG Activation protocol in the new spec, which allows apps to grab focus after a user interacts with a notification. The updated protocol is already released and documented. You can find the list of changes at the pull request that introduced the v2 notifications portal.
While there is still some technical debt remaining in this area, the STF funding allowed us to get to a more sustainable place and lay the groundwork for future extensions to the standard. There is already a list of planned features for a version 3 of the notification portal.
You can read more about this initiative in Julian’s blog post on the GNOME Shell blog.
Notifications in GNOME Shell
GNOME Shell provides the core user interface for the GNOME desktop, which includes notification banners, the notification list, and the lock screen. As part of the STF project, Julian Sparber worked on refactoring and improving this part of the GNOME Shell code base, in order to make it more feasible to extend it and support new features. Specifically, this allows us to implement the UI for the v2 notifications API:
- Notifications are now tracked per-app. We now show which app sent each notification, and this also lays the technical ground work for grouping notifications by app.
- Allowing notifications to be expanded to show the full content and buttons
- Keeping all notifications until you dismiss them, rather than only keeping the 3 most recent ones
- Grouping notifications by app
Julian was also able to clean up a bunch of legacy code, like GNOME Shell’s integration with telepathy.
Most of these changes landed in GNOME 46. Grouping landed in GNOME 48. For more detail, see Julian’s blog post.
Global Shorcuts
The global shortcuts portal allows apps to request permission to recieve certain key bindings, regardless of whether the app is currently focused. Without this portal, use cases like push-to-talk in voice chat apps are not possible due to Wayland’s anti-keylogging design.
RedHat’s Allan Day created designs for this portal a while back, which we aimed to implement as part of the STF project.
Dorota Czaplejewicz spearheaded the effort to implement the global shortcuts portal across GNOME. She started the work for this in various components all over the stack, e.g. integration into the Settings UI, the compositor backend API, the GNOME portal, and the various portal client libraries (libportal and ashpd). This work has since been picked up and finalized by Carlos Garnacho and others, and landed in GNOME 48.
XDG Desktop Portals
Portals are cross-desktop system APIs that give sandboxed apps a way to securely access system resources such as files, devices like cameras, and more.
The STF project allowed Georges Stavracas to create a new dedicated documentation website for portals, which will make it easier for apps to understand and adopt these APIs. This documentation also makes it easier for desktop environment developers to implement the backend of these APIs, so that apps have complete functionality on these desktops.
Georges and Hubert Figuiere added a new portal for USB devices, and many parts of the platform are being updated to support it. This portal allows apps to list USB devices, and then request access without opening security holes.
The document portal saw some fixes for various issues, and the file transfer portal now supports directories. The settings portal was extended to advertise a new cross-desktop high contrast setting.
Hubert also worked to improve libportal, the convenience library that wraps the portal API for apps to easily consume. It now supports the settings portal, for apps to conveniently recieve light/dark mode, system accent color, and high contrast mode settings. He also fixed various bugs and memory leaks.
WebKitGTK is GNOME’s default web engine, for rendering web content in apps. It supports modern web standards, and is used in GNOME Web, our default browser. Georges adjusted WebKitGTK to make use of portals for printing and location services. New features were added to all parts of the stack to enable this. This makes WebKitGTK and every app that uses it more secure.
Flatpak and Sandboxing
Flatpak is the standard cross-distribution app packaging format, and it also provides security through sandboxing. It’s split into a few smaller sub-projects: the Flatpak core which implements the majority of Flatpak’s functionality, the xdg-dbus-proxy which filters D-Bus traffic to enforce which services sandboxed apps can talk to, and flatpak-builder which app developers can use to build Flatpak packages conveniently.
As part of the STF project, Hubert worked on improving the maintenance situation for the Flatpak core, and fixed various bugs and memory leaks. Hubert and Georges also implemented the necessary groundwork in Flatpak’s sandboxing for the new USB portal to function.
In flatpak-builder, Hubert implemented the long-awaited feature to rename MIME files and icons, which simplifies packaging of desktop applications. He also performed some general maintenance, including various minor bug fixes.
The XDG D-Bus Proxy previously relied on some very specific implementation details of the various D-Bus client libraries, such as GLib’s gdbus and zbus. This broke when zbus changed its implementation. Thanks to work by Sophie Herold, xdg-dbus-proxy was updated to stop relying on this undefined behavior, which means that all client libraries should now work without problems.
Nautilus File Chooser Portal
The file chooser portal is used by apps to bring up a sandbox-aware file picker provided by the system. It acts as an invisible permission check for Flatpak apps. This portal powers the “Open” or “Save As” actions in apps.
Previously, GNOME’s implementation of xdg-desktop-portals used GTK’s built-in file chooser dialog widget. This, however, caused some problems. Since the dialog’s implementation lived in GTK, and GTK is a dependency of libadwaita, it couldn’t use any of libadwaita’s functionality to avoid circular dependencies. This meant that the dialog couldn’t be made to work on mobile, and didn’t look in line with modern GNOME apps. The behavior of the file chooser was similar to Natuilus, our file manager app, but not identical. This would cause confusion among users. It took lots of work to keep both projects at least somewhat in line, and even then it wasn’t perfect. The file chooser couldn’t benefit from recent performance improvements in Nautilus, and it was missing some of Nautilus’s features. For example, the file chooser couldn’t generate thumbnails and did not support multiple zoom levels.
António Fernandes extended Nautilus with a new implementation for the file chooser portal (with the help of Corey Berla and Khalid Abu Shawarib doing reviews and fixups). Nautilus can now behave like an open or save file picker dialog, handling all the edge cases this entails. This required a surprising amount of work. For example, Mutter needed improvements to handle attaching Nautilus as a modal dialog, Natuilus itself needed several refactors to support different views (saving files, opening files, normal file browser), the initial portal implementation needed to be reworked to avoid breaking dark mode, and there were several iterations on the design to deal with UX edge cases.
All of this work landed in GNOME 47.
GNOME Online Accounts
GNOME Online Accounts (GOA) is GNOME’s single sign-on framework, providing a way for users to setup online accounts to be used across the desktop and preinstalled apps. Since there was no fixed maintainer in recent years, the project fell behind in maintenance, relied on old libraries, used old tooling for tests, and was missing support for open protocols like WebDAV (including CalDAV and CardDAV). Andy Holmes took over maintenance thanks to the STF project, and put it on more stable footing.
GOA used to only have limited WebDAV support as part of its Nextcloud integration. Andy separated the WebDAV support into a standalone integration, which allows users to integrate with more open-source-friendly providers, like Fastmail. This integration was also tested with well-known self-hosted servers.
GOA previously relied on its own webview for Oauth2 login, for providers like Google. Andy replaced this with a secure exchange using the default web browser. This also allowed Andy to upgrade GOA to GTK4 (with reviews by Philp Withnall) and remove the last GTK3 dependency from GNOME Settings. As part of this rework, the backend API was refactored to be fully asynchronous.
Finally, Andy updated GOA’s test infrastructure, to use modern static analyzers and better CI tests.
Language Bindings
GLib is GNOME’s foundational C library, and includes many common-sense utilities that the C standard library lacks. It also provides a layer of platform-agnostic functionality, which means that C programs targeting GLib are easier to port to other operating systems like Windows. For instance, GLib.DateTime is a set of utilities for getting the current time (which is OS-specific), doing complex math with time, and formatting timestamps for human-readable display.
GObject introspection is GNOME’s language binding infrastructure. It allows libraries that are written in C (and, lately, Rust) to be used from other languages, including Rust, Python, JavaScript, Swift, and more! It consists of a set of coding style conventions, annotations (that appear as code comments on functions), an object-oriented type system called GObject, a tool that extracts all of this information into .gir files, a library to parse the these files, and per-language infrastructure to consume this parsed data into the language’s type system. This infrastructure enables language bindings to be relatively easy to make and maintain, which in turn enables GNOME’s large ecosystem of apps written in a diverse set of languages.
GLib and GObject introspection are tightly coupled projects. GLib defines the type system (including GObject, and the lower-level concepts underneath it), and GObject introspection heavily relies on these types. Conversely, GLib itself is accessible from language bindings, which means that GLib depends on GObject introspection. This complicated dependency situation makes it rather difficult to iterate on our language bindings, and was quite messy to maintain.
As part of the STF project, Philip Withnall started work on merging GObject introspection into GLib. Having them in the same repository means that developing them together is easier, because it can avoid dependency cycles. So far, he was able to move libgirepository, which is a large part of GObject introspection. In practice, this has allowed us to generate the .gir files for GLib as part of its build process, rather than generating them externally.
Building on this work, Evan Welsh was able to start making improvements to our language bindings. Evan added support for annotating async functions (based on work by Veena Nager), so that GObject introspection doesn’t need to use heuristics to guess which APIs are async. This allows language bindings to better integrate GNOME’s async APIs with the language’s native async/await syntax.
Evan’s work on async function calls required work across the entire language binding stack, including some more merging of GObject introspection into GLib. Most notably, these new features required new test cases, which meant that GLib’s CI needed to use the bleeding-edge version of GObject introspection, which was rather difficult due to the entangled dependencies between the two projects. Evan made these necessary changes, so now it is more feasible to extend the functionality of our language bindings.
Evan then went on to integrate this work across the rest of the stack. In Meson, there’s a pending pull request to transition from the old GObject introspection tools to the new GLib tools. In GNOME’s JavaScript bindings, Philip started integrating the GLib version of libgirepository, and Evan has since continued this work.
Evan also did some work in GTK to fix an issue that previously skipped some accessibility APIs when generating language bindings. This made it possible for apps written in languages other than C to better communicate their structure to the screen reader, improving the accessibility of those apps.
Finally, Evan worked on advancing GNOME’s TypeScript bindings by merging gi.ts and ts-for-gir into a single toolchain which can fully cover GNOME’s stack and have accurate enough types to work with existing JavaScript codebases. This was possible thanks to help by Pascal Garber, the maintainer of ts-for-gir. This will enable GNOME’s many apps implemented in JavaScript to be ported to TypeScript, allowing for static analysis and increasing code quality. For instance, GNOME Weather was recently ported to TypeScript.
GLib
After merging libgirepository into GLib, Philip was able to port GLib away from gtk-doc and to gi-docgen, GNOME’s modern API documentation generator. This brought a much faster build time for documentation, and makes the docs much more useful for users of language bindings. As part of this transition, someone has to go through API-by-API and port all of the documentation to the new gi-docgen syntax. As part of the STF project, Philip was able to port all section introductions and some API documentation comments, but there’s a huge number of APIs so more work is still required. As documentation is ported, various improvements can be made to documentation quality.
The STF project also allowed Philip to focus on various ongoing maintenance tasks for GLib, with prominent examples including:
- Reviewed and landed integration with the CHERI processor, which is an new architecture with increased memory security compared to traditional x86/ARM architectures. Having GLib running on it is an important step to bootstrapping an OS. This is the kind of work which wouldn’t get reviewed without maintenance funding for GLib, yet is important for the wider ecosystem.
- An important security bugfix for xdgmime, a core component of most desktop environments.
- A security bugfix for gdbus, GLib’s D-Bus client library, which is used by the entire GNOME stack.
- Reviewed a race-condition bugfix for GObject’s reference counting. This bug caused hard-to-debug crashes and freezes.
- Reviewed some GObject performance improvements
QA, Developer Tools, and GNOME OS
Many issues in our development process come from the fact that there’s not enough end-to-end testing with the entire stack. This was the initial motivativation for the GNOME Continuous project, which eventually became GNOME OS as we know it today. GNOME OS powers our automated QA process, and allows some limited direct testing of new GNOME features in virtual machines.
However, GNOME OS has a lot more potential beyond that as a QA and development tool. It’s 90% of the way there to being a usable OS for GNOME developers to daily drive and dogfood the latest GNOME features. This is sometimes the only way to catch bugs, especailly those relating to hardware, input, and similar situations that a VM can’t emulate. Also, since it’s a platform we control, we saw the opportunity to integrate some quality-of-life improvements for GNOME developers deep into the OS.
Transition to Sysupdate
Switching GNOME OS away from ostree and to systemd-sysupdate opened the doors to more complete integration with all of systemd’s existing and future development tools, like systemd-sysext. It also enabled us to build security features into GNOME OS, like disk encryption and UEFI secure boot, which made it suitable for daily-driver use by our developers.
This work started before the STF’s investment. Valentin David and the rest of the GNOME OS team had already created an experimental build of GNOME OS that replaced ostree with systemd-sysupdate. It coexisted with the official recommended ostree edition. At roughly the same time, Adrian Vovk was making a similar transition in his own carbonOS, when he discovered that systemd-sysupdate doesn’t have an easy way to integrate with GNOME. So, he made a patch for systemd that introduces a D-Bus service that GNOME can use to control systemd-sysupdate.
As part of the STF project, these transitions were completed. Codethink’s Tom Coldrick (with help from Jerry Wu and Abderrahim Kitouni) rebased Adrian’s D-Bus service patch, and it got merged into systemd. Jerry Wu and Adrien Plazas also integrated this new service into GNOME Software.
Adrian continued improving sysupdate itself: he added support for “optional features”, which allow parts of the OS to be enabled or disabled by the system administrator. This is most useful for optionally distributing debugging or development tools or extra drivers like the propriertary NVIDIA graphics driver in GNOME OS.
GNOME OS also needed the ability to push updates to different branches simultaneously. For instance, we’d like to have a stable GNOME 48 branch that recieves security updates, while our GNOME Nightly branch contains new unfinished GNOME features. To achieve this, Adrian started implementing “update streams” in systemd-sysupdate, which are currently pending review upstream.
Codethink wrote about the sysupdate transition in a blog post.
Immutable OS Tooling
Thanks to GNOME OS’s deep integration with the systemd stack, we were able to leverage new technologies like systemd-sysext to improve the developer workflow for low-level system components.
As part of his work for Codethink, Martín Abente Lahaye built sysext-utils, a new tool that lets you locally build your own version of various components, and then temporarily apply them over your immutable system for testing. In situations where some change you’re testing substantially compromises system stability, you can quickly return to a known-good state by simply rebooting. This work is generic enough that the basics work on any systemd-powered distribution, but it also has direct integration with GNOME OS’s build tooling, making the workflow faster and easier than on other distributions. Martín went into lots more detail on the Codethink blog.
A natural next step was to leverage sysext-utils on GNOME’s CI infrastructure. Flatpak apps enjoy the benefits of CI-produced bundles which developers, testers, and users alike can download and try on their own system. This makes it very natural and quick to test experimental changes, or confirm that a bug fix works. Martín and Sam Thursfield (with the help of Jordan Petridis and Abderrahim) worked to package up sysext-utils into a CI template that GNOME projects can use. This template creates systemd-sysext bundles that can be downloaded and applied onto GNOME OS for testing, similar to Flatpak’s workflow. To prove this concept, this template was integrated with the CI for mutter and gnome-shell. Martín wrote another blog post about this work.
Security Tracking
To become suitable for daily-driver use, GNOME OS needs to keep track of the potential vulnerabilities in the software it distributes, including various low-level libraries. Since GNOME OS is based on our GNOME Flatpak runtime, improving its vulnerability tracking makes our entire ecosystem more robust against CVEs.
To that end, Codethink’s Neill Whillans (with Abderrahim’s help) upgraded the GNOME Flatpak runtime’s CVE scanning to use modern versions of the freedesktop-sdk tooling. Then, Neill expanded the coverage to scan GNOME OS as well. Now we have reports of CVEs that potentially affect GNOME OS in addition to the GNOME Flatpak runtime. These reports show the packages CVEs come from and a description of each vulnerability.
GNOME OS Installer
To make GNOME OS more appealing to our developer community, we needed to rework the way we install it. At the moment, the existing installer is very old and limited in features: it’s incompatible with dual-boot, and the current setup flow has no support for the advanced security features we now support (like TPM-powered disk encryption).
Adrian started working on a replacement installer for GNOME OS, built around systemd’s low-level tooling. This integration allows the new installer to handle GNOME OS’s new security features, as well as provide a better UX for installing and setting up GNOME OS. Most crucially, the new architecture makes dual-boot possible, which is probably one of the most requested GNOME OS features from our developers at the moment.
Sam Hewitt made comprehensive designs and mockups for the new installer’s functionality, based on which Adrian has mostly implemented the frontend for the new installer. On the backend, we ran into some unexpected difficulties and limitations of systemd’s tools, which Adrian was unable to resolve within the scope of this project. The remaining work is mostly in systemd, but it also requires improvements to various UAPI Group Specifications and better integration with low-level boot components like the UEFI Secure Boot shim. Adrian gave an All Systems Go! talk on the subject, which goes into more details about the current status of this work, and the current blockers.
Buildstream
Buildstream is the tool used to build GNOME OS as well as the GNOME SDK and Runtime for Flathub, building the bases of all GNOME apps distributed there.
Previously, it was not possible to use dependencies originating from git repositories when working with the Rust programming language. That made it impossible to test and integrate unreleased fixes or new features of other projects during the development cycle. Thanks to work by Sophie Herold the use of git dependencies is now possible without any manual work required.
Additionally, Buildstream is also used by the Freedesktop.org project for its SDK and runtime. Most other runtimes, including the GNOME runtime are based on it. With the newly added support for git source, it has now become possible to add new components of the GStreamer multimedia framework. Components written in Rust were previously missing from the runtime. This includes the module that makes it possible to use GStreamer to show videos in GNOME apps. These functions are already used by apps like Camera, Showtime, or Warp.
OpenQA
Quality assurance (QA) testing on GNOME OS is important because it allows us to catch a whole class of issues before they can reach our downstream distributors. We use openQA to automate this process, so that we’re continuously running QA tests on the platform. Twice a day, we generate GNOME OS images containing all the latest git commits for all GNOME components. This image is then uploaded into openQA, which boots it in a VM and runs various test cases. These tests send fake mouse and keyboard input, and then compare screenshots of the resulting states against a set of known-good screenshots.
Codethink’s Neill Whillans created a script that cleans up the repository of known-good screenshots by deleting old and unused ones. He also fixed many of our failing tests. For instance, Neill diagnosed a problem on system startup that caused QA tests to sometimes fail, and triaged it to an issue in GNOME Shell’s layout code.
Building on the sysext-utils work mentioned above, Martín made a prototype QA setup where GNOME’s QA test suite can run as part of an individual project’s CI pipeline. This will make QA testing happen even earlier, before the broken changes are merged into the upstream project. You can see the working prototype for GNOME Shell here, and read more about it in this blog post.
Security
GNOME and its libraries are used in many security-critical contexts. GNOME libraries underpin much of the operating system, and GNOME itself is used by governments, corporations, journalists, activists, and others with high security needs around the world. In recent years, the freedesktop has not seen as much investment into this area as the proprietary platforms, which has led to a gap in some areas (for instance: home directory encryption). This is why it was important for us to focus on security as part of the STF project.
Home Directory Encryption
systemd-homed is a technology that allows for per-user encrypted home directories. Most uniquely, it has a mechanism to delete the user’s encryption keys from memory whenever their device is asleep but powered on. Full Disk Encryption doesn’t protect data while the machine is powered on, because the encryption key is available in RAM and can be extracted via various techniques.
systemd-homed has existed for a couple of years now, but nobody is using it yet because it requires integrations with the desktop environment. The largest change required is that homed needs any “unlock” UI to run from outside of the user session, which is not how desktop environments work today. STF funding enabled Adrian Vovk to work on resolving the remaining blockers, developing the following integrations:
- Extending systemd-homed with the ability to store large binary blobs of user metadata, like profile pictures.
- Extending the GNOME login screen service (GDM) with the ability to show a special out-of-session unlock UI, to meet the aforementioned requirement.
- Adding plumbing to systemd-logind that notifies GDM when it’s time to show the out-of-session unlock UI.
- Integrating systemd-homed with AccountsService, which currently acts as GNOME’s database of users on the system. Previously, homed users didn’t appear anywhere in GNOME’s UI.
In addition, Adrian triaged and fixed a lot of bugs across the stack, including many blockers in systemd-homed itself.
This work was completed, and a build of GNOME OS with functional homed integration was produced. However, not all of this has been merged upstream yet. Also, due to filesystem limitations in the kernel, we don’t have a good way for multiple homed-backed users to share space on a single system at the moment. This is why we disabled multi-user functionality for now.
For more details on the subject, you can watch Adrian’s All Systems Go! talk.
GNOME Keyring
The infrastructure to securely store secrets (i.e. passwords and session tokens) for apps on Linux is the FreeDesktop Secret Service API. On GNOME, this API is provided by the GNOME Keyring service. Unfourtunately, GNOME Keyring is outdated, overly complex, and cannot meet the latest security requirements. Historically, it has also provided other security-adjacent services, like authentication agents for SSH and GPG. There have been numerous efforts to gradually reduce the scope of GNOME Keyring and modernize its implementation, the most recent of which was Fedora’s Modular GNOME Keyring proposal. Unfortunately, this work was stalled for years.
As part of the STF project, Dhanuka Warusadura took over the remaining parts of the proposal. He disabled the ssh-agent implementation in GNOME Keyring, which prompted all major distributions to switch to gcr-ssh-agent, the modern replacement. He also ported the existing gnome-keyring PAM module with reworked tests, following the modern PAM module testing best practices. With this work completed, GNOME Keyring has been reduced to just a Secret Service API provider, which makes it possible for us to replace it completely.
As the replacement for this remaining part of GNOME Keyring, Dhanuka extended the oo7 Secret Service client library to also act as a provider for the API. OO7 was chosen because it is implemented in Rust, and memory safety is critical for a service that manages sensitive user secrets. This new oo7-daemon is almost ready as a drop-in replacement for GNOME Keyring, except that it can not yet automatically unlock the default keyring at login.
As part of this project, Dhanuka also took care of general maintenance and improvements to the credential handling components in GNOME. These include gnome-keyring, libsecret, gcr and oo7-client.
Key Rack
Key Rack is an app that allows viewing, creating and editing the secrets stored by apps, such as passwords or tokens. Key Rack is based on oo7-client, and is currently the only app that allows access to the secrets of sandboxed Flatpak apps.
Key Rack was previously limited to displaying the secrets of Flatpak apps, so as part of the STF Project Felix Häcker and Sophie Herold worked on expanding its feature set. It now integrates with the Secret Service, and makes management of secrets across the system easier. With this addition, Key Rack now supports most of the features of the old Seahorse (“Passwords and Keys”) app.
Glycin
Glycin is a component to load and edit images. In contrast to other solutions, it sandboxes the loading operations to provide an extra layer of security. Apps like Camera, Fractal, Identity, Image Viewer, Fotema, and Shortwave rely on glycin for image loading and editing.
Previously, it was only possible to load images in apps and components that were written in the Rust programming language. Thanks to work by Sophie Herold it is now possible to use glycin from all programming languages that support GNOME’s language binding infrastructure. The new feature has also been designed to allow Glycin to be used outside of apps, with the goal of using it throughout the GNOME platform and desktop. Most notably, there are plans to replace GdkPixbuf with Glycin.
Bluetooth
Jonas Dreßler worked on some critical, security-relevant issues in the Linux Bluetooth stack, including work in the kernel, BlueZ, and GNOME’s Bluetooth tooling (1, 2, 3).
Bug Bounty Program
In addition to the primary STF fund, STA offers other kinds of support for public interest software projects. This includes their Bug Resilience Program, which gives projects complementary access to the YesWeHack bug bounty platform. This is a place for security researchers to submit the vulnerabilities they’ve discovered, in exchange for a bounty that depends on the issue’s severity. Once the bug is fixed, the project is also paid a bounty, which makes it sustainable to deal with security reports promptly. YesWeHack also helps triage the reported vulnerabilities (i.e. by confirming that they’re reproducible), which helps further reduce the burden on maintainers.
Sonny and I did the initial setup of this program, and then handed it over to the GNOME security team. We decided to start with only a few of the highest-impact modules, so currently only GLib, glib-networking, and libsoup are participating in the program. Even with this limited scope, at the time of writing we’ve already received about 50 reports, with about 20 bounty payments so far, totaling tens of thousands of Euro.
For up to date information about reporting security vulnerabilities in GNOME, including the current status of the bug bounty program, check the GNOME Security website.
Hardware Support
GNOME runs on a large variety of hardware, including desktops, laptops, and phones. There’s always room for improvement, especially on smaller, less performant, or more power efficient devices. Hardware enablement is difficult and sometimes expensive, due to the large variety of devices in use. For this project we wanted to focus specifically on devices that developers don’t often have, and thus don’t see as much attention as they should.
Mutter and GNOME Shell
Jonas Dreßler worked on improving hardware support in Mutter and GNOME Shell, our compositor and system UI. As part of this to this he improved (and is still improving) input and gesture support in Mutter, introducing new methods for recognizing touchscreen gestures to make touch, touchpad, and mouse interactions smoother and more reliable.
Thanks to Jonas’ work we were also finally able to enable hardware encoding for screencasts in GNOME Shell, significantly reducing resource usage when recording the screen.
The GNOME Shell user interface has also gotten and optimizations for smaller screens, ensuring a better experience on a wider range of devices, with more to follow.
Thanks to work on XWayland fractional scaling in Mutter (1, 2), the support for modern high-resolution (HiDPI) monitors got more mature and works with all kinds of applications now, making GNOME adapt better to modern hardware.
Variable Refresh Rate
Variable Refresh Rate (VRR) is a technology that allows monitors to dynamically change how often the image is updated. This is useful in two different ways. First, in the context of video games, it allows the monitor to match the graphics card’s frame rate to alleviate some microstutters without introducing tearing. Second, on devices which have support for very low minimum refresh rates (such as phones), VRR can save power by only refreshing the screen when necessary.
Dor Askayo had been working on adding VRR support to mutter in their free time for several years, but due to the fast pace of development he was never able to quite get it rebased and landed in time. The STF project allowed them to work on it full-time for a few months, which made it possible to land it in GNOME 46. The feature is currently still marked as experimental due to minor issues in some rare edge cases.
GNOME Shell Performance
GNOME Shell, through its dependency on Mutter, is the compositor and window manager underpinning the GNOME desktop. Mutter does all core input, output, and window processing. When using GNOME, you’re interacting with all applications and UI through GNOME Shell.
Thus, it’s critical that GNOME Shell remains fast and responsive because any sluggishness in Shell affects the entire desktop. As part of the STF project, Ivan Molodetskikh did an in-depth performance investigation of GNOME Shell and Mutter. Thanks to this, 12 actionable performance problems were identified, 7 of which are already fixed (e.g. 1, 2, 3), making GNOME smoother and more pleasing to use. One of the fixes made monitor screencasting eight times faster on some laptops, bringing it from unusable to working fine.
Moreover, the profiling infrastructure improvements (as well as experimental integration with the Tracy profiler, and many smaller bugfixes in profiling code paths) done thanks to the STF funding will make finding and diagnosing performance problems easier in the future.
Ivan also conducted a round of hardware input latency testing for GNOME’s VTE library, which underpins GNOME’s terminal emulators. He then worked with RedHat’s Christian Hergert to address the discovered performance bottlenecks, and then retested the library to confirm the vast performance improvement. This work landed in GNOME 46. For more details, see Ivan’s blog post.
Design Support
Close collaboration between developers and designers is an important value of the GNOME project. Even though the bulk of the work we did as part of this project was low-level technical work, many of our initiatives also had a user-facing component. For these, we had veteran GNOME designer Sam Hewitt (and myself to some degree) help developers with design across all the various projects.
This included improving accessibility across the desktop shell and apps, new and updated designs for portals (e.g. global shortcuts, file chooser), designs for security features such as systemd-homed (e.g. recovery key setup) and the new installer (e.g. disk selection), as well as general input on the work of STF contributors to make sure it fits into GNOME’s overall user experience.
Planning, Coordination & Reporting
Sonny Piers and I put together the initial STF application after consulting various groups inside the community, with the goal of addressing as many critical issues in underfunded areas as possible.
Once we got the approval we needed a fiscal host to sign the actual contract, which ended up being the GNOME Foundation. I won’t go into why this was a bad choice here (see my previous blog post for more), except to say that the Foundation was not a reliable partner for us, and we’re still waiting for the people responsible for these failures to take accountability.
However, while we were stretched thin on the administrative side due to Foundation issues, we somehow made it work. Sonny’s organizing talent and experience were a major factor in this. He was instrumental in finding and hiring contractors, reaching out to new partners from outside our usual bubble (e.g. around accessibility), managing cashflow, and negotiating very friendly terms for our contracts with Igalia and Codethink. Most importantly, he helped mediate difficult socio-technical discussions, allowing us to move forward in areas that had previously been stuck for years (e.g. notifications).
On the reporting side we collected updates from contractors and published summaries of what happened to This Week in GNOME and Mastodon. We also managed all of the invoicing for the project, including monthly reports for STA and time sheets organized by project area.
What’s Next
While we got a huge amount of work done over the course of the project, some things are not quite ready yet or need follow-up work. In some cases this is because we explicitly planned the work as a prototype (e.g. the Newton accessibility architecture), in others we realized during the project that the scope was significantly larger than anticipated due to external factors (e.g. systemd’s improved TPM integration changed our plans for how the oo7-daemon service will unlock the keyring), and in others still getting reviews was more challenging or took longer than expected.
The following describes some of the relevant follow-up work from the STF project.
Wayland-Native Accessibility (Newton)
Matt Campbell’s work on Newton, our new Wayland-native accessibility stack, was successful beyond our expectations. We intended it as only a prototype, but we were able to actually already land parts of Matt’s work. For instance, Matt worked to integrate GTK with AccessKit, which will be at the core of the Newton architecture. This work has since been picked up, updated, and merged into GTK.
However, in some ways Newton is still a prototype. It intends to be a cross-desktop standard, but has not yet seen any cross-desktop discussions. Its Wayland protocol also isn’t yet rigorously defined, which is a prerequisite for it to become a new standard. The D-Bus protocol that’s used to communicate with the screen reader is ad-hoc, and currently exists only to communicate between Orca and GNOME Shell. All of these protocols will need to be standardized before apps and desktop environments can start using it.
Even once Newton is ready and standardized, it’ll need to be integrated across the stack. GTK will get support for Newton relatively easily, since Newton is built around AccessKit. However, GNOME Shell uses its own bespoke widget toolkit and this needs to be integrated with Newton. Other toolkits and Wayland compositors will also need to add support for it.
Platform
Julian Sparber’s work on the v2 notification API has landed in part, but other parts of this are still in review (e.g. GLib, portal backend). Additionally, there’s more GUI work to be done, to adjust to some of the features in Notifications v2. GNOME Shell still needs to make better use of the notification visibility settings for the lock screen, to increase user privacy. There’s also the potential to implement special UI for some types of notifications, such as incoming calls or ringing alarms. Finally, we already did some initial work towards additional features that we want to add in a v3 of the specification, such as grouping message notifications by thread or showing progress bars in notifications.
Spiel, the new text-to-speech API, is currently blocked on figuring out how to distrbute speech synthesizers and their voices. At the moment there’s a prototype-quality implementation built around Flatpak, but unfourtunately there are still a couple of limitations in Flatpak that prevent this from working seamlessly. Once we figure out how to distribute voices, Spiel will be ready to be shipped in distros. After that, we can use Spiel in a new portal API, so that apps can easily create features that use text-to-speech.
The work done on language bindings as part of this STF project focused on the low-level introspection in GLib. This is the part that generates language-agnostic metadata for the various languages to consume. However, for this work to be useful each language’s bindings need to start using this new metadata. Some languages, like Python, have done this already. Others, like JavaScript, still need to be ported. Additionally, build systems like Meson still need some minor tweaks to start using the new introspection infrastructure when available.
GNOME OS and QA
Much of the work we’ve started for GNOME OS needs some finishing touches. The new update system, powered by systemd-sysupdate, is still missing support for multiple stable branches. It also doesn’t support delta updates yet, which would save on download bandwidth and time by only downloading changes between versions. Finally, we need to finish the work we started on the new GNOME OS installer.
We’d like to finalize and deploy the prototype that runs openQA test cases directly in the CI for each GNOME component. This infrastructure would allow us to increase the QA test coverage of GNOME as a whole.
Encrypting Home Directories
The work to integrate systemd-homed into GNOME is mostly complete and functional, but parts of it have not landed yet (see this tracking issue and all the merge requests it links to).
Due to filesystem limitations in the kernel, we don’t have a good way for multiple homed-backed users to share space on a single system. For now, we simply disabled that functionality. Follow-up work would include fixing this kernel limitation, and re-enabling multi-user functionality.
Once these things are resolved, distributions can start moving forward with their adoption plans for systemd-homed.
Long-term, we’d like to deprecate the current AccountsService daemon, which provides a centralized database for users that exist on the system. We’d like to replace it with systemd-userdb, which is a more modern and more flexible alternative.
Keyring
Before the oo7-daemon can replace the GNOME Keyring service, it still needs support for unlocking the default keyring at login. An implementation that partially copies GNOME Keyring‘s solution has been merged into libsecret, but it’s still missing integration with oo7-daemon. Once this is solved, oo7-daemon will become drop-in compatible with GNOME Keyring, and distributions will be able to start transitioning.
Longer term we would like to redo the architecture to make use of systemd’s TPM functionality, which will increase the security of the user’s secrets and make it compatible with systemd-homed.
Thanks
The 2023/2024 GNOME STF project was a major success thanks to the many many people who helped to make this possible, in particular:
- The Sovereign Tech Agency, for making all of this possible through their generous investment
- Tara Tarakiyee and the rest of the STA team, for making the bureaucratic side of this very manageable for us
- All of our contractors, for doing such wonderful work
- The wider community for their reviews, input, support, and enthusiasm for this project
- Igalia and Codethink for generously donating so much of their employee time
- RedHat and the systemd project for helping with reviews
- Sonny Piers for taking the lead on applying to the STF, and running the project from a technical point of view
- Adrian Vovk for splitting the gargantuan task of editing this blog post with me
