Most modern operating systems (including iOS, Android, macOS, and ChromeOS) have been using image-based architectures for a while, and in recent years we’ve been making progress towards this on the desktop side as well. It’s a necessity if we want to provide the security, usability and reliability that people expect from their devices nowadays.
However, broadly speaking the adoption of image-based designs has been limited in the free desktop world so far because there are still a few major gaps. While Flatpak and Flathub have more or less solved the app distribution on these systems, the developer-facing story still looks much more incomplete.
Traditionally, people have been developing against the host OS they are running. Assuming you’re working on something like NetworkManager and need a new dependency, you’d just “apt install” it globally on your system, and then configure/compile against that. Similarly, if you need a command-line utility, compiler toolchain, or other build dependency, you’d also install them from the distribution repository.
In an image-based world this approach doesn’t work, and people have come up with various ways to address this.
RPM-Ostree and Package Overlays
Fedora Silverblue can be extended similar to package-based distros with rpm-ostree overlays, but the issue with this approach is that overlayed packages can completely break the system in unexpected ways or stop it from further updating. This is why the next iteration of Silverblue is based on “bootc”, and will be explicitly avoiding this paradigm. Same issue are present with other package overlay approaches as well.
Monolithic “Development” Overlay
The approach used by Android, iOS, Windows, et al is to have a single monolithic overlay with “system development tools”. Developers can install this overlay, and it provides all the utilities people developing the system itself need.
GNOME OS does something similar: There is a “developer” system extension that overlays the toolchain used for building the OS on top of the user-facing OS image. However, this is a finite list of utilities needed specifically to build the OS. As such, it can not cover the long tail of development and debugging tools developers in different areas need (e.g. kernel development).
Toolbox
Another approach is trying to replicate the same traditional package-based experience inside a container. Toolbox and distrobox are examples of this.
Unfortunately, with this approach you are still relying on the same old package infrastructure, while also being inside a more restrictive container environment that none of the tools expect to be run in. As such, you often run into limitations when developing system components and need to bypass the container layers in order to debug the system itself.
Homebrew
Homebrew provides an independent tool chain to develop against, but in order for homebrew binaries to be usable directly in the terminal, they have to be prioritized over system binaries. This means that the system can break if there is a mismatch between what the system expects and what homebrew provides. For example, you install QEMU, but it overrides the GLib your system uses for everything else.
There are other architectural issues with homebrew, but in my opinion this alone disqualifies it for system development.
Flatpak
Lastly, we have Flatpak. Like Toolbox it uses “containers”, so the same issues and limitations are also present here, but it’s even more restrictive because the assumption is that apps will use portals to access system resources such as directories and devices. Flatpak was designed specifically with desktop apps in mind, and its architecture and integration points are not well suited for command line apps.
Somewhat tangentially, there are apps that can’t be made to fully work with Flatpak as it is today, particularly debugging utilities and IDEs. While it’s technically possible (GNOME Builder and the Ptyxis terminal are proof), older applications were not designed with sandboxed resources in mind, or have not adapted to this model yet.
What Could We Do Instead?
While we would love for everything to be sandboxed and confined, that is sadly not yet possible. All the existing approaches that attempted to containerize tooling, run into the same conflicts between the desired functionality and the restrictions inherited by this approach. This is a topic we will discuss in more depth later on.
Additionally I believe there are two different use cases here. The build tooling/toolchain used by projects, and the developer utilities.
Project Build Setups
One aspect of this is the build setup used by individual project, which would ideally be something more deterministic, and containerized like Buildstream, flatpak-builder, bazel or even nix, instead of the old status quo “apt install -yqq gcc meson libone-devel libtwo-devel”. This is also a topic for another day though.
Developer Utilities
The other aspect, and what I want to focus on today, is that a lot of developer utilities that people rely on can’t realistically be run in a confined environment (such as a container) without a near complete rewrite.
We need a way to make things like strace, ripgrep, and qemu available. Shipping them in the host system is one option, but there is a real long tail issue here. You can’t (and don’t want to) provide every single utility any developer might need. Thus we need them to be shipped independently, and as such, not attached to the host OS.
Toolpak
If we were to take a fresh look at this issue and design something from scratch, what would we want it to look like? Over the past months we’ve had a number of discussions on these topics, and it feels like we’re finally converging on a solution would address most people’s concerns and needs.
These are the important properties we would want:
- Tools should be independent from the host OS, and they should not break the host OS if something goes wrong. They need to be able to access every resource in the system, much like today.
- A large catalog of existing tooling, like we find today in distribution repositories today.
- Tools work out of the box, and will be fully functional, as it’s not feasible to modify all of them. Things like shelling out to other tools should work like it does now.
Here is what I imagine an implementation, which we will call Toolpak, would look like:
- Discoverable Disk Images (UAPI.3) as the image format. This will provide us state of the art security practices, like Verity. It will also give us a good base for allowing Reproducible builds, given that the build tooling permits.
- A mount namespace for the binaries. The /usr and /app split from Flatpak is a great idea and we should also steal it much like portable services now can use Base/Runtime images! /usr should be provided by the “Runtime Image” and shared among tools. And /app will be the main contents of the Tool image.
- Tools have unrestricted access to everything else. This should also satisfy the “no need to port” requirement (Some edge cases where the mount namespace will conflict, but they are minor).
- We should prepend the tools, to the PATH of the user (ie. our binaries will override the system ones), but it should avoid breaking the system as you can only override the system binaries. Said binary will then setup the mount namespace and execute the real binary from inside the image. This avoids messing with the linker and shared libraries. Unless you overriding the system GNU Tar with the BSD Tar (or other incompatible cases of the same binary), things should work fine. More research is needed to explore what other safeguards will be required.
- Tools can not depend on other tools. One of the issues with traditional distributions is dependency resolution and package management (this is subcategory of a heavily discussed topic, I also talked about it in my LAS talk from 2025. We should avoid making individual packages or having dependencies between tools, to avoid all the unnecessary complexity that comes with it.
- Tools will bundle all their dependencies. This will ensure that they will always execute against the environment they were tested against. It additionally allows tools to bring their own versions of libraries that might be present in the system, without conflicts. We mentioned the /usr and /app split above.
- Great user experience to discover and install tools. There should be a flagship “app store” and ideally the tools should be packaged and distributed by the developers themselves, similar to Flathub. No more “download this static binary and chmod +x it”.
- Tools have complete and arbitrary access to the system, so it’s crucial that it not be simple to install random images. They will be have to be signed with a trusted key, and verity checked at runtime, and be thoroughly reviewed before appearing on the flagship “app store”, and all the other practices we should expect from distributing software securely.
- Apps that can be packaged using Flatpak should be rejected. Unless they are less functional, like IDEs on Flathub.
In order to be adopted, it will also have to come with tooling that will make it easy to build said Toolpaks. Here are some specific properties the build tooling should have:
- It should be easy to orchestrate builds of your tool and its dependencies.
- Great caching for all build artifacts in a Content Addressable Storage.
- Reproducible by default, with tooling to easily verify the output.
- Can be used for both local development and composing the final image.
- Handles all the licensing/SBOM/etc requirements.
- A Buildstream plugin or wrapper around it should satisfy all these needs.
Next Steps
While this topic has been discussed for years, there’s now concrete work towards a prototype as part of a Prototypefund project. We plan to share more on this in the coming weeks and are interested in feedback from the wider community. In the mean time, if you have any other feedback, find us in #gnome-os:gnome.org on Matrix, or leave a comment.
Great to hear that there is work being done to fix this. And thanks for this blog post.
What is the rational that it has to be separate from flatpak and flathub (from a project perspective not technical)? With the discussion around “flatpak next” it seems that it somewhat aligns with the goal of providing unconfined applications (in addition to confined GUI apps).
In terms of user experience it would be nicer to just have these tools in Bazaar (or similar stores) instead of needing two stores. And having a larger community that handles boths GUI, CLI apps and tooling would probably accelerate adoption.
Also the current split between confined GUI apps and unconfined CLI apps leaves a gap for confined CLI apps. When looking at the available tui apps (e.g. on the ratatui catalogue) you can find a large number of applications that should not need full system access. But I get this is not the focus of toolpak being only unconfined.
Initially it started from discussions we had with Flatpak developers about Flatpak next, but the main issue is that Flatpak is and will continue to be built for Dekstop Applications. The functionality we need, is only a very small subset of Flatpak and it would also require introducing new codepaths all over the existing code to skip big part of it if its about this very specific to this usecase. It would not really be practical and it’s less complex and much simpler to have a new thing tailored to this specific problem.
As swick mentioned:
> The security and trust models, the integration, etc, are all just too different.
> In terms of user experience it would be nicer to just have these tools in Bazaar (or similar stores) instead of needing two stores.
We will have to see how we could expose them in a graphical store, but conceptually what is used to install it is mostly an implementation detail. If we come up with a way to create “Store-like” listings with screenshot and the like (what do you even present for underlying utilities), they probably be listed there.
…a few thoughts popped up while reading this:
Currently the safe way not mentioned here, is to keep your host minimal, and run deviations in a guest-kernel, though I understand why this is not always ideal, it is still something to be considered in relevant decision trees at the top.
Why trust the developer ? Users shouldn’t, developers can be people|machines with an agenda, trying to infiltrate abuse your system, and normal users wouldn’t know how to tell the difference. Distribution packaging funnels like debian packages help to block bad players. I would rather that the system keeps packages locked down and granting access should be opt-in by default.
Flatpak Flathub already face serious issues. I see many packages from unverified developers who made the first package, and they have chosen the manifest, where inclusion|omission|other can impact security, and then other better implementations are not available. Flatpak –user and overrides which defaults to block everything in global is helping, and knowing when not to accept any developer’s greedy permission demands which should have been rejected by a flatpak review group, but isn’t is troublesome.
There are more concerns, … but enough there to be sorted already.
My friend Adrian has been talking about the idea of a WSL for Linux development, as supply chain attacks become more and more common. However such system would still be in addition to this, as you still want a way to distribute tools and binaries and have a somewhat defined environment.
> Distribution packaging funnels like debian packages help to block bad players.
This has been talked about at length. Do you expect the debian developers to review all the code changes in each Firefox update, cause its not happening. Everything is automated and updated without a second though, and nowdays distributions are basically a middle man.
> Flatpak Flathub already face serious issues. I see many packages from unverified developers who made the first package, and they have chosen the manifest,
Permissions are not granted unless they can be justified and there is no other way of achieving it. They are not there cause the reviewers and developer are evil, but cause its crucial to the functionality of the application. You can override whatever you don’t like.
> Everything is automated and updated without a second though, and nowdays distributions are basically a middle man.
Some people may want this middleman, with shared interests in mind, to be there.
As you mentioned Firefox, there have been cases of Debian people catching privacy-unfriendly Firefox defaults and adjusting them before shipping them to users.
Flatpak won’t protect me against the developer of an installed app with already-granted permissions starting to do something shady with a new update, as, with high probability, I won’t review the app’s code. A Debian maintainer may.
Though, having extra security layers seems nice, just without giving up the layer of a third party keeping an eye on the code before it gets to me.
> Currently the safe way not mentioned here, is to keep your host minimal, and run deviations in a guest-kernel […]
> My friend Adrian has been talking about the idea of a WSL for Linux development, as supply chain attacks become more and more common. However such system would still be in addition to this, as you still want a way to distribute tools and binaries and have a somewhat defined environment.
Cannot resist to mention the project “Bubbles” I’ve been working on for a while, which basically implements that. Prioritizing isolation, but also intending to provide a better UX and reproducibility than e.g. distroboxes. For now, it leans on Nix and apt for tooling availability.
Not perfect yet (EGL is WIP, Debian base image only, …), but I think general UX is fine already. Maybe efforts could be synced here. Published to Flathub last week: https://flathub.org/en/apps/de.gonicus.bubbles
no need for WSL for Linux, linux ecosystem already have guest-kernels, qemu, incus, …etc where it exists already, all this existed long before WSL and such came to exist.
some times it feels like people, and maybe new people joining the open source communities, try to reinvent the wheel to take control, which can feel a bit discerning, like they are playing for footprint, instead of integrating, they like to extend and embrace, which we have seen before somewhere.
innovation and evolution is healthy, when it is implemented with care, and not some quick hallucination AI inference or 160 character one-liner inspired move just trying to make a new name.
not saying that’s what’s going on here, just wondering if these things are taking into consideration.
I understand this is only at the ideation stage at this time, and whether there is a following is yet to be seen, but there is a lot of work in this area taking place, and gnome folks, yes I am one too, should be careful about the influence a change makes, and should take time to consider things carefully, and also try to leave things open for integration, rather than crowding the plate.
The selling point of WSL is not the technology, all that has existed yes. Its the integration and the seameless-ness part about it.
On the linux we have the kernel, qemu, containers, but nobody has made an App that will give you a WSL like environment, completely integrated, ready to use for development.
We have general virtual machine apps, but this is a much much different usecase.
flatpak|flathub next have potential if done carefully. it is important work.
There is a real gap for users of KDE Linux/GNOME OS that I think Toolpak can address. Currently I think the best answer for such users is actually Nix, this provides arbitrary packages that you can run outside of a container. But it’s not the most friendly UI to install and use so I don’t feel great recommending it to people.
Ptyxis works as a Flatpak because it runs partly outside the container. I think this is actually a good example of an app that could just be run as a privileged toolpak instead. It demonstrates the need for way to distribute apps like this without providing much security from containerization.
On the flip side there are is a whole world of servers that you can run in podman etc with very limited permissions. They don’t belong in Flatpak but maybe it would be nice if there was an app store and UI for finding and running them. Thinking of things like Soft Serve (a git host) or even complex servers like Immich. It is something to think about, if only to limit the scope of Toolpak.
I expect apps with some CLI-feature to be packed as a toolpack, instead of a flatpack, just to make it easer to run it from the commandline. How about the developer having to install Docker Desktop in their enterprise setting and expecting a working CLI to come with it. Or a user with Scribus (desktop publishing) but using a community commandline script that calls it from the command line?
I appreciate the work on the issue of command line apps, but worry about these hybrid cases.
I might be biased but what is wrong with the bootc approach and composefs? i know a discussion was had your Main critism you want dmverify not fsverify so you wanted image based security guarantees. but as you build more and more of the surrounding systems around this image paradigm, the case for superiority of approach gets thinner and the duplication and bandwidth story gets unconvincing for user.
fsverity requires parsing the filesystem before being able to authenticate the image, where with dmverity it happens on the blocks. There are very specific cases where fsverity would be preferable over dmverity (at the cost of weaker guarantees) and this is not one of them.