Category: Flatpak

System Extensions from Flatpak

I write about Sysprof here quite often. Mostly in hopes of encouraging readers to use it to improve Linux as a whole.

An impediment to that is the intrusiveness to test out new features as they are developed. If only we had a Flatpak which you could install to test things right away.

One major hurdle is how much information a profiler needs to be useful. The first obvious “impossible to sandbox” API you run into is the perf subsystem. It provides information about all processes running on the system and their memory mappings which would make snooping on other processes trivial. Both perf and ptrace are disabled in the Flatpak sandbox.

After that, you still need unredacted access to the kernel symbols and their address mappings (kallsyms). You also need to be in a PID namespaces that allows you to see all the processes running on the system and their associated memory mappings which essentially means CAP_SYS_ADMIN.

Portable Services

Years ago, portable services were introduced into systemd through portablectl. I had high-hopes for this because it meant that I could perhaps ship a squashfs and inject it as a transient service on the host.

However, Sysprof needs more integration than could be provided by this because portable services are still rather isolated from the host. We need to own a D-Bus name, policy-kit action integration, in addition to the systemd service.

Even if that were all possible with portable services it wouldn’t get us access to some of the host information we need to properly decode symbols.

System Extensions

Then came along systemd-sysext. It provides a way to “layer” extensions on top of the host system’s /usr installation rather than in an isolated mount namespace.

This sounds much more promising because it would allow us to install .policy for policy-kit, .service files for Systemd and D-Bus, or even udev rules.

Though, with great power comes excruciating pain, or something like that.

So if you need to provide binaries that run on the host you need to either static-link (rust, go, zig perhaps?) or use something you can reasonably expect to be there (python?).

In the Sysprof case, everything is C so it can statically link almost everything by being clever with how it builds against glibc. Though this still requires glibc and quite frankly I’m fine with that. Potentially, one could use MUSL or ucLibc if they had high enough pain threshold for build tooling.

Bridging Flatpak and System Extensions

The next step would be to find a way to bridge system extensions and Flatpak.

In the wip/chergert/sysext branch of Sysprof I’ve made it build a number of things statically so that I can provide a system extension directory tree at /app/lib/extensions. We can of course choose a different path for this but that seemed similar to /var/lib/extensions.

Here we see the directory tree laid out. To do this right for systemd-sysext we also need to install an extension point file but I’ll save that for another day.

The Directory Tree

$ find /app/lib/extensions -type f
/app/lib/extensions/usr/lib/systemd/system/sysprof3.service
/app/lib/extensions/usr/share/polkit-1/actions/org.gnome.sysprof3.policy
/app/lib/extensions/usr/share/dbus-1/system-services/org.gnome.Sysprof3.service
/app/lib/extensions/usr/share/dbus-1/system.d/org.gnome.Sysprof3.conf
/app/lib/extensions/usr/libexec/sysprofd

Registering the Service

First we need to symlink our system extension into the appropriate place for systemd-sysext to pick it up. Typically /var/lib/extensions is used for transient services so if this were being automated we might use another directory for this.

# mkdir -p /var/lib/extensions
# ln -s /var/lib/flatpak/org.gnome.Sysprof.Devel/current/active/files/lib/extensions/ org.gnome.Sysprof.Devel

Now we need to merge the extension so it overlays into /usr. We must use --force because we didn’t yet provide an appropriate extension point file for systemd.

# systemd-sysext merge --force
Using extensions 'org.gnome.Sysprof.Devel'.
Merged extensions into '/usr'.

And now make sure our service was installed to the approriate location.

# ls /usr/lib/systemd/systemd/sysprof3.service
-rw-r--r-- 2 root root 115 Dec 31 1969 /usr/lib/systemd/system/sysprof3.service

Next we need to reload the systemd daemon, but newer versions of systemd do this automatically.

# systemctl daemon-reload

Here is where things are a bit tricky because they are somewhat specific to the system. I think we should make this better in the appropriate upstream projects to avoid this altogether but also easily handled with a flatpak installation trigger.

First make sure that policy-kit reloads our installed policy file.

# systemctl restart polkit.service

With dbus-broker, we also need to reload configuration to pick up our new service file. I’m not sure if dbus-daemon would require this, I haven’t tested that. Though I wouldn’t be surprised if this is related to inotify file-monitors and introducing a merged /usr.

# gdbus call -y -d org.freedesktop.DBus \
-o /org/freedesktop/DBus \
-m org.freedesktop.DBus.ReloadConfig

At this point, the service should be both systemd and D-Bus activatable. We can verify that with another gdbus call quick.

# gdbus call -y -d org.gnome.Sysprof3 \
-o /org/gnome/Sysprof3 \
-m org.freedesktop.DBus.Peer.Ping
()

Now I can run the Flatpak as normal and it should be able to use the system extension to get profiling and system data from the host as if it were package installed.

$ flatpak run org.gnome.Sysprof.Devel

The following screenshots come from GNOME OS using yesterdays build with what I’ve described in this post. However, it also works on Fedora Rawhide (and probably Fedora 40) if you boot with selinux=0. More on that in the FAQ below.

Flatpak Integration

So obviously nobody would want to do all the work above just to make their Flatpak work. The user-facing goal here would be for the appropriate triggers to be provided by Flatpak to handle this automatically.

Making this happen in an automated fashion from flatpak installation triggers on the --system installation does not seem terribly out-of-scope. It’s possible that we might want to do it from within the flatpak binary itself but I don’t think that is necessary yet.

FAQ

What about non-system installations?

It would be expected that system extensions require installing to a system installation.

It does not make sense to allow for a --user installation, controllable by an unprivileged user or application, to be merged onto the host.

Does SELinux affect this?

In fact it does.

While all of this works out-of-the-box on GNOME OS, systems like Fedora will need work to ensure their SELinux policy to not prevent system extentions from functioning. Of course you can boot with selinux=0 but that is not viable/advised on end-user installations.

In the Sysprof case, AVC denials would occur when trying to exec /usr/libexec/sysprofd.

Does /usr become read-only?

If you have systemd <= 255 then system-sysext will most definitely leave /usr read-only. This is problematic if you want to modify your system after merging but makes sense because sysext was designed for immutable systems.

For example, say you wanted to sudo dnf install a-package on Fedora. That would fail because /usr becomes read-only after systemd-sysext merge.

In systemd >= 256 there is effort underway to make /usr writable by redirecting writes to the top-most writable layer. Though my early testing of Fedora Rawhide with systemd 256~rc1 still shows this is not yet working.

So why not a Portal?

One could write a portal for profilers alone but that portal would essentially be sysprofd and likely to be extremely application specific.

Can I use this for udev rules?

You could.

Though you might be better served by using the new Device and/or USB portals which will both save you code and systems integration hassle.

Can I have different binaries per OS?

Yes.

The systemd-sysext subsystem has a directory layout which allows for matching on some specific information in /etc/os-release. You could, for example, have a different system extension for specific Debian or CentOS Stream versions.

Can they be used at boot?

If we choose to symlink into a persistent systemd-sysext location (perhaps /etc/extensions) then they would be available at boot.

Can services run independent of user app?

Yes.

It would be possible to have a system service that could run independently of the user facing application.

Builder GTK 4 Porting, Part VII

It’s been another couple weeks of porting, along with various distractions.

The big work this time around has been deep surgery to Builder’s “Foundry”. This is the sub-system that is responsible for build-systems, pipelines, external-devices, SDKs, toolchains, deployments-strategies and more. The sub-system was starting to show it’s age as it was one of the first bits of Builder to organically emerge.

One of the things that become so difficult over the years is dealing with all the container layers we have to poke holes through. Running a command is never just running a command. We have to setup PTYs (and make sure the TTY setup ioctl()s happen in the right place), pass environment variables (but to only the right descendant process), and generally a lot more headaches.

What kicked off this work was my desire to remove a bunch of poorly abstracted bits and we’re almost there. What has helped considerably is creating a couple new objects to help manage the process.

The first is an IdeRunContext. It is sort of like a GSubprocessLauncher but allows you to create layers. At the end you can convert those layers into a subprocess launcher but only after each layer is allowed to rewrite the state as you pop back to the root. In practice this has been working quite well. I finally have control without crazy amounts of argument rewriting and guesswork.

To make that possible, I’ve introduced an IdeUnixFDMap which allows to manage source↔dest FD translations for FDs that will end up in the subprocess. It has a lot of helpers around it to make it fit well into the IdeRunContext world.

All of this has allowed the new IdeRunCommand to really shine. We have various run command providers (e.g. plugins) all of which can seamlessly be used across the sub-systems supporting IdeRunContext. Plugins such as meson can even export unit tests as run commands.

The shellcmd plugin has also been rewritten upon these foundations. You can create custom commands and map them to keyboard shortcuts. The commands, like previous version of Builder, can run in various localities. A subprocess, from the build pipeline, as an app runner, or on the host. What has improved, however, is that they can also be used in surrogate of your projects run command. These two features combined means you can make Builder work for a lot of scenarios it never did before by configuring a few commands.

There aren’t a lot of screenshots for things like this, because ideally it doesn’t look too different. But under the hood it’s faster, more reliable, and far more extensible than it was previously. Hopefully that helps us cover a number of highly requested use-cases.

a screenshot of the debugger

a screenshot of the build menu with debug selected

a screenshot of the run command selection selection dialog

a screenshot showing the location of the select run command menu item

a screenshot editing a command

Builder GTK 4 Porting, Part II

Another week of work towards porting Builder to GTK 4. Since I can’t add to TWIG from IRC, I’ll try harder to drop some occasional updates here.

GtkSourceView

  • Merged fixes for highlighting unicode literals for C
  • Improved parsing of language values in snippet bundles
  • GtkSourceGutter will not correctly mark prelit and selection quarks within GtkSourceGutterLines.
  • Fixed a bunch of little mouse pointer annoyances when using GtkSourceHover interactive tooltips.
  • GtkSourceGutterRenderers can now opt-out of signal emission for GtkSourceGutterRenderer::query_data(). Signal emission with GObject is rather slow, so avoiding it on every line can be helpful. Just set the virtual method pointer to NULL. The signal was only ever added to make porting easier anyway.

libpanel

  • Merged fixes to be used as a subproject with static libraries only
  • CSS styling matches recent changes in libadwaita, particularly around making navigation tabs and panel frame headers more unified.
  • PanelWidget no longer uses a GtkBinLayout so that it’s easier for subclasses to integrate with popovers from size_allocate() to call gtk_popover_present().

Builder

  • Georges did a live coding stream where they ported a bunch of the “buildui” plugin. That is merged to the GTK 4 port now. It brings a number of features back to the UI including the build terminal, run terminal, build panel (with pipeline stages, warnings, and errors), and project information in the omnibar popover.
  • Günther did a bunch of work porting our old snippet files to the new XML-based snippet bundle format upstream in GtkSourceView. Along with that came porting of the snippets plugin for Builder’s new GTK 4-based editor.
  • Workspace windows have a bit better predictability when restoring sizes.
  • The project creation workflow was ported, albeit needs to have the redesign implemented still.
  • Lots of resiliency fixes for clang and symbol-tree plugins to improve life-cycle management.
  • The Valgrind plugin was ported to C. This was also back-ported to main because it fixed a number of oddities (crashes) occurring in PyGObject.
  • Builder’s “omni-gutter” GtkSourceGutterRenderer was ported to GTK 4 and got a lot of cleanups along the way. I believe there is still some outstanding things to fix such as handling rendering for symbolic icons as I’m pretty sure that’s not correct yet.
  • The “debuggerui” plugin has been ported to GTK 4 and appears to be working well now. This plugin is responsible for bridging the internal IdeDebugger interfaces to the UI interface.
  • Style schemes were updated for GtkSourceView 5
  • Tabs are now the default navigation interface for pages. There is likely still a lot to do around how we want empty frames to look and behave though.
  • The project-tree is now most ported, and with some workarounds to allow making GtkPopover work you can actually display popovers and activate menu items.
  • A long-standing plugin ordering issue has been fixed so that gtk/menus.ui embedded in plugin resources have menu-merging in the proper order.
  • Some incremental work landed to support per-project configuration of languages.
  • The “ls” plugin (directory views) supports “humanized” timestamps again and columns are resizable once more.
  • You can place panels in the right-side-bar now.

Upcoming

There are a bunch of foundational things to still get landed before I feel I can get Builder flipped over to our Nightly builds. In particular we need to land support for things like:

  • Keyboard shortcuts using GtkShortcutController. This was all done with libdazzle previously.
  • Allow plugins to define custom keyboard shortcuts and merge them into the controller.
  • Port “shellcmd” plugin and integrate keyboard shortcuts to apply those commands.
  • Finish rewrite of the search popover. I’m trying to delay this until GtkListView with sections is supported, as it would provide us a much greater path for performance.
  • A lot of our problems would be simpler if we could make GtkActionMuxer use an alternate action muxer parent from another (non-descendant) widget.
  • Configuration editing in the project configuration window. This is a new design so the port is not 1:1.

A screenshot of Builder with the project-tree context menu and debugger on display

A screenshot of Builder with various panels moved around to non-standard locations

A GTK 4 based Text Editor

It started as an application for me to verify the correctness of the GtkSourceView 5 API (which targets GTK 4). After that it helped me implement JIT support for GtkSourceView languages. Once that was done it became my test case while I wrote the GTK 4 macOS backend and revamped the GL renderer.

It is a simple and humble text editor. It does not have all the corner cases you’d expect from a text editor yet. It does not have aspirations to be a programmers text editor.

Now that you know this is very much a technology preview release only, you might be tempted to keep your important data away from it.

What it can do

  • Simple interface designed by the GNOME design team. You can find the mockups in the traditional places
  • Search and Replace
  • Typical GtkSourceView features
  • Quick access to recent documents
  • Multiple windows
  • Automatic discovery of .editorconfig and modelines settings
  • Light and dark mode
  • Automatically save files to drafts, restored in case of crash
  • Printing
  • Can be run from within a Flatpak sandbox and uses document portal for access to host files

What it cannot do

  • It doesn’t protect you from trying to open really large files
  • Support your custom GTK 4 theme
  • Auto-completion or snippets
  • Plugins
  • Custom file encodings
  • Spell check
  • Change style schemes beyond light and dark
  • Translations or Help of any kind

Building

Here is a release tarball.

If you’d like to test it out, one way is to clone the repository from GNOME Builder and click Run. Additionally, you can find a Flatpak in the gnome-nightly Flatpak repository.

Rust and GNOME Builder

I still spend most of my day writing C and I doubt that is going to change any time soon. But that doesn’t mean you should have to!

Builder got a number of late arriving improvements around Rust support, so now would be a good time to go test them before the 40 release is out.

Yesterday I landed a long awaited feature that will find the common Flatpak SDK ancestors. This was needed to resolve the branch name for SDK extensions like org.freedesktop.Sdk.Extension.rust-stable. Your project might use org.gnome.Sdk//3.38 but the branch for rust-stable is 20.08 (coming from org.freedesktop.Sdk//20.08). Terribly annoying, but hey, now it’s fixed.

Furthermore, Builder can use rust-analyzer¹ as bundled by the org.freedesktop.Sdk.Extension.rust-stable SDK so that is one less thing you need to install or manage. It will pick up all the same dependencies as your project because it will run from within your projects build container. It will see everything your build system does, and in the same way.

Just create a new project using the “GNOME Application” template, select “Rust” as the language, and Run.

¹ rust-analyzer provides diagnostics, auto-completion, and more.

Improving the Container Workflow

As I mentioned in my talk at Scale 17x, if you aren’t using containers for building your application yet, it’s likely you will in the not-so-distant future. Moving towards an immutable base OS is a very likely future because the security advantages are so compelling. With that, comes a need for a mutable playground, and containers generally fit the bill. This is something I saw coming when I started making Builder so we have had a number of container abstractions for some time.

With growing distributions, like Fedora’s freshly released Silverblue, it becomes even more necessary to push those container boundaries soon.

This week I started playing with some new ideas for a terminal workspace in Builder. The goal is to be a bit of a swiss-army knife for container oriented development. I think there is a lot we can offer by building on the rest of Builder, even if you’re primary programming workhorse is a terminal and the IDE experience is not for you. Plumbing is plumbing is plumbing.

I’m interested in getting feedback on how developers are using containers to do their development. If that is something that you’re interested in sharing with me, send me an email (details here) that is as concise as possible. It will help me find the common denominators for good abstractions.

What I find neat, is that using the abstractions in Builder, you can make a container-focused terminal in a couple of hours of tinkering.

I also started looking into Vagrant integration and now the basics work. But we’ll have to introduce a few hooks into the build pipeline based on how projects want to be compiled. In most cases, it seems the VMs are used to push the app (and less about compiling) with dynamic languages, but I have no idea how pervasive that is. I’m curious how projects that are compiling in the VM/container deal with synchronizing code.

Another refactor that needs to be done is to give plugins insight into whether or not they can pass file-descriptors to the target. Passing a FD over SSH is not possible (although in some cases can be emulated) so plugins like gdb will have to come up with alternate mechanisms in that scenario.

I will say that trying out vagrant has been a fairly disappointing experience compared to our Flatpak workflow in Builder. The number of ways it can break is underwhelming and the performance reminds me of my days working on virtualization technology. It makes me feel even more confident in the Flatpak architecture for desktop developer tooling.

Flatpaking Terminals

One thing Builder has done for a long time is make terminals work seamlessly even if distributed using container technologies. Because pseudo-terminals are steeped in esoteric UNIX history, it can be non-obvious how to make this work.

I’m in a place to help you not have to deal with that pain because I’ve already gone through it. So I created some utility code and a demo application that can be packaged with Flatpak. If it detects it’s running under Flatpak it will use a few techniques to get a user-preferred shell executed on the host with a PTY controlled by application.

Check out the code.

Edit: The flatterm repository has been updated to use the brand new VTE_PTY_NO_CTTY flag that was added in response to this blog post. Users of Vte from git (what will be 0.58) get to enjoy writing even less code.

Podman Support in Builder

For years now, Builder has had rich abstractions for containers built right into the core of the IDE. Builder even knows the difference between build and runtime containers which naturally maps with Flatpak SDKs like org.gnome.Sdk vs org.gnome.Platform.

With the advent of operating systems focused on immutability, like Fedora Silverblue, developers are going to be increasingly developing in containers.

The technology underlying projects like Toolbox is podman. It provides a command-line tool to manage containers without a daemon by using the various container APIs afforded to us in modern Linux.

Bridging Builder’s container APIs to support podman was pretty painless on my part. A couple hours to choose the right abstractions and implementing them led me to a missing piece in podman; passing FDs to the container.

The reason that Builder requires this is that we often need to communicate across containers. An easy way to do that is over a pair of pipe() since it is anonymous. By anonymous, I mean we don’t need to share any file-system hierarchy, IPC or network namespaces, or even PTY namespace.

The most important piece that requires this in Builder is our GDB-based debugger. We use GDB inside the container so it has native access to things like build sources, libraries, symbols, and more. This is all orchestrated using GDB’s mi2 interface over a PTY, with a second PTY for the target process. When GDB lands on a breakpoint, we know how to translate paths between Builder’s container (usually Flatpak) and the target container (in this case, podman). Doing so ensures that we open the right file and line number to the user. Fundamentals, of course.

So a couple weeks later and podman exec has gained the --preserve-fds=N option, available on Rawhide and Fedora 30 (currently in updates-testing). If you have all the necessary bits in place, Builder will allow you to select a podman container from Build Preferences, and you’re off to the races.

Furthermore, you can even seamlessly get a terminal in the build environment with Control+Alt+Shift+T which can prove useful if you have to install dependencies.

Since we don’t know much about the container, we don’t have the ability to install dependencies on your behalf. But if someone were to work on Dockerfile support, I don’t see that as an intractable problem.

Here is a quick test command-line program debugging in Builder using the GDB backend to prove it all works.

Designing for Sandboxes

One of the things I talked about in my talk at Scale 17x is that there are a number of platform features coming that are enevitable.

One of those is application sandboxing.

But not every component within an application is created equal or deserves equal access to user data and system configuration. Building the next big application is increasingly requiring thinking about how you segment applications into security domains.

Given the constraints of our current operating systems, that generally means processes. Google’s Chrome was one of the first major applications to do this. The Chrome team had created a series of processes focused on different features. Each of those processes had capabilities removed (such as network, or GPU access) from the process space to reduce the damage of an attack.

Recently Google released sandboxed-api, which is an interesting idea around automatically sandboxing libraries on Linux. While interesting, limiting myself to designs that are Linux only is not currently realistic for my projects.

Since I happen to work on an IDE, one of the technologies I’ve had to become familiar with is Microsoft’s Language Server Protocol. It’s a design for worker processes to provide language-specific features.

It usually works like this:

  • Spawn a worker process, with a set of pipe()s for stdin/stdout you control
  • Use JSONRPC over the pipe()s with some well-formatted JSON commands

This design can be good for sandboxing because it allows you to spawn subprocesses that have reduced system capabilities, easily clean up after them, and provides an IPC format. Despite having written jsonrpc-glib and a number of helpers to make writing JSON from C rather clean, I’m still unhappy with it for a number of reasons. Those reasons range from everything from performance to correctness to brittleness of nonconforming implementations.

I’d like to use this design in more than just Builder but those applications are more demanding. They require passing FDs across the process boundary. (Also I’m sick of hand writing JSON RPCs and I don’t want to do that anymore).

Thankfully, we’ve had this great RPC system for years that fits the bill if you reuse the serialization format: DBus.

  • No ties to a DBus daemon
  • GDBus in GLib has a full implementation that plays well with async/sync code
  • gdbus-codegen can generate our RPC stubs and proxies
  • Well defined interfaces in XML files
  • Generated code does type enforcement to ensure contracts
  • We can easily pass FDs across the process boundary, useful for memfd/tmpfs/shm

To setup the sandboxes, we can use tools like flatpak-spawn or bwrap on Linux to restrict process capabilities before launching the target process. Stdin/stdout is left untouched so that we can communicate with the subprocess even after capabilities are dropped.

Before I (re)settled on DBus, I tried a number of other prototypes. That included writing an interface language/codegen for JSONRPC, using libvarlink, Thrift’s c_glib compiler and protobufs. I’m actually surprised I was happiest with the DBus implementation, but that’s how it goes sometimes.

While I don’t expect a lot of sandboxing around our Git support in Builder, I did use it as an opportunity to prototype what this multi-process design looks like. If you’re interested in checking it out, you can find the worker sources here.

What excites me about the future is how this type of design could be used to sandbox image loaders like GdkPixbuf. One could quite trivially have an RPC that passes a sealed memfd for compressed image contents and returns a memfd for the decoded framebuffer or pre-compressed GPU textures. Keep that process around a little while to avoid fork()/exec() overhead, and we gain a bit of robustness with very little performance drawbacks.