Sysprof Developments

Earlier this month, Matthias and I teamed up to push through some of our profiling tooling for GTK and GNOME. We took the occasional work I had done on Sysprof over the past few years and integrated that into the GTK-4.x tree.

Sysprof uses a binary log file to store information about execution in a manner that is easy to write-buffer and read-back using positioned reads. It helps keep the sampling overhead of sysprof low. But it’s too detail oriented for each application supporting the format to write. To make this stuff reusable I created a libsysprof-capture-3.a static library we embed from various layers of the platform.

GTK-4.x is now using this. Builder itself uses it to log internal statistics, tracing data, and counters for troubleshooting. I’ve also put forward patches for GJS to integrate with it. Georges revamped and pushed forward a prototype by Jonas to integrate with Mutter/Shell and get us frame timings and Cogl pipeline data. With some work we can finish off the i915 data sources that Eric Anholt did to correlate GPU commands too.

What this means for developers is that soon we’ll be able to capture system information from various layers in the stack and correlate them using similar clocks. We’re only scratching the surface right now, but it’s definitely promising. It’s already useful to quantify the true performance improvements of merge-requests in Mutter and Shell.

To help achieve this goal during the 3.34 cycle, I’ve started the GNOME Profiling Initiative to track integration of various problem spaces. If you’re an application developer, you can use sysprof_capture_writer_new_for_env() to create a new SysprofCaptureWriter if Sysprof is profiling you (otherwise you’ll get NULL back). Use that to write marks, logs, metadata, files, or anything else you would like to capture.

If you’re interested in helping to write more data collectors, that would be appreciated. Data sources like battery voltage/wattage consumption seem like good candidates so that we can better optimize our platform for battery-based devices.

I have a Sysprof Copr repository for Rawhide and F30 if you’d like to try stuff out and submit issues.

Many thanks to Red Hat for sponsoring all the work I do on GNOME and my amazing manager Matthias for visiting Portland to make this stuff happen even sooner.

As always, follow my grumpy ramblings on birdsite for early previews of my work.

Improving the Container Workflow

As I mentioned in my talk at Scale 17x, if you aren’t using containers for building your application yet, it’s likely you will in the not-so-distant future. Moving towards an immutable base OS is a very likely future because the security advantages are so compelling. With that, comes a need for a mutable playground, and containers generally fit the bill. This is something I saw coming when I started making Builder so we have had a number of container abstractions for some time.

With growing distributions, like Fedora’s freshly released Silverblue, it becomes even more necessary to push those container boundaries soon.

This week I started playing with some new ideas for a terminal workspace in Builder. The goal is to be a bit of a swiss-army knife for container oriented development. I think there is a lot we can offer by building on the rest of Builder, even if you’re primary programming workhorse is a terminal and the IDE experience is not for you. Plumbing is plumbing is plumbing.

I’m interested in getting feedback on how developers are using containers to do their development. If that is something that you’re interested in sharing with me, send me an email (details here) that is as concise as possible. It will help me find the common denominators for good abstractions.

What I find neat, is that using the abstractions in Builder, you can make a container-focused terminal in a couple of hours of tinkering.

I also started looking into Vagrant integration and now the basics work. But we’ll have to introduce a few hooks into the build pipeline based on how projects want to be compiled. In most cases, it seems the VMs are used to push the app (and less about compiling) with dynamic languages, but I have no idea how pervasive that is. I’m curious how projects that are compiling in the VM/container deal with synchronizing code.

Another refactor that needs to be done is to give plugins insight into whether or not they can pass file-descriptors to the target. Passing a FD over SSH is not possible (although in some cases can be emulated) so plugins like gdb will have to come up with alternate mechanisms in that scenario.

I will say that trying out vagrant has been a fairly disappointing experience compared to our Flatpak workflow in Builder. The number of ways it can break is underwhelming and the performance reminds me of my days working on virtualization technology. It makes me feel even more confident in the Flatpak architecture for desktop developer tooling.

Translucent Completion

Sometimes completion windows get in the way of reading the surrounding code. With Builder Nightly, you can press and release Left Control to toggle translucency of the completion window.

Flatpaking Terminals

One thing Builder has done for a long time is make terminals work seamlessly even if distributed using container technologies. Because pseudo-terminals are steeped in esoteric UNIX history, it can be non-obvious how to make this work.

I’m in a place to help you not have to deal with that pain because I’ve already gone through it. So I created some utility code and a demo application that can be packaged with Flatpak. If it detects it’s running under Flatpak it will use a few techniques to get a user-preferred shell executed on the host with a PTY controlled by application.

Check out the code.

Edit: The flatterm repository has been updated to use the brand new VTE_PTY_NO_CTTY flag that was added in response to this blog post. Users of Vte from git (what will be 0.58) get to enjoy writing even less code.

Builder 3.33.1

Our first 3.33 release has landed as we move towards 3.34. There is a lot to do this cycle in case you’re interested in contributing. The best way to get started is to dive into the code. We can help you with that on IRC.

Lots of this release is code behind the scenes, so screenshots won’t do them justice. But there are some visible goodies too.

We got a D-Bus Inspector inspired by D-feet. The long term goal is to merge that new code into D-feet itself.

Occasionally we got issues filed about not being able to half-tile Builder. The smaller your screen size, the more unrealistic of an expectation that is, but we can certainly do a little better. The new search box takes up less space and animates in as necessary.

We gained some initial Podman support so long as your system has support for podman exec --preserve-fds. You can even debug inside the containers if they have gdb in the container.

Some users were surprised by how Builder auto-downloaded SDKs and Runtimes to build projects. So we’ve added a confirmation dialog to that process so the user can make informed decisions.

I’ve also moved Git integration out of process. Both the Git and Flatpak plugins previously used git in process which had a number of drawbacks. We created some new API and the gnome-builder-git daemon which provides access to common git operations using libgit2-glib and D-Bus serialization binary format over stdin/stdout. We have push, stage, and signed commit support, but that currently lacks UI in this release.

GtkSourceView moved to Meson

The master branch of GtkSourceView (what will become 4.4) has moved to meson for development. I branched gtksourceview-4-2 for patch releases of 4.2.x which will remain autotools. Today’s release of gtksourceview-4.3.1 contains both autotools and meson. However 4.3.2 will remove autotools entirely.

I also landed some code to speed up line number drawing which was a non-trivial amount of the render cost while kinetic scrolling.

Podman Support in Builder

For years now, Builder has had rich abstractions for containers built right into the core of the IDE. Builder even knows the difference between build and runtime containers which naturally maps with Flatpak SDKs like org.gnome.Sdk vs org.gnome.Platform.

With the advent of operating systems focused on immutability, like Fedora Silverblue, developers are going to be increasingly developing in containers.

The technology underlying projects like Toolbox is podman. It provides a command-line tool to manage containers without a daemon by using the various container APIs afforded to us in modern Linux.

Bridging Builder’s container APIs to support podman was pretty painless on my part. A couple hours to choose the right abstractions and implementing them led me to a missing piece in podman; passing FDs to the container.

The reason that Builder requires this is that we often need to communicate across containers. An easy way to do that is over a pair of pipe() since it is anonymous. By anonymous, I mean we don’t need to share any file-system hierarchy, IPC or network namespaces, or even PTY namespace.

The most important piece that requires this in Builder is our GDB-based debugger. We use GDB inside the container so it has native access to things like build sources, libraries, symbols, and more. This is all orchestrated using GDB’s mi2 interface over a PTY, with a second PTY for the target process. When GDB lands on a breakpoint, we know how to translate paths between Builder’s container (usually Flatpak) and the target container (in this case, podman). Doing so ensures that we open the right file and line number to the user. Fundamentals, of course.

So a couple weeks later and podman exec has gained the --preserve-fds=N option, available on Rawhide and Fedora 30 (currently in updates-testing). If you have all the necessary bits in place, Builder will allow you to select a podman container from Build Preferences, and you’re off to the races.

Furthermore, you can even seamlessly get a terminal in the build environment with Control+Alt+Shift+T which can prove useful if you have to install dependencies.

Since we don’t know much about the container, we don’t have the ability to install dependencies on your behalf. But if someone were to work on Dockerfile support, I don’t see that as an intractable problem.

Here is a quick test command-line program debugging in Builder using the GDB backend to prove it all works.

Designing for Sandboxes

One of the things I talked about in my talk at Scale 17x is that there are a number of platform features coming that are enevitable.

One of those is application sandboxing.

But not every component within an application is created equal or deserves equal access to user data and system configuration. Building the next big application is increasingly requiring thinking about how you segment applications into security domains.

Given the constraints of our current operating systems, that generally means processes. Google’s Chrome was one of the first major applications to do this. The Chrome team had created a series of processes focused on different features. Each of those processes had capabilities removed (such as network, or GPU access) from the process space to reduce the damage of an attack.

Recently Google released sandboxed-api, which is an interesting idea around automatically sandboxing libraries on Linux. While interesting, limiting myself to designs that are Linux only is not currently realistic for my projects.

Since I happen to work on an IDE, one of the technologies I’ve had to become familiar with is Microsoft’s Language Server Protocol. It’s a design for worker processes to provide language-specific features.

It usually works like this:

  • Spawn a worker process, with a set of pipe()s for stdin/stdout you control
  • Use JSONRPC over the pipe()s with some well-formatted JSON commands

This design can be good for sandboxing because it allows you to spawn subprocesses that have reduced system capabilities, easily clean up after them, and provides an IPC format. Despite having written jsonrpc-glib and a number of helpers to make writing JSON from C rather clean, I’m still unhappy with it for a number of reasons. Those reasons range from everything from performance to correctness to brittleness of nonconforming implementations.

I’d like to use this design in more than just Builder but those applications are more demanding. They require passing FDs across the process boundary. (Also I’m sick of hand writing JSON RPCs and I don’t want to do that anymore).

Thankfully, we’ve had this great RPC system for years that fits the bill if you reuse the serialization format: DBus.

  • No ties to a DBus daemon
  • GDBus in GLib has a full implementation that plays well with async/sync code
  • gdbus-codegen can generate our RPC stubs and proxies
  • Well defined interfaces in XML files
  • Generated code does type enforcement to ensure contracts
  • We can easily pass FDs across the process boundary, useful for memfd/tmpfs/shm

To setup the sandboxes, we can use tools like flatpak-spawn or bwrap on Linux to restrict process capabilities before launching the target process. Stdin/stdout is left untouched so that we can communicate with the subprocess even after capabilities are dropped.

Before I (re)settled on DBus, I tried a number of other prototypes. That included writing an interface language/codegen for JSONRPC, using libvarlink, Thrift’s c_glib compiler and protobufs. I’m actually surprised I was happiest with the DBus implementation, but that’s how it goes sometimes.

While I don’t expect a lot of sandboxing around our Git support in Builder, I did use it as an opportunity to prototype what this multi-process design looks like. If you’re interested in checking it out, you can find the worker sources here.

What excites me about the future is how this type of design could be used to sandbox image loaders like GdkPixbuf. One could quite trivially have an RPC that passes a sealed memfd for compressed image contents and returns a memfd for the decoded framebuffer or pre-compressed GPU textures. Keep that process around a little while to avoid fork()/exec() overhead, and we gain a bit of robustness with very little performance drawbacks.