In case you missed it, I was on the Lunduke Hour last week talking about Builder. In reality it turned into a discussion about everything from why Gtk 4, efficient text editor design, creating UI designers, Flatpak, security implications of the base OS, and more.
Now that 3.24 is almost here, I want to take a little downtime and write some information about pieces of the design underneath Builder. Today, let’s lift the hood and take a look at Runtimes.
In Builder, an IdeRuntime is an abstraction around an environment to run programs. That could be build tooling, SDK management, your application, unit tests, and more. Unsurprisingly, it’s naming is similar to Flatpak runtimes because they have a very 1:1 relationship in the Flatpak plugin. But it’s not an uncommon term by any means.
Today, Builder supports 3 types of runtimes: host, jhbuild, and flatpak.
The host runtime simply executes processes on the host system. This is as close to if you just opened a shell with
gnome-terminal as we can give you. We actually go through great pains to do so when the application is distributed via Flatpak. For example, we use the HostCommand² developer-mode interface of Flatpak to execute the subprocess from the host session¹ yet using a PTY master/slave from within the sandbox.
The jhbuild runtime is similar to the host runtime except that we first enter the jhbuild environment using
jhbuild run. This ensures that we are using dependencies that you may have built locally. Psuedo terminals are also run this way, so
ctrl+alt+shift+t will naturally give you the equivalent of
jhbuild shell inside a new embedded terminal.
The flatpak runtime uses
flatpak build to enter the runtime environment. Our build pipeline will have had to advance to the point at least the
IDE_BUILD_PHASE_PREPARE phase to ensure that the manifest and build directories are setup. So there is just a bit extra complexity involved here. But the result is pretty nice. Your subprocess will see
/app and such just like during the application build process. This makes exploring the build environment very simple. Just open a new terminal with
Running your application naturally uses runtimes too. This allows us to setup the environment, DBus, network namespaces, Wayland access, and more in the way your application needs. We use information from the build configuration (such as your org.gnome.Foo.json flatpak manifest) to determine what that is.
So what’s next?
I’m relatively happy with this design so far and I think it could work for a few more scenarios. One might be a runtime accessible via
adb to your phone. Or possibly a remote container. Or far more interesting to me, inside a GNOME virtual machine providing a simulator with GNOME continuous.
Until next time…
- Obviously this means Builder has a more relaxed sandboxing story than a general purpose application. This comes with the territory of developer tooling of this magnitude.
- If you want to see this feature land in MonoDevelop/Xamarin Studio, go help fix https://github.com/mono/dbus-sharp/issues/57
One thing we are striving for in 3.24 is to make it as simple as possible for newcomers to get their development environment setup. Hopefully in time so that our next round of Outreachy and GSoC interns have an easier time getting started.
A common installation issue we’ve seen is that people have flatpak, but not flatpak-builder. Without it, Builder can’t do builds inside of the target mount namespace with all your proper dependencies. So now Builder will detect this and install it for you if you like.
It’s a rainy afternoon in Portland so I’m cozy with an espresso watching the rain. After a short hacking session you can now export your application as a Flatpak bundle quickly and easily. Just select the workbench menu in the top right corner of the workbench, followed by Flatpak, and then Export as Bundle.
A normal build will proceed but also includes a finalization step to populate the OSTree repository (
flatpak build-export) and then create the bundle (
flatpak build-bundle). Finally, Files will be opened with the bundle selected for your copy-and-paste ease. If you double-click the bundle, GNOME Software will be opened and allow you to quickly install the bundle.
Additionally, we’ve split off the update dependencies phase of the build pipeline for Flatpak-based configurations so you can work offline while still updating dependencies at your convenience. Just activate the menu item and the build pipeline will progress to download updates if the network is enabled.
In Builder, we landed a new feature for 3.24 that allows you to create a new terminal inside the application runtime. If you’re building against your host system, then this is nothing special. If you’re building against jhbuild you’ll get a shell inside of that (but again, nothing really special).
However, if you’re building for a flatpak runtime, such as
org.gnome.Platform, your shell will look completely different from your host system. Instead, it will be what the application sees at runtime. (Okay, it’s actually what it sees at build time, but the SDK is just the Platform with compiler bits, headers, pkg-config, and such).
ctrl+shift+tfor a terminal on your host
ctrl+alt+shift+tfor a terminal inside your application runtime
So here is an image of a shell in the host system on left and the runtime on the right. This makes it very convenient to figure out what your application is seeing during execution and audit the files you need to remove in your cleanup phase. Or maybe you just want to explore!
If you are not using Builder, you can explore a SDK or runtime with
flatpak run org.gnome.Sdk. Additionally, if you’re interested in exactly what your application sees after you have packaged and installed it, try:
flatpak run --command=bash com.example.MyApp
-d to the above will give you the SDK instead of the runtime. This is much closer to what Builder will give you.
flatpak run -d --command=bash com.example.MyApp
A decade ago I lived on the west side of Los Angeles. One of my favorite conferences was Southern California Linux Expo. Much like Karen, this is the conference where I performed my first technical talk. It’s also where I met and became friends with great people like Jono, Ted, Jeff, the fantastic organizing staff, and so many more.
I was happy to come back again this year and talk about what I’ve been working on in GNOME. The combination of Flatpak and Builder (and Sysprof).
The event was live streamed on youtube, and you can watch it here. I expect the rooms to be cut and uploaded as individual talks but I don’t know the timeline for that. I’ll update this if/when I discover that so you can
youtube-dl if you prefer.
We are really starting to be able to do cool things with Builder these days. Just install our Builder Nightly and you can build plenty of GNOME apps.
I should note that Builder doesn’t yet have the ability to install your application but I intend for this to land before 3.24.
Here is a screenshot sequence of me using Builder installed from flatpak to build and test GNOME ToDo, also in flatpak.
One of the great powers of namespace APIs on Linux (mount namespaces, user namespaces, etc) is that you can create a new view into the world of your computer that is very different from the host. This can make traditional profiling tools difficult.
To begin with, we need to ensure that we have access to
perf infrastructure. Easy enough, just don’t drop those privileges before calling
execve(). This is the
--allow=devel option to
flatpak run. But after that, we need to do the detailed phase of translating instruction pointers to a function name.
Generally, the translation between an instruction pointer and a function name requires looking up what file is mapped over the address. Then you open that file with an ELF reader and locate the file containing debug information (which may be the same file). Then open that file and locate what function contains that instruction pointer. (We subtract the beginning of the map from the instruction pointer to get a relative offset).
Now, here is the trouble with mount namespaces. The “path” of the map might be something like “
/newroot/usr/lib/libgtk-3.so“. Of course, “
/newroot/” doesn’t actually exist.
So… what to do.
Well, we can find information about the mounts in the process by looking at
/proc/$pid/mountinfo. Just look for the longest common prefix to get the translation from the mount namespace into the host. One caveat though. The translated path is relative to the root of the partition mounted. So the next step is to locate the file-system mount on the host (via
If you put this all together, you can profile and extract symbols from applications running inside of containers.
Now, most applications aren’t going to ship debug symbols with them. But we’ll need those for proper symbol extraction. So if you install the
.Debug variant of a Flatpak runtime (
org.gnome.Sdk.Debug for example) then we can setup symbol resolution specifically for
/usr/lib/debug. Builder now does this.
I’m not sure, but I think this makes Builder and Sysprof one of the first (if not the first) profiler on Linux to support symbol extraction while profiling containerized applications from outside the container.
I’ve only tested with Flatpak, but I don’t see why this code can’t work with other tooling using mount namespaces.
I wanted a bit of a distraction this evening, so I put together a quick valgrind plugin. Just select “Run with Valgrind” from the drop down menu and it will make sure your program is run using the venerable valgrind.
If valgrind isn’t available in your runtime (host, jhbuild, flatpak, etc) you wont see the menu item. But thankfully the org.gnome.Sdk runtime we use to build GNOME apps has it, so for many of you it should Just Work™.
One of the core features of Builder is, well, building. So we need to provide the best experience we can. That means we need wide support for build systems and languages. How we get from source code to a product can vary in a multitude of ways.
For example, you might need to alter how you launch programs to run inside a container. You might need access to headers that only exist in the SDK and not the application runtime. You might even need to download or build dependencies for your project at a given point. Furthermore, this could all change based on what plugins you’ve chosen to enable in your Builder environment.
When I set out to create Builder, I knew that I wouldn’t be able to see all the possibilities that Builder needs to handle. So I explicitly avoided designing an abstract solution until I understood the problem better. But I think we understand the problem well enough now. So over the winter holiday I started putting together the design of IdeBuildPipeline.
The pipeline is a one-directional series of build phases. Each phase can have stages attached to it. Builds progress in the order of their attached stages, each of which can be an asynchronous operation. We only need to advance the build up to the requested phase. If the IDE detects something has changed, it can invalidate various phases of the pipeline causing them to be executed again at the next build request.
Plugins should implement the IdeBuildPipelineAddin interface to register any necessary pipeline stages.
Generally, build stages are an IdeSubprocessLauncher, but you can implement custom IdeBuildStage subclasses to do more fancy things. For example, if you need to download something, you might add an IdeBuildStageTransfer and you’ll see the transfer show up in the transfers window.
Of course, some problems are complex enough where we don’t know if they are invalidated upfront and the build stage may query some external resource. For that, the IdeBuildStage class can implement the query signal. Of course we should strive to perform blocking operations asynchronously so consider pausing the stage until your asynchronous request has completed.
We know how difficult it is for newcomers to get started so we’ve taught Builder how to download Flatpak runtimes and SDKs based on your project configuration. That means that you can omit those flatpak install commands from your “project setup” guide. Just clone the project URL and click Build, we’ll take care of the rest.
Thanks to this new design, implementing a build system is much easier. To prove this, I finally implemented a basic cmake+ninja plugin in a matter of an hour. Check it out here. If you’d like to write CFLAG extraction for cmake, we could use that still!