Improving the Container Workflow

As I mentioned in my talk at Scale 17x, if you aren’t using containers for building your application yet, it’s likely you will in the not-so-distant future. Moving towards an immutable base OS is a very likely future because the security advantages are so compelling. With that, comes a need for a mutable playground, and containers generally fit the bill. This is something I saw coming when I started making Builder so we have had a number of container abstractions for some time.

With growing distributions, like Fedora’s freshly released Silverblue, it becomes even more necessary to push those container boundaries soon.

This week I started playing with some new ideas for a terminal workspace in Builder. The goal is to be a bit of a swiss-army knife for container oriented development. I think there is a lot we can offer by building on the rest of Builder, even if you’re primary programming workhorse is a terminal and the IDE experience is not for you. Plumbing is plumbing is plumbing.

I’m interested in getting feedback on how developers are using containers to do their development. If that is something that you’re interested in sharing with me, send me an email (details here) that is as concise as possible. It will help me find the common denominators for good abstractions.

What I find neat, is that using the abstractions in Builder, you can make a container-focused terminal in a couple of hours of tinkering.

I also started looking into Vagrant integration and now the basics work. But we’ll have to introduce a few hooks into the build pipeline based on how projects want to be compiled. In most cases, it seems the VMs are used to push the app (and less about compiling) with dynamic languages, but I have no idea how pervasive that is. I’m curious how projects that are compiling in the VM/container deal with synchronizing code.

Another refactor that needs to be done is to give plugins insight into whether or not they can pass file-descriptors to the target. Passing a FD over SSH is not possible (although in some cases can be emulated) so plugins like gdb will have to come up with alternate mechanisms in that scenario.

I will say that trying out vagrant has been a fairly disappointing experience compared to our Flatpak workflow in Builder. The number of ways it can break is underwhelming and the performance reminds me of my days working on virtualization technology. It makes me feel even more confident in the Flatpak architecture for desktop developer tooling.

Flatpaking Terminals

One thing Builder has done for a long time is make terminals work seamlessly even if distributed using container technologies. Because pseudo-terminals are steeped in esoteric UNIX history, it can be non-obvious how to make this work.

I’m in a place to help you not have to deal with that pain because I’ve already gone through it. So I created some utility code and a demo application that can be packaged with Flatpak. If it detects it’s running under Flatpak it will use a few techniques to get a user-preferred shell executed on the host with a PTY controlled by application.

Check out the code.

Edit: The flatterm repository has been updated to use the brand new VTE_PTY_NO_CTTY flag that was added in response to this blog post. Users of Vte from git (what will be 0.58) get to enjoy writing even less code.

Podman Support in Builder

For years now, Builder has had rich abstractions for containers built right into the core of the IDE. Builder even knows the difference between build and runtime containers which naturally maps with Flatpak SDKs like org.gnome.Sdk vs org.gnome.Platform.

With the advent of operating systems focused on immutability, like Fedora Silverblue, developers are going to be increasingly developing in containers.

The technology underlying projects like Toolbox is podman. It provides a command-line tool to manage containers without a daemon by using the various container APIs afforded to us in modern Linux.

Bridging Builder’s container APIs to support podman was pretty painless on my part. A couple hours to choose the right abstractions and implementing them led me to a missing piece in podman; passing FDs to the container.

The reason that Builder requires this is that we often need to communicate across containers. An easy way to do that is over a pair of pipe() since it is anonymous. By anonymous, I mean we don’t need to share any file-system hierarchy, IPC or network namespaces, or even PTY namespace.

The most important piece that requires this in Builder is our GDB-based debugger. We use GDB inside the container so it has native access to things like build sources, libraries, symbols, and more. This is all orchestrated using GDB’s mi2 interface over a PTY, with a second PTY for the target process. When GDB lands on a breakpoint, we know how to translate paths between Builder’s container (usually Flatpak) and the target container (in this case, podman). Doing so ensures that we open the right file and line number to the user. Fundamentals, of course.

So a couple weeks later and podman exec has gained the --preserve-fds=N option, available on Rawhide and Fedora 30 (currently in updates-testing). If you have all the necessary bits in place, Builder will allow you to select a podman container from Build Preferences, and you’re off to the races.

Furthermore, you can even seamlessly get a terminal in the build environment with Control+Alt+Shift+T which can prove useful if you have to install dependencies.

Since we don’t know much about the container, we don’t have the ability to install dependencies on your behalf. But if someone were to work on Dockerfile support, I don’t see that as an intractable problem.

Here is a quick test command-line program debugging in Builder using the GDB backend to prove it all works.

Designing for Sandboxes

One of the things I talked about in my talk at Scale 17x is that there are a number of platform features coming that are enevitable.

One of those is application sandboxing.

But not every component within an application is created equal or deserves equal access to user data and system configuration. Building the next big application is increasingly requiring thinking about how you segment applications into security domains.

Given the constraints of our current operating systems, that generally means processes. Google’s Chrome was one of the first major applications to do this. The Chrome team had created a series of processes focused on different features. Each of those processes had capabilities removed (such as network, or GPU access) from the process space to reduce the damage of an attack.

Recently Google released sandboxed-api, which is an interesting idea around automatically sandboxing libraries on Linux. While interesting, limiting myself to designs that are Linux only is not currently realistic for my projects.

Since I happen to work on an IDE, one of the technologies I’ve had to become familiar with is Microsoft’s Language Server Protocol. It’s a design for worker processes to provide language-specific features.

It usually works like this:

  • Spawn a worker process, with a set of pipe()s for stdin/stdout you control
  • Use JSONRPC over the pipe()s with some well-formatted JSON commands

This design can be good for sandboxing because it allows you to spawn subprocesses that have reduced system capabilities, easily clean up after them, and provides an IPC format. Despite having written jsonrpc-glib and a number of helpers to make writing JSON from C rather clean, I’m still unhappy with it for a number of reasons. Those reasons range from everything from performance to correctness to brittleness of nonconforming implementations.

I’d like to use this design in more than just Builder but those applications are more demanding. They require passing FDs across the process boundary. (Also I’m sick of hand writing JSON RPCs and I don’t want to do that anymore).

Thankfully, we’ve had this great RPC system for years that fits the bill if you reuse the serialization format: DBus.

  • No ties to a DBus daemon
  • GDBus in GLib has a full implementation that plays well with async/sync code
  • gdbus-codegen can generate our RPC stubs and proxies
  • Well defined interfaces in XML files
  • Generated code does type enforcement to ensure contracts
  • We can easily pass FDs across the process boundary, useful for memfd/tmpfs/shm

To setup the sandboxes, we can use tools like flatpak-spawn or bwrap on Linux to restrict process capabilities before launching the target process. Stdin/stdout is left untouched so that we can communicate with the subprocess even after capabilities are dropped.

Before I (re)settled on DBus, I tried a number of other prototypes. That included writing an interface language/codegen for JSONRPC, using libvarlink, Thrift’s c_glib compiler and protobufs. I’m actually surprised I was happiest with the DBus implementation, but that’s how it goes sometimes.

While I don’t expect a lot of sandboxing around our Git support in Builder, I did use it as an opportunity to prototype what this multi-process design looks like. If you’re interested in checking it out, you can find the worker sources here.

What excites me about the future is how this type of design could be used to sandbox image loaders like GdkPixbuf. One could quite trivially have an RPC that passes a sealed memfd for compressed image contents and returns a memfd for the decoded framebuffer or pre-compressed GPU textures. Keep that process around a little while to avoid fork()/exec() overhead, and we gain a bit of robustness with very little performance drawbacks.

Builder 3.32 Sightings

We just landed the largest refactor to Builder since it’s inception. Somewhere around 100,000 lines of code where touched which is substantial for a single development cycle. I wrote a few tools to help us do that work, because that’s really the only way to do such a large refactor.

Not only does the refactor make things easier for us to maintain but it will make things easier for contributors to write new plugins. In a future blog post I’ll cover some of the new design that makes it possible.

Let’s take a look at some of the changes in Builder for 3.32 as users will see them.

First we have the greeter. It looks similar as before, although with a design refresh. But from a code standpoint, it no longer shares it’s windowing with the project workspace. Taking this approach allowed us to simplify Builder’s code and allows for a new feature you’ll see later.

Builder now gives some feedback about what files were removed when cleaning up old projects.

Builder gained support for more command-line options which can prove useful in simplifying your applications setup procedure. For example, you can run gnome-builder --clone https://gitlab.gnome.org/GNOME/gnome-builder.git to be taken directly to the clone dialog for a given URL.

The clone activity provides various messaging in case you need to debug some issues during the transfer. I may hide this behind a revealer by default, I haven’t decided yet.

Creating a new project allows specifying an application-id, which is good form for desktop applications.

We also moved the “Continue” button out of the header bar and placed it alongside content since a number of users had difficulty there.

The “omni-bar” (center of header bar) has gained support for moving through notifications when multiple are present. It can also display buttons and operational progress for rich notifications.

Completion hasn’t changed much since last cycle. Still there, still works.

Notifications that support progress can also be viewed from our progress popover similar to Nautilus and Epiphany. Getting that circle-pause-button aligned correctly was far more troublesome than you’d imagine.

The command-bar has been extracted from the bottom of the screen into a more prominent position. I do expect some iteration on design over the next cycle. I’ve also considered merging it into the global search, but I’m still undecided.

Also on display is the new project-less mode. If you open Builder for a specific file via Nautilus or gnome-builder foo.c you’ll get this mode. It doesn’t have access to the foundry, however. (The foundry contains build management and other project-based features).

The refactoring not only allowed for project-less mode but also basic multi-monitor support. You can now open a new workspace window and place it on another monitor. This can be helpful for headers, documentation, or other references.

The project tree has support for unit tests and build targets in addition to files.

Build Preferences has been rebuilt to allow plugins to extend the view. That means we’ll be able to add features like toggle buttons for meson_options.txt or toggling various clang/gcc sanitizers from the Meson plugin.

The debugger has gone through a number of improvements for resilience with modern gdb.

When Builder is full-screen, the header bar slides in more reliably now thanks to a fix I merged in gtk-3-24.

As previewed earlier in the cycle, we have rudimentary glade integration.

Also displayed here, you can select a Build Target from the project tree and run it using a registered IdeRunHandler.

Files with diagnostics registered can have that information displayed in the project tree.

The document preferences have been simplified and extracted from the side panel.

The terminal now can highlight filename:line:column patterns and allow you to ctrl+click to open just like URLs.


In a future post, we’ll cover some of what went into the refactoring. I’d like to discuss how the source tree is organized into a series of static libraries and how internal plugins are used to bridge subsystems to avoid layering violations. We also have a number of simplified interfaces for plugin authors and are beginning to have a story around ABI promises to allow for external plugins.

If you just can’t wait, you can play around with it now (and report bugs).

flatpak install https://gitlab.gnome.org/GNOME/gnome-apps-nightly/raw/master/gnome-builder.flatpakref

Until next time, Happy Hacking!

Introducing deviced

Over the past couple of weeks I’ve been heads down working on a new tool along with Patrick Griffis. The purpose of this tool is to make it easier to integrate IDEs and other tooling with GNU-based gadgets like phones, tablets, infotainment, and IoT devices.

Years ago I was working on a GNOME-integrated home router with davidz which sadly we never finished. One thing that was obvious to me in that moment of time was that I will not do another large scale project until I have better tooling. That was Builder’s genesis and device integration is what will make it truly useful to myself and others who love playing with GNU-friendly gadgets.

Now, building an IDE is a long process. There is a ton of code to write, trade-offs to work through, and persistence beyond what any reasonable programmer would voluntarily sign up for. But the ends justify the slog.

So what we’ve created is uninterestingly called “deviced”. It currently has three components. A deviced daemon lives on the target device that we’re interested in writing software for. A GObject-based libdeviced library provides access to discover and connect to devices and do interesting things on them. Lastly, devicectl is a readline-based command line tool that allows you to interact with these devices without having to write a program using libdeviced.

The APIs in libdeviced are appropriately abstracted so that we can provide different transports in the future. Currently, we only have network-based communication but we will implement a USB transport in the not-too-distant future. Other protocols such as SSH or custom micro-controllers can be added. Although something like SSH is more complex because it’d be the combination of both a protocol and how to run commands to get the intended effect, which is non-portable. It will be possible to support devices that do not run deviced, but that is currently out of scope.

To allow devices to be discover-able, deviced will broadcast it’s presence using mDNS on networks it is configured to listen (based on network-manager connection UUID). Long term my goal is that you can configure deviced access in Control Center, similar to “Sharing and Privacy”. The network protocol is rather simple as it’s just JSON-RPC over TLS with self-signed certificates. When a client connects to the daemon, a gnome-shell notification is presented allowing you to accept the connection. At that point, the client certificate is saved for future validation.

Our libdeviced library is GObject introspectable and should therefore work with a number of languages.

Right now, only Flatpak applications are supported, but we have abstractions to allow for contributions to support additional application layers like docker or plain old .desktop files. Currently you can push flatpak applications and runtimes to the device and install them and run them. If you have a new enough Flatpak, you can do delta updates.

It can even bridge multiple PTY devices for a shell, which isn’t really meant to be an SSH replacement, but more of a single abstraction we can use to be able to control a debugger and inferior from the IDE tooling.

There are still lots of little bugs to shake out and more bits to implement, but this is a pretty sweet 2-week proof of concept.

https://gitlab.gnome.org/chergert/deviced/

Here is a 20 second demo running on a single machine. It’s the same when using multiple machines except you get the notification on the programmable device rather than on your workstation. Obviously for IoT devices we’d need to create some sort of freedesktop notification bridge or alternate notification mechanism.

Anytime you work on a new project people will inevitably ask “why not just use XYZ”. In this case, I would expect both SSH and ADB to fall into that category. Most importantly, libdeviced is going to be about providing a single “remote device” abstraction for us in Builder. So it’s reasonable that we could abstract both of those systems from libdeviced. But neither of those provide the work-flow I envision for out-of-box experience, hence the deviced daemon. In the ADB case, it will be very difficult to get code upstream and released to distributions as it is increasingly unlikely our use-case is interesting to upstream. There were experimental patches to ADB a couple years ago to support flatpak so we didn’t take on this effort without considering our options. Ultimately, this prototype was to see the feasibility of making something that solves our problems while not locking us out of supporting other systems in the future.

Builder happenings for January

I’ve been very busy with Builder since returning from the holidays. As mentioned previously, we’ve moved to gitlab. I’m very happy about it. I can see how this is going to improve the engagement and communication between our existing community and help us keep new contributors.

I made two releases of Builder so far this month. That included both a new stable build (which flatpak users are already using) and a new snapshot for those on developer operating systems like Fedora Rawhide.

The vast majority of my work this month has been on stabilization efforts. Builder is already a very large project. Every moving part we add makes this Rube Goldberg machine just a bit more difficult to maintain. I’ve tried to focus my time on things that are brittle and either improve or replace the designs. I’ve also fixed a good number of memory leaks and safety issues. However, the memory overhead of clang sort of casts a large shadow on all that work. We really need to get clang out of process one of these days.

Over the past couple years, our coding style evolved thanks to new features like g_autoptr() and friends. Every time I come across old-style code during my bug hunts, I clean it up.

Builder learned how to automatically install Flatpak SDK Extensions. These can save you a bunch of time when building your application if you have a complex stack. Things like Rust and Mono can just be pulled in and copied into your app rather than compiled from source on your machine. In doing so, every app that uses the technology can share objects in the OSTree repository, saving disk space and network transfer.

That allowed me to create a new template, a GNOME C♯ template. It uses the Mono SDK extension and gtk-sharp for 3.x. If you want to help here, work on a omni-sharp language server plugin for us!

A new C++ template using Gtkmm was added. Given that I don’t have a lot of recent experience with Gtkmm, it’d be nice to have someone from that community come in and make sure things are in good shape.

I also did some cleanup on our code-indexer to avoid threading in our API. Creating plugins on threads turned out to be rather disastrous, so now we try extra hard to keep things on the main thread with the typical async/finish function pairs.

I created a new messages panel to elevate warnings to the user without them having to run Builder from a terminal. If you want an easy project to work on, we need to go find interesting calls to g_warning() and use ide_context_warning() instead.

Our flatpak plugin now tries extra hard to avoid downloads. Those were really annoying for people when opening builder. It took some troubleshooting in flatpak-builder, and that is fixed now.

In the process of fixing the extraneous downloading I realized we could start bundling flatpak-builder with Builder. After a couple of fixes to flatpak-builder Builder Nightly no longer requires flatpak-builder on the host. That’s one less thing to go wrong for people going through the newcomers work-flow.

We just landed the beginning of a go-langserver plugin. It seems like the language server for Go is pretty new though. We only have a symbol resolver thus far.

I found a fun bug in Vala that caused const gchar * const * parameters to async functions to turn into gchar **, int. It was promptly fixed upstream for us (thanks Rico).

Some 350 commits have landed this month so far, most of them around stabilizing Builder. It’s a good time to start playing with the Nightly branch if you’re into that.

Oh, and after some 33 years on Earth, I finally needed glasses. So I look educated now.

Debugging

I’ve been quiet since I got back from GUADEC. It’s been a busy summer, but I’ve managed to sneak away and build this in-between my other maintainer/GSoC duties.

There is still plenty to do, but this gets the basic plumbing in place for a debugger.

It can also debug flatpak-based applications (although you’ll need .Debug runtimes for good symbols in gtk/glib/etc).