Category: Builder

Profiling Flatpak’d applications

One of the great powers of namespace APIs on Linux (mount namespaces, user namespaces, etc) is that you can create a new view into the world of your computer that is very different from the host. This can make traditional profiling tools difficult.

To begin with, we need to ensure that we have access to ptrace or perf infrastructure. Easy enough, just don’t drop those privileges before calling execve(). This is the --allow=devel option to flatpak run. But after that, we need to do the detailed phase of translating instruction pointers to a function name.

Generally, the translation between an instruction pointer and a function name requires looking up what file is mapped over the address. Then you open that file with an ELF reader and locate the file containing debug information (which may be the same file). Then open that file and locate what function contains that instruction pointer. (We subtract the beginning of the map from the instruction pointer to get a relative offset).

Now, here is the trouble with mount namespaces. The “path” of the map might be something like “/newroot/usr/lib/libgtk-3.so“. Of course, “/newroot/” doesn’t actually exist.

So… what to do.

Well, we can find information about the mounts in the process by looking at /proc/$pid/mountinfo. Just look for the longest common prefix to get the translation from the mount namespace into the host. One caveat though. The translated path is relative to the root of the partition mounted. So the next step is to locate the file-system mount on the host (via /proc/mounts).

If you put this all together, you can profile and extract symbols from applications running inside of containers.

Now, most applications aren’t going to ship debug symbols with them. But we’ll need those for proper symbol extraction. So if you install the .Debug variant of a Flatpak runtime (org.gnome.Sdk.Debug for example) then we can setup symbol resolution specifically for /usr/lib/debug. Builder now does this.

I’m not sure, but I think this makes Builder and Sysprof one of the first (if not the first) profiler on Linux to support symbol extraction while profiling containerized applications from outside the container.

I’ve only tested with Flatpak, but I don’t see why this code can’t work with other tooling using mount namespaces.

Valgrind Integration in Builder

I wanted a bit of a distraction this evening, so I put together a quick valgrind plugin. Just select “Run with Valgrind” from the drop down menu and it will make sure your program is run using the venerable valgrind.

If valgrind isn’t available in your runtime (host, jhbuild, flatpak, etc) you wont see the menu item. But thankfully the org.gnome.Sdk runtime we use to build GNOME apps has it, so for many of you it should Just Work™.

Builder’s Build Pipeline

One of the core features of Builder is, well, building. So we need to provide the best experience we can. That means we need wide support for build systems and languages. How we get from source code to a product can vary in a multitude of ways.

For example, you might need to alter how you launch programs to run inside a container. You might need access to headers that only exist in the SDK and not the application runtime. You might even need to download or build dependencies for your project at a given point. Furthermore, this could all change based on what plugins you’ve chosen to enable in your Builder environment.

When I set out to create Builder, I knew that I wouldn’t be able to see all the possibilities that Builder needs to handle. So I explicitly avoided designing an abstract solution until I understood the problem better. But I think we understand the problem well enough now. So over the winter holiday I started putting together the design of IdeBuildPipeline.

The pipeline is a one-directional series of build phases. Each phase can have stages attached to it. Builds progress in the order of their attached stages, each of which can be an asynchronous operation. We only need to advance the build up to the requested phase. If the IDE detects something has changed, it can invalidate various phases of the pipeline causing them to be executed again at the next build request.

Plugins should implement the IdeBuildPipelineAddin interface to register any necessary pipeline stages.

Generally, build stages are an IdeSubprocessLauncher, but you can implement custom IdeBuildStage subclasses to do more fancy things. For example, if you need to download something, you might add an IdeBuildStageTransfer and you’ll see the transfer show up in the transfers window.

Of course, some problems are complex enough where we don’t know if they are invalidated upfront and the build stage may query some external resource. For that, the IdeBuildStage class can implement the query signal. Of course we should strive to perform blocking operations asynchronously so consider pausing the stage until your asynchronous request has completed.

We know how difficult it is for newcomers to get started so we’ve taught Builder how to download Flatpak runtimes and SDKs based on your project configuration. That means that you can omit those flatpak install commands from your “project setup” guide. Just clone the project URL and click Build, we’ll take care of the rest.

Thanks to this new design, implementing a build system is much easier. To prove this, I finally implemented a basic cmake+ninja plugin in a matter of an hour. Check it out here. If you’d like to write CFLAG extraction for cmake, we could use that still!

We now have support for autotools, cargo, cmake, and meson build systems.

Contribute to Polari with this one simple trick!

I’ve been rather quiet recently working on new features for Builder. But we managed to just release Builder 3.22.3 which is full of bug fixes and a really new important feature. You can now meaningfully target flatpak when building your application. Matthew Leeds has done this outstanding work and it is really going to simplify how you contribute to GNOME applications going forward.

I’m really happy with the quality of this feature because it has shown me where our LibIDE design has done well, and where it has not. Of course, we will address that for 3.24 to help make some of the UI less confusing.

Without further ado, how to clone, build, run, and hack on Polari without so much as a toolchain installed on your host system. The only prerequisite is to get GNOME Builder 3.22.3 from the GNOME flatpak repository (or your distribution if it is up to date).

Edit: Your system might require the installation of flatpak-builder if it is in a separate package (such as on Fedora 25).

# Get things ready on Fedora 25
sudo dnf install flatpak-builder

# Download GNOME's nightly SDK for development.
# We'll automate this for GNOME 3.24.
flatpak --user remote-add gnome-nightly \
https://sdk.gnome.org/gnome-nightly.flatpakrepo
flatpak --user install gnome-nightly org.gnome.Sdk master
flatpak --user install gnome-nightly org.gnome.Platform master

Open Builder, select Clone from the buttons in the header bar.
Open Builder, select Clone from the buttons in the header bar.
Set the URL for Polari to git://git.gnome.org/polari (optionally use user@git.gnome.org:/git/polari.git if you have commit access to GNOME)
Set the URL for Polari to git://git.gnome.org/polari (optionally use user@git.gnome.org:/git/polari.git if you have commit access to GNOME)
Click on the blue Clone button
Click on the blue Clone button and wait a few moments while we download the repository
You'll be presented with the Workbench like so, click on “Build Preferences” in the perspective selector.
You’ll be presented with the Workbench like so, click on “Build Preferences” in the perspective selector.
Now select “org.gnome.Polari.json” as the build runtime. (We expect this to be treated as a build configuration in 3.24)
Now select “org.gnome.Polari.json” as the build runtime. (We expect this to be treated as a build configuration in 3.24 instead of a runtime)
Click the Build button on the header bar or in the build popover
Click the Build button on the header bar or in the build popover
On the first build, flatpak-builder is used to build all the necessary dependencies for Polari (such as telepathy). After it has built, click on the “Run” button in the header bar.
On the first build, flatpak-builder is used to build all the necessary dependencies for Polari (such as telepathy). After it has built, click on the “Run” button in the header bar.
Hey! Look at that, a Polari window connected to freenode!
Hey! Look at that, a Polari window connected to freenode!
To make sure I'm not fooling you, let's add a printf() to polari.c (the application entry point). Save, then click “Run” again.
To make sure I’m not fooling you, let’s add a printf() to polari.c (the application entry point). Save, then click “Run” again.
How about that, we can see the output in the “Run Output” panel and Polari still seems to work. Yay!
How about that, we can see the output in the “Run Output” panel and Polari still seems to work. Yay!

Accelerators, a history lesson

It’s good to dive into our shared history every now and again to learn something new. We want to build a customized shortcut engine for Builder and that means we need to have a solid understanding of all the ways to activate shortcuts in GTK+. So the following is a list of what I’ve found, as of GTK+ 3.22. I’ve included some pros and cons of each based on my experience using them in particularly large applications.

GtkAccelGroup and GtkAccelMap

The GtkAccelGroup class is remnant from the days before GtkUIManager was deprecated. It is a structure that is attached to a top-level GtkWindow and maps “paths” to closures. It’s purpose was to be used with a GtkAccelMap to ultimately bind accelerators (such as “<control>q”) to a closure (such as gtk_main_quit).

The GtkAccelMap is a singleton that maps accelerators to paths. When activating a keybinding, the accelerator machinery will try to match the path with one registered in the GtkAccelGroup on the window.

Pros

  • Widgets can be activated no matter what the focus widget. However, the window must be focused.
  • It is simple to display keyboard accelerators next to menu items.
  • Creating an accelerator editor is straight forward. Lookup accel path, map to accelerator.

Cons

  • Not deprecated, but much of the related machinery is deprecated.
  • Fairly complex API usage. It doesn’t seem to be designed to be used by application developers.

Signal Actions

Signal actions are GObject signals that contain the G_SIGNAL_ACTION bit set. By marking this bit, we allow the GTK+ machinery to map an accelerator to a particular signal on the GtkWidget (or any ancestor widget in the hierarchy if it matches).

In the GTK+ 2.x days, we would control these keybindings using a gtkrc file. But today, we use GTK+ CSS to create binding sets and attach them to widgets using selectors. This is how the VIM implementation in Builder is done.

It should be noted that signal actions (and binding them from gtkrc/css) were meant to provide a way to provide key themes to GTK+ such as an emacs mode. It was not meant to be the generic way that applications should use to activate keybindings.

Pros

  • You can use CSS selectors to apply binding sets (groups of accelerators) to widgets.
  • Accelerators can include parameters to signals, meaning you can be quite flexible in what gets activated. With enough bong hits you’ll eventually land at vim.css.

Cons

  • Only the highest-priority CSS selector can attach binding sets to the widget. So the highest-priority needs to know about all binding sets that are necessary. That reduces the effectiveness of this as a generic solution.
  • This relies on the focus chain for activation. That means you can’t have a signal activate if another widget focused such as a something in the header bar.
  • It feels like a layer violation. I could make arguments on both side of the fence for this, which is sort of indicative of the point.
  • We have to “unbind” various default accelerators on widgets that collide with our key themes. This means that “key themes” are not only additive, but destructive as well.

GAction

Years back, we wanted the ability to activate actions inside of a program in a structured way. This allowed for external menu items as well as simplifying the process of single-instance applications. This was integrated into the GtkWidget hierarchy as well. Using gtk_application_set_accels_for_action() you can map an accelerator to a GAction. Using the current widget focus, the GTK+ machinery will work its way up the widget hierarchy until it finds a matching action. For example, if a widget has a “foo” GActionGroup associated with it, and that group contains an action named “bar”, then mapping “<control>b” to “foo.bar” would activate that action.

Pros

  • Actions can have state, meaning that we can attach them to things like toggle buttons.
  • Application level actions can be activated no matter what window is focused.
  • It’s fairly easy to reason about what actions are discoverable and activatable from a node in the widget tree. Especially with GTK+ Inspector integration.
  • Theoretically we could rebuild a11y on top of GAction and fix a lot of synchronous IPC to be async similar to how GMenu integrates with external processes. This would require a lot of external collaboration on various stacks though, so it isn’t an immediate “Pro”.

Cons

  • Action activation requires the focus hierarchy to activate the action. This means to get something activatable from the entire window we need to attach the actions to the top-level. This can be a bit inconvenient, especially in multi-document scenarios which do not share widgetry.
  • Actions are rather verbose to implement. Doing them cleanly means implementing proper API to perform the action, and then wrapping those using GActionEntry or similar. Additionally, to propagate state, the base API needs to know about the action so it can update action state. Coming up with a strategy for who is the “owner” of state is challenging for newcomers.
  • The way to attach accelerators is via gtk_application_set_accels_for_action() which is a bit weird for widgets that would want to provide accelerators. It means the applications are given more freedom to control how accelerators work at the cost of having to setup a lot of their own mechanics for activation.

What do we need in Builder?

Builder certainly lies on the “more complex” side of things in terms of what is needed from the toolkit for accelerators. Let’s go over a few of those necessities to help create a wishlist for whatever code gets written.

VIM and Emacs Keybindings

These will never be 100% perfect, but we need reasonable implementations that allow Emacs and VIM users not scream at their computer while trying to switch to a new tool. That means we need to hit the 80% really well, even if that last 20% is diminishing returns. Today, we’ve found a way to mostly do this using G_SIGNAL_ACTION and a lot of custom GTK+ CSS. The con mentioned above of GTK+ CSS requiring that the -gtk-key-bindings: CSS property know about all binding sets that need to be applied from the selector make it unrealistic for us to keep pushing this further.

Custom Accelerators

We want the user to be able to start from a basic “key theme” such as Gedit, VIM, or Emacs and apply their own overrides. This means that our keybinding registration cannot be static, but flexible to account for changes while the application is running. We also need to know when there are collisions with other accelerators so that we can let the user know what they are doing may have side-effects.

GtkShortcutsWindow Integration

In Builder we do not account for the “key theme” in the shortcuts window. This can be confusing to users of Emacs and Vim mode as what they see may not actually activate the desired action. We would like a way to map the accelerators available (grouped and with proper sections) to be automatically reflected in our GtkShortcutsWindow.

Elevating Accelerator Visibility

The shortcuts window is nice, but making the accelerators available to the user while they are exploring the interface is far more beneficial (based on my experiences). So whatever we end up implementing needs to make it fairly trivial to display accelerators in the UI. Today, we do that manually by hard-coding these accelerators into the GtkBuilder UI files. That is less than ideal.

Window-wide Activation

If I’m focused on the source code editor, I may want to activate an accelerator from a panel that is visible. For example, <control><shift>F to activate the documentation search.

Using actions to do this today would require that all panel plugins register their actions on the top-level as well as keybindings using gtk_application_set_accels_for_action(). While this is acceptable for the underlying machinery, it’s almost certainly not the API we want to expose to plugin developers in Builder. It’s tedious and prone to breakage when multiple “key themes” are in play. So we need an abstraction layer here that can use the appropriate strategy for what the plugin has in mind and also allows for different accelerators based on the “key theme”.

Re-usability

There are other large applications in the GNOME eco-system that I care about. GIMP is one of those projects that has a hard time moving to GTK+ 3.x due to a few reasons, some of which are out of our control upstream. The sheer size of the application makes it difficult. Graphical tools often are full of custom widgetry which necessarily dives into implementation details. Another reason is how important accelerators are to an immersive, creativity-based application. I would like whatever we create to be useful to other projects (after a few API iterations). GIMP’s GTK+ 3 port is one obvious possibility.

Builder Rust

With Federico’s wonderful post on Rust’ifying librsvg I guess it makes sense to share what I’ve been doing the last couple of days.

I’ve been keeping my eye on Rust for quite a while. However, I’ve been so heads down with Builder the last two years that I haven’t really gotten to write any or help on integration into our platform. Rust appears to take a very pragmatic stance on integration with systems code (which is primarily C). The C calling convention is not going anywhere, so at some point, you will be integrating with some part of a system that is “C-like”. Allowing us to piecemeal upgrade the “Safety” of our systems is much smarter than rewrite-the-universe. This pragmatism is likely due to the realities of Rust’s birth at Mozilla. It’s a huge code-base, and incrementally modernizing it is the only reality that is approachable.

We too have a lot of code. And like many other projects, we care about being language agnostic to a large degree. In the early days we might have chosen C because it was the only toolchain that worked reliably on Linux (remember C++ pre-2000? or LinuxThreads?) but today we still care about “Language Interopability” and C is the undeniable common denominator.

One way in which we can allow the interopability that we desire and the Safety we need is to start approaching some of our problems like Federico has done. The C calling convention is “Safe”. That is not where zero-day bugs come from. They come from hastily written C code. It is perfectly fine to write our interfaces in C (which can be our basis for GObject Introspection) but implement Safety critical portions in Rust.

This is exactly what I’d like to see from the hundreds of thousands of lines of C I’ve written over the years. We need to identify and fix the Satefy Critical portions. GStreamer is already on the right track here by looking at codec and plugin implementations in Rust. I’m not sure how far they will go with adapting to Rust, but it will be one of the best case-studies we will have to learn from.

So because of this desire to look at building a Safer Platform for developers and users alike, I’ve decided to start adding Rust support to Builder. Thanks to the hard work of the Rust team, it’s a fairly easy project from our side. There is a new Language Server Protocol 2.0 that was worked on by various people at Microsoft, Red Hat, and elsewhere. The new rustls was announced last week and uses this protocol. So I implemented support for both as quick as I could, and now we have something to play with.

Because of the Language Server Protocol, our Rust plugin is tiny. It is essentially a glorified supervisor to handle subprocess crashes and some binding code to connect our stdin/stdout streams to the Language Server Protocol client in Builder. See for yourself.

There is a bunch more work for us in Builder to make it a great Rust IDE. But if people start playing with the language and are willing to work on Builder and GNOME to improve things as a community, we can build a Modern, Safe, and Elegant developer platform.

The big ticket next steps for those that want to contribute to Rust support would include:

  • Cargo build system support. I do believe that ebassi started on something here. But I need to circle back around and sync up.
  • Symbol Tree needs improvements.
  • Semantic highlighter (which we can implement using fuzzy symbols from the symbol tree until a real protocol comes along).
  • Upstream rustls needs work too to get us the features we want. So Rustaceans might want to spend some time helping out upstream.
  • We need to simplify the glue code from Rust←→GObject so that it is dead simple to wrap Rust code in a GObject-based library (where we get our free language interopability).
  • … and of course all the other Builder plumbing that needs to happen in general. See our list of projects that need work.
This image depicts control+clicking on a symbol to jump to its definition. +period or :gd in Vim mode also work.
This image depicts control+clicking on a symbol to jump to its definition. <alt>+period or :gd in Vim mode also work.
This image shows diagnostics displayed over source code.
This image shows diagnostics displayed over source code.
This image shows completion of fields from a struct.
This image shows completion of fields from a struct.
This image shows the Symbol Tree on the right containing elements from the Rust document.
This image shows the Symbol Tree on the right containing elements from the Rust document.

Sysprof Plans for 3.24

The 3.24 cycle is just getting started, and I have a few plans for Sysprof to give us a more polished profiling experience in Builder. The details can be found on the mailing list.

In particular, I’d love to land support for visualizers. I expect this to happen soon, since there is just a little bit more to work through to make that viable. This will enable us to get a more holistic view of performance and allow us to drill into callgraphs during a certain problematic period of the profile.

Once we have visualizer support, we can start doing really cool things like extracting GPU counters, gdk/clutter frame-clock timing, dbus/cpu/network monitors and whatever else you come up with.

A CPU visualizer displayed above the callgraph to provide additional context to the profiled execution.
A CPU visualizer displayed above the callgraph to provide additional context to the profiled execution.

Additionally we have some work to do around getting access to symbols when we are running in binary stripped environments. This means we can upload a stripped binary to your IoT/low-power device to profile, but have the instruction-pointer-to-symbol resolver happen on the developers workstation.

As I just alluded to, I’d love to see remote profiling happen too. There is some plumbing that needs to occur here, but in general it shouldn’t be terribly complicated.