Sidebars in Libpanel

One of the more recent design trends in GNOME has been the use of sidebars. It looks great, it’s functional, and it gives separation of content from hierarchy.

A screenshot showing a number of GNOME applications which contain sidebars including Nautilus, D-Spy, Control Center, and Calendar. The image contains both light and dark variants split by a line from lower left to upper right of the image.

Builder, on the other hand, has been stuck a bit closer to the old-hat design of IDEs where the hierarchy falls strictly from the headerbar. This is simply because libpanel was designed before that design trend. Some attempt was made in Builder to make it look somewhat sidebar’ish, but that was the extent of it given available time.

A screenshot of the GNOME 45 release of Builder where the headerbar is across the top and panels, documents, and project panel below.

Last week I had a moment of inspiration on a novel way we could solve it without uprooting the applications which use libpanel. You can now insert edge widgets in PanelDockChild which are always visible even when the child is not. Combining that with being able to place a headerbar inside a PanelDockChild along with your PanelFrames means you can get something that looks more familiar in modern GNOME.

A screenshot of what will become Builder for GNOME 46 which includes the common sidebar styling.

If you’d like to improve things further, you know where to find the code.

GJS plugins in GNOME Builder

As I mentioned in my last post, Builder has switched to GJS as it’s dynamic language for plugins. We already support a number of compiled languages including C, C++, Rust, and Vala.

Previously we had used PyGObject. Do to the lack of GTypeInstance support in PyGObject, that isn’t an option currently. I already ported all of Builder’s plugins written in Python to C over the course of a week last summer. That ended up making things both more stable and allow us to ship the GTK 4 port on time.

This past year I wrote a new async/futures framework for GLib called libdex which provides Fibers, Futures, Channels, await, threadpools, io_uring support, and more. That tool heavily uses the same GTypeInstance features that GTK 4 uses.

GJS has improved a lot over the years due to how it is being maintained and it’s importance in the GNOME Shell stack. I’d like to double down on that so Builder can benefit from their hard work. Therefore, if you want to write plugins in JavaScript and maintain them upstream, that’s something I’m happy to see happen.

You can see some examples for how to write a JavaScript plugin for Builder in the examples directory.

GJS plugins for libpeas-2.0

One of the main features I want to land for the libpeas-2.0 ABI break is support for plugins in JavaScript.

With the right set of patches, you can get that. Thanks to Philip Chimento, GJS will hopefully soon land support for running code in a SpiderMonkey realm. Philip also did us a solid and wrote the code to exfiltrate enough GType information from an imported JavaScript module. That allows libpeas to correlate which GTypes are provided by a plugin.

With the GJS realm support in place, we can land the new GJS loader for libpeas-2.0.

My personal goal for this is to enable JavaScript-based plugins in GNOME Builder. With how much GJS has improved over the years to support GNOME Shell, it is probably our most-maintained language binding for a dynamic language with modern JIT features.

For example, if you wanted to make an addin in Builder which responded to changes of a file within the editor, you might write something like this as your plugin. Keep in mind I’m not a JavaScript developer and GJS developers may tell you there are fancy new language features you can use to simplify this code further.

import GObject from 'gi://GObject';
import Ide from 'gi://Ide';

export var TestBufferAddin = GObject.registerClass({
    Implements: [Ide.BufferAddin],
}, class TestBufferAddin extends GObject.Object {

    vfunc_language_set(buffer, language_id) {
        print('language set to', language_id);
    }

    vfunc_file_loaded(buffer, file) {
        print(file.get_uri(), 'loaded');
    }

    vfunc_save_file(buffer, file) {
        print('before saving buffer to', file.get_uri());
    }

    vfunc_file_saved(buffer, file) {
        print('after buffer saved to', file.get_uri());
    }

    vfunc_change_settled(buffer) {
        print('spurious changes have settled');
    }

    vfunc_load(buffer) {
        print('load buffer addin');
    }

    vfunc_unload(buffer) {
        print('unload buffer addin');
    }

    vfunc_style_scheme_changed(buffer) {
        let scheme = buffer.get_style_scheme();
        print('style scheme changed to', scheme ? scheme.get_id() : scheme);
    }
});

You can easily correlate that to the IdeBufferAddin interface definition.

Concurrency, Parallelism, I/O Scheduling, Thread Pooling, and Work-Stealing

Around 15 years ago I worked on some interesting pieces of software which are unfortunately still not part of my daily toolbox in GNOME and GTK programming. At the time, the world was going through changes to how we would write thread pools, particularly with regards to wait-free programming and thread-pooling.

New trends like work-stealing were becoming a big thing, multiple-CPUs with multiple NUMA nodes were emerging on easy to acquire computers. We all were learning that CPU frequency was going to stall and that non-heterogeneous CPUs were going to be the “Next Big Thing”.

To handle those changes gracefully, we were told that we need to write software differently. Intel pushed that forward with Threading Building Blocks (TBB). Python had been doing things with Twisted which had an ergonomic API and of course “Stackless Python” and similar was a thing. Apple eventually came out with Grand Central Dispatch. Microsoft Research had the Concurrency and Coordination Runtime (CCR) which I think came out of robotics work.

Meanwhile, we had GThreadPool which honestly hasn’t changed that much since. Eventually the _async/_finish paring we’re familiar with today emerged followed by GTask to provide a more ergonomic API on top of it.

A bit before the GTask we all know today, I had written libgtask which was more of a Python Twisted-style API which provided “deferreds” and nice ways to combine them together. That didn’t come across into GLib unfortunately. To further the pain there, what we got in the form of GTask has some serious side-effects which make it unsuitable as a general construct in my humble opinion.

After realizing libgtask was eclipsed by GLib itself, I set off on another attempt in the form of libiris. That took a different approach that tried to merge the mechanics of CCR (message passing, message ports, coordination arbiters, actors), the API ergonomics of Python Twisted’s Deferred, and Apple’s NSOperation. It provided a wait-free work-stealing scheduler to boot. But it shared a major drawback of GLib’s GTask (beyond correctness bugs that plague it today). Primarily that thread pools can only process the work queue and therefore if you need to combine poll() or GSource that attach to a GMainContext you’re going to require code-flow to repeatedly bounce between threads.

This is because you can simplify a thread pool worker to while (has_work()) do_work ();. Any GSource or I/O either needs to bounce to the main thread where the applications GMainContext exists or to another I/O worker thread if doing synchronous I/O. On Linux, for a very long time, synchronous I/O was the “best” option if you wanted to actually use the page cache provided by the kernel, so that’s largely what GLib and GIO does.

The reason we couldn’t do something else is that to remove an item from the global work queue required acquiring a GMutex and blocking until an item is available. On Linux at least, we didn’t have APIs to be able to wait on a Futex while also poll()ing a set of file-descriptors. (As a note I should mention for posterity that FD_FUTEX was a thing for a short while, but never really usable).

In the coming years, we got a lot of new features in Linux that allowed improvements to the stack. We got signalfd to be able to poll() on incoming Unix signals. We got eventfd() which allowed a rather low-overhead way to notify coordinating code with a poll()able file-descriptor. Then EFD_SEMAPHORE was added so that we can implement sem_t behavior with a file-descriptor. It even supports O_NONBLOCK.

The EFD_SEMAPHORE case is so interesting to me because it is provides the ability to do something similar to what IRIX did 20+ years ago, which is a pollable semaphore! Look for usnewpollsema() if you’re interested.

There was even some support in GLib to support epoll(), but that seems to have stalled out. And honestly, making it use io_uring might be smarter option now.

After finishing the GTK 4 port of GNOME Builder I realized how much code I’ve been writing in the GAsyncReadyCallback style. I don’t particularly love it and I can’t muster the energy to write more code in that style. I feel like I’m writing top-half/bottom-half interrupt handlers yet I lack the precision to pin things to a thread as well as having to be very delicate with ownership to tightly control object finalization. That last part is so bad we basically don’t use GLib’s GTask in Builder in favor of IdeTask which is smart about when and where to release object references to guarantee finalization on a particular thread.

One thing that all these previous projects and many hundreds of thousands of async-oriented C code written has taught me is that all these components are interlinked. Trying to solve one of them without the others is restrictive.

That brings me to 2022, where I’m foolishly writing another C library that solves this for the ecosystem I care most about, GTK. It’s goal is to provide the pieces I need in both applications as well as toolkit authoring. For example, if I were writing another renderer for GTK, this time I’d probably built it on something like this. Given the requirements, that means that some restrictions exist.

  • I know that I need GMainContext to work on thread pool threads if I have any hope of intermixing workloads I care about on worker threads.
  • I 100% don’t care about solving distributed computing workloads or HTTP socket servers. I refuse to race for requests-per-second at the cost of API ergonomics or power usage.
  • I know that I want work stealing between worker threads and that it should be wait-free to avoid lock contention.
  • Worker threads should try to pin similar work to their own thread to avoid bouncing between NUMA nodes. This increases data cacheline hits as well as reduces chances of allocations and frees moving between threads (something you want minimized).
  • I know that if I have 32 thread pool threads and 4 jobs are added to the global queue, I don’t want 32 threads waking up from poll() to try to take that those work items.
  • The API needs to be simple, composable, and obvious. There is certainly a lot of inspiration to be taken from things like std::future, JavaScript’s Promise, Python’s work on deferred execution.
  • GObject has a lot of features and because of that it goes through great lengths to provide correctness. That comes at great costs for things you want to feel like a new “primative”, so avoiding it makes sense. We can still use GTypeInstance though, following in the footsteps of GStreamer and GTK’s Scene Kit.
  • Cancellation is critical and not only should it cause the work you created to cancel, that cancellation should propagate to the things your work depended on unless non-cancelled work also depends on it.

I’ve built most of the base of this already. The primary API you interact with is DexFuture and there are many types of futures. You have futures for IO (using io_uring). Futures for unix signals. A polled semaphore where dex_semaphore_wait() is a future. It can wrap _async/_finish pairs and provide the result as a Future. Thread pools are efficient in waiting on work (so staying dormant until necessary and minimal thread wake-ups) while also coordinating to complete work items.

There is still a lot of work to go, and so far I’ve only focused on the abstractions and Linux implementations. But I feel like there is promise (no pun intended) and I’m hoping to port swaths of code in Builder to this in the coming months. If you’d like to help, I’d be happy to have you, especially if you’d like to focus on alternate DexAioBackends, DexSemaphore using something other than eventfd() on BSD/Solaris/macOS/Windows, and additional future types. Additionally, working to GLib to support GMainContext directly using io_uring would be appreciated.

You can find the code here, but it will likely change in the near future.

Builder 43.alpha0

It’s been an absolute mad dash this cycle porting Builder to GTK 4, but 43.alpha0 is out and available on GNOME Nightly.

Builder is one of the larger applications within GNOME, especially if you include the libraries I had to write and maintain to make that possible. Porting an application to a new toolkit is always a big undertaking. However, it also provides an opportunity to rethink how major components work and simplify them while you’re there.

So that is what has happened this cycle. It’s going to end up being a much more polished product due to the enormous amount of simplification going on.

GTK 4 has simplified how a lot of things work and provided APIs that feel so obvious when you use them. Of course, that also means lots of code needs to be changed (well deleted, mostly). Having focused heavily on using GListModel in previous releases also paid off massively this cycle.

Anyway, here it is. It’s still missing plenty of features as I dash towards the finish line implementing them as quick as I can.

For those that want to test it out, note that our application-id has changed so that you can install Builder’s stable branch and nightly branch side-by-side.

flatpak --user remote-add --if-not-exists gnome-nightly \
  https://nightly.gnome.org/gnome-nightly.flatpakrepo
flatpak --user install gnome-nightly org.gnome.Builder.Devel

A screenshot of Builder's new about dialog

Builder GTK 4 Porting, Part VII

It’s been another couple weeks of porting, along with various distractions.

The big work this time around has been deep surgery to Builder’s “Foundry”. This is the sub-system that is responsible for build-systems, pipelines, external-devices, SDKs, toolchains, deployments-strategies and more. The sub-system was starting to show it’s age as it was one of the first bits of Builder to organically emerge.

One of the things that become so difficult over the years is dealing with all the container layers we have to poke holes through. Running a command is never just running a command. We have to setup PTYs (and make sure the TTY setup ioctl()s happen in the right place), pass environment variables (but to only the right descendant process), and generally a lot more headaches.

What kicked off this work was my desire to remove a bunch of poorly abstracted bits and we’re almost there. What has helped considerably is creating a couple new objects to help manage the process.

The first is an IdeRunContext. It is sort of like a GSubprocessLauncher but allows you to create layers. At the end you can convert those layers into a subprocess launcher but only after each layer is allowed to rewrite the state as you pop back to the root. In practice this has been working quite well. I finally have control without crazy amounts of argument rewriting and guesswork.

To make that possible, I’ve introduced an IdeUnixFDMap which allows to manage source↔dest FD translations for FDs that will end up in the subprocess. It has a lot of helpers around it to make it fit well into the IdeRunContext world.

All of this has allowed the new IdeRunCommand to really shine. We have various run command providers (e.g. plugins) all of which can seamlessly be used across the sub-systems supporting IdeRunContext. Plugins such as meson can even export unit tests as run commands.

The shellcmd plugin has also been rewritten upon these foundations. You can create custom commands and map them to keyboard shortcuts. The commands, like previous version of Builder, can run in various localities. A subprocess, from the build pipeline, as an app runner, or on the host. What has improved, however, is that they can also be used in surrogate of your projects run command. These two features combined means you can make Builder work for a lot of scenarios it never did before by configuring a few commands.

There aren’t a lot of screenshots for things like this, because ideally it doesn’t look too different. But under the hood it’s faster, more reliable, and far more extensible than it was previously. Hopefully that helps us cover a number of highly requested use-cases.

a screenshot of the debugger

a screenshot of the build menu with debug selected

a screenshot of the run command selection selection dialog

a screenshot showing the location of the select run command menu item

a screenshot editing a command

Builder GTK 4 Porting, Part IV

This week was a little slower as I was struggling with an adjustment to my new medication. Things progress nonetheless.

Text Editor

I spent a little time this week triaging some incoming Text Editor issues and feature requests. I’d really like this application to get into maintenance mode soon because I have plenty of other projects to maintain.

  • Added support for gnome-text-editor - to open a file from standard input, even if you’re communicating to a single instance application from terminal.
  • Branch GNOME 42 so we can add new strings.
  • Fix a no-data-loss crash during shutdown.

Template-GLib

  • Fix template evaluation on macOS.
  • Make boolean expression precedence more predictable.
  • Cleanup output of templates with regards to newlines.

libpanel

  • Propagate modified page state to tabs
  • Some action tweaks to make things more keyboard shortcut friendly.

Builder

  • Merged support for configuration editing from Georges.
  • Add lots of keybindings using our new keybinding engine.
  • Track down and triage that shortcut controllers do not capture/bubble to popovers. Added workarounds for common popovers in Builder.
  • Teach Builder to load keybindings from plugins again and auto-manage them.
  • Lots of tweaks to the debugger UI and where widgetry is placed.
  • Added syntax highlighting for debugger disassembly.
  • Added menus and toggles for various logging and debugger hooks. You can get a breakpoint on g_warning() or g_critical() by checking a box.
  • Ability to select a build target as the default build target finally.
  • More menuing fixes all over the place, particularly with treeviews and sourceviews.
  • Fix keyboard navigation and activation for the new symbol-tree
  • Port the find-other-file plugin to the new workspace design which no longer requires using global search.
  • GTK 4 doesn’t appear to scroll to cells in textview as reliably as I’d like, so I dropped the animation code in Builder and we jump strait to the target before showing popovers.
  • Various work on per-language settings overrides by project.
  • Drop the Rust rls plugin as we can pretty much just rely on rust-analyzer now.
  • Lots of CSS tweaks to make things fit a bit better with upcoming GNOME styling.
  • Fix broken dialog which prevented SDK updates from occurring with other dependencies.

A screenshot of builder's find-other-file plugin

A screenshot of Builder's debugger

A screenshot showing the build target selection dialog

A screenshot of the run menu A screenshot of the logging menu

Builder GTK 4 Porting, Part II

Another week of work towards porting Builder to GTK 4. Since I can’t add to TWIG from IRC, I’ll try harder to drop some occasional updates here.

GtkSourceView

  • Merged fixes for highlighting unicode literals for C
  • Improved parsing of language values in snippet bundles
  • GtkSourceGutter will not correctly mark prelit and selection quarks within GtkSourceGutterLines.
  • Fixed a bunch of little mouse pointer annoyances when using GtkSourceHover interactive tooltips.
  • GtkSourceGutterRenderers can now opt-out of signal emission for GtkSourceGutterRenderer::query_data(). Signal emission with GObject is rather slow, so avoiding it on every line can be helpful. Just set the virtual method pointer to NULL. The signal was only ever added to make porting easier anyway.

libpanel

  • Merged fixes to be used as a subproject with static libraries only
  • CSS styling matches recent changes in libadwaita, particularly around making navigation tabs and panel frame headers more unified.
  • PanelWidget no longer uses a GtkBinLayout so that it’s easier for subclasses to integrate with popovers from size_allocate() to call gtk_popover_present().

Builder

  • Georges did a live coding stream where they ported a bunch of the “buildui” plugin. That is merged to the GTK 4 port now. It brings a number of features back to the UI including the build terminal, run terminal, build panel (with pipeline stages, warnings, and errors), and project information in the omnibar popover.
  • Günther did a bunch of work porting our old snippet files to the new XML-based snippet bundle format upstream in GtkSourceView. Along with that came porting of the snippets plugin for Builder’s new GTK 4-based editor.
  • Workspace windows have a bit better predictability when restoring sizes.
  • The project creation workflow was ported, albeit needs to have the redesign implemented still.
  • Lots of resiliency fixes for clang and symbol-tree plugins to improve life-cycle management.
  • The Valgrind plugin was ported to C. This was also back-ported to main because it fixed a number of oddities (crashes) occurring in PyGObject.
  • Builder’s “omni-gutter” GtkSourceGutterRenderer was ported to GTK 4 and got a lot of cleanups along the way. I believe there is still some outstanding things to fix such as handling rendering for symbolic icons as I’m pretty sure that’s not correct yet.
  • The “debuggerui” plugin has been ported to GTK 4 and appears to be working well now. This plugin is responsible for bridging the internal IdeDebugger interfaces to the UI interface.
  • Style schemes were updated for GtkSourceView 5
  • Tabs are now the default navigation interface for pages. There is likely still a lot to do around how we want empty frames to look and behave though.
  • The project-tree is now most ported, and with some workarounds to allow making GtkPopover work you can actually display popovers and activate menu items.
  • A long-standing plugin ordering issue has been fixed so that gtk/menus.ui embedded in plugin resources have menu-merging in the proper order.
  • Some incremental work landed to support per-project configuration of languages.
  • The “ls” plugin (directory views) supports “humanized” timestamps again and columns are resizable once more.
  • You can place panels in the right-side-bar now.

Upcoming

There are a bunch of foundational things to still get landed before I feel I can get Builder flipped over to our Nightly builds. In particular we need to land support for things like:

  • Keyboard shortcuts using GtkShortcutController. This was all done with libdazzle previously.
  • Allow plugins to define custom keyboard shortcuts and merge them into the controller.
  • Port “shellcmd” plugin and integrate keyboard shortcuts to apply those commands.
  • Finish rewrite of the search popover. I’m trying to delay this until GtkListView with sections is supported, as it would provide us a much greater path for performance.
  • A lot of our problems would be simpler if we could make GtkActionMuxer use an alternate action muxer parent from another (non-descendant) widget.
  • Configuration editing in the project configuration window. This is a new design so the port is not 1:1.

A screenshot of Builder with the project-tree context menu and debugger on display

A screenshot of Builder with various panels moved around to non-standard locations

GtkSourceView Interactive Tooltips

During the past years (and especially last cycle) I’ve worked to push a number of features upstream from Builder into GtkSourceView. Not only does this improve the ecosystem for all applications, but it reduces the number of things we need to maintain downstream in Builder as we move to GTK 4.

That included a new completion engine, a new snippet engine with tooltips based on expansion points, updated gutter and gutter renderer designs, the editor overview map, background patterns, sysprof tracing integration, and most recently interactive tooltips.

You can read our nightly generated documentation to learn about it. In particular, GtkSourceHover is attached to the GtkSourceView and can have GtkSourceHoverProviders registered with it. Implementing that interface will be familiar to those who’ve implemented completion providers, as they work in a similar way.

This came from Builder where we have interactive tooltips for things like breakpoints, documentation, and symbol information. Other applications using GtkSourceView may find it useful when implementing Language Server Protocol’s hover providers.

Rust and GNOME Builder

I still spend most of my day writing C and I doubt that is going to change any time soon. But that doesn’t mean you should have to!

Builder got a number of late arriving improvements around Rust support, so now would be a good time to go test them before the 40 release is out.

Yesterday I landed a long awaited feature that will find the common Flatpak SDK ancestors. This was needed to resolve the branch name for SDK extensions like org.freedesktop.Sdk.Extension.rust-stable. Your project might use org.gnome.Sdk//3.38 but the branch for rust-stable is 20.08 (coming from org.freedesktop.Sdk//20.08). Terribly annoying, but hey, now it’s fixed.

Furthermore, Builder can use rust-analyzer¹ as bundled by the org.freedesktop.Sdk.Extension.rust-stable SDK so that is one less thing you need to install or manage. It will pick up all the same dependencies as your project because it will run from within your projects build container. It will see everything your build system does, and in the same way.

Just create a new project using the “GNOME Application” template, select “Rust” as the language, and Run.

¹ rust-analyzer provides diagnostics, auto-completion, and more.