Happenings in GNOME – Page 6 – Details of Christian's work on GNOME

libpeas-2

Now that GNOME 44 is out the door, I took some time to do a bunch of the refactoring I’ve wanted in libpeas for quite some time. For those not in the know, libpeas is the plugin engine behind applications like Gedit and Builder.

This does include an ABI break but libpeas-1.0 and libpeas-2 can be installed side-by-side.

In particular, I wanted to remove a bunch of deprecated API that is well over a decade old. It wasn’t used for very long and causes libpeas to unnecessarily link against gobject-introspection-1.0.

Additionally, there is no need for the libpeas-gtk library anymore. With GTK 4 came much more powerful list widgets. Combine that with radically different plugin UI designs, the “one stop plugin configuration widget” in libpeas-gtk just isn’t cutting it.

Now that there is just the single library, using subdirectories in includes does not make sense. Just #include <libpeas.h> now.

Therefore, PeasEngine is now a GListModel containing PeasPluginInfo.

I also made PeasExtensionSet a GListModel which can be convenient when you want to filter which extensions use care about using something like GtkFilterListModel.

And that is one of the significant reasons for the ABI break. Previously, PeasPluginInfo was a boxed-type, incompatible with GListModel. It is now derived from GObject and thusly provides properties for all the important bits, including PeasPluginInfo:loaded to denote if the plugin is loaded.

A vestige of the old-days is PeasExtension which was really just an alias to GObject. This just isn’t needed anymore and we use GObject directly in function prototypes.

PeasActivatable is also removed because creating interfaces is so easy these days with language bindings and/or G_DECLARE_INTERFACE() that it doesn’t make sense to have such an interface in-tree. Just create the interface you want rather than shoehorning this one in.

I’ve taken this opportunity to rename our development branch to main and you can get the old libpeas-1.0 ABI from the very fresh 1.36 branch.

Smoother Scrolling of Text Views

When working on GTK 4, special care was taken to ensure that most of a GtkTextView‘s content could be rendered without GL program changes and maximal use of glDrawArrays() with vertices from a VBO.

That goes a long way towards making smooth scrolling.

In recent releases, GTK gained support for scrolling using more precise scroll units. On macOS with Apple touchpads, for example, that might map physical distance to a certain number of logical pixels within the application.

If you’re at 2x scaling, you might get values from the input system with “half pixel” values (e.g. .5) since that would map just fine to the physical pixel boundary of the underlying display server.

That’s great for our future, but not everything from GTK’s early designs around X11 have been excised from the toolkit. Currently, widget allocations are still integer based, meaning at 2x scaling they will sit on 2x physical pixel boundaries even though 1x physical pixel boundaries would be just fine to keep lines sharp (assuming you aren’t fractionally scaling afterwards).

Furthermore, GtkTextView works in logical pixels as well. That means that if you want to smooth scroll a text view and you get those 0.5 logical pixel values (1x physical pixels) you wont scroll until you get to 1.0 which then jumps you 2x physical pixels.

That can create some unsightly jitter, most noticeable during kinetic deceleration.

To fix the widget allocation situation, a future ABI break in GTK would have to move to some sort of float/double for widget coordinates. Everything underneath GTK (like GSK/GDK/Graphene/etc) is already largely doing this as GDK went through substantial improvements and simplification for GTK 4.

But with a little creativity, abstraction, and willingness to completely break ABI for a prototype, you can get an idea of what that would be like.

I put together a quick branch today which makes GtkTextView use double for coordinates so that I could push it to snap to physical pixel boundaries.

The fun thing is finding all the ways it breaks stuff. Like text underline getting into situations where it looks different as you scroll or having to allocate GtkWidget embedded within the GtkTextView on logical pixels.

Like I said earlier, it completely breaks ABI of GtkTextView, so don’t expect to replace your system GTK with it or anything.

P.S. I’m on Mastodon now.

Frame pointers and other practical near-term solutions

I’m the author/maintainer of Sysprof. I took over maintainership from Søren Sandmann Pedersen years ago so we could integrate it with GNOME Builder. In the process we really expanded it’s use cases beyond just being a sampling profiler. It is a system-wide profiler that makes it very easy to see what is going on and augment that data with counters, logs, embedded files, marks, custom data sources and more.

It provides:

Callgraphs (based on stacktraces obtained via perf, previously via a custom kernel module).
A binary capture format which makes it very convenient to work with capture files to write custom tooling, unlike perf.dat or pprof.
Accessory collection of counters (cpu, net, mem, cpufreq, energy consumption, battery charge, etc), logs, files, and marks with visualizers for them.
A “control fd” that can be sendmsg()d to peers or inherited by subprocesses w/ SYSPROF_CONTROL_FD=n set allowing them to request a ring buffer (typically per-thread) to send accessory information which is muxed into the capture file.
Memory profilers (via LD_PRELOAD) which use the above to collect allocation records and callgraphs to visualize temporaries, leaks, etc.
Integration with language runtimes like GJS (GNOME’s SpiderMonkey used by GNOME shell and various applications) to use timers and SIGPROF to unwind/collect samples and mux into captures.
Integration with platform libraries like GLib, GTK, Mutter, and Pango which annotate the recordings with information about the GPU, input events, background operations, etc.
The ability to decode symbols at the tail of a recording and insert those mappings (including kallsyms) into the snapshot. This allows users to give me a .bz2 recording I can open locally without the same binary versions or Linux distribution.

This has been incredibly useful for us in GNOME because we can take someone who is having a problem, run Sysprof, and they can paste/upload the callgraph and we can address it quickly. It’s a major contributor to why we’ve been able to make GNOME so much faster in recent releases.

The Breakdown

Where this all breaks down is when we have poor/unreliable stack traces.

For example, most people want to view a callgraph upside down, starting from application’s entry points. If you can only unwind a few frames, you’re out of luck here because you can’t trace deep enough to reach the instruction pointer for main() (or similar).

You can, of course, ask perf to give you some 8 Kb of stack data on every sample. Sysprof does thousands of samples per second, so this grows quickly. Even more so the time to unwind it takes longer than reading this post. Nobody does this unless someone shows up with a
pile of money.

It’s such a problem that I made GLib/GTK avoid using libffi marshalers internally (which has exception unwind data but no frame pointers) so that it wouldn’t break Linux’s frame-pointer unwinder.

Beyond that, we started building the Flatpak org.freedesktop.Sdk with -fno-omit-frame-pointers so that we could profile software while writing it. (what a concept!) GNOME OS also is compiled with frame pointers so when really tricky things come up, many of us just use that instead of Fedora.

Yes there are cases where leaf functions are missed or not 100% correct, but it hasn’t been much of an issue compared to truncated stacks or stacks with giant holes at library boundaries because the Linux kernel frame-pointer unwinder fails.

Practical Solutions

It’s not that frame pointers are great or anything, it’s that reliability tends to be the most important characteristic. So many parts of our platform can cause profiling to give inaccurate results.

I’m not advocating for frame pointers, I’m advocating for “Works OOTB”. Fixing Python and BoringSSL to emit better instructions in the presence of frame pointers is minuscule compared to the alternatives suggested thus far.

Compiling our platforms with frame pointers is the single easiest thing we could do to ensure that we can make big wins going forward until we have a solution that reliably works across a number of failure scenarios I’ll layout below.

Profiling Needs

One necessity we have when doing desktop engineering is that some classes of problems occur in the interaction of components rather than one badly behaved component on it’s own.

Seeing profiling data across all applications, which may already be running, but also may be spawned by D-Bus or systemd during the profiling session is a must. Catching problematic processes during their startup (or failure to startup) is critical.

That means that pre-computing unwind tables is inherently unreliable for us unless we can stall any process until unwind tables are generated and uploaded for an eBPF unwinder. This may be possible, but in and of itself will skew some sorts of profiling results due to the additional latency as well as memory pressure for unwind tables (which are considerably larger than just emitting the frame pointers in the binaries .text section).

I suspect that doing something like QEMU’s live migration strategy may be an option, but again with all the caveats that it is going to perturb some sorts of results.

Load eBPF program to cause all remapping of pages that are X^W to SIGSTOP. Notify an agent to setup unwind tables.
Load unwind or DWARF data for all X^W pages mapped, generate system-wide tables
Handle incoming requests for SIGSTOP’d processes
Upload new unwind table data
Repeat from #3

But, even if this were a solution today, it has a number of situations that it flat out doesn’t handle well.

Current Hurdles

Startup of new processes incur latency, which for some workloads relying on fork()/exec() may perturb results especially for file-based work queue processing.
Static binaries are a thing now. Even beyond C/C++ both Rust and golang are essentially statically linked and increasing in use across all our tooling (podman, toolbx, etc) as well as desktop applications (gtk-rs).

This poses a huge issue. The amount of unwind data we need to load increases significantly because we can’t rely on MAP_SHARED from shared libraries to reduce the total footprint.

Again, we’re looking for whole system profiling here.

The tables become so large that they push out resident memory from the things you’re trying to profile to the point that you’re really not profiling what you think you are.

Containers, if allowed to version skew or if we’re unable to resolve mappings to the same inode, present a similar challenge (more below in Podman and how that is a disaster on Fedora today).

Thankfully from the Flatpak perspective, it’s very good at sharing inodes across the hard-link farm. However application binaries are increasingly Rust.

ORC overhead in the kernel comes in about 5Mb of extra memory at a savings of a few hundred Kb of frame pointer instructions. The value, of course, is that your instruction cache is tighter. Imagine fleets that are all static binaries and how intractable this becomes quickly. Machines at capacity will struggle to profile when you need it most.

Unwinding with eBPF appears to currently require exfiltrating the data via side-channels (BPF maps) rather than from the perf event stream. This can certainly be fixed with the right unwinder hooks in the kernel, but currently requires agents to not only setup unwind tables but to provide access to the unwind stacks too. My personal viewpoint is that these stacks should be part of the perf data stream, not a secondary data stream to be merged.

If we can’t do this thousands of times per second, it’s not fast enough.

If an RPM is upgraded, you lose access to the library mapped into memory from outside the process as both the inode and CRC will have changed on disk. You can’t build unwind tables, so accounting in that process breaks.

ELF/Dwarf Parsing and Privileged Processes

Parsing DWARF/.eh_frame data is very much an under researched problem by the security community. The process that sets up perf and/or BPF programs would need to do this so that unwind tables can be uploaded. You probably want the agent to be in control of that, but also very much want it sandboxed with something like bwrap (Bubblewrap) at minimum to protect the privileged agent.

Generating Missing .eh_frame Data

A fantastic entry from Oopsla 2019 talks about both validating .eh_frame data as well as synthesizing when missing by analyzing assembly instructions. This is very neat, but it also means that compiler tooling should be doing things like this to automatically generate proper .eh_frame data in the presence of inline assembly. Currently, you must get that correct by manually writing DWARF data in your assembly. Notable issues in both LLVM and glibc have created issues here.

Read more from the incredibly well written and implemented Oopsla 2019 submission [PDF].

libffi and .eh_frame

Libffi does dynamically generate enough information to unwind a stack in process across C++ exceptions. However, this is a lot more problematic if you have an agent generating unwind tables out of process. To get that data you have to map in user-space memory from the application (say /dev/$pid/mem) to access those pages and then trust that the memory isn’t malicious to the agent.

Blown FUSEs

Podman does this thing (at least for user-namespace containers on Fedora) where all the image content is served via FUSE. That means when you try to resolve the page table mappings in user-space to find the binary to locate symbols from, you are basically out of luck.

Sysprof goes through great pains by parsing layers of Podman’s JSON image information to discover what the mapping should have been. This is somewhat limited because we can only do it for the user that is running the sysprof client (as those stack frame instruction pointers are symbolized client-side in libsysprof). Doing this from an agent would require uploading that state to the agent to request integration into the unwind tables.

We have to do the same for flatpak, but thankfully /.flatpak-info contains everything we need to do that symbol resolution.

Subvolumes and OSTree deployments further complicate this matter because of indirection of mounts/mountinfo. We have to resolve through subvol=* for example to locate the correct file for the path provided in /proc/$pid/maps.

Again, since we need to build unwind tables up-front, this needs to be resolved when the profiler starts up. We can’t rely on /proc/$pid/mem because there is no guarantee the section will be mapped or that we’ll be able discover which map it was without the ELF header (which too may no longer be mapped). Since the process will likely have closed the FD after mmap(), we need to locate the proper files on disk.

Thankfully, in Sysprof we store the inode/CRC so that we can symbolize something useful if they’re incorrect, even if it’s a giant “Hey this is broken” callgraph entry.

In a world without frame-pointers, you have very little luck at making profilers reliably work in production unless you can resolve all of these issues.

There has been a lot of talk about how we can do new unwinders, and that is seriously great work! The issues above are real ones today and will become even bigger issues in the upcoming years and we’ll need all the creativity we can get to wrangle these things together.

Again, I don’t think any of us like frame pointers, just that they are generally lower effort today while also being reasonably reliable.

It might be the most practical near term solution to enable frame-pointers across Fedora today, while we push to get the rest of the system integrated to robustly support alternative unwinding capabilities.

Fiber examples and Windows support

I added a bunch of examples to help refine some of the libdex APIs.

Additionally, I added support for the Windows Fibers API which brings our support matrix up to Linux, macOS, FreeBSD (possibly some other BSDs too), Illumos, Hurd, and now Windows.

I know there are a few things I’d like to still add and change the APIs, but I guess now it’s at the point where it’s time to start building things with it. I’m likely to target pieces of Builder which have a lot of complex async callback chains. Those are likely to benefit the most from fibers.

Threading Fibers

Previously on Futures, Work-Stealing, and such.

One thing I admired during the early days of Python Twisted was how it took advantage of the language to write asynchronous code which read as synchronous. Pretty much all the modern languages now have some form of this in stackless or stackfull form.

C has been able to do it for ages but it generally doesn’t play well outside of constrained applications. The primary things you have to worry about are functional tooling like debuggers (thread apply all bt wont show you all your fibers) and features like thread-local-storage.

If you’re careful about when you suspend your fiber, the later isn’t so much of an issue. Where it can become a serious issue is if you do something like call an intermediate function which uses callbacks and the callback suspends. In this case, the intermediate function (out of your control) might have some TLS state cached on the stack, which of course could be modified before resume is called.

But either way, dex (wip title newlib) has support for fibers now which can be spawned using dex_scheduler_spawn(). Fiber stacks are given a guard page so that stack overflows are still guarded. It tries to do a bit of madvise() when it makes sense to.

That means I can finally write async code for GNOME Builder similar to:

/* Suspend fiber until read_bytes future completes */
g_autoptr(GBytes) bytes = dex_await_boxed (dex_input_stream_read_bytes (input, count, 0), &error);

/* Use bytes from above after resuming and suspend until write completes */
gssize n_written = dex_await_int64 (dex_output_stream_write_bytes (output, bytes, 0), &error);

There is also API to do this using plain file-descriptors which is backed by io_uring on Linux. However, that is still private API because I’m not sure how I want to expose it (if at all).

You can await any DexFuture sub-type which allows for complex composition. Fibers themselves are a DexFuture, which means you can await on another fiber completing as part of the composition.

Fibers can run on any scheduler of your choosing. They integrate into the GMainContext on that scheduler which may defer to a sub-scheduler. For example, the thread pool scheduler will pin the fiber to a sub-scheduler on a single worker thread.

I’ve tested it on Linux, macOS, FreeBSD 13 and Illumos (OpenIndiana) with success so far. The fiber suspend/resume scheduling is done with makecontext()/swapcontext() APIs but uses assembly form Russ Cox’s libtask for the places where that would otherwise incur an unnecessary syscall.

While porting to FreeBSD 13 I noticed that eventfd() is supported there now. Cool! Same goes for Illumos which apparently also supports epoll (we use io_uring, but still nice to see Linux APIs proliferate to ease porting efforts).

Concurrency, Parallelism, I/O Scheduling, Thread Pooling, and Work-Stealing

Around 15 years ago I worked on some interesting pieces of software which are unfortunately still not part of my daily toolbox in GNOME and GTK programming. At the time, the world was going through changes to how we would write thread pools, particularly with regards to wait-free programming and thread-pooling.

New trends like work-stealing were becoming a big thing, multiple-CPUs with multiple NUMA nodes were emerging on easy to acquire computers. We all were learning that CPU frequency was going to stall and that non-heterogeneous CPUs were going to be the “Next Big Thing”.

To handle those changes gracefully, we were told that we need to write software differently. Intel pushed that forward with Threading Building Blocks (TBB). Python had been doing things with Twisted which had an ergonomic API and of course “Stackless Python” and similar was a thing. Apple eventually came out with Grand Central Dispatch. Microsoft Research had the Concurrency and Coordination Runtime (CCR) which I think came out of robotics work.

Meanwhile, we had GThreadPool which honestly hasn’t changed that much since. Eventually the _async/_finish paring we’re familiar with today emerged followed by GTask to provide a more ergonomic API on top of it.

A bit before the GTask we all know today, I had written libgtask which was more of a Python Twisted-style API which provided “deferreds” and nice ways to combine them together. That didn’t come across into GLib unfortunately. To further the pain there, what we got in the form of GTask has some serious side-effects which make it unsuitable as a general construct in my humble opinion.

After realizing libgtask was eclipsed by GLib itself, I set off on another attempt in the form of libiris. That took a different approach that tried to merge the mechanics of CCR (message passing, message ports, coordination arbiters, actors), the API ergonomics of Python Twisted’s Deferred, and Apple’s NSOperation. It provided a wait-free work-stealing scheduler to boot. But it shared a major drawback of GLib’s GTask (beyond correctness bugs that plague it today). Primarily that thread pools can only process the work queue and therefore if you need to combine poll() or GSource that attach to a GMainContext you’re going to require code-flow to repeatedly bounce between threads.

This is because you can simplify a thread pool worker to while (has_work()) do_work ();. Any GSource or I/O either needs to bounce to the main thread where the applications GMainContext exists or to another I/O worker thread if doing synchronous I/O. On Linux, for a very long time, synchronous I/O was the “best” option if you wanted to actually use the page cache provided by the kernel, so that’s largely what GLib and GIO does.

The reason we couldn’t do something else is that to remove an item from the global work queue required acquiring a GMutex and blocking until an item is available. On Linux at least, we didn’t have APIs to be able to wait on a Futex while also poll()ing a set of file-descriptors. (As a note I should mention for posterity that FD_FUTEX was a thing for a short while, but never really usable).

In the coming years, we got a lot of new features in Linux that allowed improvements to the stack. We got signalfd to be able to poll() on incoming Unix signals. We got eventfd() which allowed a rather low-overhead way to notify coordinating code with a poll()able file-descriptor. Then EFD_SEMAPHORE was added so that we can implement sem_t behavior with a file-descriptor. It even supports O_NONBLOCK.

The EFD_SEMAPHORE case is so interesting to me because it is provides the ability to do something similar to what IRIX did 20+ years ago, which is a pollable semaphore! Look for usnewpollsema() if you’re interested.

There was even some support in GLib to support epoll(), but that seems to have stalled out. And honestly, making it use io_uring might be smarter option now.

After finishing the GTK 4 port of GNOME Builder I realized how much code I’ve been writing in the GAsyncReadyCallback style. I don’t particularly love it and I can’t muster the energy to write more code in that style. I feel like I’m writing top-half/bottom-half interrupt handlers yet I lack the precision to pin things to a thread as well as having to be very delicate with ownership to tightly control object finalization. That last part is so bad we basically don’t use GLib’s GTask in Builder in favor of IdeTask which is smart about when and where to release object references to guarantee finalization on a particular thread.

One thing that all these previous projects and many hundreds of thousands of async-oriented C code written has taught me is that all these components are interlinked. Trying to solve one of them without the others is restrictive.

That brings me to 2022, where I’m foolishly writing another C library that solves this for the ecosystem I care most about, GTK. It’s goal is to provide the pieces I need in both applications as well as toolkit authoring. For example, if I were writing another renderer for GTK, this time I’d probably built it on something like this. Given the requirements, that means that some restrictions exist.

I know that I need GMainContext to work on thread pool threads if I have any hope of intermixing workloads I care about on worker threads.
I 100% don’t care about solving distributed computing workloads or HTTP socket servers. I refuse to race for requests-per-second at the cost of API ergonomics or power usage.
I know that I want work stealing between worker threads and that it should be wait-free to avoid lock contention.
Worker threads should try to pin similar work to their own thread to avoid bouncing between NUMA nodes. This increases data cacheline hits as well as reduces chances of allocations and frees moving between threads (something you want minimized).
I know that if I have 32 thread pool threads and 4 jobs are added to the global queue, I don’t want 32 threads waking up from poll() to try to take that those work items.
The API needs to be simple, composable, and obvious. There is certainly a lot of inspiration to be taken from things like std::future, JavaScript’s Promise, Python’s work on deferred execution.
GObject has a lot of features and because of that it goes through great lengths to provide correctness. That comes at great costs for things you want to feel like a new “primative”, so avoiding it makes sense. We can still use GTypeInstance though, following in the footsteps of GStreamer and GTK’s Scene Kit.
Cancellation is critical and not only should it cause the work you created to cancel, that cancellation should propagate to the things your work depended on unless non-cancelled work also depends on it.

I’ve built most of the base of this already. The primary API you interact with is DexFuture and there are many types of futures. You have futures for IO (using io_uring). Futures for unix signals. A polled semaphore where dex_semaphore_wait() is a future. It can wrap _async/_finish pairs and provide the result as a Future. Thread pools are efficient in waiting on work (so staying dormant until necessary and minimal thread wake-ups) while also coordinating to complete work items.

There is still a lot of work to go, and so far I’ve only focused on the abstractions and Linux implementations. But I feel like there is promise (no pun intended) and I’m hoping to port swaths of code in Builder to this in the coming months. If you’d like to help, I’d be happy to have you, especially if you’d like to focus on alternate DexAioBackends, DexSemaphore using something other than eventfd() on BSD/Solaris/macOS/Windows, and additional future types. Additionally, working to GLib to support GMainContext directly using io_uring would be appreciated.

You can find the code here, but it will likely change in the near future.

GNOME Builder 43.0

After about 5 months of keeping myself very busy, GNOME Builder 43 is out!

This is the truly the largest release of Builder yet, with nearly every aspect of the application improved. It’s pretty neat to see all this come together after having spent the past couple years doing a lot more things outside of Builder like modernizing GTKs OpenGL renderer, writing the new macOS GDK backend, shipping a new Text Editor for GNOME, and somehow getting married during all that.

Modern and Expressive Theming

The most noticeable change, of course, is the port to GTK 4. Builder now uses WebKit, VTE, libadwaita, libpanel, GtkSourceView, and many other libraries recently updated to support GTK 4.

Like we did for GNOME Text Editor, Builder will restyle the application window based on the syntax highlighting scheme. In practice this feels much less jarring as you use the application for hours.

The Foundry

Behind the scenes, the “Foundry” behind Builder has been completely revamped to make better use of SDKs and runtimes. This gives precise control over how processes are created and run. Such control is important when doing development inside container technologies.

Users can now define custom “Commands” which are used to run your project and can be mapped to keyboard shortcuts. This allows for the use of Builder in situations where it traditionally fell short. For example, you can open a project without a build system and use commands to emulate a build system.

Furthermore, those commands can be used to run your application and integrate with tooling such as the GNU debugger, Valgrind, Sysprof, and more. Controlling how the debugger was spawned has been a long requested feature by users.

You can control what signal is sent to stop your application. I suspect that will be useful for tooling that does cleanup on signals like SIGHUP. It took some work but this is even plugged into “run tools” so things like Sysprof can deliver the signal to the right process.

If you’re using custom run commands to build your project you can now toggle-off installation-before-run and likely still get what you want out of the application. This can be useful for very large projects where you’re working on a small section and want to cheat a little bit.

Unit Testing

In previous version of Builder, plugins were responsible for how Unit Tests were run. Now, they also use Run Commands which allows users to run their Unit Tests with the debugger or other tooling.

Keyboard Shortcuts

Keyboard shortcuts were always a sore spot in GTK 3. With the move to GTK 4 we redesigned the whole system to give incredible control to users and plugin authors. Similar to VS Code, Builder has gained support for a format similar to “keybindings.json” which allows for embedding GObject Introspection API scripting. The syntax matches the template engine in Builder which can also call into GObject Introspection.

Command Bar and Project Search

We’ve unified the Command Bar and Project Search into one feature. Use Ctrl+Enter to display the new Global Search popover.

We do expect this feature to be improved and expanded upon in upcoming releases as some features necessary are still to land within a future GTK release.

Movable Panels and Sessions

Panels can be dragged around the workspace window and placed according to user desire. The panel position will persist across future openings of the project.

Additionally, Builder will try to save the state of various pages including editors, terminals, web browsers, directory listings, and more. When you re-open your project with Builder, you can expect to get back reasonably close to where you left off.

Closing the primary workspace will now close the project. That means that the state of secondary workspaces (such as those created for an additional monitor) will be automatically saved and restored the next time the project is launched.

GtkSourceView

Core editing features have been polished considerably as part of my upstream work on maintaining GtkSourceView. Completion feels as smooth as ever. Interactive tooltips are polished and working nicely. Snippets too have been refined and performance improved greatly.

Not all of our semantic auto-indenters have been ported to GtkSourceIndenter, but we expect them (and more) to come back in time.

There is more work to be done here, particularly around hover providers and what can be placed in hover popovers with expectation that it will not break input/grabs.

Redesigned Preferences

Preferences have been completely redesigned and integrated throughout Builder. Many settings can be tweaked at either the application-level as a default, or on a per-project basis. See “Configure Project” in the new “Build Menu” to see some of those settings. Many new settings were added to allow for more expressive control and others improved open.

Use Ctrl+, to open application preferences, and Alt+, to open your project’s preferences and configurations.

Document Navigation

Since the early versions of Builder, users have requested tabs to navigate documents. Now that we’re on GTK 4 supporting that in a maintainable fashion is trivial and so you can choose between tabs or the legacy “drop down” selector. Navigation tabs are enabled by default.

Some of the UI elements that were previously embedded in the document frame can be found in the new workspace statusbar on the bottom right. Additionally, controls for toggling indentation, syntax, and encoding have been added.

Switching between similar files is easy with Ctrl+Shift+O. You’ll be displayed a popover with similarly named files to the open document.

The symbol tree is also still available, but moved to the statusbar. Ctrl+Shift+K will bring it up and allow for quick searching.

WebKit

A new web browser plugin was added allowing you to create new browser tabs using Ctrl+Shift+B. It is minimal in features but can be useful for quick viewing of information or documentation.

Additionally, the html-preview, markdown-preview, and sphinx-preview plugins have been rewritten to build upon this WebKit integration.

Plugin Removals

Some features have been removed from Builder due to the complexity and time necessary for a proper redesign or port. The Glade plugin (which targets GTK 3 only) has been removed for obvious reasons. A new designer will replace it and is expected as part of GNOME 44.

Devhelp has also been removed but may return after it moves to supporting GTK 4. Additionally, other tooling may supersede this plugin in time.

The code beautifier and color-picker were also removed and will likely return in a different form in future releases. However, language servers providing format capabilities can be enabled in preferences to format-on-save.

Project Templates

Project templates have been simplified and improved for GTK 4 along with a new look and feel for creating them. You’ll see the new project template workflow from the application greeter by clicking on “Create New Project”.

Top Matches

Heavy users of code completion will notice a new completion result which contains a large star (★) next to it. This indicates that the proposal is a very close match for the typed text and is getting resorted to the top of the completion results. This serves as an alternative to sorting among completion providers which is problematic due to lack of common scoring algorithms across different data sources.

Sysprof Integration

Tooling such as Sysprof went through a lot of revamp too. As part of this process I had to port Sysprof to GTK 4 which was no small task in it’s own right.

Additionally, I created new tooling in the form of sysprof-agent which allows us to have more control when profiling across container boundaries. Tools which need to inject LD_PRELOAD (such as memory profilers) now work when combined with an appropriate SDK.

Language Servers

Language servers have become a part of nearly everyone’s development toolbox at this point. Builder is no different. We’ve added support for a number of new language servers including jdtls (Java), bash-language-server (Bash), gopls (Golang) and improved many others such as clangd (C/C++), jedi-language-server (Python), ts-language-server (JavaScript/Typescript), vls (Vala), rust-analyzer (Rust), blueprint, and intelephense (PHP).

Many language servers are easier to install and run given the new design for how cross-container processes are spawned.

Quick Settings

From the Run Menu, many new quick settings are available to tweak how the application runs as well as well as configure tooling.

For example, you can now toggle various Valgrind options from the Leak Detector sub-menu. Sysprof integration also follows suit here by allowing you to toggle what instruments will be used when recording system state.

To make it easier for developers to ensure their software is inclusive, we’ve added options to toggle High Contrast themes, LTR vs RTL, and light vs dark styling.

Refactory

For language tooling that supports it, you can do things like rename symbols. This has been in there for years, but few knew about it. We’ve elevated the visibility a bit now in the context menu.

Vim Emulation

In GTK 3, we were very much stuck with deep hacks to make something that looked like Vim work. Primarily because we wanted to share as much of the movements API with other keybinding systems.

That changed with GtkSourceView 5. Part of my upstream maintainer work on GtkSourceView included writing a new Vim emulator. It’s not perfect, by any means, but it does cover a majority of what I’ve used in more than two decades as a heavy Vim user. It handles registers, marks, and tries to follow some of the same pasteboard semantics as Vim (“+y for system clipboard, for example).

I made this available in GNOME Text Editor for GNOME 42 as well. Those that wonder why we didn’t an external engines to synchronize with, can read the code to find out.

Plugins

We have been struggling with our use of PyGObject for sometime. It’s a complex and difficult integration project and I felt like I spent more time debugging issues than I was comfortable with. So this port also included a rewrite of every Python-based plugin to C. We still enable the Python 3 plugin loader from libpeas (for third-party plugins), but in the future we may switch to another plugin language.

Maintainers Corner

So…

A special thanks to all those that sent me merge requests, modernized bits of software I maintain, fixed bugs, or sent words of encouragement.

I’m very proud of where we’ve gotten. However, it’s been an immense amount of work. Builder could be so much more than it is today with your help with triage of bugs, designing and writing features, project and product management, writing documentation, maintaining plugins, improving GNOME OS, and everything in-between.

The biggest lesson of this cycle is how a strong design language is transformative. I hope Builder’s transformation serves as an example for other GNOME applications and the ecosystem at large. We can make big leaps in short time if we have the right tooling and vision.

Builder 43.alpha0

It’s been an absolute mad dash this cycle porting Builder to GTK 4, but 43.alpha0 is out and available on GNOME Nightly.

Builder is one of the larger applications within GNOME, especially if you include the libraries I had to write and maintain to make that possible. Porting an application to a new toolkit is always a big undertaking. However, it also provides an opportunity to rethink how major components work and simplify them while you’re there.

So that is what has happened this cycle. It’s going to end up being a much more polished product due to the enormous amount of simplification going on.

GTK 4 has simplified how a lot of things work and provided APIs that feel so obvious when you use them. Of course, that also means lots of code needs to be changed (well deleted, mostly). Having focused heavily on using GListModel in previous releases also paid off massively this cycle.

Anyway, here it is. It’s still missing plenty of features as I dash towards the finish line implementing them as quick as I can.

For those that want to test it out, note that our application-id has changed so that you can install Builder’s stable branch and nightly branch side-by-side.

flatpak --user remote-add --if-not-exists gnome-nightly \
  https://nightly.gnome.org/gnome-nightly.flatpakrepo
flatpak --user install gnome-nightly org.gnome.Builder.Devel

Builder GTK 4 Porting, Part VII

It’s been another couple weeks of porting, along with various distractions.

The big work this time around has been deep surgery to Builder’s “Foundry”. This is the sub-system that is responsible for build-systems, pipelines, external-devices, SDKs, toolchains, deployments-strategies and more. The sub-system was starting to show it’s age as it was one of the first bits of Builder to organically emerge.

One of the things that become so difficult over the years is dealing with all the container layers we have to poke holes through. Running a command is never just running a command. We have to setup PTYs (and make sure the TTY setup ioctl()s happen in the right place), pass environment variables (but to only the right descendant process), and generally a lot more headaches.

What kicked off this work was my desire to remove a bunch of poorly abstracted bits and we’re almost there. What has helped considerably is creating a couple new objects to help manage the process.

The first is an IdeRunContext. It is sort of like a GSubprocessLauncher but allows you to create layers. At the end you can convert those layers into a subprocess launcher but only after each layer is allowed to rewrite the state as you pop back to the root. In practice this has been working quite well. I finally have control without crazy amounts of argument rewriting and guesswork.

To make that possible, I’ve introduced an IdeUnixFDMap which allows to manage source↔dest FD translations for FDs that will end up in the subprocess. It has a lot of helpers around it to make it fit well into the IdeRunContext world.

All of this has allowed the new IdeRunCommand to really shine. We have various run command providers (e.g. plugins) all of which can seamlessly be used across the sub-systems supporting IdeRunContext. Plugins such as meson can even export unit tests as run commands.

The shellcmd plugin has also been rewritten upon these foundations. You can create custom commands and map them to keyboard shortcuts. The commands, like previous version of Builder, can run in various localities. A subprocess, from the build pipeline, as an app runner, or on the host. What has improved, however, is that they can also be used in surrogate of your projects run command. These two features combined means you can make Builder work for a lot of scenarios it never did before by configuring a few commands.

There aren’t a lot of screenshots for things like this, because ideally it doesn’t look too different. But under the hood it’s faster, more reliable, and far more extensible than it was previously. Hopefully that helps us cover a number of highly requested use-cases.

Builder GTK 4 Porting, Part VI

Short update this week given last Monday was Memorial Day in the US. I had a lovely time relaxing in the yard and running errands with my wife Tenzing. We’ve been building such a beautiful home together that it’s nice to just sit back and enjoy it from time to time.

GTK

Merged some work on debug features for testing RTL vs LTR from Builder. There is a new GTK_DEBUG=invert-text-dir to allow rudimentary testing with alternate text directions.

Builder

Landed a new clone design using libadwaita.
Fixed rendering of symbolic icons in the gutter for diagnostics, etc
Fixed error underlines for spellcheck when dealing with languages where the glyph baseline may change
Added a new IdeVcsCloneRequest which can do most of the clone work so the UI bits can be very minimal.
Added interfaces to allow for retrieving a list of branches on a remote before you’ve cloned it. Useful to help selecting an initial branch, but do to how libgit2 works, we have to create a temporary directory to make it work (and then unlink it). Handy nonetheless.
Make gnome-builder --clone work again.
Make cloning newcomer applications automatically work again.
Made a lot of our popover’s use menu styling, despite being backed by GListModel and GtkListView.
Even more menuing cleanups. Amazing how each pass of this really tends to clarify things from a user perspective.
Made all of the editor menu buttons in the statusbar functional now.
New gsetting and preference toggle to set default license for new projects.
A new IdeWebkitPage page implementation which is a very rudimentary web-browser. This will end up being re-used by the html-preview, markdown-preview, and sphinx plugins.
Removed the glade plugin
Fixed presentation of clang completion items.

I’m pretty satisfied with the port of the cloning workflow, but it really needs to have a PTY plumbed through to the peer process so we can get better/more complete information. We’ll see if there is time before 43 though given how much else there is to get done.

All of this effort is helping me get a more complete vision of what I’d like to see out of a GTK 5. Particularly as we start attacking things from a designer tooling standpoint.