Builder GTK 4 Porting, Part VII

It’s been another couple weeks of porting, along with various distractions.

The big work this time around has been deep surgery to Builder’s “Foundry”. This is the sub-system that is responsible for build-systems, pipelines, external-devices, SDKs, toolchains, deployments-strategies and more. The sub-system was starting to show it’s age as it was one of the first bits of Builder to organically emerge.

One of the things that become so difficult over the years is dealing with all the container layers we have to poke holes through. Running a command is never just running a command. We have to setup PTYs (and make sure the TTY setup ioctl()s happen in the right place), pass environment variables (but to only the right descendant process), and generally a lot more headaches.

What kicked off this work was my desire to remove a bunch of poorly abstracted bits and we’re almost there. What has helped considerably is creating a couple new objects to help manage the process.

The first is an IdeRunContext. It is sort of like a GSubprocessLauncher but allows you to create layers. At the end you can convert those layers into a subprocess launcher but only after each layer is allowed to rewrite the state as you pop back to the root. In practice this has been working quite well. I finally have control without crazy amounts of argument rewriting and guesswork.

To make that possible, I’ve introduced an IdeUnixFDMap which allows to manage source↔dest FD translations for FDs that will end up in the subprocess. It has a lot of helpers around it to make it fit well into the IdeRunContext world.

All of this has allowed the new IdeRunCommand to really shine. We have various run command providers (e.g. plugins) all of which can seamlessly be used across the sub-systems supporting IdeRunContext. Plugins such as meson can even export unit tests as run commands.

The shellcmd plugin has also been rewritten upon these foundations. You can create custom commands and map them to keyboard shortcuts. The commands, like previous version of Builder, can run in various localities. A subprocess, from the build pipeline, as an app runner, or on the host. What has improved, however, is that they can also be used in surrogate of your projects run command. These two features combined means you can make Builder work for a lot of scenarios it never did before by configuring a few commands.

There aren’t a lot of screenshots for things like this, because ideally it doesn’t look too different. But under the hood it’s faster, more reliable, and far more extensible than it was previously. Hopefully that helps us cover a number of highly requested use-cases.

a screenshot of the debugger

a screenshot of the build menu with debug selected

a screenshot of the run command selection selection dialog

a screenshot showing the location of the select run command menu item

a screenshot editing a command

Builder GTK 4 Porting, Part IV

This week was a little slower as I was struggling with an adjustment to my new medication. Things progress nonetheless.

Text Editor

I spent a little time this week triaging some incoming Text Editor issues and feature requests. I’d really like this application to get into maintenance mode soon because I have plenty of other projects to maintain.

  • Added support for gnome-text-editor - to open a file from standard input, even if you’re communicating to a single instance application from terminal.
  • Branch GNOME 42 so we can add new strings.
  • Fix a no-data-loss crash during shutdown.

Template-GLib

  • Fix template evaluation on macOS.
  • Make boolean expression precedence more predictable.
  • Cleanup output of templates with regards to newlines.

libpanel

  • Propagate modified page state to tabs
  • Some action tweaks to make things more keyboard shortcut friendly.

Builder

  • Merged support for configuration editing from Georges.
  • Add lots of keybindings using our new keybinding engine.
  • Track down and triage that shortcut controllers do not capture/bubble to popovers. Added workarounds for common popovers in Builder.
  • Teach Builder to load keybindings from plugins again and auto-manage them.
  • Lots of tweaks to the debugger UI and where widgetry is placed.
  • Added syntax highlighting for debugger disassembly.
  • Added menus and toggles for various logging and debugger hooks. You can get a breakpoint on g_warning() or g_critical() by checking a box.
  • Ability to select a build target as the default build target finally.
  • More menuing fixes all over the place, particularly with treeviews and sourceviews.
  • Fix keyboard navigation and activation for the new symbol-tree
  • Port the find-other-file plugin to the new workspace design which no longer requires using global search.
  • GTK 4 doesn’t appear to scroll to cells in textview as reliably as I’d like, so I dropped the animation code in Builder and we jump strait to the target before showing popovers.
  • Various work on per-language settings overrides by project.
  • Drop the Rust rls plugin as we can pretty much just rely on rust-analyzer now.
  • Lots of CSS tweaks to make things fit a bit better with upcoming GNOME styling.
  • Fix broken dialog which prevented SDK updates from occurring with other dependencies.

A screenshot of builder's find-other-file plugin

A screenshot of Builder's debugger

A screenshot showing the build target selection dialog

A screenshot of the run menu A screenshot of the logging menu

Builder GTK 4 Porting, Part II

Another week of work towards porting Builder to GTK 4. Since I can’t add to TWIG from IRC, I’ll try harder to drop some occasional updates here.

GtkSourceView

  • Merged fixes for highlighting unicode literals for C
  • Improved parsing of language values in snippet bundles
  • GtkSourceGutter will not correctly mark prelit and selection quarks within GtkSourceGutterLines.
  • Fixed a bunch of little mouse pointer annoyances when using GtkSourceHover interactive tooltips.
  • GtkSourceGutterRenderers can now opt-out of signal emission for GtkSourceGutterRenderer::query_data(). Signal emission with GObject is rather slow, so avoiding it on every line can be helpful. Just set the virtual method pointer to NULL. The signal was only ever added to make porting easier anyway.

libpanel

  • Merged fixes to be used as a subproject with static libraries only
  • CSS styling matches recent changes in libadwaita, particularly around making navigation tabs and panel frame headers more unified.
  • PanelWidget no longer uses a GtkBinLayout so that it’s easier for subclasses to integrate with popovers from size_allocate() to call gtk_popover_present().

Builder

  • Georges did a live coding stream where they ported a bunch of the “buildui” plugin. That is merged to the GTK 4 port now. It brings a number of features back to the UI including the build terminal, run terminal, build panel (with pipeline stages, warnings, and errors), and project information in the omnibar popover.
  • Günther did a bunch of work porting our old snippet files to the new XML-based snippet bundle format upstream in GtkSourceView. Along with that came porting of the snippets plugin for Builder’s new GTK 4-based editor.
  • Workspace windows have a bit better predictability when restoring sizes.
  • The project creation workflow was ported, albeit needs to have the redesign implemented still.
  • Lots of resiliency fixes for clang and symbol-tree plugins to improve life-cycle management.
  • The Valgrind plugin was ported to C. This was also back-ported to main because it fixed a number of oddities (crashes) occurring in PyGObject.
  • Builder’s “omni-gutter” GtkSourceGutterRenderer was ported to GTK 4 and got a lot of cleanups along the way. I believe there is still some outstanding things to fix such as handling rendering for symbolic icons as I’m pretty sure that’s not correct yet.
  • The “debuggerui” plugin has been ported to GTK 4 and appears to be working well now. This plugin is responsible for bridging the internal IdeDebugger interfaces to the UI interface.
  • Style schemes were updated for GtkSourceView 5
  • Tabs are now the default navigation interface for pages. There is likely still a lot to do around how we want empty frames to look and behave though.
  • The project-tree is now most ported, and with some workarounds to allow making GtkPopover work you can actually display popovers and activate menu items.
  • A long-standing plugin ordering issue has been fixed so that gtk/menus.ui embedded in plugin resources have menu-merging in the proper order.
  • Some incremental work landed to support per-project configuration of languages.
  • The “ls” plugin (directory views) supports “humanized” timestamps again and columns are resizable once more.
  • You can place panels in the right-side-bar now.

Upcoming

There are a bunch of foundational things to still get landed before I feel I can get Builder flipped over to our Nightly builds. In particular we need to land support for things like:

  • Keyboard shortcuts using GtkShortcutController. This was all done with libdazzle previously.
  • Allow plugins to define custom keyboard shortcuts and merge them into the controller.
  • Port “shellcmd” plugin and integrate keyboard shortcuts to apply those commands.
  • Finish rewrite of the search popover. I’m trying to delay this until GtkListView with sections is supported, as it would provide us a much greater path for performance.
  • A lot of our problems would be simpler if we could make GtkActionMuxer use an alternate action muxer parent from another (non-descendant) widget.
  • Configuration editing in the project configuration window. This is a new design so the port is not 1:1.

A screenshot of Builder with the project-tree context menu and debugger on display

A screenshot of Builder with various panels moved around to non-standard locations

GtkSourceView Interactive Tooltips

During the past years (and especially last cycle) I’ve worked to push a number of features upstream from Builder into GtkSourceView. Not only does this improve the ecosystem for all applications, but it reduces the number of things we need to maintain downstream in Builder as we move to GTK 4.

That included a new completion engine, a new snippet engine with tooltips based on expansion points, updated gutter and gutter renderer designs, the editor overview map, background patterns, sysprof tracing integration, and most recently interactive tooltips.

You can read our nightly generated documentation to learn about it. In particular, GtkSourceHover is attached to the GtkSourceView and can have GtkSourceHoverProviders registered with it. Implementing that interface will be familiar to those who’ve implemented completion providers, as they work in a similar way.

This came from Builder where we have interactive tooltips for things like breakpoints, documentation, and symbol information. Other applications using GtkSourceView may find it useful when implementing Language Server Protocol’s hover providers.

Rust and GNOME Builder

I still spend most of my day writing C and I doubt that is going to change any time soon. But that doesn’t mean you should have to!

Builder got a number of late arriving improvements around Rust support, so now would be a good time to go test them before the 40 release is out.

Yesterday I landed a long awaited feature that will find the common Flatpak SDK ancestors. This was needed to resolve the branch name for SDK extensions like org.freedesktop.Sdk.Extension.rust-stable. Your project might use org.gnome.Sdk//3.38 but the branch for rust-stable is 20.08 (coming from org.freedesktop.Sdk//20.08). Terribly annoying, but hey, now it’s fixed.

Furthermore, Builder can use rust-analyzer¹ as bundled by the org.freedesktop.Sdk.Extension.rust-stable SDK so that is one less thing you need to install or manage. It will pick up all the same dependencies as your project because it will run from within your projects build container. It will see everything your build system does, and in the same way.

Just create a new project using the “GNOME Application” template, select “Rust” as the language, and Run.

¹ rust-analyzer provides diagnostics, auto-completion, and more.

GtkSourceView Next

Earlier this year I started a branch to track GTK 4 development which is targeted for release by end-of-year. I just merged it which means that our recently released gtksourceview-4-8 branch is going to be our LTS for GTK 3. As you might remember from the previous maintainer, GtkSourceView 4.x is the continuation of the GtkSourceView 3.x API with all the deprecated API removed and a number of API improvements.

Currently, GtkSourceView.Next is 5.x targeting the GTK 4.x API. It’s a bit of an unfortunate number clash, but it’s been fine for WebKit so we’ll see how it goes.

It’s really important that we start getting solid testing because GtkSourceView is used all over the place and is one of those “must have” dependencies when moving to a new GTK major ABI.

Preparations in GTK 4

Since I also spend time contributing to GTK, I decided to help revamp GtkTextView for GTK 4. My goal was to move various moving parts into GtkTextView directly so that we could make them more resilient.

Undo Support

One feature was undo support. GTK 4 now has native support for undo by implementing text history in a compact form within GTK itself. You can now set the enable-undo properties to TRUE on GtkTextView, GtkEditable widgets like GtkText or GtkEntry, and others.

GPU Rendered Text (sort of)

Matthias Clasen and I sat down one afternoon last year and wrote a new PangoRenderer for GSK using render nodes and the texture atlas provided by the OpenGL and Vulkan renderers. Since then, GtkTextView gained a GtkTextLineDisplay cache so that we can keep these immutable render nodes around across multiple snapshots.

Text is still rendered on the CPU into a texture atlas, which is uploaded to the GPU and re-used when possible. Maybe someday things like pathfinder will provide a suitable future.

GtkTextView and Widgets

Previously, the gutters for GtkTextView were simply a GdkWindow which could be rendered to with Cairo. This didn’t fit well into the “everything should be a widget” direction for GTK 4. So now you can pack a widget into each of the 4 gutters around the edges of a GtkTextView. This means you can handle input better too using GtkGesture and GtkEventControllers. More importantly, though, it means you can improve performance of gutter rendering using snapshots and cached render nodes when it makes sense to do so.

Changes in GtkSourceView Next

Moving to a new major ABI is a great time to do cleanups too as it will cause the least amount of friction. So I took this opportunity to revamp much of the GtkSourceView code. We follow more modern GObject practices and have bumped our compiler requirements to closely match GTK 4 itself. This still means no g_autoptr() usage from within GtkSourceView sadly thanks to MSVC being … well the worse C compiler still in wide use.

GtkSourceGutterRenderer is now a GtkWidget

Now that we have margins which can contain widgets and contribute to the render node tree, both GtkSourceGutter and GtkSourceGutterRenderer are GtkWidget. This will mean you need to change custom gutter renderers a bit, but in practice it means a lot less code than they previously contained. It also makes supporting HiDPI much easier.

GtkSourceCompletion Revamp

I spent a lot of time making completion a pleasing experience in GNOME Builder and that work has finally made it upstream. To improve performance and simplicity of implementation, this has changed the GtkSourceCompletionProvider and GtkSourceCompletionProposal interfaces in significant ways.

GtkSourceCompletionProposal is now a mostly superfluous type used to denote a specialized GObject. It doesn’t have any functions in the vtable nor any properties currently and the goal is to avoid adding them. Simply G_IMPLEMENT_INTERFACE (GTK_SOURCE_TYPE_COMPLETION_PROPOSAL, NULL) when defining your proposal object GType.

This is because all of the completion provider implementation can now be performed from GtkSourceCompletionProvider. This interface focus on using interfaces like GListModel (like the rest of GTK 4) and how to asynchronously generate and refine the results with additional key-presses.

The completion window has been revamped and now allows proposals to fill a number of columns including an icon, return-type (Left Hand Side), Typed Text, and supplementary text. It resizes with content and ensures that we only inflate the number of GObjects necessary to view the current set. A fixed number of widgets are also created to reduce CSS and measurement costs.

Further, proposals may now have “alternates” which allows for providers to keep all of the DoSomething() proposals with 20 overloaded forms for each base type in whatever language of the day is being used from clogging up the suggestions.

The new GtkSourceCompletionCell widget is a generic container used throughout completion for everything from containing icons, text, or even custom widgetry for the completion details popover.

Completion Preview

GtkSourceGutterLines

A new abstraction, GtkSourceGutterLines, was added to help reduce overhead in generation of content in the gutter. The design of gutters lead to an exorbitant amount of measurement work on every frame. This was actually the biggest hurdle in making GTK 3 applications scroll smoothly. The new design allows for all the renderers to collect information about lines in one pass (along with row height measurements) and then snapshot in their second pass. Combined with the ability to cache render nodes, gutter renderers should have what they need to remain fast even in HiDPI environments.

The implementation of this also has a few nice details to further reduce overhead, but I’ll leave that to those interested in reading the code.

GtkSourceBuffer::cursor-moved

GtkSourceBuffer now has a cursor-moved signal. This seemed to be something implemented all over the place so we might as well have it upstream.

Reduce signal emission overhead

A number of places have had signal emission overhead reduced. Especially in property notifications.

Spaces Drawing

The GtkSourceSpaceDrawer now caches render nodes for drawing spaces. This should improve the performance in the vast majority of cases. However, one case still could be improved upon: tabs when the tab width changes (generally when used after text or spaces).

New Features

Snippets

A new snippet engine has landed based on a much improved version from GNOME Builder. You can provide bundles using an XML snippets file. You can also create them dynamically from your application and insert them into the GtkSourceView. In fact, many completion providers are expected to do this.

The snippet language is robust and shares many features and implementation details from GNOME Builder.

Assistants

A new subsystem, GtkSourceAssistant is used to provide accessory information in a GtkSourceView. Currently this type is private and an implementation detail. However, GtkSourceCompletion and GtkSourceSnippet build upon it to provide some of their features. In the long term, we expect hover providers to also take advantage of this subsystem.

Sysprof Support

GtkSourceView now uses the Sysprof collector API just like GTK 4 does (among many other GNOME projects). This means you can get profiling information about renderings right in the Sysprof visualizer along other data.

Future Work

PCRE2

With GRegex on the chopping block for deprecation, it’s time to start moving to PCRE2 much like VTE did. Doing so will not only make us more deprecation safe, but ensure that we can actually use the JIT feature of the regex engine. With how much regexes are used by the highligting engine, this should be a fairly sizable improvement.

This has now been implemented.

Hover Providers

In GNOME Builder, we added an abstraction for “Hover Providers”. This is also a thing in the Language Server Protocol realm. Nothing exists upstream in GtkSourceView for this and that should probably change. Otherwise all the trickyness in making transient popovers work is put on application authors.

Style Schemes

I would like to remove or revamp some of our default style schemes. They do not handle the world of dyanmic GTK themes so well and become a constant source of bug reports by applications that want a “one size fits all” style scheme. I’m not sure yet on the complete right answer long term here, but my expectation is that we’d want to move toward a default style scheme that is mostly font changes rather than color changes which eventually fall apart on the more … interesting themes.

Anyway, that’s all for now!

GtkSourceView Snippets

I’m trying to blog about every week now this year, so here we go again.

The past week I’ve been pushing hard on finishing up the snippets work for the GTK 4 port. It’s always quite a bit more work to push something upstream because you have to be so much more complete while being generic at the same time.

I think at this point though I can move on to other features and projects as the branch seems to be in good shape. I’ve fixed a number of bugs in the GTK 4 port along the way and made tests, documentation, robustness fixes, style-scheme integration, a completion provider, file-format and parser, and support for layering snippet files the same way style-schemes and language-specs work.

As part of the GTK 4 work I’ve spent a great deal time modernizing the code-base. Now that we can depend on the same things that GTK 4 will depend on, we can use some more modern compiler features. Additionally, GObject has matured so much since most of the library was written and we can use that to our advantage.

GtkSourceView branching

Branching

We’re currently finishing up the cycle towards GNOME 3.36, which means it’s almost time to start branching and thinking about what we want to land early in the 3.37 development cycle. My goal is to branch gtksourceview-4-6 which will be our long-term stable branch for gtksourceview-4.x (similar to how the gnome-3-24 branch is our long-term stable for the gtksourceview-3.x series. After that, master will move to GTK 4 as we start to close in on GTK 4 development. The miss-alignment in version numbers is an unfortunate reality, but a reality I inherited so we’ll keep on keepin’ on.

That means if you are not setting a branch in your flatpak manifests, you will want to start doing that when we branch (probably in the next couple of weeks) or your builds will start to fail. Presumably, this only will affect your Nightly builds, because who targets upstream master in production builds, not you surely!

Snippets

I’ve started moving some features from Builder into GtkSourceView. However, I’m limiting those to the GTK 4 port because I don’t particularly want to add new ABI right before putting a branch into long-term stability mode. The first to be uplifted into GtkSourceView is Builder’s snippet engine. It also went through quite a bit of rewrite and simplification as part of this process to make it more robust. Furthermore, having moved undo/redo into GtkTextBuffer directly has done wonders from a correctness standpoint. The snippet engine used to easily be confused by the undo/redo engine.

The most difficult part of the snippet engine is dealing with GtkTextMark that are adjacent. In particular if you have each snippet focus-position wrapped in marks. Adjacent, empty mark ranges can end up overlapping each other and you have to be particularly diligent to prevent that. But the code in the branch has a pretty good handle on that, much better than what I had done in the past inside of Builder.

The bits I still need to do to finish up the snippet engine:

  • Land the GTK 4 port on master.
  • Add various style tags to bundled style schemes.
  • Documentation of course.
  • A new snippet manager, file format, and parser. We’ll probably switch to XML for this so it matches language-spec and style-scheme instead of our adhoc format from Builder.

Future work

Completion

After the GTK 4 port and snippet engine has landed, I’ll probably turn my work towards updating the completion APIs to take advantage of various GLib/GIO advancements from recent years (similar to what we did in Builder). We can also do a bit of style refreshing there based on what Builder did too.

Indenters

If we can find a nice API for it, I’d like to land a basic indenter API as well. The one we have in Builder has worked, but it’s not exactly simple to write new indenters. Then we can start having contributions upstream which can be tied to a specific language-spec. Doing language specific stuff is always nice.

Movements Engine

How you move through a GtkTextView is rather simplistic. There are a number of keyboard accelerators for common movements but they are far more restrictive than what you’d expect from a code editor. In Builder we’ve had custom signals to do more robust movements which the VIM, Emacs, and Sublime emulation builds off.

We can make this more robust in the future using GTK 4’s widget actions. I plan to move a lot of the custom movements from Builder into GtkSourceView so that we can eventually have keyboard shortcut emulation upstream.

Event Controllers

One of the trickiest (and dirtiest) bits of code in Builder is our VIM keybindings emulation for the editor. It always was meant to be a hack to get us to GTK 4, and it did that fine. But we should let it end with that as the constraints from a gtkrc-based (GTK 2.x) system ported to CSS (GTK 3.x) is simply too much pain to bare.

In GTK 4 we have event controllers, especially those for handling keyboard input. I think it is possible for us to move much of a VIM emulation layer (for input) into an event controller (or GtkGesture even). Given that this would use C code instead of CSS, I’d have a much easier time dealing with all the hundreds of corner cases where VIM is internally inconsistent, but expected behavior.

Hover Providers

The Language Server Protocol has had success at abstracting a number of things, including the concept of “Hover Providers”. These are essentially interactive tooltips. Builder has support for them built upon transient GtkPopover. This seems like a possible candidate to move up to GtkSourceView too.

Other Possibilities

Some other possibilities, given enough interest, could be our OmniGutterRenderer from Builder (with integrated debug breakpoint and diagnostic integration), line-change gutter renderers (which can be connected to Git, SVN, etc), reformatting, semantic highlighting, and multiple cursors. However that last one is incredibly difficult to do from a completeness standpoint as it might need some level of plumbing down in GtkTextView for robustness sake.

Sysprof Developments

Earlier this month, Matthias and I teamed up to push through some of our profiling tooling for GTK and GNOME. We took the occasional work I had done on Sysprof over the past few years and integrated that into the GTK-4.x tree.

Sysprof uses a binary log file to store information about execution in a manner that is easy to write-buffer and read-back using positioned reads. It helps keep the sampling overhead of sysprof low. But it’s too detail oriented for each application supporting the format to write. To make this stuff reusable I created a libsysprof-capture-3.a static library we embed from various layers of the platform.

GTK-4.x is now using this. Builder itself uses it to log internal statistics, tracing data, and counters for troubleshooting. I’ve also put forward patches for GJS to integrate with it. Georges revamped and pushed forward a prototype by Jonas to integrate with Mutter/Shell and get us frame timings and Cogl pipeline data. With some work we can finish off the i915 data sources that Eric Anholt did to correlate GPU commands too.

What this means for developers is that soon we’ll be able to capture system information from various layers in the stack and correlate them using similar clocks. We’re only scratching the surface right now, but it’s definitely promising. It’s already useful to quantify the true performance improvements of merge-requests in Mutter and Shell.

To help achieve this goal during the 3.34 cycle, I’ve started the GNOME Profiling Initiative to track integration of various problem spaces. If you’re an application developer, you can use sysprof_capture_writer_new_for_env() to create a new SysprofCaptureWriter if Sysprof is profiling you (otherwise you’ll get NULL back). Use that to write marks, logs, metadata, files, or anything else you would like to capture.

If you’re interested in helping to write more data collectors, that would be appreciated. Data sources like battery voltage/wattage consumption seem like good candidates so that we can better optimize our platform for battery-based devices.

I have a Sysprof Copr repository for Rawhide and F30 if you’d like to try stuff out and submit issues.

Many thanks to Red Hat for sponsoring all the work I do on GNOME and my amazing manager Matthias for visiting Portland to make this stuff happen even sooner.

As always, follow my grumpy ramblings on birdsite for early previews of my work.