Author: chergert

GtkSourceView branching

Branching

We’re currently finishing up the cycle towards GNOME 3.36, which means it’s almost time to start branching and thinking about what we want to land early in the 3.37 development cycle. My goal is to branch gtksourceview-4-6 which will be our long-term stable branch for gtksourceview-4.x (similar to how the gnome-3-24 branch is our long-term stable for the gtksourceview-3.x series. After that, master will move to GTK 4 as we start to close in on GTK 4 development. The miss-alignment in version numbers is an unfortunate reality, but a reality I inherited so we’ll keep on keepin’ on.

That means if you are not setting a branch in your flatpak manifests, you will want to start doing that when we branch (probably in the next couple of weeks) or your builds will start to fail. Presumably, this only will affect your Nightly builds, because who targets upstream master in production builds, not you surely!

Snippets

I’ve started moving some features from Builder into GtkSourceView. However, I’m limiting those to the GTK 4 port because I don’t particularly want to add new ABI right before putting a branch into long-term stability mode. The first to be uplifted into GtkSourceView is Builder’s snippet engine. It also went through quite a bit of rewrite and simplification as part of this process to make it more robust. Furthermore, having moved undo/redo into GtkTextBuffer directly has done wonders from a correctness standpoint. The snippet engine used to easily be confused by the undo/redo engine.

The most difficult part of the snippet engine is dealing with GtkTextMark that are adjacent. In particular if you have each snippet focus-position wrapped in marks. Adjacent, empty mark ranges can end up overlapping each other and you have to be particularly diligent to prevent that. But the code in the branch has a pretty good handle on that, much better than what I had done in the past inside of Builder.

The bits I still need to do to finish up the snippet engine:

  • Land the GTK 4 port on master.
  • Add various style tags to bundled style schemes.
  • Documentation of course.
  • A new snippet manager, file format, and parser. We’ll probably switch to XML for this so it matches language-spec and style-scheme instead of our adhoc format from Builder.

Future work

Completion

After the GTK 4 port and snippet engine has landed, I’ll probably turn my work towards updating the completion APIs to take advantage of various GLib/GIO advancements from recent years (similar to what we did in Builder). We can also do a bit of style refreshing there based on what Builder did too.

Indenters

If we can find a nice API for it, I’d like to land a basic indenter API as well. The one we have in Builder has worked, but it’s not exactly simple to write new indenters. Then we can start having contributions upstream which can be tied to a specific language-spec. Doing language specific stuff is always nice.

Movements Engine

How you move through a GtkTextView is rather simplistic. There are a number of keyboard accelerators for common movements but they are far more restrictive than what you’d expect from a code editor. In Builder we’ve had custom signals to do more robust movements which the VIM, Emacs, and Sublime emulation builds off.

We can make this more robust in the future using GTK 4’s widget actions. I plan to move a lot of the custom movements from Builder into GtkSourceView so that we can eventually have keyboard shortcut emulation upstream.

Event Controllers

One of the trickiest (and dirtiest) bits of code in Builder is our VIM keybindings emulation for the editor. It always was meant to be a hack to get us to GTK 4, and it did that fine. But we should let it end with that as the constraints from a gtkrc-based (GTK 2.x) system ported to CSS (GTK 3.x) is simply too much pain to bare.

In GTK 4 we have event controllers, especially those for handling keyboard input. I think it is possible for us to move much of a VIM emulation layer (for input) into an event controller (or GtkGesture even). Given that this would use C code instead of CSS, I’d have a much easier time dealing with all the hundreds of corner cases where VIM is internally inconsistent, but expected behavior.

Hover Providers

The Language Server Protocol has had success at abstracting a number of things, including the concept of “Hover Providers”. These are essentially interactive tooltips. Builder has support for them built upon transient GtkPopover. This seems like a possible candidate to move up to GtkSourceView too.

Other Possibilities

Some other possibilities, given enough interest, could be our OmniGutterRenderer from Builder (with integrated debug breakpoint and diagnostic integration), line-change gutter renderers (which can be connected to Git, SVN, etc), reformatting, semantic highlighting, and multiple cursors. However that last one is incredibly difficult to do from a completeness standpoint as it might need some level of plumbing down in GtkTextView for robustness sake.

GtkSourceView on GTK 4

I spent some time this cycle porting GtkSourceView to GTK 4. It was a good opportunity to help me catch up on how GTK 4’s internals have changed into something modern. It gave me a chance to fix a few pot-holes along the way too.

One of the pot-holes was one I left in GtkTextView years ago. When I plumbed the pixelcache into GTK 3’s TextView I had only cached the primary text content. It seemed fine at the time because the gutters (used for line numbers) is just not that many pixels. So if we have to re-generate that every frame, so be it.

However, in a HiDPI world and 4k monitors on our laps things start to get… warm. So while changing the drawing model in GtkTextView we decided to make the GtkTextView gutters real widgets. Doing so means that GtkSourceGutterRenderer will be real GtkWidget‘s going forward and can do all sorts of neat stuff widgets can do.

But to address the speed of rendering we needed a better way to avoid walking the text btree linearly so many times while rendering the gutter. I’ve added a new class GtkSourceGutterLines to allow collecting information about the text buffer in one-pass. The renderers can then use that information when creating render nodes to avoid further tree scans.

I have some other plans for what I’d like to see before a 5.0 of GtkSourceView. I’ve already written a more memory-compact undo/redo engine for GTK’s GtkTextView, GtkEntry, GtkText, and friends which allowed me to delete that code from the GtkSourceView port. Better yet, you get undo/redo in all the places you would, well, expect it.

In particular I would like to see the async+GListModel based API for completion from Builder land upstream. Builder also has a robust snippet engine which could be reusable from GtkSourceView as that is a fairly useful thing across editors. Perhaps we could extract Builder’s indenter APIs and movements engine too. These are used by Builder’s Vim emulation quite heavily, for example.

If you like following development of stuff I’m doing you can always get that fix here on Twitter given my blogging infrequency.

Introducing Bonsai

TL;DR: Pair your Linux devices, developer APIs to share files, create object graphs with partial sync between devices, transactions, secondary indexes, rebasing, and more built upon GVariant and LMDB. Tooling to build cloudless multi-device services.

A picture of a Bonsai tree and a gnomeI’ve been spending a great deal of time thinking about what types of products I’d like to see in GNOME and what is missing to make that happen.

One observation is that I want access to my files and application data on all my computing devices but I don’t want to store that data on other peoples computers. I have computers, they have internet access, I shouldn’t have to use a multi-tenancy cloud if I’m running as much Free Software as I do. But if that is going to be competitive it needs to be easier than the alternatives.

But to build this I need a few fundamental layers to build applications atop. I’ll need access to files using all the GIO file APIs we love (GFile, GFileEnumerator, GIOStream, etc). I’ll also need the ability to read and write application data in a way that can be shared between devices which may not always be connected to my home Wi-Fi. In particular, we need to give developers great tools to make applications that natively support device synchronization.

What I’ve built to experiment with this all is Bonsai. It is very much an experiment at this phase but it is getting interesting enough to collaborate with others who would like to join me.

Bonsai consists of a daemon that you run on your “mostly connected” computer. Although that could easily be a raspberry pi quality computer in your home. That computer hosts the “upstream” storage space for files and application content.

Other devices like laptops, phones, or IoT can be paired with that primary device. They communicate using TLS connections using pinned self-signed certificates with point-to-point D-Bus serialization on top. The D-Bus serialization makes it convenient to use gdbus-codegen to generate proxies and services.

One service available to devices is the storage service. It can be consumed from libbonsai-storage to allow applications to browse, create, move, modify and stream file content.

Applications are much better when they can communicate between devices. So a Data-Access-Object library, aptly named libbonsai-dao, provides serializable object storage built upon GVariant and LMDB. It supports primary and secondary indexes, queries, cursors, transactions, and incremental sync between devices. It has the ability to rebase local changes atop changes pulled from the primary Bonsai device.

That last bit is neat because it means if an application is running on two devices which create new content they don’t clobber each others history.
The primary issue here is dealing with merge conflicts but libbonsai-dao provides some design for data objects to do the right thing.

Bonsai could also could serve as a base to build interesting services like backup, VPN, media sharing and casting, news, notes, calendars, contacts, and more. But honestly, it can only do that if people are actually interested in something like this. If so, let me know and see if you can lend your time or ideas for what you’d want this to become.

Sysprof Updates

I just uploaded the sysprof-3.33.4 tarball as we progress towards 3.34. This alpha release has some interesting new features that some of you may find interesting as you continue your quests to improve the performance of your system by improving the software running upon it.

An image of Sysprof with various performance graphs

For a while, I’ve been wondering about various ways to move GtkTextView forward in GTK 4. It’s of particular interest to me because I spent some time in the GTK 3 days making it faster for smooth scrolling. However, the designs that were employed there work better on the traditional Xorg setup than they do on GTK 3’s Wayland backend. Now that GTK 4 can have a proper GL pipeline, there is a lot of room for improvement.

Thanks to the West Coast Hackfest, I had a chance to sit down with Matthias and work through that design. GtkLabel was already using some accelerated text rendering so we started by making that work for GtkTextView. Then we extended the GSK PangoRenderer to handle the rest of the needs of GtkTextView and Matthias re-implemented some features to avoid cairo fallbacks.

After the hackfest I also found time to implement layout caching of PangoLayout. It helps reduce some of the CPU overhead in calculating line layouts.

As we start using the GPU more it becomes increasingly important to keep the CPU usage low. If we don’t it’s very likely to raise overall energy usage. To help keep us honest, I’ve added some RAPL energy statistics to Sysprof.

Sysprof design work

Since my last post, I’ve been working on a redesign of Sysprof (among other things) to make it a bit more useful and friendly to newcomers.

Many years ago, I worked on a small profiler project called “Perfkit” that never really went anywhere. I had already done most of my UI research for this years ago, so it was pretty much just a matter of applying that design to the Sysprof code-base.

Now you can individually show extra detail rows and much more. Same great Sysprof 3-part callgraph breakdown.

A screenshot of Sysprof

I’ll get some delayed 3.33.3 tarballs out this week.

GTK 3 Frame Profiler

I back-ported the GTK 4 frame-profiler by Matthias to GTK 3 today. Here is an example of a JavaScript application using GJS and GTK 3. The data contains mixed native and JS stack-traces along with compositor frame information from gnome-shell.

What is going on here is that we have binary streams (although usually captured into a memfd) that come from various systems. GJS is delivered a FD via GJS_TRACE_FD environment variable. GTK is delivered a FD via GTK_TRACE_FD. We get data from GNOME Shell using the org.gnome.Sysprof3.Profiler D-Bus service exported on the org.gnome.Shell peer. Stack-traces come from GJS using SIGPROF and the SpiderMonkey profiler API. Native stack traces come from the Linux kernel’s perf_event_open system. Various data like CPU frequency and usage, memory and such come from /proc.

Muxing all the data streams uses sysprof_capture_writer_cat() which knows how to read data frames from supplemental captures and adjust counter IDs, JIT-mappings, and other file-specific data into a joined capture.

A quick reminder that we have a Platform Profiling Initiative in case you’re interested in helping out on this stuff.

Sysprof Developments

This week I spent a little time fixing up a number of integration points with Sysprof and our tooling.

The libsysprof-capture-3.a static library is now licensed under the BSD 2-clause plus patent to make things easier to consume from all sorts of libraries and applications.

We have a MR for GJS to switch to libsysprof-capture-3.a and improve plumbing so Sysprof can connect automatically.

We also have a number of patches for GLib and GTK that improve the chances we can get useful stack-traces when unwinding from the Linux kernel (which perf_event_open does).

A MR for GNOME Shell automatically connects the GJS profiler which is required as libgjs is being used as a library here. The previous GJS patches only wire things up when the gjs binary is used.

With that stuff in place, you can get quite a bit of data correlated now.

# Logout, Switch to VT2
sysprof-cli -c "gnome-shell --wayland --display-server" --gjs --gnome-shell my-capture.syscap

If you don’t want mixed SpiderMonkey and perf stack-traces, you can use --no-perf. You can’t really rely on sample rates between two systems at the same time anyway.

With that in place, you can start correlating more frame data.

Sysprof Developments

Earlier this month, Matthias and I teamed up to push through some of our profiling tooling for GTK and GNOME. We took the occasional work I had done on Sysprof over the past few years and integrated that into the GTK-4.x tree.

Sysprof uses a binary log file to store information about execution in a manner that is easy to write-buffer and read-back using positioned reads. It helps keep the sampling overhead of sysprof low. But it’s too detail oriented for each application supporting the format to write. To make this stuff reusable I created a libsysprof-capture-3.a static library we embed from various layers of the platform.

GTK-4.x is now using this. Builder itself uses it to log internal statistics, tracing data, and counters for troubleshooting. I’ve also put forward patches for GJS to integrate with it. Georges revamped and pushed forward a prototype by Jonas to integrate with Mutter/Shell and get us frame timings and Cogl pipeline data. With some work we can finish off the i915 data sources that Eric Anholt did to correlate GPU commands too.

What this means for developers is that soon we’ll be able to capture system information from various layers in the stack and correlate them using similar clocks. We’re only scratching the surface right now, but it’s definitely promising. It’s already useful to quantify the true performance improvements of merge-requests in Mutter and Shell.

To help achieve this goal during the 3.34 cycle, I’ve started the GNOME Profiling Initiative to track integration of various problem spaces. If you’re an application developer, you can use sysprof_capture_writer_new_for_env() to create a new SysprofCaptureWriter if Sysprof is profiling you (otherwise you’ll get NULL back). Use that to write marks, logs, metadata, files, or anything else you would like to capture.

If you’re interested in helping to write more data collectors, that would be appreciated. Data sources like battery voltage/wattage consumption seem like good candidates so that we can better optimize our platform for battery-based devices.

I have a Sysprof Copr repository for Rawhide and F30 if you’d like to try stuff out and submit issues.

Many thanks to Red Hat for sponsoring all the work I do on GNOME and my amazing manager Matthias for visiting Portland to make this stuff happen even sooner.

As always, follow my grumpy ramblings on birdsite for early previews of my work.

Improving the Container Workflow

As I mentioned in my talk at Scale 17x, if you aren’t using containers for building your application yet, it’s likely you will in the not-so-distant future. Moving towards an immutable base OS is a very likely future because the security advantages are so compelling. With that, comes a need for a mutable playground, and containers generally fit the bill. This is something I saw coming when I started making Builder so we have had a number of container abstractions for some time.

With growing distributions, like Fedora’s freshly released Silverblue, it becomes even more necessary to push those container boundaries soon.

This week I started playing with some new ideas for a terminal workspace in Builder. The goal is to be a bit of a swiss-army knife for container oriented development. I think there is a lot we can offer by building on the rest of Builder, even if you’re primary programming workhorse is a terminal and the IDE experience is not for you. Plumbing is plumbing is plumbing.

I’m interested in getting feedback on how developers are using containers to do their development. If that is something that you’re interested in sharing with me, send me an email (details here) that is as concise as possible. It will help me find the common denominators for good abstractions.

What I find neat, is that using the abstractions in Builder, you can make a container-focused terminal in a couple of hours of tinkering.

I also started looking into Vagrant integration and now the basics work. But we’ll have to introduce a few hooks into the build pipeline based on how projects want to be compiled. In most cases, it seems the VMs are used to push the app (and less about compiling) with dynamic languages, but I have no idea how pervasive that is. I’m curious how projects that are compiling in the VM/container deal with synchronizing code.

Another refactor that needs to be done is to give plugins insight into whether or not they can pass file-descriptors to the target. Passing a FD over SSH is not possible (although in some cases can be emulated) so plugins like gdb will have to come up with alternate mechanisms in that scenario.

I will say that trying out vagrant has been a fairly disappointing experience compared to our Flatpak workflow in Builder. The number of ways it can break is underwhelming and the performance reminds me of my days working on virtualization technology. It makes me feel even more confident in the Flatpak architecture for desktop developer tooling.