Happenings in GNOME – Page 3 – Details of Christian's work on GNOME

Debug Builds and GPUs

Decades ago, when you wanted to run debug builds for UI applications, things were incredibly slow.

First you’d wait minutes for the application to present a window. Then wait tens of seconds for each frame to render. You were extremely lucky if Valgrind caught the issue while you exercised the UI.

Things have gotten much better due to movement in two different directions.

In one direction GCC and Clang got compiler integration for sanitizers like ASAN. Instead of relying on extreme amounts of emulation in Valgrind compilers can insert the appropriate checks, canaries, and memory mapping tricks to catch all sorts of behavior.

In the other direction we’ve started drawing modern UI toolkits with the GPU. The idea here is that if the work is dispatched to the GPU, there is less for the CPU to run and therefore less work for the sanitizers and/or Valgrind to do.

Don’t let that fool you though. A lot of specialized work is done on the CPU still to allow those GPUs to go fast. You trade off framebuffer updates and huge memory bus transfers for more complex diffing, batching and reordering operations, state tracking, occasional texture uploads, and memory bandwidth for vertex buffers and the likes.

Here I’ve compiled all the hot parts of a GTK application with the address sanitizer. That includes GLib/GIO/GObject, Harfbuzz, Pango, and GTK. The application is also running with GSK_DEBUG=full-redraw to ensure we redraw the entire window every single frame with full damage. We use GDK_DEBUG=no-vsync to let it run as fast as it can rather than block waiting for the next vblank.

And still, GTK can produce hundreds of frames per second.

Truly magical.

BOLT’ing Libraries

I did a little experimenting with BOLT today to optimize libraries post-link.

I’m not an expert on it or anything, but it seems to allow you to reorder functions in your executable/library based on feedback from perf record and some special post-processing. You can merge multiple runs together in case you have different workloads you’d like to optimize for. But in the end, hot functions get placed near each other to reduce instruction cache pressure.

In all, it says you can expect gains up-to about 7% which fits in line with my experiment. For example, I open gnome-text-editor with a large C file, the overview map enabled, and syntax highlighting on. Then hold down Page Down to the bottom, Page Up to the top, and then Page Down back to the bottom.

The first pass through the source code is usually a little slower because you’re doing the incremental syntax-highlighting process.

After using BOLT on Pango, I saw roughly a 6% reduction in time spent measuring text (which is one of the most expensive parts of the overview map).

To test this out, I did have to play with my CFLAGS to have -Wl,--emit-relocs linker option. After that and a meson setup --wipe $SRCDIR things seem to work as expected.

Trying it For Yourself

sudo dnf install llvm-bolt perf


perf record -e cycles:u -j any,u -o perf.data -- gnome-text-editor
# you can do this for any of the binaries

perf2bolt -p perf.data -o perf.fdata ~/.jhbuild/lib/libpango-1.0.so
llvm-bolt ~/.jhbuild/lib/libpango-1.0.so -o libpango-1.0.so.bolt -data=perf.fdata -reorder-blocks=ext-tsp -reorder-functions=hfsort -split-functions -split-all-cold -split-eh -dyno-stats

mv libpango-1.0.so.bolt ~/.jhbuild/lib/libpango-1.0.so

Rinse and repeat.

Frame Pointers in the Media

BPF Performance Tools author and all around profiling expert Brendan Gregg wrote a blog post that sums up what was in my Fedora Magazine article quite well.

Though he has this to say on the topic of Fedora who made this ground breaking change and Ubuntu who followed along afterwards:

The main users of this change are enterprise Linux. Back-end servers.

Which is true in the sense of absolute numbers. But I must say it’s been extremely valuable on the desktop.

I can’t imagine having contributed to making VTE (a code-base I was unfamiliar with) twice as fast without it. Especially when that work happened over the course of about two weeks. It’s so much easier to do performance work when one monitor has usable profiler flamegraphs and the other code.

The wash/rinse/repeat cycle has gotten really good on Fedora and our performance future is bright.

Improving poll() timeout precision

Recently I was looking at a VTE performance issue so I added a bunch of Sysprof timing marks to be picked up by the profiler. I combined that with GTK frame timing information and GNOME Shell timing information because Sysprof will just do that for you. I noticed a curious thing in that almost every ClutterFrameClock.dispatch() callback was rougly 1 millisecond late.

A quick look at the source code shows that ClutterFrameClock uses g_source_set_ready_time() to specify it’s next deadline to awaken. That is in µsec using the synchronized monotonic clock (CLOCK_MONOTONIC).

Except, for various reasons, GLib still uses poll() internally which only provides 1 millisecond timeout resolution. So whatever µsec deadline was requested by the ClutterFrameClock doesn’t really matter if nothing else wakes up around the same time. And since the GLib GSource code will always round up (to avoid spinning the CPU) that means a decent amount late.

With the use of ppoll() out of question, the next thing to use on Linux would be a timerfd(2).

Here is a patch to make GLib do that. I don’t know if that is something we should have there as it will create an extra timerfd for every GMainContext you have, but it doesn’t seem insane to do it there either.

If that isn’t to be, then here is a patch to ClutterFrameClock which does the same thing there.

And finally, here is a graph of how the jitter looks when not using timerfd and when using timerfd.

Accessibility in Ptyxis

First off, what the heck is Ptyxis?

Ptyxis is the new name of what was formerly Prompt. The extremely nice people at Panic let me know they had a product that might be confused with Prompt and I agreed it could be confusing. Thankfully, their office is a few miles from me in Portland so I had a chance to meet the team face to face!

We found a lot to talk about, especially when it comes to text rendering with the GPU and people getting aggressive over their beloved fonts.

Hopefully, you like the new name. If not, feel comfort knowing that the desktop spec has support for GenericName and Keywords allowing you to type whatever you like to find the application.

In the North, Spring is about to burst which is a great reason to learn what Ptyxis is. You can even find a beautiful example of it on the cover of The Linux Programming Interface which if you’re into terminals and Linux you should already own.

Now, the new bit.

Accessibility is super important for many reasons I shouldn’t need to repeat.

I put together an implementation of the new GtkAccessibleText for VteTerminal and am bundling it with Ptyxis to get more testing.

Since this has the potential to really overload the screen reader as the final interfaces are figured out, it’s gated behind a toggle in preferences. I also am hesitant to enable it by default until we have a way in the a11y stack to be extremely lazy about initialization. I don’t want to waste a lot of CPU cycles tracking changes only for them to be sent to an accessibility D-Bus where nobody is listening.

Hopefully this is one less thing preventing Linux distributions from shipping a GTK 4 based terminal emulator by default.

Performance Profiling for Fedora Magazine

I’ve authored an article recently for Fedora Magazine on Performance Profiling in Fedora.

It covers both the basics on how to get started as well as the nitty-gritty details of how profilers work. I’d love for others to be more informed on that so I’m not the only person maintaining Sysprof.

Hopefully I was able to distill the information down a bit better than my typical blog posts. If you felt like those were maybe too difficult to follow, give this one a read.

Sidebars in Libpanel

One of the more recent design trends in GNOME has been the use of sidebars. It looks great, it’s functional, and it gives separation of content from hierarchy.

Builder, on the other hand, has been stuck a bit closer to the old-hat design of IDEs where the hierarchy falls strictly from the headerbar. This is simply because libpanel was designed before that design trend. Some attempt was made in Builder to make it look somewhat sidebar’ish, but that was the extent of it given available time.

Last week I had a moment of inspiration on a novel way we could solve it without uprooting the applications which use libpanel. You can now insert edge widgets in PanelDockChild which are always visible even when the child is not. Combining that with being able to place a headerbar inside a PanelDockChild along with your PanelFrames means you can get something that looks more familiar in modern GNOME.

If you’d like to improve things further, you know where to find the code.

Faster Numbers

The venerable GtkSourceView project provides a GtkWidget for various code languages. It has a number of features including the most basic, showing a line number next to your line of text.

It turns out that takes a lot more effort than you might think, particularly when you want to do it at 240hz with kinetic scrolling on crappy hardware that may barely have enough engine for the GL driver.

First, you need to have the line number as a string to be rendered. For a few years now, GtkSourceView has code which will optimizes the translation from number to strings with minimal overhead. If you g_snprintf(), you’re gonna be slow.

After that you need to know the X,Y coordinate of the particular line within the gutter and it’s line height when wrapped. Then you need to know the measured pixel width of the line number string. Further still you need the xalign/yalign and xpad/ypad to apply proper alignments based on application needs. You may even want to align based on first line, last wrapped line, or the entire cell.

In the GtkSourceView 5.x port I created GtkSourceGutterLines which can cache some of that information. It’s still extremely expensive to calculate but at least we only have to do it once per-frame now no matter how many GtkSourceGutterRenderer are packed into the GtkSourceGutter.

After that, we can create (well recycle) a PangoLayout to setup what we want to render. Except, that is also extremely expensive because you need to measure the contents and go through a PangoRenderer for each line you render.

If you are kinetic scrolling through a GtkSourceView with something like a touch pad there is a good chance that a decent chunk of CPU is wasted on line numbers. Nicht gut.

Astute readers will remember that I spent a little time making VTE render faster this cycle and one of the ways to do that was to avoid PangoLayout. We can do the same here as it’s extremely simple and controlled input. Just cache the PangoGlyphInfo for 0..9 and use that to build a suitable PangoGlyphString. Armed with a PangoFont and said string, we can use gsk_text_node_new() and gtk_snapshot_append_node() instead of gtk_snapshot_render_layout().

A quick hour or so later I have given you back double digit CPU percentages but more importantly, smoother and lower latency input.

Sysprof makes it easy to locate, triage, and verify performance fixes.

That said, in the future, if I were redesigning something to replace all of this I’d probably just use widgets for each line number and recycle them like GtkListView. Then you get GtkWidget render node caching for free. C’est la vie.

Prompt

Prompt is a terminal that marries the best of GNOME Builder’s seamless container support, the beauty of GNOME Text Editor, and the robustness of VTE. I like to think of it as a companion terminal to Builder.

Though it’s also useful for immutable/container-oriented desktops like Fedora Silverblue or Project Bluefin where containers are front-and-center.

This came out of a prototype I made for the GNOME Builder IDE nearly a decade ago. We already had all the container abstractions so why not expose them as a Terminal Workspace?

Prompt extracts the container portion of the code-foundry into a standalone program.

My prototype didn’t go anywhere in recent years because I was conflicted. I have a high performance bar for software I ship and VTE wasn’t there yet on Wayland-based compositors which I use. But if you frequent this blog you already know that I reached out to the meticulous VTE maintainers and helped them pick the right battles to nearly double performance this GNOME cycle. I also ported gnome-terminal to GTK 4 which provided me ample opportunity to see where and how containers would integrate from an application perspective.

I designed Prompt to be Flatpak-first. That has design implications if you want a robust feature-set. Typically an application is restricted to the PID and PTY namespace within the Flatpak sandbox even if you’re capable of executing processes on the host. That means using TTY API like tcgetpgrp() becomes utterly useless when the kernel ioctl() returns you a PID of 0 (as it’s in a different namespace). Notably, 0 is the one value tcgetpgrp() is not documented to return. How fun!

To give Prompt the best chance at tracking containers and foreground processes a prompt-agent runs from the host system. It is restricted to very old versions of GLib/GObject/GIO and JSON-GLib because we know that /usr/bin/flatpak will already require them. Using those libraries instead of certain GLibc API helps us in situations where GLibc is only backward-compatible and not forwards-compatible. Combined with point-to-point D-Bus serialization on top of a socketpair() we have a robust way to pass file-descriptors between the agent and the UI process and we’ll use that a bunch.

There are a lot of little tricks in here to keep things fast and avoid energy-drain. For example, process tracking is done with a combination of exponential-backoff and re-triggering based on either new content arriving or certain key-presses. It gives a very low-latency feeling to the sudo/ssh feature I love from Console, albeit with less overhead.

One thing I find helpful with Builder is that when I come back to it my project session is right there. So this has session support too. It will restore your tabs/containers how they were before. So if you have a similar workflow you might find that useful. If not? Just turn it off in Preferences.

I want to have a bit of fun because right now I’m stuck indoors caring for my young, paraplegic dog. So it’s packed full of palettes you can play with. Who doesn’t like a little color!

There are some subtle performance choices that make for a better experience in Prompt. For example, I do like having a small amount of padding around the terminal so that rounded corners look nice and also avoids an extra framebuffer when rendering on the GPU. However, that looks odd with scrollback. So Prompt rewrites the snapshot from VTE to remove the background and extraneous clipping. We already have a background from window recoloring anyway. It’s a small detail that just feels good when using it.

Another subtle improvement is detecting when we are entering the tab overview. Currently, libadwaita uses a GtkWidgetPaintable to represent the tab contents. This works great for the likes of Epiphany where the contents are backed by GL textures. But for a terminal application we have a lot of text and we don’t want to redraw it scaled as would happen in this case. That puts a lot of pressure on the glyph cache. So instead, we create a single texture upfront and scale that texture. Much smoother.

For people writing terminal applications there is a little inspector you can popup to help you out. It can be difficult to know if you’re doing the right thing or getting the right behavior so this might be something we can extend going forward to make that easier for you. GTK’s inspector already does so much so this is just an extension of what you could do there.

Creating Prompt has elevated problems we should fix.

Podman creates an additional PTY which sort of breaks the whole notion of foreground processes. Filed an issue upstream and it seems likely we can get that addressed for personal containers. That will improve what happens when you close your terminal tab with something running or if you SSH’d into another host from the container.
Container tracking is currently limited to Fedora hosts because toolbox only emits the container-tracking escape sequences when the host is Fedora. The current goal I’ve discussed with VTE maintainers is that we’ll use a new “termprop” feature in VTE that will be silently dropped on terminal applications not interested in it. That way toolbox and the likes can safely emit the escape sequence.
Currently podman will exit if you pass a --user or --workdir that does not exist in the container. That isn’t a problem with toolbox as it is always your user and fails gracefully for directories. So we need a good strategy to see if both of those are available to inherit when creating new tabs.
This does have transparency support, but it’s hidden in a GSetting for now. Once we have libadwaita with CSS variable support we can probably make this look better during transitions which is where it falls down now. We also need some work on how AdwTabOverview does snapshots of tabs and inserts a background.
I have a .bashrc snippet to treat jhbuild as a container which is helpful for those of us GNOME developers still using it.
Accessibility is an extremely important piece of our infrastructure in GNOME. So part of this work will inevitably tie into making sure the a11y portion of VTE works with the soon-to-land a11y improvements in GTK. That has always been missing on GTK 4-based VTE and therefore every terminal based upon it.

$ flatpak install --user --from https://nightly.gnome.org/repo/appstream/org.gnome.Prompt.Devel.flatpakref

If you like software that I write, consider donating to a pet shelter near you this holiday season. We’re so lucky to have great pet care in Oregon but not everywhere is so lucky.

Happy Holiday Hacking!

Toby is Recovering in ER ICU

Normally I’m posting about code here, but for the past two weeks most of my time has been spent taking care of our 4 year old Australian Shepherd. Toby is very special to me and we even share the same birthday!

Toby recently lost control of his hind legs, which was related to a herniated disk and likely IVDD. For the past two weeks my wife and I have been on full-time care duty. Diapers, sponge baths, the whole gamut.

Previously we had X-Rays done but that type of imaging is not all that conclusive for spinal injuries. So yesterday he had a doggy MRI then rushed into surgery for his L1/L2 discs and a spinal tap. The spinal tap is to make sure the situation wasn’t caused by meningitis. It seems to have gone well but this morning he still hasn’t regained control of his hind legs (and that’s to be expected so soon after surgery). He does have feeling in them though, so that’s a positive sign.

He’s still a very happy boy and we got to spend a half hour with him this morning in the ICU.

Thanks to everyone who has been supporting us through this, especially with watching our 5 month kitten June who can be a bit of a rascal during the day. We wouldn’t be able to get any sleep without y’all.

To keep his mind active, Tenzing started to teach him to sing. He’s already got part of La Bouche’s “Be My Lover” down which is just too adorable not to share with you.