Bisect’ing outside of the box

There is a sort of thrill to a bug hunt once you dig your heels in deep enough. This is one of those stories.

Earlier in the week a fellow Red Hatter messaged me about a bug in Ptyxis where if you right click on a tab, the tab stays in the active state. Clearly not a very good look. My immediate thought was that we don’t do anything special there, so maybe the issue is lower in the stack.

A screenshot of Ptyxis with 4 terminal tabs open. Two of which appear "focused" due to an overzealous active state on the tab widget.



The first layer beneath that is of course libadwaita and so naturally I’ll test Text Editor to see if I get the same behavior. Lo-and-behold it exists there. Cool, file an issue in libadwaita while I simultaneously see if I can fix it.

Nothing stands out too obvious, but I try a few things which make it slightly better but I can still get into this state with enough attempts. Bummer. Also, Alice thinks the real issue is in GTK because we’ve seen this in a number of places. It appears to be something wrong with active state tracking.


No worries, I know that code-base well, so we hop on over there and see what is going on. First things first though, where does active state tracking happen? A quick git grep later we see it is largely in gtkmain.c in response to incoming GdkEvent.


Having written a large portion of the macOS back-end for GTK 4, I know that events can be extremely platform specific. So lets cut this problem in half by determining if the issue really is in the GTK-side or GDK-side. So we run the test again with GDK_BACKEND=x11 ptyxis -s and see if we can reproduce.

The issue appears to go away, lovely! Although, let’s test it on macOS too just to be certain. And again, it does not reproduce there. So that must mean the issue lies somewhere in gdk/wayland/ or related.


Another important detail that Alice mentioned was that it seemed to be a regression. That would be great because if so, it would mean that we can bisect to find an answer. However, since both gdk/wayland/ and compositors move forward at a somewhat lock-step pace, that can be quite difficult to prove reliably enough for bisect.

So instead I move on to something a bit whacky. I have a CentOS 7 installation here that I use to test that Ptyxis crazy GLibc hacks work for ptyxis-agent. That has GNOME 3.28 which can run as a Wayland compositor if you install the appropriate session file. Thus I do, and what do you know, it has the same issue there. That was surely to hit the “old wayland protocol” code-paths which are a bit different than the newer extensions, which was what I had hoped to test.

Either way, very old Mutter still equals broken. Womp.

Weston Enters the Ring

We have a “reference compositor” in the form of Weston which can help shake out protocol issues (assuming the diligence to Weston correctness is upheld). So I try running Ptyxis on my local machine inside a nested Weston instance (which appears to be 13.0.0). Surprising, things actually work!

Okay, but I still have that CentOS 7 VM running, I wonder if it has Weston available? It does not. Bummer. But I also have an Alma Linux 9 VM around that I use for testing occasionally and that must have it. Let’s try there.

In fact it does, but it’s version 8.0.0. Let’s see if it works there too!

A screenshot of GNOME Boxes running Alma Linux 9 in a GNOME session with a nested Weston compositor containing Ptyxis with 2 of 3 tabs broken.

Bisecting Weston

So now we know Weston 8.0.0 is broken and 13.0.0 works. Now we have something I can bisect!

So I fire up meson locally and ensure that my local build fails when it is 8.0.0. Indeed it does. And 13.0.0 appears to work. Bisect commence!

We find the commit which “fixes” things from a Weston point of view to be “Set grab client before calling grab functions“.

That gives me enough information to start commenting out lines of code in Mutter like the uncivilized monkey I am. Eventually, Shakespeare has been written and the bug goes away. Though certainly not in an upstream-able fashion. Bugs filed, lines of code pointed to, and here I hope for our lovely Mutter maintainers to take over.

Back to GDK

Lots of amazing things have happened since we got Wayland compositors. One of the more undeniable difficulties though has been equal event ordering amongst them. It’s almost certainly the case that other compositors are doing things in slightly different ways and it may not even be possible to change that based on their use-case/designs.

So some sort of mitigation will be required on the GDK side.

After some interpretation of what is really going on here, a very small patch to drop certain leave events.

And that’s the story of how I bisected Weston to find an issue in Mutter (and GDK).

Manuals on Flathub

Manuals contains the documentation engine from Builder as a standalone application. Not only does it browse documentation organized by SDK but can install additional SDKs too. This is done using the same techniques Builder uses to manage your project SDKs.

It should feel very familiar if you’re already using the documentation tooling in Builder.

In the past, we would just parse all the *.devhelp2 files up-front when loading. GMarkupParseContext is fast enough that it isn’t too much overhead at start-up for a couple hundred files.

However, once you start dealing with SDKs and multiple versions of all these files the startup performance can take quite a hit. So Manuals indexes these files into SQLite using GOM and performs queries using that instead. It conveniently makes cross-referencing easy too so you can jump between SDK revisions for a particular piece of documentation.


Red Hat Day of Learning

Occasionally at Red Hat we have a “Day of Learning” where we get to spend time learning about technology of our choice.

I spent some time listening to various AI explanations which were suggested readings for the day. Nothing too surprising but also not exactly engaging to me. Maybe that’s because I grew up with a statistics professor for a father.

So while that was playing I spent a little time learning how the GitLab API works. Immediately it stood out that one of the primary challenges in presenting UI for such an API would be in bridging GListModel to their implementation.

So I spent a little time on the architecture for how you might want to do that in the form of a Gitlab-GLib library.


There are essentially two modes of pagination supported by GitLab depending on the result set size. Both methods use HTTP headers to denote information about the result set.

Importantly, any design would want to have an indirection object (and in this case GitlabListItem) which can have it’s resource dynamically loaded.

Resources (such as a GitlabProject, or GitlabIssue) are “views” into the JSON result set. They are only used for reading, not for mutating or creating resources on the API server.

Offset-based Pagination

This form is somewhat handy because you can know the entire result-set size up front. Then you can back-fill entries as accessed by the user using bucketed lazy-loading.

When the bucketed page loads, the indirection objects are supplied with their resource object which provides a typed API over the JsonNode backing it.

This is the “ideal” form from the consuming standpoint but can put a great deal of load on the GitLab server instance.

Progressive Pagination

The other form of pagination lets you fetch the next page after each subsequent request. The header provides the next page number.

One might expect that you could use this to still jump around to the appropriate page. However if you are not provided the “number of pages” header from the server then there is not much you can do to clamp your page range.


This was a fun little side-project to get to know some of the inner workings of the API backing what those of us in GNOME use every day. I have no idea if anything will come of it, but it certainly could be useful from Builder if anyone has time to run with it.

For example, imagine having access to common GitLab operations from the header bar.

A screenshot of GNOME Builder with a new GtkMenuButton in the headerbar containing a GitLab icon and menu.

Ptyxis on Flathub

You can get Ptyxis on Flathub now if you would like to run the stable version rather than Nightly. Unless you’re interested in helping QA Ptyxis or contributing that is probably the Flatpak you want to have installed.

Nightly builds of Ptyxis use the GNOME Nightly SDK meaning GTK from main (or close to it). Living on “trunk” can be a bit painful when it goes through major transitions like is happening now with a move to Vulkan-by-default.


System Extensions from Flatpak

I write about Sysprof here quite often. Mostly in hopes of encouraging readers to use it to improve Linux as a whole.

An impediment to that is the intrusiveness to test out new features as they are developed. If only we had a Flatpak which you could install to test things right away.

One major hurdle is how much information a profiler needs to be useful. The first obvious “impossible to sandbox” API you run into is the perf subsystem. It provides information about all processes running on the system and their memory mappings which would make snooping on other processes trivial. Both perf and ptrace are disabled in the Flatpak sandbox.

After that, you still need unredacted access to the kernel symbols and their address mappings (kallsyms). You also need to be in a PID namespaces that allows you to see all the processes running on the system and their associated memory mappings which essentially means CAP_SYS_ADMIN.

Portable Services

Years ago, portable services were introduced into systemd through portablectl. I had high-hopes for this because it meant that I could perhaps ship a squashfs and inject it as a transient service on the host.

However, Sysprof needs more integration than could be provided by this because portable services are still rather isolated from the host. We need to own a D-Bus name, policy-kit action integration, in addition to the systemd service.

Even if that were all possible with portable services it wouldn’t get us access to some of the host information we need to properly decode symbols.

System Extensions

Then came along systemd-sysext. It provides a way to “layer” extensions on top of the host system’s /usr installation rather than in an isolated mount namespace.

This sounds much more promising because it would allow us to install .policy for policy-kit, .service files for Systemd and D-Bus, or even udev rules.

Though, with great power comes excruciating pain, or something like that.

So if you need to provide binaries that run on the host you need to either static-link (rust, go, zig perhaps?) or use something you can reasonably expect to be there (python?).

In the Sysprof case, everything is C so it can statically link almost everything by being clever with how it builds against glibc. Though this still requires glibc and quite frankly I’m fine with that. Potentially, one could use MUSL or ucLibc if they had high enough pain threshold for build tooling.

Bridging Flatpak and System Extensions

The next step would be to find a way to bridge system extensions and Flatpak.

In the wip/chergert/sysext branch of Sysprof I’ve made it build a number of things statically so that I can provide a system extension directory tree at /app/lib/extensions. We can of course choose a different path for this but that seemed similar to /var/lib/extensions.

Here we see the directory tree laid out. To do this right for systemd-sysext we also need to install an extension point file but I’ll save that for another day.

The Directory Tree

$ find /app/lib/extensions -type f

Registering the Service

First we need to symlink our system extension into the appropriate place for systemd-sysext to pick it up. Typically /var/lib/extensions is used for transient services so if this were being automated we might use another directory for this.

# mkdir -p /var/lib/extensions
# ln -s /var/lib/flatpak/org.gnome.Sysprof.Devel/current/active/files/lib/extensions/ org.gnome.Sysprof.Devel

Now we need to merge the extension so it overlays into /usr. We must use --force because we didn’t yet provide an appropriate extension point file for systemd.

# systemd-sysext merge --force
Using extensions 'org.gnome.Sysprof.Devel'.
Merged extensions into '/usr'.

And now make sure our service was installed to the approriate location.

# ls /usr/lib/systemd/systemd/sysprof3.service
-rw-r--r-- 2 root root 115 Dec 31 1969 /usr/lib/systemd/system/sysprof3.service

Next we need to reload the systemd daemon, but newer versions of systemd do this automatically.

# systemctl daemon-reload

Here is where things are a bit tricky because they are somewhat specific to the system. I think we should make this better in the appropriate upstream projects to avoid this altogether but also easily handled with a flatpak installation trigger.

First make sure that policy-kit reloads our installed policy file.

# systemctl restart polkit.service

With dbus-broker, we also need to reload configuration to pick up our new service file. I’m not sure if dbus-daemon would require this, I haven’t tested that. Though I wouldn’t be surprised if this is related to inotify file-monitors and introducing a merged /usr.

# gdbus call -y -d org.freedesktop.DBus \
-o /org/freedesktop/DBus \
-m org.freedesktop.DBus.ReloadConfig

At this point, the service should be both systemd and D-Bus activatable. We can verify that with another gdbus call quick.

# gdbus call -y -d org.gnome.Sysprof3 \
-o /org/gnome/Sysprof3 \
-m org.freedesktop.DBus.Peer.Ping

Now I can run the Flatpak as normal and it should be able to use the system extension to get profiling and system data from the host as if it were package installed.

$ flatpak run org.gnome.Sysprof.Devel

The following screenshots come from GNOME OS using yesterdays build with what I’ve described in this post. However, it also works on Fedora Rawhide (and probably Fedora 40) if you boot with selinux=0. More on that in the FAQ below.

Flatpak Integration

So obviously nobody would want to do all the work above just to make their Flatpak work. The user-facing goal here would be for the appropriate triggers to be provided by Flatpak to handle this automatically.

Making this happen in an automated fashion from flatpak installation triggers on the --system installation does not seem terribly out-of-scope. It’s possible that we might want to do it from within the flatpak binary itself but I don’t think that is necessary yet.


What about non-system installations?

It would be expected that system extensions require installing to a system installation.

It does not make sense to allow for a --user installation, controllable by an unprivileged user or application, to be merged onto the host.

Does SELinux affect this?

In fact it does.

While all of this works out-of-the-box on GNOME OS, systems like Fedora will need work to ensure their SELinux policy to not prevent system extentions from functioning. Of course you can boot with selinux=0 but that is not viable/advised on end-user installations.

In the Sysprof case, AVC denials would occur when trying to exec /usr/libexec/sysprofd.

Does /usr become read-only?

If you have systemd <= 255 then system-sysext will most definitely leave /usr read-only. This is problematic if you want to modify your system after merging but makes sense because sysext was designed for immutable systems.

For example, say you wanted to sudo dnf install a-package on Fedora. That would fail because /usr becomes read-only after systemd-sysext merge.

In systemd >= 256 there is effort underway to make /usr writable by redirecting writes to the top-most writable layer. Though my early testing of Fedora Rawhide with systemd 256~rc1 still shows this is not yet working.

So why not a Portal?

One could write a portal for profilers alone but that portal would essentially be sysprofd and likely to be extremely application specific.

Can I use this for udev rules?

You could.

Though you might be better served by using the new Device and/or USB portals which will both save you code and systems integration hassle.

Can I have different binaries per OS?


The systemd-sysext subsystem has a directory layout which allows for matching on some specific information in /etc/os-release. You could, for example, have a different system extension for specific Debian or CentOS Stream versions.

Can they be used at boot?

If we choose to symlink into a persistent systemd-sysext location (perhaps /etc/extensions) then they would be available at boot.

Can services run independent of user app?


It would be possible to have a system service that could run independently of the user facing application.

Guitar Builderings

I’m from a small town in Washington State that is well-known for top-tier guitars. It was home to Boogie Bodies, co-created by Lynn Ellsworth previously of Charvel fame. You might remember some of the guitars they worked on including a number of EVH’s iconic striped guitars.

When the company split Jim Warmoth took his half to create Warmoth Guitars, which is still in Puyallup to this day.

My cousin happened to work for Lynn doing paint and so I was always enthralled with their custom guitars from childhood. A few years back I put a “parts-caster” strat together with Warmoth parts and just now did another iteration on it.

The body consists of Alder from Washington State with a 5A Flame Maple laminated top from the north. It is dyed green with a small amount of burst to the edges.

The electronics come from Klein Pickups who in my opinion are doing some of the best out there right now. These are the mid-scoops but I’m going to swap them out soon for some 1965 recreations matched to SRV’s Lenny this summer.

The potentiometers are your normal strat 250k setup though this is a 5-way switch so you can run additional combinations out-of-phase to cancel out some of that infamous “single coil hum”.

The bridge comes from John Mann which should be familiar to the PRS players out there. Unless you’re like me and you have a PRS Core 24 with a Floyd Rose tremolo instead.

The tuners are locking Schaller M6 which are my go-to choice. Honestly, I like them more than the top screw lock that my PRS has. Don’t know why, just do.

The most recent change here since I originally assembled this is a new neck. It is a super curly 3A Flame Maple with Ebony fret-board. You can’t get the same sort of Ebony you used to for scarcity reasons, so now days you just have to be really selective about which cut you use if you want it to look black.

The back of the neck really demonstrates the beauty of a curly flame maple. I prefer nitro finishes for necks so that is what you see. One thing I should note here is that I wish I had opted for a contoured cut at the neck bolt-on position for a bit more arm space.

For the nut I decided to do something different than I’ve played before and used the Earvana nut. So far I’m pretty happy with it but I haven’t noticed too much of a difference.

That’s about it!

Here are a few tone samples which are just quick single takes so try not to hamper too much on my mistakes.

Some clean-ish samples (compressor, chorus, reverb essentially)

Some dirty samples (lead channel at about 1/3 drive, compressor and reverb)

Thanks for reading and listening!

Collaborating on Builder

It’s no secret I have way more projects to manage than hours in the day.

I hope to rectify this by sharing more knowledge on how my projects are built. The most important project, Builder, is quite a large code-base. It is undoubtedly daunting to dive in and figure out where to start.

A preview of a few pages of the book including the cover, and two pages from the table of contents.

Here is a sort of engineers journal (PDF) in book form about how the components fit together. The writing tries to be brief and to the point. You can always reference the source code for finer points.

There is so much more that can happen with Builder if we have regular contributors and subsystem maintainers.

We still need to bring back a new designer. We still need real strong git commit integration. We still need a physical device simulator to go with our architecture emulator. Our debugger API could use data visualizers and support for the debug-adapter-protocol. Container integration has a long tail of desired features.

You get the idea.

Lots of exciting features will happen once people dive into the code-base to learn more. Hopefully this book helps you along that journey.

Builder of Things

Sometimes I build stuff other than software and this is a post about that.

My wife and I had to postpone our honeymoon for a couple years due to COVID. Last spring we were able to take a trip to Thailand and really enjoyed it. So much so that when we got back we had a desire recreate that sort of relaxed urban yet tropical feel we enjoyed so much in various cities.

We don’t have a lot of extra space at our house but we did have some in the back which was vacated by recently fell Elm which were diseased.

I’m not one to shy away from projects I’ve never done before so why not build a deck and pergola to fill out that space for entertaining and hacking?

The first step was to even the grade. When we bought the house it was already sloped but the grinding of Elm tree trunks added another layer on top of that.

That is a lot of work but to make something 8’×20′ we need to clear a lot more than this. Only to be made worse by root after root of dead Elm tree remnants. Pick axe and sweat.

That’s more like it.

We don’t have much of a frost line here so you don’t need to do the whole concrete-pylon thing for the deck and integrated pergola. But what we do need to do is ensure that we have a retaining wall to keep wet ground from touching the cedar for the deck.

The 2×4 cedar topper is mostly just there to match what will be the cedar floor of the deck.

A bunch of gravel of various grades under the cinders create something stable that drains water beneath the cedar 4×4 which will be our columns.

This is built as two 8’×10′ sections which are connected. That makes the math much easier in my head but also easier to source good wood. Above you see the rim and floor joists for the first side with the blocking yet to be done.

I lucked out that the cedar supplier gave me 12′ 4×4s instead of the requested 10′. That gave me 2′ extra to use when framing without having to first get the whole 10′ columns leveled. Later one we’ll take them out one-by-one and replace them with the columns.

Now we have the columns in place and the whole thing leveled. Blocking has started and will get completed before adding the floor boards. I ended up using screws at an angle from the rounded corners of the ceder floor boards using “camo hidden fasteners”. After seeing how they handled the winter that was definitely the right decision.

Thankfully I had help from my wife and emeritus GNOMie Cosimo with all this.

Now we have all the floor boards on. The cedar appearance board is tacked on the edge of the rim joist to hide the pressure treated lumber. The beams have been connected at the top of the columns. The start of a privacy wall is in place to see how it will fit together. Next up, all those rafters which I used some simple angle cuts and a multi-tool to extract pockets to interference-fit over the beams.

Lastly my wife did her design magic after gathering all the outdoor furniture together.

This year I have jasmine and honeysuckle doing it’s thing to hopefully start getting some of this pergola covered with leaves and flowers by summer. I also extended the privacy wall a bit more since this picture to have a lattice for the vines to climb.

That’s it!

Refreshed Search

Builder got a refreshed search popover. It’s not even a GtkPopover anymore and instead uses AdwDialog.

You can use some of the typical “prefixes” to filter search results or do nothing and get everything mixed together.

A screenshot of the search popover displaying a list of filters such as @ to display symbols or ~ to filter only filenames.

For example, prefix the search with @ to limit the results to indexed symbol names. Quick preview is still presented side-by-side.

A popover is displayed filtering results to symbols using the @ prefix.

You can also search for documentation now if jumping to the search panel is too much work. Just prefix with ? and you’re ready to go.

The search popover displaying a list of documentation options from DexFuture.

Sometimes it can be handy to run various build actions using the search popover as well. Many of the menu items are searchable. Just prefix the search query with >.

The search popover showing a number of menu items and their action description along with associated keyboard shortcuts.


Documentation in Builder

A long time ago we had Devhelp integrated in Builder.

It got lost in the GTK 4 port because there was no GTK 4 version of Devhelp. Additionally, it didn’t handle the concept of SDKs at all. We went through great lengths in Builder to try to copy them around so libdevhelp could pick them up (with marginal success).

Builder now has code which can index various types of SDKs including Flatpak, the host system, and jhbuild. It does so automatically at startup into a SQLite database. That allows us to compare etags at startup and avoid a whole lot of extra work. It also serves as a convenient place to implement search going forward.

It looks like this

A screenshot of Builder showing a panel on the left with a documentation tree and a page in the document grid containing the documentation contents as a WebKitWebView. Breadcrumbs are provided in a pathbar as part of the statusbar.

The path bar at the bottom provides a convenient way to navigate around without having to go back to the documentation tree. It looks like this

A screenshot of Builder showing the documentaiton panel and page with a popover open on a pathbar breadcrumb allowing to navigate to similar entries at that level.

It also works when you’re using Builder as a “Text Editor with Plugins” replacement (e.g. the editor workspace).