Happenings in GNOME

With GNOME 48 I released libdex 0.10 on the march towards a 1.0. One of the major improved features there was around fiber cancellation.

I’m not going to go into detail about the differences between threads and fibers as wikipedia or your local CS department can probably help you there. But what I will say is that combining __attribute__((cleanup)) (e.g. g_autoptr()) with futures and fibers makes such a nicer experience when writing C.

Thread cancellation is a rather non-portable part of the threading stack across platforms. Some POSIX platforms support it, some don’t. Having safe places to cancel can be a real challenge even if you are depending on a threading implementation that can do it.

With fibers, we have a natural cancellation point due to the cooperative nature of scheduling. All (well behaved) fibers are either making progress or awaiting completion of a future. We use the natural await() points to implement cancellation. If everything that was awaiting the future of the fiber has been cancelled, then the fiber can naturally cancel too. The next time it awaits that will just happen and natural exit paths will occur.

When you don’t want cancellation to propagate, you still use dex_future_disown() like always (as the fiber itself is a future).

Just to give a quick example of how fibers and futures makes writing C code nicer, here is an excerpt from libfoundry that asynchronously implements the necessary phases to build/run your project with a specific tool, possibly on a non-local system. In the GNOME Builder IDE, this is a series of async callbacks that is extremely difficult to read/debug. But with Foundry using libdex, it’s just a few lines of code and every bit as non-blocking.

From foundry-run-manager.c.

[code lang=”c”]
g_autoptr(FoundryDeployStrategy) deploy_strategy = NULL;
g_autoptr(FoundryBuildProgress) progress = NULL;
g_autoptr(GSubprocess) subprocess = NULL;
GError *error = NULL;

if (!(deploy_strategy = dex_await_object (foundry_deploy_strategy_new (state->pipeline), &error)) ||
!dex_await (foundry_deploy_strategy_deploy (deploy_strategy, state->build_pty_fd, state->cancellable), &error) ||
!dex_await (foundry_deploy_strategy_prepare (deploy_strategy, state->launcher, state->pipeline, state->build_pty_fd, state->cancellable), &error) ||
!dex_await (foundry_run_tool_prepare (state->run_tool, state->pipeline, state->command, state->launcher, state->run_pty_fd), &error) ||
!(subprocess = foundry_process_launcher_spawn (state->launcher, &error)))
return dex_future_new_for_error (error);
[/code]

At each dex_await*() function call the fiber is suspended and we return to the main loop for additional processing.

In a better world we’d be able to do these without fibers and instead do stackless coroutines. But maybe with a little compiler help we can have that too.

Not all of my projects this December are code related. In fact a lot of them have been house maintenance things, joy of home ownership and all.

This week was spent building my new office and music space. I wanted a way to have my amplifiers and guitars more accessible while also creating a sort of “dark academia” sort of feeling for working.

The first step was to get the guitars mounted on the walls. I was looking for something blending artistic showpiece and functional use.

After that I quickly framed them in. Just quarter round with the hard edge inwards, 45° miter, some caulking, easy peasy.

My last office had Hale Navy as the color, but the sheen was too much that it made it difficult to actually see the color. This time I went flat and color drenched the space (so ceilings, trim, etc all in matching tone).

Then somewhat final result is here. I still want to have a lighting story for these that doesn’t involve a battery so some electrical fish taping is likely in my future.

I also converted the wide closet into a workstation area with the studio monitors for recording. But that is still partially finished as I need to plane all the slats for the slat wall, frame the builtin, and attach the countertop.

Early on Builder had the concept of layered settings. You had an application default layer the user could control. You also had a project layer which allowed the user to change settings just for that project. But that was about the extent of it. Additionally, these settings were just stored in your normal GSettings data repository so there is no sharing of settings with other project collaborators. Boo!

With Foundry, I’d like to have a bit more flexibility and control. Specifically, I want three layers. One layer for the user’s preferences at the application level. Then project settings which can be bundled with the project by the maintainer for needs specific to the project. Lastly, a layer of user overrides which takes maximum preference.

Of course, it should still continue to use GSettings under the hood because that makes writing application UI rather easy. As mentioned previously, we’ll have a .foundry directory we place within the project with storage for both user and project data. That means we can use a GKeyFile back-end to GSettings and place the data there.

You can git commit your project settings if you’re the maintainer and ensure that your projects conventions are shared to your collaborators.

Of course, since this is all command-line based right now, there are tab-completable commands for this which again, makes unit testing this stuff easier.

# Reads the app.devsuite.foundry.project config-id gsetting
# taking into account all layers
$ foundry settings get project config-id

# Sets the config-id setting for just this user
$ foundry settings set project config-id "'org.example.app.json'"

# Sets the config-id for the project default which might
# be useful if you ship multiple flatpak manifest like GTK does
$ foundry settings set --project project config-id "'org.example.app.json'"

# Or maybe set a default for the app
$ foundry settings set --global project stop-signal SIGKILL


That code is now wired up to the FoundryContext via foundry_context_load_settings().

Next time I hope to cover the various sub-systems you might need in an IDE and how those services are broken down in Foundry.

A core tenant of Foundry is a pleasurable command-line experience. And one of the most creature-comforts there is tab-completion.

But how you go about doing that is pretty different across every shell. In Flatpak, they use a hidden internal command called “complete” which takes a few arguments and then does magic to figure out what you wanted.

Implementing that when you have one layer of commands is not too difficult even to brute force. But imagine for a second that every command may have sub-commands and it can get much more difficult. Especially if each of those sub-commands have options that must be applied before diving into the next sub-command.

Such is the case with foundry, because I much prefer foundry config switch over foundry config-switch. Particularly because you may have other commands like foundry config list. It feels much more spatially aware to me.

There will be a large number of commands implemented over time, so keeping the code at the call-site rather small is necessary. Even more so when the commands could be getting proxied from another process or awaiting for futures to complete.

With all those requirements in mind, I came up with FoundryCliCommandTree. The tree is built as an n-ary tree using GNode where you register a command vtable with the command parts like ["foundry", "config", "switch"].

At each layer you can have GOptionEntry like you normally use with GLib-based projects but in this case they will end up in a FoundryCliOptions very similar to what GApplicationClass.local_command_line() does.

So now foundry has a builtin “complete” command like Flatpak and works fairly similarly though with the added complexity to support my ideal ergonomics.

I tend to bulk most of my vacation at the end of the year because it creates enough space and time for fun projects. Last year, however, our dog Toby went paraplegic and so were care-taking every three hours for about two months straight. Erratic sleep, erratic self-care, but in the end he could walk again so definitely worth it.

That meant I didn’t really get to do my fun end-of-year hacks beyond just polishing Ptyxis which I had just prototyped for RHEL/CentOS/Bluefin (and more recently Fedora).

This year I’m trying something I’ve wondered about for a while. What would it look like if you shoved a full-featured IDE into the terminal?

The core idea that makes this possible is using a sub-shell with a persistent parent process. So just like you might “jhbuild shell” you can “foundry enter” to enter the “IDE”.

In the JHBuild case it would exec over itself after setting things up. In the foundry case it maintains an ancestor process and spawns a sub-shell beneath that.

When running foundry commands from a sub-shell it will proxy that work to the ancestor instance. This all happens with a private D-Bus peer-to-peer process. So you can have multiple of these in place across different terminal tabs eventually.

This is all built with a “libfoundry” that I could consume from Builder in the future to provide the same feature from a full-blown GTK-based IDE too. Not to mention the IDE becomes instantly script-able from your shell. It also becomes extremely easy to unit test.

Since I originally created Builder, I wrote a library to make doing futures, concurrency, and fibers much easier in C. That is libdex. I tend to make things more concurrent while also reducing bug counts when using it. Especially for the complex logic parts which can be written in synchronous looking C even though it is asynchronous in nature.

So the first tenant of the new code is that it will be heavily based on DexFuture.

The second tenant is going to be a reversal of something I tried hard to avoid in Builder. That is a “dot” directory in projects. I never liked how IDEs would litter projects with state files in projects. But since all the others continue to do so I don’t see much value in tying our hands behind our back out of my own OCD purity. Instead, we’ll drop a .foundry directory with appropriate VCS ignore files. This gives us convenient space for a tmpdir, project-wide settings, and user settings.

The project is just getting started, but you can follow along at chergert/foundry and I’ll try to write more tidbits as I go.

Next time, we’ll cover how the command line tools are built as an N-ary tree to make tab-completion from bash easy.

The upcoming systemd v257 release will have support for a feature originating from ConEmu (a terminal emulator for Windows) which was eventually adopted by Windows Terminal.

Specifically, it is an OSC (Operating System Command) escape sequence which defines progress state.

Various systemd tools will natively support this. Terminal emulators which do not support it simply ignore the OSC sequence but those that do support it may provide additional UI to the application.

Lennart discussed this briefly in their ongoing systemd v257 features series on Mastodon and so I took up a quick attempt to implement the sequence parsing for VTE-based terminals.

That has since been iterated upon and landed in VTE. Additionally, Ptyxis now has corresponding code to support it as well.

Once GNOME CI is back up and running smoothly this will be available in the Ptyxis nightly build.

Based on some initial work by Barnabás Pőcze Sysprof gained support for symbolizing stack traces using debuginfod.

If you don’t want to install debuginfo packages for your entire system but still want really useful function names, this is for you. The system-configured debuginfod servers will provide you access to those debuginfo-enabled ELF binaries so that we can discover the appropriate symbol names.

Much like in gdb, you can cancel them if you don’t care and those symbols will fallback to the “In File lib.so+offset” you’re used to.

I expect a bit more UI work around this before GNOME 48 like preferences to disable it or configure alternate debuginfod servers.

Happy profiling!

There is this extremely neat feature in GCC 10+ and Clang 9+ that allows you to determine which function should be patched in at runtime. Procress startup code will call your user-defined function to figure out which function should be used for the environment.

For example, take the following

[code lang=”c”]
static gboolean (*choose_utf8_impl (void)) (const char *, gssize, const char **)
{
if (go_fast)
return fast_version;
else
return slow_version;
}

gboolean
g_utf8_validate (const char *str,
gssize len,
const char **end)
__attribute__((ifunc("choose_utf8_impl")));
[/code]

This can be handy when you’re dealing with complex branching based on the system. For example, imagine you need to check if you’re running under Valgrind as one does occasionally to satisfy an incorrect assumption about accessing past heap boundaries.

The typical way to handle this is to #include "valgrind.h" and branch based on if (RUNNING_ON_VALGRIND) ....

However, that inlines a bunch of assembly you probably don’t want in your ultra-fast new utf8-validation implementation. It might be better to just do that once at process startup and save yourself a bunch of CPU cycles.

One of the things that GLib-based APIs got right from early on is that generally speaking, our API boundaries expect UTF-8. When transitioning between API boundaries, UTF-8 validation is a common procedure.

Therefore, it is unsurprising that GLib does a lot of UTF-8 string validation of all sorts of string lengths.

The implementation we’ve had is fairly straight forward to read and reason about. Though it is not winning any performance awards.

UTF-8 validation performance has been on my radar for some time. Back when I was working on VTE performance it was an issue there. Recently with GVariant I had to look for ways to mitigate that.

There are many options out there which can be borrowed-from or inspired-by with varying levels of complexity.

Deterministic Finite Automaton

Björn Höhrmann wrote a branchless UTF-8 validator years ago which is the basis for many UTF-8 validators. Though once you really start making it integrate well with other API designs you’ll often find you need to add branches outside of it.

This DFA approach can be relatively fast, but sometimes slower than the venerable g_utf8_validate().

VTE uses a modified version of this currently for its UTF-8 validation. It allows some nice properties when processing streaming input like that of a PTY character stream.

Chromium uses a modified version of this which removes the ternary for even better performance.

simdutf

One of the more boundary-pushing implementations is simdutf used by the WebKit project. Simdutf is written in C++ and performance-wise it is extremely well regarded.

Being written in C++ does present challenges to integrate into a C abstraction library, though not insurmountable.

The underlying goal of a SIMD (single-instruction, multiple-data) approach is to look at more than a single character at a time using wide-registers and clever math.

c-utf8

Another implementation out there is the c-utf8 project. It is used by dbus-broker and libvarlink.

Like simdutf it attempts to take advantage of SIMD by using some very simple compiler built-ins. It is also simple C code (no assembly) with just the use of a few GCC’isms all of which can be removed or alternatives used for other compilers such as MSVC.

What to Choose

Just to throw up an idea to see if it could be shot down, I suggested we look at these for GLib (issue #3481) to replace what we have.

I haven’t been able to get as good of performance with the DFA approach as the SIMD approach. So I felt I needed to choose between c-utf8 and simdutf.

Given the ease of integration I went with the c-utf8 implementation. It required removing the case 0x01 ... case 0x7F: GCC-ism. Additionally I simplified a number of the macros which are part of sibling c-util projects in favor of GLib equivalents.

I opted to drop __builtin_assume_aligned(expr,n) and use __attribute__((aligned(n))) instead. It generates the same code on GCC but has the benefit of having an equivalent syntax on MSVC in the form of __declspec(align(n)) in the same syntax position (before the declarator). That means a simple preprocessor macro per-compiler gets the same effect.

Performance Numbers

The c-utf8 project has a nice benchmark suite available for testing UTF-8 validation performance. It will benchmark the performance against a trivial implementation which matches what GLib has at the time of writing this. It also will benchmark it against the performance of strlen() which is an extremely optimized piece of the stack.

I modeled the same tests but using g_utf8_validate() with my integration patches to ensure we were getting the same performance as c-utf8, which we are.

For long ASCII strings, you can expect more than 10x speed-ups. (On my Xeon it was 13x and on my Apple Silicon it was 12x). For long multi-byte (2-to-4 byte sequences) you can expect 4x-6x improvements.

I wouldn’t expect too much difference for small strings as you wont see much of a benefit until you can get to word-size operations. As long as we don’t regress there, I think this is a good direction to go in.

Merge Request !4319.