Happenings in GNOME

One of the larger bits of work I’ve been doing crosses many core projects. I’m motivated to switch to GNOME OS across multiple form factors, but to do that, homed and related tooling need quite a few functional gaps fixed.

TL;DR

• github.com/chergert/linux
• github.com/chergert/cryptsetup
• github.com/chergert/qemu
• github.com/chergert/systemd
• github.com/chergert/edk2

I really wanted a rather simple storage stack. I try to run XFS or ext4 in most places because they continue to serve me well. I also want to run GNOME OS on phones, where we’ll need dm-inlinecrypt for better performance and to avoid loading raw storage encryption keys into system memory.

Where I prefer something like thin provisioning is a multi-user setup, where I want encryption, integrity, layering, and reliable accounting without dedicating a fixed partition to every user.

Getting all of those properties at once required work from the filesystem down through device-mapper, the block layer, UFS, Qemu, and cryptsetup.

Keeping the storage key out of memory

Traditional full-disk encryption requires the raw volume key to enter kernel memory. That key can unlock the entire device, so extracting it from a running system is particularly valuable to an attacker.

Hardware-wrapped keys change this arrangement. The long-term key is stored as an opaque, device-bound blob. During activation, it is converted into a boot-scoped ephemeral blob and handed to the storage hardware. The hardware derives and programs the AES-XTS key without disclosing it to software.

There is still a separate 32-byte software secret for integrity and other cryptographic operations which cannot be offloaded. Knowing that secret does not reveal the inline-encryption key.

This reduces the opportunity to extract a reusable storage key, but it is not magic. It does not protect plaintext already present in memory, nor does it defeat an attacker who fully controls the running system.

The practical problem with hardware-wrapped keys is that they are difficult to develop and test without the relevant hardware. Even when hardware is available, failures across the complete stack can be difficult to reproduce and inspect.

So I started building the hardware I needed in Qemu.

A virtual UFS inline-crypto engine

The Qemu work adds an optional UFSHCI 4.1 inline-crypto profile. It supports AES-256-XTS, 512- and 4096-byte data units, 32 keyslots, 64-bit data-unit numbers, and both legacy and MCQ request formats.

It models more than just the encryption operation. Keyslots are programmed in stages before being activated, can be evicted, and are zeroized during reset. Requests take their own snapshots of key state so that concurrent eviction or reprogramming has deterministic behavior. Crypto failures are reported as storage errors rather than returning corrupted data.

There is also a test-only wrapped-key mailbox. It can import or generate a key, prepare a boot-scoped version, derive the associated software secret, program a keyslot, and evict it. The long-term and ephemeral representations use authenticated envelopes so tests can also exercise damaged or substituted blobs.

This mailbox models the API and lifecycle that the guest needs, but it is not a trusted execution environment. Qemu necessarily has access to its root secret.

Following an encrypted write

With the hardware model available, a write can be followed through the entire Linux stack.

dm-inlinecrypt attaches an encryption context to the I/O, including the key and data-unit number. The block inline-crypto layer programs a UFS keyslot, then UFS submits the request to Qemu. Only encrypted bytes reach the backing image. Reads take the reverse path.

This makes dm-inlinecrypt a useful full-device target. Device-mapper describes which blocks should be encrypted, while the actual transform stays in inline-encryption hardware.

The target distinguishes raw keys from hardware-wrapped keys. A long-term wrapped blob is prepared into a fresh ephemeral blob during activation. Only the ephemeral form is placed in an active device-mapper table.

Key replacement also makes secure suspend useful. Userspace can suspend the device and wipe the active key. Resume is refused until a replacement has been installed, at which point a new ephemeral key is prepared and programmed.

Integrity without losing the hierarchy

Encryption by itself does not prevent undetected modification, so the protected configuration places exported dm-integrity above dm-inlinecrypt.

That ordering means integrity authenticates the plaintext seen by the filesystem. Inline encryption protects the filesystem data as well as the integrity tags, superblock, and journal when they reach physical storage.

The HMAC-SHA256 integrity key is derived from the hardware-provided software secret using HKDF-SHA256. The binary LUKS UUID is used as the salt, along with a fixed domain-separation string. This keeps integrity separate from the hardware-only inline-encryption key.

I’ve added a fixed profile at 4096-byte integrity blocks, 32-byte HMAC tags, colocated metadata, and a 32 MiB journal. This is fairly intuition based so it needs more testing.

Suspend and resume follow the layering. Suspend wipes integrity first then inline encryption. Restoration is reversed. This is really hard to test with real hardware, so Qemu again really comes in handy.

Provisioning blocks before publishing them

Thin provisioning adds another problem. A filesystem may publish a logical allocation before the thin pool has assigned physical storage. Failure from lack of capacity is then deferred until too late such as when writing data, an integrity tag, or the integrity journal. All of those can be catastrophic.

I added REQ_OP_PROVISION based on earlier ideas on LKML to make persistent allocation a block-layer op. It is different from a write and is effectively the opposite of discard: it asks the storage stack to ensure that a range is physically backed.

Provisioning is carried through the block core, loop devices, device-mapper, thin volumes, dm-integrity, and ext4. Thin volumes allocate, zero, and commit their mappings. Integrity provisions every corresponding data, metadata, and journal region. Ext4 provisions new data and metadata extents before exposing mappings.

The initial ext4 support is conservative. The provision mount option implies nodelalloc, requires 4 KiB extents without bigalloc, rejects unsupported stacks, and disables online resize.

Turning it into a LUKS2 workflow

The cryptsetup work ties these pieces together.

A platform provisioner can generate or import a wrapped key, derive the optional software secret, format a LUKS2 device for hardware-wrapped encryption, and add ordinary LUKS2 keyslots using the opaque blob as the volume key.

Hardware-wrapped segments have an explicit key_type and mandatory requirements. The integrity configuration is also fixed and marked as dependent on hardware-wrapped-key integrity support. That should make older implementations reject the device.

During activation, cryptsetup retrieves the long-term blob, prepares an ephemeral one, creates dm-inlinecrypt, derives the integrity key, and finally creates dm-integrity above it.

For now, this interface is library-only which is how I’m using it from homed.

Testing all of this in a custom GNOME OS build resulted in me finding some issues in tianocore as well (edk2) which I’ve fixed in my tree to allow booting off PCI-UFS over SCSI.

A laboratory for the whole stack

The important result is that this can now be tested without specialized storage hardware.

We can run AES-XTS known-answer tests and independently inspect ciphertext in Qemu’s backing image. We can test both data-unit sizes, legacy and MCQ queues, fragmented requests, concurrency, reset, cancellation, rekeying, damaged envelopes, and storage errors.

We can also exercise thin-volume provisioning through integrity and inline encryption, reject malformed or downgraded LUKS2 metadata, and verify secure suspend, resume, and key replacement.

Each layer has tests which I tried to keep working and improve along the way.

There is plenty left to do. Namely, I’m not really interested in doing LKML type stuff while unemployed living abroad. So if this is something other people want, they’ll need to encourage their respective teams to pick up the work.

Either way, I now have something useful which is a virtual test lab for a security feature which requires each of these layers to work together.

Earlier this year as I drift abroad in France, I made a new abstraction over Avahi and systemd-resolved. It is called librebonjour and I wrote about it here.

It’s nice in that I no longer need to build Avahi to get GObject bindings to essentially call a D-Bus interface. It’s also nice to not have to care as an application developer if the system is configured with Avahi or systemd-resolved. Though, the systemd-resolved abstraction was lacking a bit compared to Avahi due to missing features.

When you are browsing for services using Avahi, you can be notified automatically of changes. This doesn’t quite work the same in systemd-resolved. Librebonjour had to set a timer and poll occasionally for updates and compare old-to-new sets to notify the application. Not very ideal.

When looking at a recent systemd checkout, I noticed that it already had support for the notification over its varlink interface. A handfull of commits later to hoist a few things and handle client disconnections/isolation properly and I can have the feature for librebonjour too.

One more dependency I can cut out of my system (there will be many more coming, I assure you, as GNOME is heavy with cruft).

I’ve been working on a bunch of things across the Linux puzzle for a product I want to build. Here is an overview of a few of those things.

LibMKS at 120hz

I wanted to get my virtual machines to 120hz so that I can start testing product features inside of VMs. In fact, I actually like doing development with virtual machines over say, trying to shove all your development tooling in a sysext which, at least to me, feels like square-peg/round-hole territory.

To get this working, a few things needed improvements.

Qemu

Qemu has a dbus display backend where it can send you DMABUF FD. But it doesn’t really handle any sort of sync and that becomes a problem as you crank up the frame rate. Additionally, it just defaulted to 75hz with no mechanism to override it.

So I have some patches which provide a new D-Bus interface which can be implemented by LibMKS. It provides something more like a Vulkan swap-chain as well as API to set the refresh rate. While this isn’t a mapping 1:1 of what a wayland protocol might do for frame rate, it does match more what the emulated graphics device expects, so it is probably fine for now and clearly an huge improvement.

A big change in the new API is that we will register all the DMABUF up front, and then tell the client just to switch to another DMABUF along with damage rectangles. Of course, I also had to make Qemu start collecting damage rectangles correctly.

• DMABUF3 Listener API for D-Bus
• Allow Clients to set Frame Rate

Linux

With those changes in place, I kept seeing damage being full-frame. The next part of the stack that can break is thus the Linux kernel virtio graphics driver. Damage rectangles come in as properties on the drm plane being submitted. So it turns out that in two places some short circuiting was preventing that from working right.

After fixing all that (and the corresponding LibMKS side) I have decent graphics performance in a VM.

Since I continue to be floating precariously abroad, this is my notice of such patches. If you are interested in seeing these upstream and work in either of those communities, feel free to crib them, improve them, and submit them upstream. I’m happy locally patching my software given the copious amount of free time I have so there is little incentive for me to collaborate with corporations.

• Virtio fix 1
• Virtio Fix 2

Combined with the LibMKS merge request !53 I can have both minimal damage rectangles all the way to host GPU scanout as well as drag windows around in the guest quite fast.

About a month ago Anthony Green added a novel “call plan” to libffi. This allows one to cache the hard work of determining what to do once and skip all of that work in subsequent calls. You can read more of the details on their article about it.

The best case for me was about a 25% reduction in amortized overhead for GClosure invocation. In some less ideal situations it was still about a 10% reduction. Not bad!

A merge request for GLib is here, though it will likely require some build system triage since it requires the newest libffi for CI to unblock it.

I also patched libffi to add frame-pointers so I can unwind across ffi boundaries from the Linux perf unwinder. Very handy if you want, you know, to profile your system in a useful manner. And repeated testing here showed about the same (or shockingly less) overhead than the counter-parts. Modern CPU sure are interesting beasts.

Writing state machines gets a bit of a bad reputation because they are often implemented in complex manners which are specific to the problem domain. I think that makes people shy away from writing them when they are truly beneficial, including myself.

Where they often go awry is when you have some sort of work that needs to be done asynchronously. This is exceedingly common in UI programming like GTK applications but just as easily found in daemons.

Because of this, I see people explicitly avoiding the state machine, or worse, implicitly avoiding its correctness by open-coding a solution across a dozen callbacks.

With DexLimiter and DexFiber I find I can write these state machines better.

You can use the limiter with a max-concurrency of 1 to get an “asynchronous Mutex” of sorts. No lock management necessary.

static void
password_daemon_init (PasswordDaemon *daemon)
{
  daemon->limiter = dex_limiter_new (1);
}

Imagine, if you will, that a limiter is a mutex plus a callback/closure which fires as soon as a slot is free. That means we need a little state to send to our transition callback.

/* Define our closure state for a transition */
DEX_DEFINE_CLOSURE_TYPE (StateTransition, state_transition,
  DEX_DEFINE_CLOSURE_OBJECT (PasswordDaemon, daemon),
  DEX_DEFINE_CLOSURE_VALUE (PasswordDaemonMode, target))

That is a nice wrapper around defining a struct with a new and free function.

Now we can request a transition of the state machine. Since our DexLimiter is an asynchronous mutex (with a single runnable slot), the fiber will not be spawned until it is the highest priority.

DexFuture *
password_daemon_transition (PasswordDaemon     *daemon,
                            PasswordDaemonMode  mode)
{
  StateTransition *transition;

  transition = state_transition_new ();
  transition->daemon = g_object_ref (daemon);
  transition->target = mode;

  return dex_limiter_run (daemon->limiter, NULL, 0,
                          password_daemon_transition_fiber,
                          transition,
                          state_transition_free);
}

That makes our transition code very clean when you combine the fiber with g_autoptr() and dex_await() to await the completion of futures. So a state machine might look like the following:

static DexFuture *
password_daemon_transition_fiber (gpointer user_data)
{
  TransitionState *state = user_data;
  GError *error = NULL;

  switch (state->target)
    {
     case PASSWORD_DAEMON_MODE_HANDOFF:
       if (state->daemon->mode != PASSWORD_DAEMON_MODE_INITIAL)
         return invalid_transition (state->daemon->mode,
                                    state->target);

       if (!password_daemon_enter_handoff (self, &error))
         return dex_future_new_for_error (&error);

       break;

      case PASSWORD_DAEMON_MODE_LOCKED:
        ...

      case PASSWORD_DAEMON_MODE_UNLOCKED:
        ...
    }

  return dex_future_new_enum (state->daemon->mode);
}

static gboolean
password_daemon_enter_handoff (PasswordDaemon  *daemon,
                               GError         **error)
{
  GSocket *control;

  if (!(control = dex_await_object (create_socket (), error)))
    return FALSE;

  ...
}

What I find nice about this is enter/leave transition components can be customized for the state machine transition. That leaves room for transitions between states which require specialization for correctness.

This is much cleaner than ad-hoc callbacks chained together because you can await in the transition fiber for asynchronous work to complete and the state machine itself is preserved. No shoving temporary state in your class instance. No testing hell to see if you caught all the failure cases. No pain with sequencing or order of main loop processing.

Hopefully that shows you can use libdex to write more correct and cleaner state machines by keeping the majority of the implementation in one place.

I occasionally see people go through great effort to do end-to-end testing of keyboard input latency. That is fantastic but it requires hardware and patience I don’t, nor will ever, have.

Here is a much simpler way to get about 90% of the value. For example, everything but driver/interrupt handler latency and display link scanout/monitor visibility latency and of course your app side (but you could theoretically rig this up to do that too, inside your app). Not that those aren’t important, but they definitely fall into the category of things I personally cannot control for you.

Keyspeed is a very simple GTK application which uses /dev/uinput to synthesize keypresses. Since it knows the time of provenance, it can compare that to when it gets the event back from compositor delivery.

Wrap all that data up in Sysprof capture marks, pull in some from the compositor (GNOME Shell/Mutter support this), tie in some callgraphs/flamegraphs, and you have a very good overview of what is going on during your keypress.

Run it like this (but remember to chmod back when you’re done less you have attack surface available).

$ sudo chmod 660 /dev/uinput
$ git clone https://gitlab.gnome.org/chergert/keypress
$ sudo dnf install sysprof-devel libinput-devel gtk4-devel
$ make
$ sysprof-cli --gtk --gnome-shell capture.syscap -- ./keyspeed
$ sysprof capture.syscap

Currently, this only shows you keypress send to receive in GTK, but if someone cared enough, you could make it take the next GtkFrameTimings and use that to get the presentation time. I don’t need that for what I’m doing, so it doesn’t.

If you go to the marks section, you can dive in to a specific keypress/release cycle. Zoom in on just that section, switch back to callgraph/flamegraph profiler and see what was going on.

Pretty simple, no special hardware needed.

You can see how long it took, where time was spent, and more importantly, how much time was empty between things that matter.

Gom is a very old object mapper I wrote to bridge GObject to SQLite. It made a lot of assumptions about the world based on when it was prototyped.

The past couple years had me using it again for the documentation search in Manuals. Typically, I would have just built Manuals to parse all the XML files on disk and hold them in memory. That’s how both Devhelp and Builder always did things. Once we started supporting Flatpak SDKs that was no longer realistic. You could have numerous SDKs all with copies of the overlapping data and it just became easier to have a query model.

One of the more performance critical limitations was the locking model. When gom-1.0 was written, it was not common for distributions to compile SQLite with locking support. So you just created a single thread and did your work over there.

Bolting fulltext search and many other missing features onto the old ABI just wasn’t realistic. Especially when I’ve wanted to make the thing properly async for years. One of my other projects, Libdex is just right over there and perfect for this sort of problem.

The landscape changed and so do our horizons.

A new informed ABI

In the years after Gom was prototyped, I worked at a commercial database company and learned a great deal about implementing the internals of both that database and more traditional RDBMS. That left a certain cringe on my mouth whenever looking at my code predating it. Knowing how things get done inside the database allows for building better APIs to interact with it.

This time everything is async. Queries are modeled like you do with a compiler. Lowered into the back-end specific implementation. There can be an entity map and real transactions which allows you to read back the same instance despite which query inflated it.

The Center

Your early stage objects are the GomRepository, GomDriver, and GomRegistry.

The registry describes the entities that can exist within the repository. This is handy because it allows us to pre-compile information into a model that is both immutable and fast at runtime. Compare that to methods like g_object_class_list_properties() which is a performance bottleneck of its own.

The driver is very obvious. It is our abstraction layer for the database engines. Currently we have support for SQLite and PostgreSQL.

The repository is the center of the center. It is how you query, insert, update, delete, transact, and more. It is likely your application instance owns one of these unless of course you use Gom as your file format in which case you’ll have one per “document”.

Two Access Models

This new version of Gom can support either the entity mapping you’re used to or; optionally, raw access to relations/projections via the GomCursor.

As the cursor moves through the resulting rows you will have access to all the projections requested in the query. Though it holds enough information to allow you to gom_cursor_materialize() the row into an GomEntity subclass.

If you want a snapshot of that cursor row without materializing, you can use GomRecord which can also conveniently be used in GListModel for integration into GTK applications.

Most of the time, you’ll use materialization. And even then it is likely to happen through automated collections rather than with a cursor directly. More on that later.

Sessions

As I mentioned, there was no concept of transactions previously.

In this iteration we have GomSession. It is your standard identity-map layer with transaction-scoping. If you perform multiple queries for the same record, the session will ensure you get the same instance back. That is essential when you do local mutations on an instance and what to see that reflected in followup queries.

Additionally, it makes it nice to have multiple views of an object with an editor or listview and needing them to stay in sync.

Relationship Modeling

Support for relationships was adhoc previously. We had some functions named in ways that made you think you could, but I assure you, they were not well tested.

This time around you can model your GomEntity with 1:1, 1:M, M:M, inverse, self-referencing, all while handling proper delete rules. Combing this with the session support mentioned previously is crucial.

So now you should be able to show related models easily in GtkListView while keeping the paginated-and-lazy model beneath it transactional.

Migrations

In the previous version migrations were dynamic, but largely controlled by Gom itself. Very inflexible.

This time around we have things broken down into Migrator and Migration.

You can use built-in implementations like the EntityMigrator or implement your own. CustomMigrator makes that easy. Especially since you can inject your own migrations at just the right point.

Internally, libgom-2 can snapshot your GomRegistry at specific versions based on the provided metadata. Then it performs a diff between two versions of the registry to determine what migration work must be done.

You can just as easily use a SqlMigration with custom SQL scripts. This stuff is all highly composable now to get exactly what you need.

Live List Models

I’ve written many ways to get live SQLite results into GTK over the past two decades. I think one of the first was a GtkTreeModel implementation for GTK 2 which could do it. With that in mind, it was still rather annoying when making Manuals so I set off to make that convenient.

We have GomRecordListModel, GomEntityListModel, GomRelatedListModel, GomQueryModel all of which have practical uses based on application needs.

But in short, most of those are lazy and support transaction-backed stable identities for entities. Very useful when you have a list of items and an editor loaded in another frame, both of which must reflect the same data.

Expression Trees

This time around I implemented proper expression trees. They model the query, relations, and projections in a manner that allows the driver to lower into a query much more accurately.

You can model things like function-calls cleanly all of which required writing manual SQL before. If you did anything outside of what gom could generate previously, it became madness to maintain.

Vectors

This version of libgom embeds the vec1 extension for SQLite. That means we can store vectors in your records and query them. GomVector makes that easier to manage as a property within your application entity.

I can think of a few things this will be useful for, maybe you can too.

Profiling

This version of libgom has profiling support with another project of mine, Sysprof. The whole library emits profiler marks about what is going on so that it is easy for you to figure out why something might be slow in your application.

Since we’ve already done the integration of Sysprof into GLib/GObject, GTK, Pango, Libdex, and GNOME Shell/Mutter you can very quickly get an idea with details of what is going on in your application. Click record, select the problem area, zoom, and it is often pretty clear. You can have flamegraphs, callgraphs, and timing marks all in one place.

Local First with Sync Coordination

One of my personal motivations for this is around building a native sync protocol for applications I’m building. I wrote numerous SQLite-based sync protocols for the now defunct catch.com before they were acquired by apple. That means I know multiple wrong ways to do it.

This time around, I want to put it right in the data-mapper at the point where you have the most insight. So libgom has the right abstractions in place to build that. The GomSyncCoordinator manages the process and GomSyncTransport is the abstraction-point for service integration.

You work with GomDelta at this layer. The application can provide you with a GomMergePolicy to help make decisions which allow for contextually doing the right thing.

This part is still very new. I’m still building the other side of it but landing the shape early allows me to mock and test things comprehensively before committing to the ABI.

My goal is building a practical, robust, and correct implementation for personal local first features.

A small personal note: as I wrote in my recent update from France, I am no longer employed by Red Hat. Work like this is currently self-funded, out of pocket, while my family and I settle into a new chapter. If you find it useful, a note of encouragement or a contribution means a lot right now. It helps make it possible to keep improving the free software infrastructure many of us rely on.

Fibers are always a nice way to keep your async C code clean while using Libdex. However, occasionally you may want a lighter option which doesn’t require a stack or saving registers for work doing little more than coordinating futures.

I’ve added Stackless Coroutines for this which still allows writing future-coordinating code. Though this will suspend/resume your coroutine by re-entering the function and jumping to the next position. Your threads stack is reused. State is saved in your closure state.

This isn’t a new concept. It is really old just like fibers. What is useful is that this style of continuation passing may still be represented as a DexFuture and therefore composed like the others.

You can place these stackless coroutines in DexTaskGroup alongside fibers, threadpool work, and others. Cancellation will propagate to a clean exit point of the coroutine just like it would with a fiber.

Overhead is a bit lower than fibers in synthetic benchmarks depending on use. I was actually impressed our fiber implementation performed as well as it did head-to-head.

To make building your coroutine continuation easier, libdex provides a handy macro to create your typedef struct, _new(), and _free() helpers in a single macro expansion using DEX_DEFINE_CLOSURE_TYPE().

You use it like this:

DEX_DEFINE_CLOSURE_TYPE (MyTaskState, my_task_state,
  DEX_DEFINE_CLOSURE_VALUE (gsize, bytes),
  DEX_DEFINE_CLOSURE_POINTER (GBytes *, bytes_obj, g_bytes_unref),
  DEX_DEFINE_CLOSURE_OBJECT (GSocketConnection, conn))

Coroutines cannot use the exact syntax that fibers do for awaiting, which is a bummer, but a side-effect of trying to make something that works across Linux, Windows, FreeBSD, Solaris, macOS, etc. Particularly because the implementation must use switch/case to stay portable without address-of-label support on MSVC nor clang-cl.exe.

So awaiting is a bit more clear you’re suspending/resuming the stackless coroutine.

DEX_DEFINE_CLOSURE_TYPE (LoadState, load_state,
  DEX_DEFINE_CLOSURE_OBJECT (GFile, file),
  DEX_DEFINE_CLOSURE_OBJECT (GFileInputStream, input),
  DEX_DEFINE_CLOSURE_OBJECT (GFileInfo, info),
  DEX_DEFINE_CLOSURE_VALUE (int, io_priority))

static DexFuture *
do_something (DexCoroutineContext *context,
              gpointer             user_data)
{
  LoadState *state = user_data;
  g_autoptr(GError) error = NULL;

  DEX_COROUTINE_BEGIN (context);

  DEX_COROUTINE_SUSPEND_OBJECT (
    &state->input, &error,
    dex_file_read (state->file, state->io_priority));

  if (error != NULL)
    return dex_future_new_for_error (g_steal_pointer (&error));

  DEX_COROUTINE_SUSPEND_OBJECT (
    &state->info, &error,
    dex_file_input_stream_query_info (
      state->input,
      G_FILE_ATTRIBUTE_STANDARD_SIZE,
      state->io_priority));

  if (error != NULL)
    return dex_future_new_for_error (g_steal_pointer (&error));

  /* maybe do something useful here */

  return dex_future_new_int64 (g_file_info_get_size (state->info));

  DEX_COROUTINE_END;
}

You do need to be careful about placing things on the stack, because they wont be there on the other side of that DEX_COROUTINE_SUSPEND_* macro expansion. That is because when the scheduler jumps back into your stackless coroutine, it will use a switch/case to jump to the next bit of code. Don’t fear though, just add your state to your continuation which we’ve established is easy to do now.

If you don’t like these macros, you can do things the manual way using dex_coroutine_context_suspend() and dex_coroutine_context_resume() who’s APIs are not terrible either. They do require you make up your own program-counter regime though which for the macro case is basically just __COUNTER__.

You can spawn your coroutine using dex_scheduler_spawn_coroutine() or as part of a work-queue in DexLimiter with dex_limiter_run_coroutine().

dex_scheduler_spawn_coroutine (
  dex_thread_pool_scheduler_get_default (),
  my_coroutine,
  my_coroutine_new (),
  (GDestroyNotify) my_coroutine_free);

I hope you've enjoyed this attempt to make another 1970s technology useful in a modern world.

libdex 1.2 is still in pre-alpha phase but it is also far enough along that it is worth talking about the direction: libdex is growing from a library of future and fiber helpers into a more complete concurrency toolkit.

The most important 1.2 theme is that applications can now describe not just what work should happen concurrently, but how that work should be bounded and owned. DexLimiter lets a workload run with a fixed concurrency budget, with dex_limiter_run() handling the common fiber case by acquiring a permit before work starts and releasing it after the fiber completes. For larger workflows, DexTaskGroup gives related futures a structured scope that can be closed, awaited, or cancelled as one unit.

That combination makes cleanup much easier to reason about when a workflow has many moving pieces. A loader can start many subtasks, keep only a useful number active at once, and return a single future representing the whole operation. If the window closes, the project changes, or the operation times out, the group gives the application one place to cleanly shut the work down.

static DexFuture *
load_many_files (GPtrArray *files)
{
  g_autoptr(DexTaskGroup) group = dex_task_group_new (0);
  g_autoptr(DexLimiter) limiter = dex_limiter_new (8);

  for (guint i = 0; i < files->len; i++)
    {
      GFile *file = g_ptr_array_index (files, i);

      dex_task_group_add (group,
                          dex_limiter_run (limiter,
                                           NULL,
                                           0,
                                           load_one_file,
                                           g_object_ref (file),
                                           g_object_unref));
    }

  return dex_future_with_timeout_seconds (dex_task_group_close (group), 10);
}

There is also a new DexThreadPool for the cases that are not naturally fiber-shaped. Fibers and schedulers are still the right fit for cooperative async work, but many applications need to integrate blocking libraries, database clients, filesystem helpers, or other foreign code. A fixed pool of reusable OS threads, dex_thread_pool_submit(), and asynchronous dex_thread_pool_close() give that integration story a bounded queue and an explicit shutdown path.

Deadlines are another practical piece of the same story. The new timeout wrappers, including dex_future_with_timeout_seconds() and dex_future_with_deadline(), turn time limits into ordinary future composition. Instead of open-coded timeout state spread across an application, a future can resolve normally, reject normally, or reject with DEX_ERROR_TIMED_OUT when the deadline wins.

On the I/O side, 1.2 continues filling in the operations that make responsiveness easier to preserve. dex_aio_open() and dex_aio_close() matter because even operations that look small can stall when they touch the kernel, storage, or network-backed filesystems. Keeping those calls in libdex’s file-descriptor AIO model makes it easier to keep them off the UI thread, using io_uring where it is available and the fallback AIO backend elsewhere.

The broader GIO coverage is intentionally less surprising, but still important. More app launching, GFile, stream, socket, resolver, proxy, TLS, DTLS, permission, subprocess, and Unix-facing APIs now have future-first wrappers. That is the kind of coverage people should expect from libdex over time: not every wrapper needs a release headline, but each one reduces the pressure to leave the future model for common GNOME application work.

I made another weird side project while unemployed. In fact I’ve wanted it for a while but once I learned that “Rebonjour” is the word for “hello again” I just had to finish the library.

librebonjour is an asynchronous DNS-SD and mDNS client library for GLib applications. Or, more practically, it is a small GObject API over the two local service-discovery providers you are likely to find on a Linux system: Avahi and systemd-resolved.

It does not link against either of them. It only talks to them over D-Bus.

The reason for that is mostly boring, which is usually where the useful things are. Applications should not need to care if a machine has Avahi running, or if it is using systemd-resolved for mDNS. They should be able to discover a service, resolve it, maybe advertise something, and get on with whatever they were actually trying to do.

So RebonjourClient selects a backend internally. If org.freedesktop.Avahi is available on the system bus, it uses Avahi. If not, it falls back to systemd-resolved’s org.freedesktop.resolve1 API. If neither is around, availability checks fail like you would expect.

The public API stays the same either way.

What It Does

There are three common things I wanted to make pleasant.

First, one-shot discovery. Ask for the service types in local, ask for instances of something like _ipp._tcp, then resolve one of those instances into addresses and TXT metadata.

Second, browser-style discovery. A RebonjourBrowser owns a stable GListModel of RebonjourService objects. That fits nicely into GTK code because the model object can stay the same while the contents change underneath it.

Third, registration. You can describe a local service with RebonjourServiceDescription, register it, and keep the returned RebonjourRegistration alive for as long as the service should be advertised.

Resolving a service gives you a RebonjourResolvedService. That contains the SRV result, TXT data, priority, weight, and a model of RebonjourEndpoint objects. The endpoints hold the GSocketAddress you would actually use to connect.

Why Two Backends

Avahi is the nicer backend for browsing. Its D-Bus API gives you long-lived browser objects and emits signals when services appear and disappear. That maps very naturally to GListModel changes.

systemd-resolved is different. It has useful DNS-SD and mDNS operations over D-Bus, but the browsing side is lookup-based. That means you can ask what is there, but you do not get the same live add/remove signal stream that Avahi provides.

I did not want applications to have to care about that distinction unless they really want to. So the browser has auto-refresh and refresh-interval properties. With Avahi, auto-refresh is effectively harmless because the model is already live. With systemd-resolved, it starts an internal refresh loop and updates the model for you.

It is not magic. It is just putting the backend-shaped unpleasantness in one place so application code can stay boring.

Asynchronous with libdex

The whole thing is built on libdex. Anything that might touch D-Bus or the network returns a DexFuture.

That means construction, availability checks, service-type lookup, instance lookup, resolving, registration, browser refresh, and unregistering are all future-based. If you are already writing fiber-style code with libdex, the API fits into that directly:

[code language=”c”]
g_autoptr(RebonjourClient) client = NULL;
g_autoptr(GListModel) services = NULL;
g_autoptr(GError) error = NULL;

if (!(client = dex_await_object (rebonjour_client_new (), &error)))
g_error ("%s", error->message);

services = dex_await_object (rebonjour_client_lookup_instances (client,
0,
"_ipp._tcp",
NULL,
REBONJOUR_LOOKUP_FLAGS_NONE),
&error);
[/code]

The 0 there means any interface. Passing NULL for the domain uses local. The common case should not require looking up interface indexes which I’m pretty sure most people reading this have never even done before.

Advertising

Advertising is where things get more system-policy-oriented.

With Avahi, registration goes through Avahi’s D-Bus API. With systemd-resolved, registration uses RegisterService and UnregisterService, which are polkit-protected. Also, resolved needs full mDNS enabled with MulticastDNS=yes; MulticastDNS=resolve is enough to browse and resolve, but not enough to respond as a service.

So librebonjour can expose one API for registration, but it cannot make host policy disappear. Applications still need to handle authorization failure, missing mDNS responder support, sandbox boundaries, or whatever policy the system administrator has decided is appropriate.

That seems like the right way to demarcate things. The library should hide the provider mechanics, not the permissions of the platform.

Why

Mostly because I wanted this to exist.

DNS-SD is handy. Local-network service discovery is still useful. But using it from a GLib application means either caring too much about the provider or writing just enough glue that every application gets to have its own slightly different version of the same code.

And even worse is having to bundle things to build projects like Avahi for Flatpak when you only use the library which calls into D-Bus anyway.

This is not a grand platform initiative. It is not something I am employed to maintain. So you know, use wisely.