Status Week 43

Got a bit side-tracked with life stuff but lets try to get back to it this week.

Libdex

  • D-Bus signal abstraction iteration from swick

  • Merge documentation improvements for libdex

  • I got an email from the Bazaar author about a crash they’re seeing when loading textures into the GPU for this app-store.

    Almost every crash I’ve seen from libdex has been from forgetting to transfer ownership. I tried hard to make things ergonomic but sometimes it happens.

    I didn’t have any cycles to really donate so I just downloaded the project and told my local agent to scan the project and look for improper ownership transfer of DexFuture.

    It found a few candidates which I looked at in detail over about five minutes. Passed it along to the upstream developer and that-was-that. Their fixu-ps seem to resolve the issue. Considering how bad agents are at using the proper ownership transfer its interesting it can also be used to discover them.

  • Add some more tests to the testsuite for future just to give myself some more certainty over incoming issue reports.

Foundry

  • Added a new Gir parser as FoundryGir so that we can have access to the reflected, but not compiled or loaded into memory, version of Gir files. This will mean that we could have completion providers or documentation sub-system able to provide documentation for the code-base even if the documentation is not being generated.

    It would also mean that we can perhaps get access to the markdown specific documentation w/o HTML so that it may be loaded into the completion accessory window using Pango markup instead of a WebKitWebView shoved into a GtkPopover.

    Not terribly different from what Builder used to do in the Python plugin for code completion of GObject Introspection.

  • Expanding on the Gir parser to locate gir files in the build, system, and installation prefix for the project. That allows trying to discover the documentation for a keyword (type, identifier, etc), which we can generate as something markdowny. My prime motivation here is to have Shift+K working in Vim for real documentation without having to jump to a browser, but it obviously is beneficial in other areas like completion systems. This is starting to work but needs more template improvements.

  • Make FoundryForgeListing be able to simplify the process of pulling pages from the remote service in order. Automatically populates a larger listmodel as individual pages are fetched.

  • Start on Gitlab forge implementation for querying issues.

    Quickly ran into an issue where gitlab.gnome.org is not servicing requests for the API due to Anubis. Bart thankfully updated things to allow our API requests with PRIVATE-TOKEN to pass through.

    Since validating API authorization tokens is one of the most optimized things in web APIs, this is probably of little concern to the blocking of AI scrapers.

  • Gitlab user, project, issues abstractions

  • Start loading GitlabProject after querying API system for the actual project-id from the primary git remote path part.

  • Support finding current project

  • Support listing issues for current project

  • Storage of API keys in a generic fashion using libsecret. Forges will take advantage of this to set a key for host/service pair.

  • Start on translate API for files in/out of SDKs/build environments. Flatpak and Podman still need implementations.

  • PluginGitlabListing can now pre-fetch pages when the last item has been queries from the list model. This will allow GtkListView to keep populating in the background while you scroll.

  • mdoc command helper to prototype discover of markdown docs for use in interesting places. Also prototyped some markdown->ansi conversion for nice man-like replacement in console.

  • Work on a new file search API for Foundry which matches a lot of what Builder will need for its search panel. Grep as a service basically with lots of GListModel and thread-pool trickery.

  • Add foundry grep which uses the foundry_file_manager_search() API for testing. Use it to try to improve the non-UTF-8 support you can run into when searching files/disks where that isn’t used.

  • Cleanup build system for plugins to make it obvious what is happening

  • Setup include/exclude globing for file searches (backed by grep)

  • Add abstraction for search providers in FoundryFileSearchProvider with the default fallback implementation being GNU grep. This will allow for future expansion into tooling like rg (ripgrep) which provides some nice performance and tooling like --json output. One of the more annoying parts of using grep is that it is so different per-platform. For example, we really want --null from GNU grep so that we get a \0 between the file path and the content as any other character could potentially fall within the possible filename and/or encoding of files on disk.

    Where as with ripgrep we can just get JSON and make each search result point to the parsed JsonNode and inflate properties from that as necessary.

  • Add a new IntentManager, IntentHandler, and Intent system so that we can allow applications to handle policy differently with a plugin model w/ priorities. This also allows for a single system to be able to dispatch differently when opening directories vs files to edit.

    This turned out quite nice and might be a candidate to have lower in the platform for writing applications.

  • Add a FoundrySourceBuffer comment/uncomment API to make this easily available to editors using Foundry. Maybe this belongs in GSV someday.

Ptyxis

  • Fix shortcuts window issue where it could potentially be shown again after being destroyed. Backport to 48/49.

  • Fix issue with background cursor blink causing transparency to break.

Builder

  • Various prototype work to allow further implementation of Foundry APIs for the on-foundry rewrite. New directory listing, forge issue list, intent handlers.

Other

  • Write up the situation with Libpeas and GObject Introspection for GNOME/Fedora.

Research

  • Started looking into various JMAP protocols. I’d really like to get myself off my current email configuration but it’s been around for decades and that’s a hard transition.

Libpeas and Introspection

One of the unintended side-effects of writing applications using language bindings is that you inherit the dependencies of the binding.

This made things a bit complicated when GIRepository moved from gobject-introspection-1.0 to girepository-2.0 as we very much want language bindings to move to the new API.

Where this adds great difficulty on maintainers is in projects like Libpeas which provides plug-in capabilities for GTK application developers across multiple programming languages.

In practice this has allowed applications like Gedit, Rhythmbox, and GNOME Builder to be written in C but load plugins from languages such as Python, Lua, JavaScript, Rust, C, C++, Vala, or any other language capable of producing a .so/.dylib/.dll.

A very early version of Libpeas, years before I took over maintaining the library, had support for GObject Introspection baked in. This allowed really interesting (at the time) tooling to perform dynamic method call dispatching using a sort of proxy object implemented at runtime. Practically nobody is using this feature from what I can tell.

But maintaining ABI being what it is, the support for it has long been part of the libpeas-1.x ABI.

A couple years ago I finally released a fresh libpeas-2.x ABI which removed a lot of legacy API. With objects implementing GListModel and PleasPluginInfo becoming a GObject, the need for libpeas-gtk dwindled. It’s extremely easy for your application to do everything provided by the library. Additionally, I removed the unused GObject Introspection support which means that libpeas-2.x no longer needs to link against gobject-introspection-1.0 (nor girepository-2.0).

One area where those are still used is the testsuite. This can complicate testing because we want to make sure that APIs work across language bindings but if the language binding uses a specific version of GObject Introspection that does not align with what the libpeas project is using for tests, it will of course lead to runtime disasters.

Such is the case with some language bindings. The Lua LGI project is scarcely maintained right now and still uses gobject-introspection-1.0 instead of girepository-2.0. I submitted patches upstream to port it over, but without an official maintainer well versed in C and language bindings there isn’t anyone to review and say “yes merge this”.

There is a fork now that includes some of the patches I submitted upstream, but the namespace is different so it isn’t a 1:1 replacement.

The PyGObject project has moved to girepository-2.0 upstream and that caused some breakage with applications in Fedora 42 that were still using the legacy libpeas-1.x ABI. For that reason, I believe the older PyGObject was released With Fedora 42.

If you are using libpeas-2.x in your application, you have nothing to fear with any of the language runtimes integrated with libpeas. Since libpeas doesn’t link against either introspection libraries, it can’t hurt you. Just make sure your dependencies and language bindings are all in sync and you should be fine.

If you are using libpeas-1.x still (2.x was released a little over 2 years ago) then you are in a much worse shape. Language bindings are moving (or have moved) to girepository-2.0 while libpeas cannot realistically be ported and maintain ABI. Too much is exposed as part of the library itself.

It is imperative that if you want to keep your application working that you are either on libpeas-2.x or you’re bundling your application in such a way that you can guarantee your dependencies are all on the same version of GObject Introspection.

Halfway ABI

There exists a sort of “half-way-ABI” that someone could work on with enough motivation which is to break ABI as a sort of libpeas-1.38. It would move to girepository-2.0 and all the ABI side-effects that come with it. Since the introspection support in libpeas-1.x is rarely used there should be little side-effects other than recompiling against the new ABI (and the build system transitions that go along with that).

In my experience maintaining the largest application using libpeas (being Builder), that is really a lot more effort than porting your applications to libpeas-2.x.

Is my app effected?

So in short, here are a few questions to ask yourself to know if you’re affected by this.

  • Does my application only use embedded plug-ins or plug-ins from shared-modules such as *.so? If so, then you are all set!
  • Do I use libpeas-1.x? If no, then great!
  • Does my libpeas-1.x project use Python for plug-ins? If yes, port to libpeas-2.x (or alternatively work suggested halfway-ABI for libpeas).
  • Does my libpeas-1.x or libpeas-2.x project use Lua for plug-ins? If yes, make sure all your dependencies are using gobject-introspection-1.0 only. Any use of girepository-2.0 will end in doom.

Since JavaScript support with GJS/MozJS was added in libpeas-2.x, if you’re using JavaScript plug-ins you’re already good. GJS recently transitioned to girepository-2.0 already and continues to integrate well with libpeas. But do make sure your other dependencies have made the transition.

How this could have been avoided?

Without a time machine there are only three options besides what was done and they all create their own painful side-effects for the ecosystem.

  1. Never break ABI even if your library was a stop gap, never change dependencies, never let dependencies change dependencies, never fix anything.
  2. When pulling GObject Introspection into the GLib project, rename all symbols to a new namespace so that both libraries may co-exist in process at the same time. Symbol multi-versioning can’t fix overlapping type name registration in GType.
  3. Don’t fix any of the glaring issues or inconsistencies when pulling GObject Introspection into GLib. Make gobject-introspection-1.0 map to the same thing that girepository-2.0 does.

All of those have serious side-effects that are equal to if not worse than the status-quo.

Those that want to “do nothing” as maintainers of their applications can really just keep shipping them on Flatpak but with the Runtime pinned to their golden age of choice.

Moral of the story is that ABI’s are hard even when you’re good at it. Doubly so if your library does anything non-trivial.

mi2-glib

At Red Hat we are expected to set, and meet, goals each quarter if we want our “full” bonus. One of those is around introducing AI into our daily work. You’ve probably seen various Red Hat employees talking about using AI. It’s hard to deny there is financial incentive to do so.

Astute students of behavioral science know that humans work harder to not lose something than to gain something new. Arguably it’s only a “mandate to use AI” if you are entitled to the revenue so attaching it to your bonus is a convenient way to both “not be a mandate” and take advantage of the human behavior to not lose something.

La mujer del César no solo debe ser honesta, sino parecerlo.

Conveniently, I got pretty fed up with Debug Adapter Protocol (DAP for short) this week. So I wondered how hard it would be to just talk the MI2 protocol like Builder does (using the GPLv3+ gdbwire) but with a libdex-oriented library. It could potentially make integrating GDB into Foundry much easier.

Red Hat has so generously given us “ephemeral credits” at Cursor.com to play with so I fired up the cursor-agent command line tool and got started.

My goal here is just to make a minimal API that speaks the protocol and provides that as DexFuture with a very shallow object model. The result for that is mi2-glib. This post is mostly about how to go about getting positive results from the bias machine.

Writing a fully featured library in a couple hours is certainly impressive, regardless of how you feel about AI’s contributions to it.

Before I continue, I want to mention that this is not used in Foundry and the GDB support which landed today is using DAP. This was just an experiment to explore alternatives.

Create Project Scaffolding

To get the project started I used Foundry’s shared library template. It creates a C-based project with pkg-config, GObject Introspection support, Vala *.vapi generation, versioning with ABI mechanics, and gi-doc based documentation. Basically all the tricky things just work out of the box.

You could probably teach the agent to call this command but I choose to do it manually.

foundry template create library
Location[/home/christian]: .

The name for your project which should not contain spaces
Project Name: mi2-glib

The namespace for the library such as "Mylib"
Namespace: Mi2
 1: No License
 2: AGPL 3.0 or later
 3: Apache 2.0
 4: EUPL 1.2
 5: GPL 2.0 or later
 6: GPL 3.0 or later
 7: LGPL 2.1 or later
 8: LGPL 3.0 or later
 9: MIT
10: MPL 2.0
License[7]:
Version Control[yes]: 
mi2-glib/meson.options
mi2-glib/meson.build
mi2-glib/lib/meson.build
mi2-glib/testsuite/meson.build
mi2-glib/lib/mi2-glib.h
mi2-glib/lib/mi2-version.h.in
mi2-glib/lib/mi2-version-macros.h
mi2-glib/testsuite/test-mi2-glib.c
mi2-glib/doc/mi2-glib.toml.in
mi2-glib/doc/urlmap.js
mi2-glib/doc/overview.md
mi2-glib/doc/meson.build
mi2-glib/README.md
mi2-glib/LICENSE
mi2-glib/.foundry/.gitignore
mi2-glib/.foundry/project/settings.keyfile
mi2-glib/.git/objects
mi2-glib/.git/refs/heads
mi2-glib/.git/HEAD

Building Context

After that I added an AGENTS.md file that described how I want code to be written. I gave it some examples of my C style (basically opinionated GTK styling). That is things like preferring autoptr over manual memory management, how to build, how to test, etc. Consider this the “taste-making” phase.

The next thing I needed to do was to ensure that it has enough examples to be able to write libdex-based GObject code. So I copied over the markdown docs from libdex and some headers. Just enough for the agent to scan and find examples to prime how it should be used. Update your AGENTS.md to note where that documentation can be found and what it can expect to find there. That way the agent knows to look for more details.

Just as a matter of practicality, it would be great if we could have gi-doc generate markdown-formatted descriptions of APIs rather than just HTML.

Now that is all good and fun, but this is about mi2 now isn’t it? So we need to teach the agent about what mi2 is and how it works. Only then will it be able to write something useful for us.

GDB conveniently has a single-file version of the documentation so we need not piss off any network admins with recursive wget.

That left me with a big HTML document with lots of things that don’t matter. I wanted something more compact that will better fit into the context window of the agent. So I asked the agent to look over the HTML file and amalgamate a single gdb.md containing the documentation in a semi-structured format with markdown. That is of course still a lot of information.

Next, I went another level deeper where I asked it to extract from gdb.md information about the mi2 protocol. Any command in the mi2 protocol or related command in GDB is important and therefore should be included in a new mi2.md document. This will serve as the basis of our contextual knowledge.

Knowing the Problem Domain

Using an agent is no replacement for knowing the problem domain. Thankfully, I’ve written dozens of socket clients/servers using GIO primitives. So I can follow my normal technique.

First we create a client class which will manage our GIOStream. That may be a GSocketConnection or perhaps even a GSimpleIOStream containing memory streams or file streams to enable unit testing.

Then we move on to a GDataInputStream which can read the format. We ask the agent to subclass appropriately and provide a DexFuture based wrapper which can read the next message. It should have enough context at this point to know what needs to be read. Start simple, just reading the next message as a bag of bytes. We parse it into structured messages later.

After that do the same for a GDataOutputStream subclass. Once we have that we can ask the agent to wrap the input/output streams inside the client. All pretty easy stuff for it to get right.

Where I found it much easier to write things myself was dealing with the mechanics of a read-loop that will complete in-flight operations. Small price to pay to avoid debugging the slop machine.

After these mechanics are in place we can start on the message type. I knew from my previous debugger work in Builder that there would be a recursive result type and a toplevel container for them. I decided to name these Message and Result and then let the agent generate them. Even the first try had something “usable” even if I would say not very tasteful. It kept trying to do manual memory management. It left lots of “TODO” in place rather than actually doing the thing.

Instead of trying to describe to the agent how its implementation was ugly and incomplete I realized my time would be better spent letting it figure it out. So I shifted my focus towards a test suite.

Setting up a Test Suite

I saw the biggest improvement from the agent after I setup a test suite. This allowed the agent to run ninja test to check the results. That way it wasn’t just about satisfying GCC compiler warnings but also that it was functional.

I would start this as early as you can because it pays dividends. Make sure you provide both good and bad inputs to help guide it towards a better implementation.

The biggest win was creating tests for the input stream. Create different files with good and bad input. Since they wrap a base stream it is pretty easy to mock that up with a GFileInputStream or GMemoryOutputStream and verify the results. If the tests provide enough context through debug messages the agent can feed that back into why the error occurred. This made the iteration loop remarkably fast.

Another extremely valuable thing to do is to setup some type of sanitizer in your build configuration. I often build with meson configure -Db_sanitize=address to help catch memory errors. But also do things like assert object finalization in your testsuite to make sure that leaks don’t hide them.

Anyway, writing this post took about as long as designing the core parts of the library and that, I guess in some small way, is pretty neat.