Libdex Futures from asyncio

One of my original hopes for Libdex was to help us structure complex asynchronous code in the GNOME platform. If my work creating Foundry and Sysprof are any indicator, it has done leaps and bounds for my productivity and quality in that regard.

Always in the back of my mind I hoped we could make those futures integrate with language runtimes.

This morning I finally got around to learning enough of Python’s asyncio module to write the extremely minimal glue code. Here is a commit that implements integration as a PyGObject introspection override. It will automatically load when you from gi.repository import Dex.

This only integrates with the asyncio side of things so your application is still responsible for integrating asyncio and GMainContext or else nothing will be pumping the DexScheduler. But that is application toolkit specific so I must leave that to you.

I would absolutely love it if someone could work on the same level of integration for GJS as I have even less experience with that platform.

Status Week 47

Ptyxis

  • Issue filed about highlight colors. Seems like a fine addition but again, I can’t write features for everyone so it will need a MR.

  • Walk a user through changing shortcuts to get the behavior they want w/ ctrl+shift+page up/down. Find a bug along the way.

  • Copy GNOME Terminal behavior for placement of open/copy links from the context menu.

  • Add a ptyxis_tab_grab_focus() helper and use that everywhere instead of gtk_widget_grab_focus() on the tab (which then defers to the VteTerminal). This may improve some state tracking of keyboard shift state, which while odd, is a fine enough workaround.

  • Issue about adding waypipe to the list of “network” command detection. Have to nack for now just because it is so useful in situations without SSH.

  • Issue about growing size of terminal window (well shrinking back) after changing foreground. Triaged enough to know this is still a GDK Wayland issue with too-aggressively caching toplevel sizes and then re-applying them even after first applying a different size. Punted to GTK for now since I don’t have the cycles for it.

Foundry

  • Make progress icon look more like AdwSpinner so we can transition between the two paintables for progress based on if we get any sort of real fractional value from the operation.

  • Paintable support for FoundryTreeExpander which is getting used all over Foundry/Builder at this point.

  • Now that we have working progress from LSPs we need to display it somewhere. I copied the style of Nautilus progress operations into a new FoundryOperationBay which sits beneath the sidebar content. It’s not a perfect replicate but it is close enough for now to move on to other things.

    Sadly, we don’t have a way with the Language Server Protocol to tie together what operation caused a progress object to be created so cancellation of progress is rather meaningless there.

  • Bring over the whole “snapshot rewriting” we do in Ptyxis into the FoundryTerminal subclass of VTE to make it easy to use in the various applications getting built on Foundry.

  • Fix FoundryJsonrpcDriver to better handle incoming method calls. Specifically those lacking a params field. Also update things to fix reading from LF/Nil style streams where the underlying helper failed to skip bytes after read_upto().

  • It can’t be used for much but there is a foundry mcp server now that will expose available tools. As part of this make a new Resources type that allows providing context in an automated fashion. They can be listed, read, track changes to list, subscribe to changes in a resource, etc.

    There are helpers for JSON style resources.

    For example, this makes the build diagnostics available at a diagnostics:// URI.

  • Make PluginFlatpakSdk explicitly skip using flatpak build before the flatpak build-init command in the pipeline should have run.

  • Allow locating PluginDevhelpBook by URI so that clicking on the header link in documentation can properly update path navigators.

  • Make FoundryPtyDiagnostics auto-register themselves with the FoundryDiagnosticManager so that things like foundry_diagnostic_manager_list_all() will include extracted warnings that came from the build pipeline automatically rather than just files with diagnostic providers attached.

  • Support for “linked pipelines” which allows you to insert a stage into your build pipeline which then executes another build pipeline from another project.

    For example, this can allow you to run a build pipeline for GTK or GtkSourceView in the early phase of your projects build. The goal here is to simplify the effort many of us go through working on multiple projects at the same time.

    Currently there are limitations here in that the pipeline you want to link needs to be setup correctly to land in the same place as your application. For jhbuild or any sort of --prefix= install prefix this is pretty simple an works.

    For the flatpak case we need more work so that we can install into a separate DESTDIR which is the staging directory of the app we’re building. I prefer this over setting up incremental builds just for this project manually as we have no configuration information to key off.

  • Add new FoundryVcsDiffHunk and FoundryVcsDiffLine objects and API necessary access them via FoundryVcsDelta. Implement the git plugin version of these too.

  • Add a new FoundryGitCommitBuilder high-level helper class to make writing commit tooling easier. Provides access to the delta, hunk, and lines as well as commit details. Still needs more work though to finish off the process of staging/unstaging individual lines and try to make the state easy for the UI side.

    Add a print-project-diff test program to test that this stuff works reasonably well.

Builder

  • Lots of little things getting brought over for the new editor implementation. Go to line, position tracking, etc.

  • Searchable diagnostics page based on build output. Also fix up how we track PTY diagnostics using the DiagnosticManager now instead of only what we could see from the PTY diagnostics.

Manuals

  • Merge support for back/forward mouse buttons from @pjungkamp

  • Merge support for updating pathbar elements when navigating via the WebKit back/forward list, also from @pjungkamp.

Systemd

  • Libdex inspired fiber support for systemd which follows a core design principle which is that fibers are just a future like any other.

    https://github.com/systemd/systemd/pull/39771

CentOS

  • Merge vte291/libadwaita updates for 10.2

GTK

  • Lots of digging through GdkWayland/GtkWindow to see what we can do to improve the situation around resizing windows. There is some bit of state getting saved that is breaking our intent to have a window recalculate it’s preferred size.

GNOME Settings Daemon

  • Took a look over a few things we can do to reduce periodic IO requests in the housekeeping plugin.

    Submitted a few merge requests around this.

GLib

  • Submitted a merge request adding some more file-systems to those that are considered “system file system types”. We were missing a few that resulted in extraneous checking in gsd-housekeeping.

    While researching this, using a rather large /proc/mounts I have on hand, and modifying my GLib to let me parse it rather than the system mounts, I verified that we can spend double-digit milliseconds parsing this file. Pretty bad if you have a system where the mounts can change regularly and thus all users re-parse them.

    Looking at a Sysprof recording I made, we spend a huge amount of time in things like strcmp(), g_str_hash(), and in hashtable operations like g_hash_table_resize().

    This is ripe for a better data-structure instead of strcmp().

    I first tried to use gperf to generate perfect hashes for the things we care about and that was a 10% savings. But sometimes low-tech is even better.

    In the end I went with bsearch() which is within spitting distance of the faster solutions I came up with but a much more minimal change to the existing code-base at the simple cost of keeping lists sorted.

    There is likely still more that can be done on this with diminishing returns. Honestly, I was surprised a single change would be even this much.

Red Hat

This week also had me spending a significant amount of time on Red Hat related things.

Status Week 46

Ptyxis

  • More back and forth on people who have issues with diacritics and ibus interaction. It’s extremely frustrating at times because the two places where stuff like this gets reported first is the text editor and terminal, when rarely those two applications have anything to do with the issue.

    In this case we might have a workaround by being a bit more explicit about focus grabs.

  • Merge support for changing profile on existing tab.

VTE

  • Back-and-forth on updating MR for PTY errno, merge to master, vte-0-82, possibly backport further for RHEL/CentOS.

  • Research some us dead key issues with diacritics and see if we can find where in VTE a problem could be. Text Editor apparently doesn’t replicate the same issue, so it’s possible we should fix something in VTE directly or in GTK IM abstractions. As mentioned in Ptyxis we can probably work around this for now.

Foundry

  • Now that foundry has support for API keys we need to have a mechanism to rotate those keys (and query for expiration). FoundryKeyRotator provides this abstraction to FoundrySecretService.

    This comes with foundry secret rotate HOST SERVICE which makes it easy to keep things up-to-date. It would be nice to do this automatically at some point though it’s rather annoying because you’ll get an email about it, at least from gitlab.

    To check the expiration, foundry secret check-expires-at is provided check again, takes a HOST SERVICE pair.

    Defaults to what the server wants for minimum key lifetime, or you can provide --expire-at=YYYY-MM-DD to specify expiration.

  • Implement symbol APIs for FoundryTextDocument which will power things like the symbol tree, what symbol is underneath my cursor, symbol path bars, and the like. Also added some command line tools for this so that it is easy to test the infrastructure when issues are inevitably filed.

    foundry symbol-tree FILE and foundry find-symbol-at FILE LN COL will quickly test the machinery for filing bug reports.

  • Updated the CTags parser (which is our usually “first implementation” symbol provider) out of simplicity. Allow it to generate data to GBytes instead of files on disk for usage on modified buffers. Also allow it to load an index from memory w/o touching disk for the other side of this index structure.

  • GCC changed the SARIF environment variable to EXPERIMENTAL_SARIF_SOCKET so track that for GCC 16 release.

  • Handy foundry_read_all_bytes(int fd) to exhaust a FD into a GBytes using the libdex AIO subsystem (io_uring, etc).

  • Prototype a tree-sitter based plugin for symbol providers so we can have some faster extractors at least for some very common languages.

  • Add FoundrySymbolLocator for locating symbols by a number of strategies that can be different based on the provider. Combine this with a new FoundrySymbolIntent to integrate into the intent system.

  • Implement FoundryPathNavigator for use in future pathbar work. Add subclasses for FoundrySymbolNavigator and FoundryDocumentationNavigator to make pathbars of those common hierarchies super easy from applications. A FoundryFileNavigator to be able to navigate the file-system. This ties in well with the intent system so activating path elements routes through intents like open-file intent, symbol intent, etc.

  • Implement FoundryPathBar and associated infrastructure for it to be abstracted in libfoundry-adw.

  • Implement LSP progress operations ($/progress and creation operations) so we can bridge them to FoundryOperation. Had to implement some missing bits of FoundryJsonrpcDriver while at it.

  • Improve support for LSP symbol APIs, particularly around support for hierarchical symbol trees. Newer revisions allow for symbols to contain other symbols rather than trying to recreate it from the containerName.

  • Discovered that there is an upper-limit to the number of GWeakRef that can be created and that number is surprisingly lower than I would expect. Sure there is extra overhead with weak refs, but having limits so low is surely a mistake.

  • Lots of work on LSP implementation to bridge things like diagnostics and symbols. It is amazing now much easier it is to do this time around now that I have fibers instead of callback noodles.

  • We have a much saner way of implementing buffer tracking this time around (especially after pushing commit notify into GTK) so the LSP integration around buffer synchronization has a cleaner implementation now.

  • Add tooltips support to the diagnostics gutter renderer.

  • A new “retained” listmodel which allows you to hold/release items in a list model and they’ll stay pinned until the hold count reaches zero. This is helpful for situations where you don’t want to affect an item while there is a mouse over something, or a popover is shown, that sort of deal. I didn’t opt for the scalable RBTree implementation yet, but someone could certainly improve it to do so.

Buider

  • Work on the new auxiliary panel design which will work a bit like it does in Text Editor but allow for panel groupings.

  • Symbols panel implementation work using the new symbol providers. Implement symbol intent handling as well to tie it all together.

  • Implement pathbar integration into text editor and documentation browser using new navigator integration.

  • Diagnostics panel for monitoring diagnostics across the active page without having to resort to scanning around either the global diagnostics or within the gutter.

  • Add annotation provider so we can show diagnostics/git-blame like we do now but in a much more optimized manner. Having diagnostics inline is new though.

  • Lots of styling work for the auxiliary panel to try to make it work well in the presence of a grid of documents. A bit harder to get right than in Text Editor.

  • Ergonomics for the messages panel (clipboard support, clearing history, etc).

  • Work on operation bay for long running operations similar to what Nautilus does. This will handle things like progress from LSPs, deployment to remote devices, etc.

CentOS

  • libadwaita backports. This one is rather frustrating because I’ve been told we can’t do sassc on the build infrastructure and since 1.6.3 libadwaita no longer generates the CSS on CI to be bundled with the tarball.

    So continue with the madness of about 60 patches layered on top of 1.6.2 to keep things progressing there. One patch won’t get in because of the CSS change which is unfortunate as it is somewhat a11y related.

    At the moment the options are (given the uncompromising no-sassc restriction), to keep back-porting and not get CSS changes, to pull in newer tarballs and generate the CSS on my machine and patch that in, or just keep doing this until we can *gestures* do something more compromising on the CentOS build infrastructure.

  • VTE backports for 0.78.6

GtkSourceView

  • Branched for 5.20 development so we can start adding new features.

  • Fix a gir annotation on GtkSourceAnnotation that had wrong transfer.

  • Make GtkSourceAnnotation right justified when it fits in the available space.

  • Add some nice styling to annotations so they are bit more pleasing to look at.

Status Week 45

Ptyxis

  • Handle some incoming issue reports which basically amounts to copying their question into google, searching, and copying the first result back. A reminder that we really need dedicated support channels that are not the issue tracker.

    But more importantly, how you move people there is still problematic. I could of course just tell them to go over “there”, but when the questions are so simple you end up taking the gentler approach and just answering it begrudgingly rather than coming off abrupt.

  • Issue reported about an async spin loop from GMainLoop after closing a window which has a really long paste actively feeding it. Looks like it needs to be addressed in VTE (more on that below), but also decided to raise priority of our fallback SIGKILL handler if SIGHUP failed.

  • Code review for incoming community feature to swap profiles on an active tab.

VTE

  • MR sent upstream to hopefully address the corner case of disposing a terminal widget while paste is ongoing.

  • Noticed that ibus-daemon is taking considerable CPU when running the ucs-decode test tool on Ptyxis/VTE. Probably something related to tracking input cursor positions w/ the text protocol or similar. Reached out to @garncho for any tricks we might be able to pull off from the VTE side of things.

Libdex

  • Looking into some strange behaviors with dex_thread_wait_for() when implementing the cross-thread semantics for libgit2 wrapping. As a result I changed where we control life-cycle for the waited upon future so that it is guaranteed longer than the DexWaiter.

  • Spend some time thinking about how we might approach a more generalized wrapper for existing GIO-like async functions. I was never satisfied with DexAsyncPair and perhaps now that we will gain a gdbus-codegen for Libdex we could gain a *.gir future wrapping codegen too.

  • Add a new DexFutureListModel which is a GListModel which is populated by a DexFuture resolving to a GListModel. Very handy when you have a future and want a GListModel immediately.

Foundry

  • Add FoundryFileRow so we have easy mechanics for the whole typing a file/directory path and browsing to it. Re-use the existing path collapse/expand stuff in Foundry to allow for niceties like ~/Directory. Fix a bunch of obnoxiousness that was in the Builder implementation of this previously.

  • Add foundry_operation_is_cancelled() to simplify integrating with external blocking/threaded code such as libgit2. This allows breaking out of a clone operation by checking for cancellation in the progress callbacks from the server but works for any sort of blocking API that itself has callback functions in its state machine.

  • Add new CLI formatter for GFlags to use value_nick instead of the whole enumeration value in UP_CASE.

  • Add FoundryBuildPipeline:build-system with discover from the FoundryBuildAddin before full addin loading occurs. This replicates the IdeBuildSystemDiscovery from Builder without having to use a secondary addin object.

  • Add priorities to FoundryBuildAddin so that we can pre-sort by priority before build-system discovery. That fixes loading the WebKit project out of the box so that CMakeLists.txt will be matched before Makefile if you configured in tree.

  • Implement write-back for FoundrySdkManager:sdk to the active configuration after checking of the config supports the SDK.

  • Implement write-back for buildconfig files.

  • Implement mechanics for write-back PluginFlatpakConfig. Also start on implementation for all the FlatpakSerializable types. This is made much more complicated by needing to keep track of the different subtle ways lists can be deserialized as well as referenced files which can be included. It will mean write-back may have a bucket of files to update. Though right now we only will do this for a few select fields so we can probably punt a bit.

  • Meson cleanup to all our tools that test various aspects of Foundry outside the confines of an entire IDE.

  • Non-destructive editing of flatpak manifests works for the core feature-set we need. It requires storing a lot of state while parsing manifests (and recursively their includes) but it seems to work well enough.

  • Setup needs-attention tracking for panels and pages.

  • Write a new foundry.1 manpage for inclusion in distributions that really want that provided.

  • Add :author property to FoundryForgeIssue and FoundryForgeMergeRequest so we can start presenting peer information in the Builder UI. Implement this for gitlab plugin.

  • Improve handling to changes in implicit-trailing-newline from file settings getting loaded/applied. That way we don’t run into weird behavior with file-change monitoring with ligbit2 getting something different than we’d expect.

  • Support for keyword search when listing issues/merge-requests through the forge interfaces.

  • Add FoundryDocumentationIntent for intent to read documentation.

  • Very basic support for Justfile using just to build. Currently it is sort of a cop-out because it uses build and clean instead of trying to track down [default] and what not. Contributions welcome.

Builder

  • Iteration on the new clone dialog based on Foundry including wiring up all the PTY usage, page navigation, etc.

  • Update builder-dark GtkSourceView style to fit in better with updated Adwaita palette. Even though it’s not using Tango palette for all the colors, we still want it to fit in.

  • Iteration on forge listings

  • Wire up needs attention for panels/pages

  • Setup forge splitbutton so we can jump to gitlab (or other forge) quickly or alternatively list issues/etc in app as we gain more forge support.

  • Implement manuals panel and page on top of the Foundry service FoundryDocumentationManager.

  • Implement browser page and port over old urlbar implementation. Still slogging through all the intricacies of navigation policy which I didn’t handle so well in Builder originally.

  • Work on bringing over path bar from Manuals so we can use it more generically for things like symbol paths and what not.

  • A lot of little things here and there that just need plumbing now that we’re in a world of futures and listmodels.

  • Lots of work on the updated greeter using Foundry. It looks mostly the same, just less madness in implementation.

Flathub

  • Investigate why older Builder is what is shown as published.

  • Merge PR to update Ptyxis to 49.2

  • Update Manuals to 49.1 since I missed the .0. Had a lot of libfoundry things to bring over as well. Needs an exceptions update to flathub linter.

Text Editor

  • Update builder style scheme

Libpanel

  • Update needs-attention style for the panel switcher buttons

Manuals

  • Make ctrl+k select all the existing text in the search entry as part of the focus-search action.

Status Week 44

Red Hat

In October I celebrated 10 years at Red Hat. I don’t think I ever worked somewhere for more than a few years before that. Considering I started working professionally in tech after I barely escaped high school, that’s nearly half of my career.

It is also a reminder to reflect on the people who helped me get there. All the people on IRC that helped me learn programming in the early days of #gtk and #gnome-hackers IRC channels. And Miguel specifically for all the random demo code he would send my way to read how something could be done.

I know I’m one of the lucky ones that was able to do this without a formal education. I don’t necessarily recommend that but it has its benefits too.

Ptyxis

  • Look into what app-id is used on Ubuntu to help users who can’t figure out where to put custom palettes. To make this a bit easier make sure we always have APP_ID in the troubleshooting data. Turned out to be missing [Light] section in their custom .palette.

    I can’t help but lament that this would be less of an issue if the org.gnome.* namespace where more allowed for people who are Foundation members. I’m both informed on why it is like this (a single situation that occurred by a former community member) and also disagree with the direction that was taken as a result of that.

  • Look into diacritics issue for input. Usually this ends up being a system configuration issue. There is an oddity about this one in that changing focus and coming back fixes it. Also using ibus directly instead of the wayland text protocol fixes it. Neither of these are ideal so it may have something to do with state tracking either in the GTK Wayland backend or in Mutter.

    Thankfully the great Carlos Garnacho is helping us track it down.

  • Look into why Silverblue has some users not seeing their containers which appear to be toolbox/podman related.

    Installed F43 Silverblue beta on my old testing laptop. Turns out I had an overzealous filter for the containers list. If you had more than one you’d be fine. This only affected 49 since that is the first release that gained support for the search filter box in the popover that otherwise looks like a GtkPopoverMenu.

    Made a release to get the fix out to F43 users quickly.

    Quickly I want to mention a feature that keyboard navigators will love. If you type Alt+comma the containers dialog will be displayed. If you hit Return you’ll land in the default host system (aka “My Computer”). If you type a few characters and hit Return you’ll land on the first container/profile that matches your keyword search.

Foundry

  • New API to simplify navigating directory listings since you may want “..” in certain contexts (like Builder’s directory page).

  • Add a FoundryFileSearchReplacement API to coordinate with the FoundryFileSearchMatch API. This makes it easy to do replacements based on search results in a manner like we do in GNOME Builder where you can tick on/off + replacement string w/ back-references. Of course to test this infrastructure I added a --replace= option to foundry grep so I can refactor quickly from Ptyxis.

  • Write documentation for most of libfoundry exposed classes.

  • Fix string array output for --format=json and improve it for regular text output.

  • Fix grep plugin to add multiple matches on a single line as separate matches in the result set. This fixes search/replace to handle all matches in a file.

  • New Terminal Intent for opening terminals within the application such as Builder. Add a TerminalService for exposing actions to the intent system to applications easily.

  • Setup using intents for search results because it will allow us to abstract things like “activate action”, “open file”, “spawn terminal”, “browse to symbol”, “read documentation”, etc.

  • Iteration on panel bar to try to give us a bit better interaction with the panel machinery in libfoundry-adw applications.

  • Improve fallback search when there is not a VCS available to accelerate index building. (We use git file listing which is almost instantaneous to build our initial fuzzy search index instead of trying to iterate the file tree).

  • Start on documentation bridge so doc search can show up in the regular search dialog.

  • Host SDK gained support for running processes through a systemd-run with scope/unit like Ptyxis does for new tabs (when systemd-run is discovered on the host). This just helps identify things easier.

  • Implement the whole PanelWidget::presented() machinery from libpanel into libfoundry-adw so that we can have Workspaces but also still be able to delay certain panel/page work until the user will actually see the widget. For example, the test suite panel needs to advance the build pipeline far enough to get introspection data which you don’t want to do unnecessarily.

  • Add a new way to track current phase of a build pipeline. It is now dynamically calculated based on the maximum of the stages which have :completed set to true (so log as others in the same phase have the same value).

  • Add new :eol property to FoundryDocumentationBundle so we can pass that information from SDKs into Manuals

  • Build system updates so that we still have more control over feature flags for compiling in/out what you need in Foundry. This is quite a burden for sure, but it allows more flexibility in where Foundry can be used. For example, Manuals doesn’t need the text subsystem or Git for example.

Builder

  • Trying a new design for various types of listings in the revamp. Directory listings, diagnostics listings, log panel, file search results, unit test panel, test output, etc.

  • Copied how I did shortcuts in Ptyxis for the revamp because it works a lot nicer from a configuration standpoint and visibility. It also makes it much easier to bind from .ui files into controllers as well as update GMenu with appropriate shortcuts.

  • Use g_file_trash() instead of delete when removing configurations.

GLib

  • Pull in a year old patch from Ulrich Drepper into a MR that can be merged to make g_filename_from_uri() handle some valid hostnames according to RFC 1123.

  • Extend the above patch to also support . later in labels so some questionable hostname like dev-ubu-25.10 in Ptyxis work.

Manuals

  • Show EOL information in documentation bundle installation dialog.

  • Allow showing/hiding EOL documentation bundles.

Releases

  • libpanel

  • Foundry

  • Builder

  • D-Spy

  • Manuals

  • Gom