Happenings in GNOME

I know that I will never convince everyone to use Builder for development. Even I have a hard time getting myself out of the terminal at times.

Foundry comes in two forms, an executable and a library. The executable is just a bunch of commands and libfoundry is meant for building your own IDE (like Builder) or specialized tooling.

# initialize .foundry directory if never done before
cd my-project/
foundry init

If you go into most GNOME applications, they likely have a Flatpak manifest. Builder uses this to auto-discover a lot about a project. Foundry is no different.

# Build the project, just like Builder would do
foundry build

You can run this from any directory in your project as it will scan upwards to locate the .foundry/ directory that contains project state. It will also do incremental building just like Builder does.

# Run the project, just like Builder would do
foundry run

# Run a specific command in place of the default program.
# Useful to test something with "runtime" instead of
# build environment.
foundry run -- gtk4-demo

Running the project will setup the necessary incremental flatpak pipelines and auxiliary tooling like an a11y bus, font integration, and what not. Your application will be run in the Flatpak build container based on the finish-args.

This does require building the application to ensure the runtime environment is setup correctly.

# Pull updated project dependencies.
foundry dependencies update

Updating your deps is pretty painless. This is not done automatically unless the dependency is missing so that we don’t have to constantly hammer remote servers on every build request.

# Run a shell in the build pipeline, launches in builddir
foundry devenv

# Run a specific command in build pipeline
foundry devenv -- ps aux

You can get a terminal shell in the build pipeline (much like ctrl+alt+shift+t in Builder.

# Lots of commands to inspect the build pipeline
foundry pipeline info

# Alias to foundry build
foundry pipeline build

# Invalidate all pipeline stages
foundry pipeline invalidate

# Purge (remove build files/artifacts/etc) to force
# a clean build
foundry pipeline purge

# clean, configure, export, rebuild, which, install
# all provide additional pipeline operations

You can interact with the active build pipeline using the above commands.

Sometimes you might want to know what compiler flags will get used with tooling on a specific file. That is made available via the pipeline as well.

foundry pipeline flags path/to/file.c

Maybe you need a language server for integrating with another editor. That is pretty easy to do for the supported languages by using the lsp command.

$ foundry lsp list 
Name                  Languages
zls                   'zig'                                   
vhdl-language-server  'vhdl'                                  
vala-language-server  'vala' 'genie'                          
ts-language-server    'js' 'jsx' 'typescript' 'typescript-jsx'
sourcekit-lsp         'swift'                                 
serve-d               'd'                                     
rust-analyzer         'rust'                                  
ruff                  'python3' 'python'                      
pylsp                 'python3' 'python'                      
mesonlsp              'meson'                                 
lua-language-server   'lua'                                   
jedi-language-server  'python'                                
jdtls                 'java'                                  
intelephense          'php'                                   
gopls                 'go'                                    
glsl-language-server  'glsl'                                  
elixir-ls             'elixir'                                
clangd                'c' 'cpp' 'chdr' 'cpphdr' 'objc'        
blueprint             'blueprint'                             
bash-language-server  'sh'                                    

# use stdin/stdout to communicate with an LSP for python3
$ foundry lsp run python3

If you have multiple project configurations, you can look through them and select the active one. Just switch configs and run, Foundry will do the rest.

foundry config list
...
foundry config switch flatpak:app.devsuite.Manuals.Devel.json
foundry run

Want to get a .flatpak you can copy to another system to test?

foundry export
...
Artifacts:
  file:///.../apps.devsuite.Manuals.Devel.flatpak

You can switch devices in case you have deviced running somewhere like a phone or tablet. Then running should deploy to that system and run it there.

foundry device list 
...
foundry device switch some_device
foundry run

You can get the URI to documentation which could allow you to integrate with other editors. Like many commands, you can use --format=json to get the output in an easy-to-parse format for editor plugins.

# list available doc bundles to install
foundry doc bundle list
...

# install docs for GNOME 48
foundry doc bundle install flatpak/org.gnome.Sdk.Docs/48/user

# Find GtkWidget documentation
foundry doc query --format=json GtkWidget

Many settings can be tweaked using foundry settings set .... Just tab-complete around to explore.

Anyway, hopefully that serves as a quick intro into some of the things you can do with Foundry already as it progresses towards being a fully-fledged replacement for Builder’s internals.

I finally got around this past week to getting Manuals ported to libfoundry.

That was something I wanted to do early so that I can rapidly get away from 3 versions of the documentation engine and Flatpak SDK management code. Currently it existed in Manuals, Builder, and Foundry, and soon we’ll get it to just Foundry.

That also makes Manuals the first application to use libfoundry which resulted in lots of tree-shaking and things are looking bright! I very much look forward to getting Builder rebased on libfoundry.

While I was at it, I ticked off a number of requested design issues as part of the Incubator submission. This afternoon it also got support for narrow views meaning you can run it on a GNOME-enabled phone.

Probably not how most people will use it, but hey, maybe you have actually good public transportation where you are and some free time.

Back in December (before I caught the flu working at a farmers market, then Covid two weeks later, then two months of long-Covid) I mentioned that we’d discuss the various subsystems needed in libfoundry to build an IDE as a library.

I used the little bit of energy I had to work on some core abstractions. In an effort to live up to my word lets talk a bit about what went into libfoundry last night.

There is now a DocumentationManager sub-system which handles documentation installed on the host system, chroots, and Flatpak SDKs. It’s a bit tricky to make this all work without blurring the lines of abstraction so lets cover how that works.

Generally speaking, we try to avoid plugins depending on other plugins. Sometimes it happens but usually it is an opportunity to make a better abstraction in libfoundry. Lets look at what is needed around documentation.

• We have many SDKs and they all might have documentation available at different locations.
• We primarily have one format we need to support in GNOME, which is the venerable Devhelp2 XML format serving as an index.
• SDKs might contain the same documentation but at different versions (Nightly vs GNOME 48 vs jhbuild for example)
• There may be more formats that matter in the future especially as we look at pulling in support for new languages.
• Adding new search capabilities shouldn’t break the API.
• Querying needs to be fast enough to update as you type.

So lets dive into the abstractions.

DocumentationManager

This is the core abstraction you start interfacing with. It is a service of the FoundryContext and therefore can be accessed with Foundry.Context:documentation-manager property.

The documentation manager manages the Foundry.DocumentationProvider plug-in abstractions. Plug-ins that which to contribute to the documentation pipeline must subclass this in their plug-in.

To query documentation, use Foundry.DocumentationManager.query(). As I noted earlier, I don’t want new capabilities to break the API so a Foundry.DocumentationQuery object is used rather than a sequence of parameters which would need to be modified.

Avoiding Formats in the API

Since we want to be able to support other documentation formats in the future, it is important that we do not force anything about devhelp2 XML into the core abstraction.

The core result object from queries is a simple Foundry.Documentation object. Like above, we want to avoid breaking API/ABI when new capabilities are added so this object serves as our abstraction to do so. Navigating a tree structure will live here and can be implemented by plug-ins through subclassing.

Additionally, a “devhelp” plug-in provides support for crawling the devhelp2-style directories on disk. But this plug-in knows nothing about where to find documentation as that is relevant only to the SDKs.

This is where the Foundry.DocumentationRoot object becomes useful. SDK plug-ins can implement DocumentationProvider in their plug-in to expose documentation roots. The host-sdk, jhbuild, and Flatpak plug-ins all do this to expose the location of their documentation.

Now the devhelp plug-in can be provided the information it needs for crawling without any knowledge of SDKs.

Fast Querying

The old adage is that the only way to go faster on a computer is to do less work. This is particularly important in search systems where doing an entire query of a database means a lot of wasted CPU, memory, and storage I/O.

To make querying fast the devhelp plug-in indexes information about SDKs in SQLite. Way back in Builder we’d avoid this and just make an optimized fuzzy search index, mmap that, and search it. But now days we’ve gone from one set of documentation to multiple sets of documentation across SDK versions. The problem domain explodes quite a bit. SQLite seemed like a nice way to do this while also allowing us to be lazy in our searching.

By lazy what I mean is that while we’ll start your query, we only retrieve the first few results from the cursor. The rest are lazily fetched as the GListModel is scanned by scrolling. As that is not a very common operation compared to typing, you can throw away a lot of work naturally while still sitting behind the comfortable GListModel interface.

What now?

Since libfoundry already supports SDK management (including Flatpak) you could probably re-implement Manuals in a week-end. Hopefully this also breaks down a bit of the knowledge used to build such an application and the deceptive complexity behind doing it well.

This should also, hopefully soon, allow us to share a documentation implementation across Builder, Manuals, and an upcoming project I have which will benefit from easy access to documentation of object properties.

With GNOME 48 I released libdex 0.10 on the march towards a 1.0. One of the major improved features there was around fiber cancellation.

I’m not going to go into detail about the differences between threads and fibers as wikipedia or your local CS department can probably help you there. But what I will say is that combining __attribute__((cleanup)) (e.g. g_autoptr()) with futures and fibers makes such a nicer experience when writing C.

Thread cancellation is a rather non-portable part of the threading stack across platforms. Some POSIX platforms support it, some don’t. Having safe places to cancel can be a real challenge even if you are depending on a threading implementation that can do it.

With fibers, we have a natural cancellation point due to the cooperative nature of scheduling. All (well behaved) fibers are either making progress or awaiting completion of a future. We use the natural await() points to implement cancellation. If everything that was awaiting the future of the fiber has been cancelled, then the fiber can naturally cancel too. The next time it awaits that will just happen and natural exit paths will occur.

When you don’t want cancellation to propagate, you still use dex_future_disown() like always (as the fiber itself is a future).

Just to give a quick example of how fibers and futures makes writing C code nicer, here is an excerpt from libfoundry that asynchronously implements the necessary phases to build/run your project with a specific tool, possibly on a non-local system. In the GNOME Builder IDE, this is a series of async callbacks that is extremely difficult to read/debug. But with Foundry using libdex, it’s just a few lines of code and every bit as non-blocking.

From foundry-run-manager.c.

g_autoptr(FoundryDeployStrategy) deploy_strategy = NULL;
g_autoptr(FoundryBuildProgress) progress = NULL;
g_autoptr(GSubprocess) subprocess = NULL;
GError *error = NULL;

if (!(deploy_strategy = dex_await_object (foundry_deploy_strategy_new (state->pipeline), &error)) ||
    !dex_await (foundry_deploy_strategy_deploy (deploy_strategy, state->build_pty_fd, state->cancellable), &error) ||
    !dex_await (foundry_deploy_strategy_prepare (deploy_strategy, state->launcher, state->pipeline, state->build_pty_fd, state->cancellable), &error) ||
    !dex_await (foundry_run_tool_prepare (state->run_tool, state->pipeline, state->command, state->launcher, state->run_pty_fd), &error) ||
    !(subprocess = foundry_process_launcher_spawn (state->launcher, &error)))
  return dex_future_new_for_error (error);

At each dex_await*() function call the fiber is suspended and we return to the main loop for additional processing.

In a better world we’d be able to do these without fibers and instead do stackless coroutines. But maybe with a little compiler help we can have that too.

Not all of my projects this December are code related. In fact a lot of them have been house maintenance things, joy of home ownership and all.

This week was spent building my new office and music space. I wanted a way to have my amplifiers and guitars more accessible while also creating a sort of “dark academia” sort of feeling for working.

The first step was to get the guitars mounted on the walls. I was looking for something blending artistic showpiece and functional use.

After that I quickly framed them in. Just quarter round with the hard edge inwards, 45° miter, some caulking, easy peasy.

My last office had Hale Navy as the color, but the sheen was too much that it made it difficult to actually see the color. This time I went flat and color drenched the space (so ceilings, trim, etc all in matching tone).

Then somewhat final result is here. I still want to have a lighting story for these that doesn’t involve a battery so some electrical fish taping is likely in my future.

I also converted the wide closet into a workstation area with the studio monitors for recording. But that is still partially finished as I need to plane all the slats for the slat wall, frame the builtin, and attach the countertop.

Early on Builder had the concept of layered settings. You had an application default layer the user could control. You also had a project layer which allowed the user to change settings just for that project. But that was about the extent of it. Additionally, these settings were just stored in your normal GSettings data repository so there is no sharing of settings with other project collaborators. Boo!

With Foundry, I’d like to have a bit more flexibility and control. Specifically, I want three layers. One layer for the user’s preferences at the application level. Then project settings which can be bundled with the project by the maintainer for needs specific to the project. Lastly, a layer of user overrides which takes maximum preference.

Of course, it should still continue to use GSettings under the hood because that makes writing application UI rather easy. As mentioned previously, we’ll have a .foundry directory we place within the project with storage for both user and project data. That means we can use a GKeyFile back-end to GSettings and place the data there.

You can git commit your project settings if you’re the maintainer and ensure that your projects conventions are shared to your collaborators.

Of course, since this is all command-line based right now, there are tab-completable commands for this which again, makes unit testing this stuff easier.

# Reads the app.devsuite.foundry.project config-id gsetting
# taking into account all layers
$ foundry settings get project config-id

# Sets the config-id setting for just this user
$ foundry settings set project config-id "'org.example.app.json'"

# Sets the config-id for the project default which might
# be useful if you ship multiple flatpak manifest like GTK does
$ foundry settings set --project project config-id "'org.example.app.json'"

# Or maybe set a default for the app
$ foundry settings set --global project stop-signal SIGKILL


That code is now wired up to the FoundryContext via foundry_context_load_settings().

Next time I hope to cover the various sub-systems you might need in an IDE and how those services are broken down in Foundry.

A core tenant of Foundry is a pleasurable command-line experience. And one of the most creature-comforts there is tab-completion.

But how you go about doing that is pretty different across every shell. In Flatpak, they use a hidden internal command called “complete” which takes a few arguments and then does magic to figure out what you wanted.

Implementing that when you have one layer of commands is not too difficult even to brute force. But imagine for a second that every command may have sub-commands and it can get much more difficult. Especially if each of those sub-commands have options that must be applied before diving into the next sub-command.

Such is the case with foundry, because I much prefer foundry config switch over foundry config-switch. Particularly because you may have other commands like foundry config list. It feels much more spatially aware to me.

There will be a large number of commands implemented over time, so keeping the code at the call-site rather small is necessary. Even more so when the commands could be getting proxied from another process or awaiting for futures to complete.

With all those requirements in mind, I came up with FoundryCliCommandTree. The tree is built as an n-ary tree using GNode where you register a command vtable with the command parts like ["foundry", "config", "switch"].

At each layer you can have GOptionEntry like you normally use with GLib-based projects but in this case they will end up in a FoundryCliOptions very similar to what GApplicationClass.local_command_line() does.

So now foundry has a builtin “complete” command like Flatpak and works fairly similarly though with the added complexity to support my ideal ergonomics.

I tend to bulk most of my vacation at the end of the year because it creates enough space and time for fun projects. Last year, however, our dog Toby went paraplegic and so were care-taking every three hours for about two months straight. Erratic sleep, erratic self-care, but in the end he could walk again so definitely worth it.

That meant I didn’t really get to do my fun end-of-year hacks beyond just polishing Ptyxis which I had just prototyped for RHEL/CentOS/Bluefin (and more recently Fedora).

This year I’m trying something I’ve wondered about for a while. What would it look like if you shoved a full-featured IDE into the terminal?

The core idea that makes this possible is using a sub-shell with a persistent parent process. So just like you might “jhbuild shell” you can “foundry enter” to enter the “IDE”.

In the JHBuild case it would exec over itself after setting things up. In the foundry case it maintains an ancestor process and spawns a sub-shell beneath that.

When running foundry commands from a sub-shell it will proxy that work to the ancestor instance. This all happens with a private D-Bus peer-to-peer process. So you can have multiple of these in place across different terminal tabs eventually.

This is all built with a “libfoundry” that I could consume from Builder in the future to provide the same feature from a full-blown GTK-based IDE too. Not to mention the IDE becomes instantly script-able from your shell. It also becomes extremely easy to unit test.

Since I originally created Builder, I wrote a library to make doing futures, concurrency, and fibers much easier in C. That is libdex. I tend to make things more concurrent while also reducing bug counts when using it. Especially for the complex logic parts which can be written in synchronous looking C even though it is asynchronous in nature.

So the first tenant of the new code is that it will be heavily based on DexFuture.

The second tenant is going to be a reversal of something I tried hard to avoid in Builder. That is a “dot” directory in projects. I never liked how IDEs would litter projects with state files in projects. But since all the others continue to do so I don’t see much value in tying our hands behind our back out of my own OCD purity. Instead, we’ll drop a .foundry directory with appropriate VCS ignore files. This gives us convenient space for a tmpdir, project-wide settings, and user settings.

The project is just getting started, but you can follow along at chergert/foundry and I’ll try to write more tidbits as I go.

Next time, we’ll cover how the command line tools are built as an N-ary tree to make tab-completion from bash easy.

The upcoming systemd v257 release will have support for a feature originating from ConEmu (a terminal emulator for Windows) which was eventually adopted by Windows Terminal.

Specifically, it is an OSC (Operating System Command) escape sequence which defines progress state.

Various systemd tools will natively support this. Terminal emulators which do not support it simply ignore the OSC sequence but those that do support it may provide additional UI to the application.

Lennart discussed this briefly in their ongoing systemd v257 features series on Mastodon and so I took up a quick attempt to implement the sequence parsing for VTE-based terminals.

That has since been iterated upon and landed in VTE. Additionally, Ptyxis now has corresponding code to support it as well.

Once GNOME CI is back up and running smoothly this will be available in the Ptyxis nightly build.