Happenings in GNOME – Page 19 – Details of Christian's work on GNOME

Builder Rust

With Federico’s wonderful post on Rust’ifying librsvg I guess it makes sense to share what I’ve been doing the last couple of days.

I’ve been keeping my eye on Rust for quite a while. However, I’ve been so heads down with Builder the last two years that I haven’t really gotten to write any or help on integration into our platform. Rust appears to take a very pragmatic stance on integration with systems code (which is primarily C). The C calling convention is not going anywhere, so at some point, you will be integrating with some part of a system that is “C-like”. Allowing us to piecemeal upgrade the “Safety” of our systems is much smarter than rewrite-the-universe. This pragmatism is likely due to the realities of Rust’s birth at Mozilla. It’s a huge code-base, and incrementally modernizing it is the only reality that is approachable.

We too have a lot of code. And like many other projects, we care about being language agnostic to a large degree. In the early days we might have chosen C because it was the only toolchain that worked reliably on Linux (remember C++ pre-2000? or LinuxThreads?) but today we still care about “Language Interopability” and C is the undeniable common denominator.

One way in which we can allow the interopability that we desire and the Safety we need is to start approaching some of our problems like Federico has done. The C calling convention is “Safe”. That is not where zero-day bugs come from. They come from hastily written C code. It is perfectly fine to write our interfaces in C (which can be our basis for GObject Introspection) but implement Safety critical portions in Rust.

This is exactly what I’d like to see from the hundreds of thousands of lines of C I’ve written over the years. We need to identify and fix the Satefy Critical portions. GStreamer is already on the right track here by looking at codec and plugin implementations in Rust. I’m not sure how far they will go with adapting to Rust, but it will be one of the best case-studies we will have to learn from.

So because of this desire to look at building a Safer Platform for developers and users alike, I’ve decided to start adding Rust support to Builder. Thanks to the hard work of the Rust team, it’s a fairly easy project from our side. There is a new Language Server Protocol 2.0 that was worked on by various people at Microsoft, Red Hat, and elsewhere. The new rustls was announced last week and uses this protocol. So I implemented support for both as quick as I could, and now we have something to play with.

Because of the Language Server Protocol, our Rust plugin is tiny. It is essentially a glorified supervisor to handle subprocess crashes and some binding code to connect our stdin/stdout streams to the Language Server Protocol client in Builder. See for yourself.

There is a bunch more work for us in Builder to make it a great Rust IDE. But if people start playing with the language and are willing to work on Builder and GNOME to improve things as a community, we can build a Modern, Safe, and Elegant developer platform.

The big ticket next steps for those that want to contribute to Rust support would include:

Cargo build system support. I do believe that ebassi started on something here. But I need to circle back around and sync up.
Symbol Tree needs improvements.
Semantic highlighter (which we can implement using fuzzy symbols from the symbol tree until a real protocol comes along).
Upstream rustls needs work too to get us the features we want. So Rustaceans might want to spend some time helping out upstream.
We need to simplify the glue code from Rust←→GObject so that it is dead simple to wrap Rust code in a GObject-based library (where we get our free language interopability).
… and of course all the other Builder plumbing that needs to happen in general. See our list of projects that need work.

This image depicts control+clicking on a symbol to jump to its definition. +period or :gd in Vim mode also work. — This image depicts control+clicking on a symbol to jump to its definition. <alt>+period or :gd in Vim mode also work.

This image shows diagnostics displayed over source code.

This image shows completion of fields from a struct.

This image shows the Symbol Tree on the right containing elements from the Rust document.

JCON – Part 2

I just finished the JCON_EXTRACT() support and it is already making my consumption of JSON easier. Here is an example:

g_autoptr(JsonNode) node = NULL;
JsonArray *ar = NULL;
gboolean success;

node = JCON_NEW (
  "foo", "{",
    "bar", "[", JCON_INT (1), JCON_INT (2), "]",
  "}"
);

success = JCON_EXTRACT (node,
  "foo", "{",
    "bar", JCONE_ARRAY (ar),
  "}"
);

And for now, you can just copy/paste the jcon.c and jcon.h files into your project, but I’d expect to come up with a patch we can push into json-glib at some point. It really belongs there.

JCON

Years ago during my tenure at MongoDB I worked on a couple libraries. Notably the libbson and mongo-c-driver libraries. One neat feature we had was this concept called BCON which stood for BSON C Object Notation. It was a succinct format for creating BSON documents that made it easier to reason about what you were creating.

So I decided to do the same thing this evening around json-glib because I found I was writing a lot of code to create objects/arrays/etc.

It looks something like this

g_autoptr(JsonNode) params = NULL;

params = JCON_NEW (                                                      
  "changes", "["                                                         
    "{",                                                                 
      "uri", JCON_STRING (src_uri),                                      
      "type", JCON_INT (FILE_CHANGE_TYPE_DELETED),                       
    "}",                                                                 
    "{",                                                                 
      "uri", JCON_STRING (dst_uri),                                      
      "type", JCON_INT (FILE_CHANGE_TYPE_CREATED),                       
    "}",                                                                 
  "]"                                                                    
);

Since this uses va_list it’s technically less type safe than your other options. But it uses some magic struct initializers to get things in a situation where we can bail at runtime if you did something wrong.

The other half that I’m currently still missing is the extraction support. If you replace JCON_NEW() with JCON_EXTRACT(node) and the values with pointers to values, you can quickly extract documents.

But that’s not done yet… so for next time…

jcon.h jcon.c

Contribute to Builder!

I started putting together a list of things that are on my mind that I’d like to see done but don’t necessarily need to be done by me. If any of them interest you, come hack with us on irc.gimp.net in #gnome-builder and we’ll help you get started.

See the list of things here.

Sysprof Visualizers

Visualizers are almost ready to land. This week I got smooth scrolling, zoom, and selections working. The handy part about selections is that it allows you to update the callgraph to limit stack samples to those falling within a given time range.

Sysprof Plans for 3.24

The 3.24 cycle is just getting started, and I have a few plans for Sysprof to give us a more polished profiling experience in Builder. The details can be found on the mailing list.

In particular, I’d love to land support for visualizers. I expect this to happen soon, since there is just a little bit more to work through to make that viable. This will enable us to get a more holistic view of performance and allow us to drill into callgraphs during a certain problematic period of the profile.

Once we have visualizer support, we can start doing really cool things like extracting GPU counters, gdk/clutter frame-clock timing, dbus/cpu/network monitors and whatever else you come up with.

A CPU visualizer displayed above the callgraph to provide additional context to the profiled execution.

Additionally we have some work to do around getting access to symbols when we are running in binary stripped environments. This means we can upload a stripped binary to your IoT/low-power device to profile, but have the instruction-pointer-to-symbol resolver happen on the developers workstation.

As I just alluded to, I’d love to see remote profiling happen too. There is some plumbing that needs to occur here, but in general it shouldn’t be terribly complicated.

Builder Nightly Flatpak

First off, I’ll be in Portland at the first ever LAS giving demos and starting on Builder features for 3.24.

For a while now, you’ve been able to get Builder from the gnome-apps-nightly Flatpak repository. Until now, it had a few things that made it difficult to use. We care a whole lot about making our tooling available via Flatpak because it is going to allow us to get new code into users hands quicker, safer, and more stable.

So over the last couple of weeks I’ve dug in and really started polishing things up. A few patches in Flatpak, a few patches in Builder, and a few patches in Sysprof start getting us towards something refreshing.

Python Jedi should be working now. So you can autocomplete in Python (including GObject Introspection) to your hearts delight.

Jhbuild from within the Flatpak works quite well now. So if you have a jhbuild environment on your host, you can use the Flatpak nightly and still target your jhbuild setup.

One of the tricks to being a module maintainer is getting other people to do your work. Thankfully the magnificent Patrick Griffis came to the rescue and got polkit building inside of Flatpak. Combined with some additional Sysprof patches, we have a profiler that can run from Flatpak.

Another pain point was that the terminal was inside of a pid/mount/network namespace different than that of the host. This meant that /usr/lib was actually from the Flatpak runtime, not your system /usr/lib. This has been fixed using one of the new developer features in Flatpak.

Flatpak now supports executing programs on the host (a sandbox breakout) for applications that are marked as developer tools. For those of you building your own Flatpaks, this requires --allow=devel when running the flatpak build-finish command. Of course, one could expect UI/UX flows to make this known to the user so that it doesn’t get abused for nefarious purposes.

Now that we have access to execute a command on the host using the HostCommand method of the org.freedesktop.Flatpak.Development interface, we can piggy back to execute our shell.

The typical pty dance, performed in our program.

/* error handling excluded */
int master_fd = vte_pty_get_fd (pty);
grantpt (master_fd);
unlockpt (master_fd);
char *name = ptsname (master_fd);
int pty_fd = open (name, O_RDWR, 0);

Then when executing the HostCommand method we simply pass pty_fd as a mapping to stdin (0), stdout (1), and stderr (2).

On the Flatpak side, it will check if any of these file-descriptors are a tty (with the convenient isatty() function. If so, it performs the necessary ioctl() to make our spawned process the controller of the pty. Lovely!

So now we have a terminal, whose process is running on the host, using a pty from inside our Flatpak.

Quick Highlight

Martin Blanchard put together a new “quick highlight” plugin for Builder this last week. It was a great example of how to submit a new feature, so I just wanted to highlight it here. Post to bugzilla, attach a patch, and we will review quickly and help with any additional integration that might be necessary.

Thanks Martin!

Highlights words matching the current selection

Sysprof + Builder

After the GNOME 3.20 cycle completed I started revamping Sysprof. More here, here, and here. The development went so smoothly that I did a 3.20 release a couple of weeks later.

A primary motivation of that work was rebuilding Sysprof into a set of libraries for building new tools. In particular, I wanted to integrate Sysprof with Builder as our profiler of choice.

On my flight back from GUADEC I laid the groundwork to integrate these two projects. As of Builder 3.21.90 (released yesterday) you can now profile your project quite easily. There are more corner cases we need to handle but I consider those incremental bugs now.

Some of our upcoming work will be to integrate the Sysprof data collectors with Python and Gjs. The Gjs implementation is written, it just needs polish and integration with upstream. I think it will be fantastic once we have a compelling profiling story weather you are writing C, C++, Vala, Python, or Gjs.

We’ve also expanded the architectures supported by Sysprof. So I expect by time 3.22 is released, Sysprof will support POWER8, ARM, ARM64, mips, and various others as long as you have an up to date Linux kernel. That is an important part of our future plans to support remote profiling (possibly over USB or TCP). If you’re interested in working on this, contact me! The plumbing is there, we just need someone with time and ambition to lead the charge.

Builder Happenings

Over the last couple of weeks I’ve started implementing Run support for Builder. This is somewhat tricky business since we care about complicated manners. Everything from autotools support to profiler/debugger integration to flatpak and jhbuild runtime support. Each of these complicates and contorts the problem in various directions.

Discovering the “target binary” in your project via autotools is no easy task. We have a few clever tricks but more are needed. One thing we don’t support yet is discovering bin_SCRIPTS. So launching targets like python or gjs applications is not ready yet. Once we discover .desktop files that should start working. Pretty much any other build system would be easier to implement this.

So much work behind the scenes for a cute little button.

While running, you can click stop to kill the spawned application’s process group.

But that is not all that is going on. Matthew Leeds has been working on search and replace recently (as well as a whole bunch of paper cuts). That work has landed and it is looking really good!

Also exciting is that thanks to Sebastien Lafargue we have a fancy new color picker that integrated with Builder’s source editor. You can visualize colors in your source files and change them using the dropper or numerous sliders based on how you’d like to tweak the color.

I still have a bunch of things I want to land before GUADEC, so back to the hacker den for me.