Builder 3.27 Progress

We are a couple of months into Builder’s 3.28 development. We have fewer big ticket features scheduled this cycle when compared to 3.26. However that is replaced by a multitude of smaller features and details. Let’s take a look at some of what has been done already.

Flatpak Improvements

Early in the cycle we merged a feature upstream in flatpak-builder to emit escape sequences to set the terminal title as we progress through the build pipeline. Users of jhbuild are probably familiar with this type of thing as it does something similar. We can now consume this information from Builder to show more detailed progress about your Flatpak as it builds.

With yesterdays Flatpak 0.10.1 release, we got a feature we needed to access /usr/include of the host from a Flatpak. This means Builder can more easily develop against your host platform when using Builder from flatpak. It’s not a common request, but one we can support now.

Also yesterday, was the release of flatpak-builder 0.10.5. It has a new feature allowing us to specify --state-dir. If we detect a new enough flatpak-builder, we’ll use this to share dependency checkouts among various projects. When combined with shallow clones, I expect this to help reduce downloads for people who contribute to multiple projects.

Pseudo-Terminal Integration

We now depend on libvte directly from libide. This allows us to use a pseudo-terminal (PTY) in the build pipeline and show a terminal for the build output. This is both faster than our previous GtkTextView implementation and also adds support for colors and fixed scroll-back. If you have something other than a subprocess generating a build logs, we merge those into the terminal too!

Simplified Newcomers

As seen previously, we have a simpler process for newcomers wanting to explore an existing GNOME project. Just click on the icon and hit run!

Improved To-Do

By increasing our guarantees of thread-safety, we were able to speed up our scanning for todo items. We also fixed a few bugs along the way.

Improved Editor Search

Our editor search is some of the trickiest code in Builder. This is because we have to try to emulate various systems such as Vim. We refactored quite a bit of it to make it more resilient and handle all those tricky corner cases better.

More Code Indexers

Patrick contributed a GJS code indexer which can make it easier to jump around to classes and functions in your GJS-based project. I did the same for Vala. If you’re part of either of these language communities, we could really use your help improving our support for them.

Three-Finger-Swipe

As seen previously, the editor gained three-finger-swipe support to move editor panels left or right. You need Wayland for this feature for proper three-finger-swipe support for the lower layers of the stack.

Improved Meson and CMake Integration

Both the Meson and CMake build system plugins have been ported to C to get some type safety on our side. The architecture was also changed a bit to make it easier to extract compiler flags without needlessly advancing the build pipeline.

Unit Testing

The basics of unit testing have landed. We still have lots to do here before 3.28 like running under gdb and getting failure logs.

Find-Other-File Improvements

The find-other-file plugin was improved to support using the global search to list alternate files. This can be handy when switching between source, headers, and ui files.

Compile Commands Database

Builder now has a helper for compile_commands.json style files made popular by Clang. This can simplify the implementation of CFLAGS extraction by build systems that support it.

Build Target Providers

Creating and IDE that natively supports such a wide variety of project types and packaging technologies can be quite a challenge. There is often no clear abstraction for where a piece of information should be extracted. For example, does the build system know about installed build targets and how to run them? Is it the packaging technology, or a .desktop file? How about when containers are used?

This harsh reality means that sometimes we need to be very specific about our extension points. The new build target provider allows various system components to give us information about build artifacts. This has made it easier to run applications even when the build system has limited support. Long story short, if you use flatpak, things should mostly Just Work™, even when you use less well supported build systems like CMake.

Happy hacking!

Code indexing in Builder

Anoop, one of Builder’s GSoC students this past summer, put together a code-index engine built upon Builder’s fuzzy search algorithm. It shipped with support for C and C++. Shortly after the 3.27 cycle started, Patrick added support for GJS. Today I added support for Vala which was rather easy given the other code we have in Builder.

It looks something like this:

A screenshot of Builder display the code search results for Vala

Happy Hacking!

Simplifying contributions

Every release of both GNOME and Builder, we try to lower the barrier a bit more for new contributions. Bastian mentioned to me at GUADEC that we could make things even simpler from the Builder side of things. After a few mockups, I finally found some time to start implementing it.

With the upcoming Nightly build of Builder, you’ll be able to warp right through cloning and building of an application that is ready for newcomer contributions. Just open Builder and click on the application’s icon.

The greeter now shows a grid of icons so newcomers can simply click on the given icon to clone and build.

There is still more to do here, like adding a language emblem and such. Of course, if you want to work on that, do get in touch.

Closures with Async Operations

Way back in 2011 people were discussing usage of modern GCC features like __attribute__((cleanup())). A few years later it found it’s way into our API’s in GLib with one small caveat, only GCC/Clang support (so no MSVC/Xlc/SunProC). Since I couldn’t care less about MSVC I’ve been using it for years (and really Microsoft, you could contribute more to the mental health of open source programmers by modernizing MSVC).

I want to give a few examples of patterns I use to make tracking down issues easier.

Using GTask

static void
my_async_cb (GObject      *object,
             GAsyncResult *result,
             gpointer      user_data)
{
  // take ownership of task from caller
  g_autoptr(GTask) task = user_data;
  g_autoptr(GError) error = NULL;

  g_assert (G_IS_TASK (task));
  g_assert (G_IS_ASYNC_RESULT (result));

  if (!do_something_finish (result, &error))
    // explicitly pass ownership of error to GTask
    g_task_return_error (task, g_steal_pointer (&error));
  else
    g_task_return_boolean (task, TRUE);
}

void
my_obj_frob_async (MyObj               *self,
                   GCancellable        *cancellable,
                   GAsyncReadyCallback  callback,
                   gpointer             user_data)
{
  g_autoptr(GTask) task = NULL;

  g_return_if_fail (MY_IS_OBJ (self));
  g_return_if_fail (!cancellable || G_IS_CANCELLABLE (cancellable));

  task = g_task_new (self, cancellable, callback, user_data);
  g_task_set_source_tag (task, my_obj_frob_async);

  // pass task ownership to callback
  do_something_async (cancellable,
                      my_async_cb,
                      g_steal_pointer (&task));
}

The nice thing about this style is that all ownership transfers are explicit. I hope that in the future we can get some automatic checking of this via coverity or gcc/clang plugins. But we’re not quite there yet. Either way, it simplifies the auditing case.

Using Idle Callbacks

State tracking during idle callbacks can very easily turn into security issues. So make sure your function always has access to a reference, and simplify your releasing of the data by allowing the GSource to own the closure. For example, with a GObject it is pretty simple.

static gboolean
frob_from_idle_cb (gpointer data)
{
  MyObj *self = data;

  my_obj_frob (self);

  return G_SOURCE_REMOVE;
}

gdk_threads_add_idle_full (G_PRIORITY_LOW,
                           frob_from_idle_cb,
                           g_object_ref (obj),
                           g_object_unref);

The GSource which is registered and calls frob_from_idle_cb() will automatically call g_object_unref() after the function returns G_SOURCE_REMOVE. This also ensures your object isn’t finalized before the callback has occurred.

This also works with g_timeout_add_full(), gdk_threads_add_timeout_full().

Creating Custom Closures

Sometimes you might have state that is more complex than passing around a single GObject. In that case, create a closure structure and define a cleanup function so you can use g_autoptr().

typedef struct
{
  MyObj *self;
  guint  count;
} FrobState;

static void
frob_state_free (FrobState *state)
{
  g_clear_object (&state->self);
  g_free (state);
}

G_DEFINE_AUTOPTR_CLEANUP_FUNC (FrobState, frob_state_free)

With the above definition, you can use g_autoptr(FrobState) state = user_data; like you would for objects. This also works with the idle functions, just use (GDestroyNotify)frob_state_free as your cleanup function.

Improving Builder docs

We use reStructuredText/sphinx for Builder’s documentation because, quite frankly, I found it the easiest for writing massive amounts of documentation in short order. I’m not sure if it’s what I want to stick with long term, but it’s doing the job short term.

However, one thing we don’t get (and I really want) out of any documentation system is support for cross-referencing the user documentation and API docs. This would be useful for us in Builder because we are the use-case for turning users into contributors. Reading our documentation on writing plugins and then locating the API docs should be seamless.

So last week I put together a prototype to generate reStructuredText/Sphinx API docs from .gir files. It will use the provided .gir, including any located dependencies, and generate API docs for them. It can’t be 100% complete for C because the .gir is missing some information that are only found in the sgml files. But it does an okay job.

Mostly, this was just a prototype to see what the state of the documentation systems are. I’m still fairly dissatisfied and am leaning towards the path of “more prototyping necessary”. For example, some things I still want from a modern documentation platform:

  • Cross-referencing among user and API docs
  • Switch between C, Python, GJS, etc
  • Jump to function/class source code
  • Find example usage of API in GNOME git
  • Partial gettext/i18n support so we don’t have to translate API docs, but can still cross-reference them.
  • Fast lookup indexes for use in IDE auto-completion docs using only read-only mmap()‘d files.
  • We might want surrogate links to API docs when cross-referencing so end-user documentation stays small.

Anyway, if you’re bored, play around with it here and give me constructive feedback.

A new gutter for Builder

The GtkSourceView library has this handy concept of a GtkSourceGutterRenderer. They are similar in concept to a GtkCellRenderer but for the gutter to the left or right of the text editor.

Like a GtkCellRenderer, you pack it into a container and they are placed one after another with some amount of optional spacing in-between. This is convenient because you can start quickly by mixing and matching what you need from existing components. Those include text (such as line numbers), pixbuf rendering, or even code folding regions.

However, there is a cost to this sort of composition. One is function call overhead, but that isn’t particularly interesting to me because there are ways to amortize that away (like we did with the pixel cache). The real problem is one of physical space. Each time a renderer is added, the width of the gutter is increased.

Builder 3.26.0 added a new column for breakpoints, and so we increased our width by another 18 pixels or so. Enough to be cumbersome. It looked like the following which has 4 renderers displayed.

  • Breakpoints renderer
  • Diagnostics renderer
  • Line numbers
  • Line changes (git)

Once you reach some level of complexity, you need to bite the bullet and implement a single renderer that has all the features you want in one place. It allows you to overlap content for density and use the background itself as a component. We just did that for Builder and here is what it looks like.

There are a couple other nice points performance-wise by implementing the gutter as a single renderer. We can take a number of “shortcuts” in the render path that a generic renderer cannot without sacrificing flexibility. Since the gutter is not pixel cached, this has improved the performance of kinetic scrolling on various HiDPI displays. There is always more performance work to do, but I’m rather happy with the result so far.

You’ll find this in the upcoming 3.26.1 release of Builder and is already available in Builder’s Nightly flatpak.

Builder 3.26 Sightings

We’re getting close to 3.26 and a number of features have landed. Let’s take a quick screenshot tour to see what you’re likely to see in 3.26.

Most of us have seen the new visual design by now

Visual refresh

A modest debugger

A debugger for Builder

Integrated Symbol Search by GSoC student Anoop Chandu

symbol search

Inline Documentation by GSoC student Lucie Charvát

Inline documentation

Word Completion based on distance from cursor by GSoC student Umang Jain

word completion

I expect the word completion to gain some fancy features like following #include files and custom sort ordering which will look and feel similar to Vim users.

This week in Builder

I’ve been aggressively pushing forward our new layout branch to integrate the new design work we’ve been iterating on. I don’t think we need to have it all done to merge to master, so we are rapidly approaching the time-frame for the branch to land.

There is much from the design still missing, but every day more pieces come together. I expect to have a good portion of it done by GUADEC.

A lot happened this week on the wip/chergert/layout branch. Here are some highlights.

  • Georges landed a “focus mode” fullscreen mode for Builder.
  • We’ve landed a patch in flatpak-builder to update terminal titles. This will allow us to scrape that info via the PTY in Builder for better progress messages.
  • Document titlebars now match the primary-color of the content. We also fade between states. This took some craftiness to avoid cascading the entire CSS tree. Video example below.
  • DzlPropertiesGroup is a convenient GActionGroup implementation for exporting multiple object properties from a GObject. This is more convenient than GPropertyAction when you are exposing lots of properties together.
  • The TODO plugin was ported to C and employs various techniques to use reduce memory overhead.
  • Dazzle gained convenience API to do insertion sort on a GtkListStore in O(log n) time by accessing the GSequence directly. This should be a safe layer-violation on Gtk+ 3.x.
  • Terminal, Devhelp, and HTML-Preview were ported to the new layout design.
  • IdeSymbolResolver gained a “find nearest scope” API which is used to update the scope name in the document titlebar. Both C and Vala are supported. Other languages need an implementation still.
  • The project tree was tweaked to look more like the mockup. While this won’t be our default project-tree for 3.26, we will keep it around and therefore it should match the design.
  • Overview map was ported to the new layout design.
  • Plugins should use the new IdeBuffer::change-settled API to be notified of when they should update their state based on buffer content (unless alternate API is provided via the plugin interface).
  • We’ve added type icons to the devhelp search that match the language-feature icons used elsewhere in Builder.
  • Scrolling to the insertion point on buffer load has been vastly improved.
  • Performance of buffer loading has also improved.
  • Todo and build diagnostics have been ported to the new design.

Reflowing text in GtkTreeView

You can do some pretty evil things with GtkTreeView if you put your mind to it. One of the most common questions over the years has been how to reflow (wrap) text. The short answer is you can’t. The answer for those willing to hack around it is something like:

So the real question is “Why not use GtkListBox?”

GtkListBox is a fantastic widget. But there are some things it does not deal well with today. I presume that once it can deal with only keeping a minimum number of widgets in memory (and the size-request magic that goes with it) it would be a better solution.

Builder keeps a lot of widgets active during the lifetime of the application, and that makes style propagation particularly slow. The fewer widgets we can use, the more responsive we can keep things. (And of course, there are multiple ways to do this).

evil tree view
evil cell renderer

Troubleshooting Builder

If you find yourself needing to troubleshoot Builder (hopefully just during our development cycles) you can now run the command “counters” from the command bar (Ctrl+Enter). This gives you access to some internal counters.

You can get these out of process too, using dazzle-list-counters PID which cracks open the shared memory page and dumps the counters.

A screenshot of the counters displayed for the program.

As you might imagine, the counters are in libdazzle now so you can have the same sort of feature in your (C-based) applications quite easily.