Using Leak Sanitizer with JHBuild

For a subset of GNOME modules, I’m still using jhbuild. I also spend a great deal of time tracking down memory bugs in various libraries. So it is very handy to have libasan.so working with meson -Db_sanitize=address.

To make things work, you currently need to:

  • dnf install libasan liblsan (or similar for your OS).
  • Use meson from git (0.48 development), for this bug fix.
  • Configure your meson projects with -Db_sanitize=address.
  • Create a suppression file for leaks out of our control.
  • Set some environment variables in ~/.config/jhbuildrc.

Here is an example of what I put in ~/.config/lsan_suppressions.txt.

leak:FcCharSetCreate
leak:FcLangSetCreate
leak:__nptl_deallocate_tsd
leak:_g_info_new_full
leak:dconf_engine_watch_fast
leak:g_get_language_names_with_category
leak:g_intern_string
leak:g_io_module_new
leak:g_quark_init
leak:libfontconfig.so.1

And add this to ~/.config/jhbuildrc.

import os
os.environ['LSAN_OPTIONS'] = 'suppressions=' + \
    os.path.expanduser('~/.config/lsan_suppressions.txt')

This has helped me track down a number of bugs in various modules this week and it would be useful if other people were doing it too.

Engineering Journals vs User Support

A major thanks to everyone involved in the gitlab migration. It’s no doubted a huge leap forward for GNOME on so many fronts.

Before we lose that momentum, I’d like to bring up in the collective minds of our project, what I consider, a separate problem. That is Bug Tracking versus User Support.

We’ve gotten a lot of flack over the years for brash or abrupt comments in our bug trackers. And while I can see that from a certain angle, I think in many cases it’s due to the miss-use of bug trackers as a user support mechanism.

For example, I’ve always seen a bug tracker as an engineering journal: a way to keep track of symptoms, potential solutions and trade-offs, and finally, the chosen solution. One valuable aspect to this is that you don’t have to go mucking through unrelated information or prose to pick out the valuable bits. In other words, terseness.

Where as user support has a different focus. The focus is on the user and their symptom, making them feel heard, and do our best to convert that information into a succinct chain of information for the engineering journal.

Treating engineering journals and user support as equivalent is preventing us from doing either with proficiency.

We generally don’t have this focus in F/OSS. It requires a set of skills that many of us have not cultivated and probably should. In addition, we should encourage those that already have these skills to join us.

But that raises the question: is gitlab the right place to do user support?

If GNOME were to advance Free Software by taking user support to the next level, what would that look like and what tooling do we need? Is that worth investing in now that we have many applications to support in addition to the desktop plumbing?

Hopefully we can have some discussion about that on the beach in Almería, Spain for GUADEC 2018.

More compiler fun

Yesterday’s post about -fstack-protector challenges leads us to today’s post. If you haven’t read it, go back and read it first.

Basically, the workaround I had at the time was to just disable -fstack-protector for the get_type() functions. It certainly made things faster, but it was a compromise. The get_type() functions can have user-provided code inserted into them via macros like G_DEFINE_TYPE_EXTENDED() and friends.

A real solution should manage to return the performance of the hot-path back to pre-stack-protector performance without sacrificing the the protection gained by using it.

So I spent some time today to make that happen. In bugzilla #795180 I’ve added some patches which break the get_type() functions into two. One containing the hot path, and a second that does the full type registration which is protected by our g_once_init_enter(). If we add the magic __attribute__((noinline)) to the function doing the full type registration, it can’t be inline’d into the fast path allowing it to pass the stack-protector sniff test (and therefore, not incur the stack checking wrath).

The best part is that applications and libraries only need to be recompiled to get the speedup (due to macro expansion).

Not a bad experience diving into some compiler bits this week.

Device Integration

I’ve been working on some groundwork features to sneak into Builder 3.28 so that we can build upon them for 3.30. In particular, I’ve started to land device abstractions. The goal around this is to make it easier to do cross-architecture development as well as support devices like phones, tablets, and IoT.

For every class and kind of device we want to support, some level of integration work will be required for a great experience. But a lot of it shares some common abstractions. Builder 3.28 will include some of that plumbing.

For example, we can detect various Qemu configurations and provide cross-architecture building for Flatpak. This isn’t using a cross-compiler, but rather a GCC compiled for aarch64 as part of the Flatpak SDK. This is much slower than a proper cross-compiler toolchain, but it does help prove some of our abstractions are working correctly (and that was the goal for this early stage).

Some of the big things we need to address as we head towards 3.30 include running the app on a remote device as well as hooking up gdb. I’d like to see Flatpak gain the support for providing a cross-compiler via an SDK extension too.

I’d like to gain support for simulators in 3.30 as well. My personal effort is going to be based around GNU-based systems. However, if others join in and want to provide support for other sorts of devices, I’d be happy to get the contributions.

Anyway, Builder 3.28 can do some interesting things if you know where to find them. So for 3.30 we’ll polish them up and make them useful to vastly more people.

Musings on bug trackers

Over the past couple of weeks we migraged jsonrpc-glib, template-glib, libdazzle, and gnome-builder all to gitlab.gnome.org. It’s been a pretty smooth process, thanks to a lot of hard work by a few wonderfully accommodating people.

I love bugzilla, I really do. I’ve used it nearly my entire career in free software. I know it well, I like the command line tool integration. But I’ve never had a day in bugzilla where I managed to resolve/triage/close nearly 100 issues. I managed to do that today with our gitlab instance and I didn’t even mean to.

So I guess that’s something. Sometimes modern tooling can have a drastic effect rather immediately.

Grow your skills with GNOME

Another year of GNOME development is coming to a close so it’s time to look back as we forge into 2018. This is going to be more verbose than I generally write. I hope you’ll have a warm drink and take the time to read through because I think this is important.

Twenty years of GNOME is a monumental achievement. We know so much more about software development than we did when we started. We regularly identify major shortcomings and try to address them. That is a part of our shared culture I enjoy greatly.

GNOME contributors have a wide variety of interests and direction when it comes to a computer’s role in our lives. That naturally creates an ever-expanding set of goals. As our goals expand we must become more organized if our quality is to maintain or improve.

Traditionally, we have a very loosely organized project. People spend their time on things that interest them, which does not put the focus on the end product. I intend to convince you that is now holding us back. We’re successful not because of our engineering focus but despite it. This results in overworking our contributors and we can do better.


Those that have not worked in larger engineering companies may be less familiar with some of the types of roles there are in software development. So let’s take a moment to describe these roles so everyone is on the same page.

Programmers are responsible for the maintenance of the code-base and implementing new features. All of us familiar with this role in GNOME because it’s what a large number of our contributors do.

Designers are responsible for thinking through the current and planned features to find improved ways for users to solve their problems.

Graphic Designers can often overlap with Design, but not necessarily. They’re responsible for creating the artwork used in the given project.

Quality Assurance ensures that you don’t ship a product that is broken. You don’t wait until the freezes to do this, but do it as features are developed so that the code is fresh in the programmers minds while addressing the issues. The sooner you catch issues, the less likely a code or design failure reaches users.

User Support is your front-line defense to triage incoming issues by your users. Without users your project is meaningless. Finding good people for this role can have a huge impact on keeping your users happy and your developers less stressed. If your bug tracker is also your user support, you might want to ask yourself if you really have user support. When you have a separate support system and bug-tracker, user support is responsible for converting user issues into detailed bug reports.

Security Engineers look for trust, privacy, and other safety issues in products and infrastructure. They take responsibility to ensure issues are fixed in a timely manner and work with others when planning features to help prevent issues in the first place.

User and Developer Advocates are liaisons between your team and the people using (or developing third-party tools with) your product. They amplify the voices of those speaking important truths.

User Testing is responsible for putting your product in-front of users and seeing how well they can perform the given tasks. Designers use this information to refine and alter designs.

Tech writers are responsible for writing technical documentation and help guides. They also help refine programmer authored API documentation. This role often fulfills the editor role to ensure a unified voice to your project’s written word.

Build engineers ensure that your product can be packaged, built reliably, and distributed to users.

Operations and “DevOps” ensure that your product is working day-to-day. They provide and facilitate the tooling that these roles need to do their jobs well.

Internationalization and localization ensure that your software is available to a group of users who might otherwise not be able to use your software. It enables your software to have global impact.

Release management is your final check-point for determining when and what you release based on the goals of the project. They don’t necessarily determine road-maps, but they do keep you honest.

Product managers are responsible for taking information and feedback from all these roles and converting that into a coherent project direction and forward looking vision. They analyze common issues, bug velocity, and features to ensure reasonable milestones that keeps the product functional as it transforms into it’s more ideal state. Most importantly, this role is leadership.

There are other roles involved in the GNOME project. If I didn’t include your role here, it is by no means of any lesser value.


For the past 3 years I’ve been working very hard because I fulfill a number of these roles for Builder. It’s exhausting and unsustainable. It contributes to burnout and hostile communication by putting too much responsibility on too few people’s shoulders.

I believe that communication breakdown is a symptom of a greater problem, not the problem itself.

To improve the situation, we need to encourage more people to join us in non-programming roles. That doesn’t mean that you can’t program, but simply a recognition that these other roles are critical to a functioning and comprehensive software project.


There are a few strategies used by companies with how to structure teams. But they can be generalized into three forms, as follows.

Teams based on product contain the aforementioned roles, but are assembled in a tight-knit group where people are focused on a single product. This model can excel at ensuring all of the members are focused on a single vision.

Teams based on role contain the aforementioned roles, but are assembled by the role. Members of the team work on different projects. This model can accel in cross-training newer team members.

A hybrid approach tries to balance the strengths of both team-based and role-based so that your team members get long-term mentorship but stick around in a project long enough to benefit from contextual knowledge.

To some degree we have teams based on role even though it’s very informal. I think we could really gain from increasing our contributors in these roles and taking a hybrid approach. For the hybrid approach to work, there needs to be strong mentor-ship for each role.

My current opinion is that with a strong focus on individual products, we can improve our depth of quality and address many outstanding user issues.

Due to how loosely assembled our teams are I think it is very difficult for someone to join GNOME and provide one of these non-programming roles in an existing project. That is because they not only need to fulfill the role but also define what that role should be and how it would contribute to the project. Then they need to convince the existing team members it’s needed.


With stronger inclusion of these roles into our software process we can begin to think about the long-term skill development of our contributors.

A good manager shepherds their team by ensuring they refine existing skills while expanding to new areas of interest.

I want people to know that by joining GNOME they can feel assured that they will be part of something greater than themselves. They will both refine and develop new skills. In many ways, we provide an accelerator for career development. We can provide an opportunity that might otherwise be unapproachable.


If contributing to GNOME in one or more of these roles sounds interesting to you, then please come join us. We need to learn to rely on each other in new ways. For that to happen, self-organization to fulfill these roles must become a priority.

https://www.gnome.org/get-involved/

Builder 3.27 Progress

We are a couple of months into Builder’s 3.28 development. We have fewer big ticket features scheduled this cycle when compared to 3.26. However that is replaced by a multitude of smaller features and details. Let’s take a look at some of what has been done already.

Flatpak Improvements

Early in the cycle we merged a feature upstream in flatpak-builder to emit escape sequences to set the terminal title as we progress through the build pipeline. Users of jhbuild are probably familiar with this type of thing as it does something similar. We can now consume this information from Builder to show more detailed progress about your Flatpak as it builds.

With yesterdays Flatpak 0.10.1 release, we got a feature we needed to access /usr/include of the host from a Flatpak. This means Builder can more easily develop against your host platform when using Builder from flatpak. It’s not a common request, but one we can support now.

Also yesterday, was the release of flatpak-builder 0.10.5. It has a new feature allowing us to specify --state-dir. If we detect a new enough flatpak-builder, we’ll use this to share dependency checkouts among various projects. When combined with shallow clones, I expect this to help reduce downloads for people who contribute to multiple projects.

Pseudo-Terminal Integration

We now depend on libvte directly from libide. This allows us to use a pseudo-terminal (PTY) in the build pipeline and show a terminal for the build output. This is both faster than our previous GtkTextView implementation and also adds support for colors and fixed scroll-back. If you have something other than a subprocess generating a build logs, we merge those into the terminal too!

Simplified Newcomers

As seen previously, we have a simpler process for newcomers wanting to explore an existing GNOME project. Just click on the icon and hit run!

Improved To-Do

By increasing our guarantees of thread-safety, we were able to speed up our scanning for todo items. We also fixed a few bugs along the way.

Improved Editor Search

Our editor search is some of the trickiest code in Builder. This is because we have to try to emulate various systems such as Vim. We refactored quite a bit of it to make it more resilient and handle all those tricky corner cases better.

More Code Indexers

Patrick contributed a GJS code indexer which can make it easier to jump around to classes and functions in your GJS-based project. I did the same for Vala. If you’re part of either of these language communities, we could really use your help improving our support for them.

Three-Finger-Swipe

As seen previously, the editor gained three-finger-swipe support to move editor panels left or right. You need Wayland for this feature for proper three-finger-swipe support for the lower layers of the stack.

Improved Meson and CMake Integration

Both the Meson and CMake build system plugins have been ported to C to get some type safety on our side. The architecture was also changed a bit to make it easier to extract compiler flags without needlessly advancing the build pipeline.

Unit Testing

The basics of unit testing have landed. We still have lots to do here before 3.28 like running under gdb and getting failure logs.

Find-Other-File Improvements

The find-other-file plugin was improved to support using the global search to list alternate files. This can be handy when switching between source, headers, and ui files.

Compile Commands Database

Builder now has a helper for compile_commands.json style files made popular by Clang. This can simplify the implementation of CFLAGS extraction by build systems that support it.

Build Target Providers

Creating and IDE that natively supports such a wide variety of project types and packaging technologies can be quite a challenge. There is often no clear abstraction for where a piece of information should be extracted. For example, does the build system know about installed build targets and how to run them? Is it the packaging technology, or a .desktop file? How about when containers are used?

This harsh reality means that sometimes we need to be very specific about our extension points. The new build target provider allows various system components to give us information about build artifacts. This has made it easier to run applications even when the build system has limited support. Long story short, if you use flatpak, things should mostly Just Work™, even when you use less well supported build systems like CMake.

Happy hacking!

Code indexing in Builder

Anoop, one of Builder’s GSoC students this past summer, put together a code-index engine built upon Builder’s fuzzy search algorithm. It shipped with support for C and C++. Shortly after the 3.27 cycle started, Patrick added support for GJS. Today I added support for Vala which was rather easy given the other code we have in Builder.

It looks something like this:

A screenshot of Builder display the code search results for Vala

Happy Hacking!

Simplifying contributions

Every release of both GNOME and Builder, we try to lower the barrier a bit more for new contributions. Bastian mentioned to me at GUADEC that we could make things even simpler from the Builder side of things. After a few mockups, I finally found some time to start implementing it.

With the upcoming Nightly build of Builder, you’ll be able to warp right through cloning and building of an application that is ready for newcomer contributions. Just open Builder and click on the application’s icon.

The greeter now shows a grid of icons so newcomers can simply click on the given icon to clone and build.

There is still more to do here, like adding a language emblem and such. Of course, if you want to work on that, do get in touch.

Closures with Async Operations

Way back in 2011 people were discussing usage of modern GCC features like __attribute__((cleanup())). A few years later it found it’s way into our API’s in GLib with one small caveat, only GCC/Clang support (so no MSVC/Xlc/SunProC). Since I couldn’t care less about MSVC I’ve been using it for years (and really Microsoft, you could contribute more to the mental health of open source programmers by modernizing MSVC).

I want to give a few examples of patterns I use to make tracking down issues easier.

Using GTask

static void
my_async_cb (GObject      *object,
             GAsyncResult *result,
             gpointer      user_data)
{
  // take ownership of task from caller
  g_autoptr(GTask) task = user_data;
  g_autoptr(GError) error = NULL;

  g_assert (G_IS_TASK (task));
  g_assert (G_IS_ASYNC_RESULT (result));

  if (!do_something_finish (result, &error))
    // explicitly pass ownership of error to GTask
    g_task_return_error (task, g_steal_pointer (&error));
  else
    g_task_return_boolean (task, TRUE);
}

void
my_obj_frob_async (MyObj               *self,
                   GCancellable        *cancellable,
                   GAsyncReadyCallback  callback,
                   gpointer             user_data)
{
  g_autoptr(GTask) task = NULL;

  g_return_if_fail (MY_IS_OBJ (self));
  g_return_if_fail (!cancellable || G_IS_CANCELLABLE (cancellable));

  task = g_task_new (self, cancellable, callback, user_data);
  g_task_set_source_tag (task, my_obj_frob_async);

  // pass task ownership to callback
  do_something_async (cancellable,
                      my_async_cb,
                      g_steal_pointer (&task));
}

The nice thing about this style is that all ownership transfers are explicit. I hope that in the future we can get some automatic checking of this via coverity or gcc/clang plugins. But we’re not quite there yet. Either way, it simplifies the auditing case.

Using Idle Callbacks

State tracking during idle callbacks can very easily turn into security issues. So make sure your function always has access to a reference, and simplify your releasing of the data by allowing the GSource to own the closure. For example, with a GObject it is pretty simple.

static gboolean
frob_from_idle_cb (gpointer data)
{
  MyObj *self = data;

  my_obj_frob (self);

  return G_SOURCE_REMOVE;
}

gdk_threads_add_idle_full (G_PRIORITY_LOW,
                           frob_from_idle_cb,
                           g_object_ref (obj),
                           g_object_unref);

The GSource which is registered and calls frob_from_idle_cb() will automatically call g_object_unref() after the function returns G_SOURCE_REMOVE. This also ensures your object isn’t finalized before the callback has occurred.

This also works with g_timeout_add_full(), gdk_threads_add_timeout_full().

Creating Custom Closures

Sometimes you might have state that is more complex than passing around a single GObject. In that case, create a closure structure and define a cleanup function so you can use g_autoptr().

typedef struct
{
  MyObj *self;
  guint  count;
} FrobState;

static void
frob_state_free (FrobState *state)
{
  g_clear_object (&state->self);
  g_free (state);
}

G_DEFINE_AUTOPTR_CLEANUP_FUNC (FrobState, frob_state_free)

With the above definition, you can use g_autoptr(FrobState) state = user_data; like you would for objects. This also works with the idle functions, just use (GDestroyNotify)frob_state_free as your cleanup function.