Speed up your GitLab CI

GNOME GitLab has AWS runners, but they are used only when pushing code into a GNOME upstream repository, not when you push into your personal fork. For personal forks there is only one (AFAIK) shared runner and you could be waiting for hours before it picks your job.

But did you know you can register your own PC, or a spare laptop collecting dust in a drawer, to get instant continuous integration (CI) going? It’s really easy to setup!

1. Install docker

apt install docker.io

2. Install gitlab-runner

Follow the instructions here:
https://gitlab.com/gitlab-org/gitlab-runner/blob/master/docs/install/linux-repository.md#installing-the-runner

(Note: The Ubuntu 18.04 package doesn’t seem to work.)

3. Install & start the GitLab runner service

sudo gitlab-runner install
sudo gitlab-runner start

4. Find the registration token

Go to your gitlab project page, settings -> CI/CD -> expand “runners”

5. Register your runner

sudo gitlab-runner register --non-interactive --url https://gitlab.gnome.org --executor docker --docker-image fedora:27 --registration-token **

You can repeat step 5 with the registration token of all your personal forks in the same GitLab instance. To make this easier, here’s a snippet I wrote in my ~/.bashrc to register my “builder.local” machine on a new project. Use it as gitlab-register .

function gitlab-register {
  host=$1
  token=$2

  case "$host" in
    gnome)
      host=https://gitlab.gnome.org
      ;;
    fdo)
      host=https://gitlab.freedesktop.org
      ;;
    collabora)
      host=https://gitlab.collabora.com
      ;;
    *)
      host=https://gitlab.gnome.org
      token=$1
  esac

  cmd="sudo gitlab-runner register --non-interactive --url $host --executor docker --docker-image fedora:27 --registration-token $token"

  #$cmd

  ssh builder.local -t "$cmd"
}

Not only will you now get faster CI, but you’ll also reduce the queue on the shared runner for others!

State of Meson in GLib/GStreamer

During the last couple of months I’ve been learning the Meson build system. Since my personal interests in Open Source Software are around GLib and GStreamer, and they both have Meson and Autotools build systems in parallel, I’ve set as personal goal to list (and try to fix) blocker bugs preventing from switching them to Meson-only. Note that I’m neither GLib nor GStreamer maintainer, so it’s not my call whether or not they will drop Autotools.

GLIB

I opened bug #790954, a meta-bug depending on all the bugs related to meson I’ve found and are bad enough that we cannot drop Autotools unless we fix them first. Amount those bugs, I’ve been personally working on those:

Bug 788773 meson does not install correct pc files

pkg-config files for glib/gobject/gio are currently generated from a .pc.in template but some flags (notably -pthread) are hidden internally in Meson and cannot be written in the .pc file. To fix that bug I had to use Meson’s pkg-config generator which has access to those internal compiler flags. For example the -pthread flag is hidden behind the dependency(‘threads’) object in a meson.build file. Unfortunately such object cannot be passed to the pkg-config generator, so I opened a pull request making that generator smarter:

  • When generating a pc file for a library, it automatically take all dependencies and link_with values from that library and add them into the Libs.private and Requires.private fields.
  • Extra dependencies (such as ‘threads’) can be added explicitly if needed. One common case is explicitly adding a public dependency that the generator would have added in private otherwise.

Bug 786796 gtk-doc build fails with meson

Pretty easy one: gobject’s API documentation needs to pass an extra header file when compiling gtkdoc-scangobj from a non standard location. I made a patch to have include_directories argument to gnome.gtkdoc() method and use it in glib.

Bug 790837 Meson: missing many configure options

Compared to Autotools’ configure, glib’s meson build system was missing many build options. Also existing options were not following the GNOME guideline. tl;dr: we want foo_bar instead of enable-foo-bar, and we want to avoid automatic options as much as possible.

Now configure options are on par with Autotools, except for the missing –with-thread which has a patch pending on bug #784995.

GStreamer

Both static and shared library

For GStreamer static builds are important. The number of shared libraries an Android application can link to is limited, and dlopen of plugins is forbidden on IOS (if I understood correctly). On those platforms GStreamer is built as one big shared library that statically link all its dependencies (e.g. glib).

Autotools is capable of generating both static and shared libraries and compile C files only once. Doing so with Meson is possible but requires unnecessary extra work. I created a pull request that adds both_library() method to meson, and add a global project option that turns all library() calls to build both shared and static.

Static build of gio modules

This one is not directly related to Meson, but while working on static builds, I’ve noticed that GStreamer patched glib-networking to be able to static build them. Their patch never made it upstream and it has one big downside: it needs to be built twice for static and dynamic. GStreamer itself recently fixed their plugins  ABI to be able to do a single compile and produce both shared and static libraries.

The crux is you cannot have the same symbol defined in every plugin. Currently GIO modules must all define g_io_module_load/unload/query() symbols which would clash if you try to static link more than one GIO module. I wrote patches for gio and glib-networking to rename those symbols to be unique. The symbol name is derived from the shared module filename. For example when gio loads libgiognutls.so extension it will remove “libgio” prefix and “.so” suffix to get “gnutls” plugin name. Then lookup for g_io_gnutls_load/unload/query() symbols instead (and fallback to old names if not found).

The second difficulty is GIO plugins uses G_DEFINE_DYNAMIC_TYPE which needs  a GTypeModule to be able to create its GType. When those plugins are static linked we don’t have any GTypeModule object. I made a patch to allow passing NULL in that case to turn a G_DEFINE_DYNAMIC_TYPE into a static GType.

Bug 733067 cerbero: support python3

Meson being python3-only and Cerbero python2-only, if we start building meson projects in cerbero it means we require installing both pythons too. It also adds problems with PYTHONPATH environment variable because it cannot differentiate between 2 and 3 (seriously why is there no PYTHONPATH3?).

I ran 2to3 script against the whole cerbero codebase and then fixed a few remaining bugs manually. All in all it was pretty easy, the most difficult part is to actually test all build variants (linux, osx, windows, cross-android, cross-windows). it’s waiting for 1.14 to be released before merging this into master.

Bug 789316 Add Meson support in cerbero

Cerbero already had a recipe to build meson and ninja but they were broken. I made patches to fix that and also add the needed code to be able to build recipes using meson. It also makes use of meson directly to build gst-transcoder instead of the wrapper configure and makefile it ships. Later more recipes will be able to be converted to Meson (e.g. glib). Blocking on the python3 port of cerbero.

Choose between static or shared library

One use case Olivier Crête described on Meson issue #2765 is he wants to make the smallest possible build of a GStreamer application, for IoT. That means static link everything into the executable (e.g. GStreamer, glib) but dynamic link on a few libraries provided by the platform (e.g. glibc, openssl).

With Autotools he was doing that using .la files: Libraries built inside cerbero has a “.la” file, and libraries provided by the platform don’t. Autotools has a mode to static link the former and dynamic link the latter, meson don’t.

I think the fundamental question in meson is what to do when a dependency can be provided by both a static and a shared library. Currently meson takes the decision for you and always use the shared library, unless you explicitly set static: true when you declare your dependency, with no project-wide switch.

In this pull request I fixed this by added 2 global options:

  • default_link: Tells whether we prefer static or shared when both are available.
  • static_paths: List of path prefixes where it is allowed to use static
    libraries. By default it has “/” which means all paths are allowed. It can be set to the path where cerbero built GStreamer (e.g. /home/…)  and it will static link only them, using shared library from /usr/lib for libraries not built within cerbero.

Conclusion

There is a long road ahead before getting meson build system on par with Autotools for GLib and GStreamer. But bugs and missing features are relatively easy to fix. Meson code base is easy and pleasant to hack, unlike m4 macros I’ve never understood in the past 10 years I’ve been writing Autotools projects.

I think droping Autotools from GLib is a key milestone. If we can achieve that, it proves that all weird use-cases people has been relying on can be done with Meson.

I’ve been working on this on my personal time and on Collabora’s “2h/week for personal projects” policy. I’ll continue working on that goal when possible.

git-phab: more updates

git-phab got tons of improvements last days, thanks to Thibault Saunier.

  • New “browse” command to open tasks and differentials links. With no arguments it opens the task if your current branch is in the form “TXXX-description”. Otherwise it can be passed a list of task ids, differential ids or commits. For example:

    git phab browse T123 D345 HEAD bc287c9

  • It uses argcomplete for bash completion, but seems to work only if writing “git-phab <command>” and not “git phab <command>”. Ideas how to improve this?
  • The “attach” command can now create a new task on phabricator when current branch is not associated to an existing one.
  • The “attach” command now proposes to create a branch with a name in the form “TXXX-description” if current branch isn’t in that format yet. That makes easier to update the patchset later since the task will be guessed from the branch name if no –task argument is passed.
  • It is now mandatory to have a “.arcconfig” file in your git repository with at least “phabricator.uri” and “project” keys. The “attach” command will use that info to set the project on new differential/tasks.
  • The default commit range for “attach” and “log” commands is now from the head of the remote tracking branch to HEAD. Instead of being hard-coded to “origin/master..HEAD”.
  • If your “.arcconfig” contains a “default-reviewers” key, it will be used when –reviewers argument is not specified.
  • More ideas for future improvements has been filled in our backlog. Help and feedback is welcome.

git-phab: updates

Many improvements on git-phab lately:

  • It is now hosted on phabricator.freedesktop.org:
  • “git phab attach” now also push the branch and links it on the Maniphest (only supported on fdo and collabora’s phabricator atm). Reviewers/testers can then easily retrieve the branch with “git phab fetch Txxx”.
  • Fixed compatibility with phabricator’s fragile commit msg parser.
  • Check the backlog, give feedback, contribute 🙂

Phabricator git integration

Recently I’ve been using Phabricator for a few projects. I wrote a small tool called git-phab to help attaching a set of patches to Phabricator for review. The idea is that it creates a new Differential for each commit in a range you provide, a bit like git-bz.

It started as a small personal helper, but lots of colleagues said it’s useful for them as well, so let’s share it: repository, README.

OpenGlucose: Again

Made progress this weekend on OpenGlucose. The GUI is still ugly but it has the info I want.

Important things on my wish-list, when I have time:

  1. Handling the units. My only device is mg/dl but other countries uses mmol/L. Since I’m living in Canada where they use mmol/L I should grab a new device with those units, so I’ll be able to compare logs and see how to know in which unit the device is configured.
  2. Make printable report, I’ve heard doctors like that. OTOH, I shouldn’t encourage using unofficial app for medical purpose.
  3. Support other FreeStyle devices. I’m pretty sure they all have the same kind of format so most of the parser should be reusable, I hope. I should be able to get a spare FreeStyle Freedom Lite in a few weeks.
  4. Publish an ubuntu package on a PPA.
  5. CSV export.
  6. Ideas?

I’m curious, does someone else own an InsuLinx and tried OpenGlucose yet? Or even tried to support another device?

 

openglucose

OpenGlucose: continued

I started working on the UI to display the results:

openglucoseIt is made using a GtkApplicationWindow containing a WebkitWebView, the content is made with HTML/CSS/JS with jquery and the chart is made using jqplot.

To make testing easier, I also added a dummy device that has random data, it can be enabled by setting OPENGLUCOSE_DUMMY_DEVICE=1 in your env.

A lot more work is needed, but that’s a start.

New project: OpenGlucose

Hello there,

I recently got diagnosed with a diabetes type 1. Like all diabetics, I got a glucometer device that comes with a windows/mac closed-source application. That’s clearly not acceptable for a freedom lover! So here is my new challenge: reverse-engineer the USB protocol of my Abbott FreeStyle InsuLinx device, and write an open source Linux application for it.

And here it is: https://github.com/xclaesse/OpenGlucose

So far it only fetch the bare minimum information from the device and print them in the terminal. More GUI/features will come later.

If you’re a geek diabetic, your help is welcome!

OTR in Empathy

It was a really popular Empathy feature request, it even has a bounty on it, and today I just did it! The patches are available in bugzilla, apply on telepathy-gabble and empathy master.

Here is a screencast of OTR in action between Empathy and Pidgin:

I hacked it on my free time, let’s see if crowd funded projects really works. If you like that feature feel free to make a donation on the bounty: https://freedomsponsors.org/core/issue/333/telepathy-should-support-otr-encryption.

Folks2 progress

I am working on Folks2 when I have a bit of free time. In short it is a rewrite from scratch of Folks in C with a daemon/db to remember who is merged with who. The goal is to be faster and being able to load individuals separately instead of having to load them all.

Recently, I’ve been writing libfolks2-roster, a GTK library providing widgets to display a contact list and contact details. It is using awesome GTK 3.10 GtkListBox, GtkStack and GtkSearchBar. The main issue I’m facing now is GtkListBox slowness, it really needs a model to create only visible widgets.

(Yes it is ubuntu theme, I’m usually using unity but video recording didn’t work)