Introducing BuildStream

Greetings fellow Gnomies :)

At Codethink over the past few months we’ve been revisiting our approach to assembly of whole-stack systems, particularly for embedded Linux systems and custom GNOME based systems.

We’ve taken inspiration, lessons and use-cases from various projects including OBS, Reproducible Builds, Yocto, Baserock, buildroot, Aboriginal, GNOME Continuous, JHBuild, Flatpak Builder and Android repo.

The result of our latest work is a new project, BuildStream, which aims initially to satisfy clear requirements from GNOME and Baserock, and grow from there. BuildStream uses some key GNOME plumbing (OSTree, bubblewrap) combined with declarative build-recipe description to provide sandboxed, repeatable builds of GNOME projects, while maintaining the flexibility and speed required by GNOME developers.

But before talking about BuildStream, lets go over what this can mean for GNOME in 2017.

Centralization of build metadata

Currently we build GNOME in various ways, including JHBuild XML, Flatpak JSON for the GNOME Runtime and SDK, and GNOME Continuous JSON for CI.

We hope to centralize all of this so that the GNOME release team need only maintain one single set of core module metadata in one repository in the same declarative YAML format.

To this end, we will soon be maintaining a side branch of the GNOME release modulesets so people can try this out early.

GNOME Developer Experience

JHBuild was certainly a huge improvement over the absolutely nothing that we had in place before it, but is generally unreliable due its reliance on host tooling and dependencies.

  • Newcomers can have a hard time getting off the ground and making sure they have satisfied the system dependencies.
  • Builds are not easily repeatable, you cannot easily build GNOME 3.6 today with a modern set of dependencies.
  • Not easy to test core GNOME components like gnome-session or the gnome-initial-setup tool.

BuildStream nips these problems at the bud with an entirely no-host-tooling policy, in fact you can potentially build all of GNOME on your computer without ever installing gcc. Instead, GNOME will be built on top of a deterministic runtime environment which closely resembles the freedesktop-sdk-images Flatpak runtime but will also include the minimal requirements for booting the results in a VM.

Building in the Swarm

BuildStream supports artifact cache sharing so that authenticated users may upload successful build results to share with their peers. I doubt that we’ll want to share all artifacts between random users, but having GNOME Continuous upload to a common artifact cache will alleviate the pain of webkit rebuilds (unless you are hacking on webkit of course).

Flatpak / Flathub support

BuildStream will also be available as an alternative to flatpak-builder.

We will be providing an easy migration path and conversion script for Flatpak JSON which should be good enough for most if not all Flatpak app builds.

As the Flathub project develops, we will also work towards enabling submission of BuildStream metadata as an alternative to the Flatpak Builder JSON.

About BuildStream

Unlike many existing build systems, BuildStream treats the problem of building and distribution as separate problem spaces. Once you have built a stack in BuildStream it should be trivial enough to deploy it as rpms, debian packages, a tarball/ostree SDK sysroot, as a flatpak, or as a bootable filesystem image which can be flashed to hardware or booted in a VM.

Our view is that build instructions as structured metadata used to describe modules and how they integrate together is a valuable body of work on its own. As such we should be able to apply that same body of work reliably to a variety of tasks – the BuildStream approach aims to prove this view while also offering a clean and simple user experience.

BuildStream is written in Python 3, has fairly good test coverage at this stage and is quite well documented.

BuildStream works well right now but still lacks some important features. Expect some churn over the following months before we reach a stable release and become a viable alternative for developing GNOME on your laptop/desktop.

Dependencies

Note that for the time being the OSTree requirement may be too recent for many users running currently stable distros (e.g. debian Jessie). This is because we use the OSTree gobject introspection bindings which require a version from August 2016. Due to this hard requirement it made little sense to include special case support for older Python versions.

However with that said; if this transitional period is too painful, we may decide to lower the Python requirement and just use the OSTree command line interface instead.

Build Pipelines

The BuildStream design in a nutshell is to have one abstract core, which provides the mechanics for sandboxing build environments (currently using bubblewrap as our default sandbox), interpreting the YAML data model and caching/sharing the build results in an artifact cache (implemented with ostree) and an ecosystem of “Element” plugins which process filesystem data as inputs and outputs.

In a very abstract view, one can say that BuildStream is like GStreamer but its extensible set of element plugins operate on filesystem data instead of audio and video buffers.

This should allow for a virtually unlimited variety of pipelines, here are some sketches which attempt to illustrate the kinds of tasks we expect to accomplish using BuildStream.

Import a custom vendor tarball, build an updated graphics stack and BSP on top of that, and use a custom export element to deploy the build results as RPMS:

Import the base runtime ostree repository generated with Yocto, build the modules for the freedesktop-sdk-images repository on top of that runtime, and then deploy both Runtime and SDK from that base, while filtering out the irrelevant SDK specific bits from the Runtime deployment:

Import an arbitrary but deterministic SDK (not your host !) to bootstrap a compiler, C runtime and linux kernel, deploy a bootable filesystem image:

Build pipelines are modular and can be built recursively. So a separate project/pipeline can consume the same base system we just built and extend it with a graphics stack:

A working demo

What follows are some instructions to try out BuildStream in its early stages.

For this demo we chose to build a popular application (gedit) in the flatpak style, however this does not yet include an ostree export or generation of the metadata files which flatpak requires; the built gedit result cannot be run with flatpak without those steps but can be run in a `build-stream shell` environment.

# Installing BuildStream

# Before installing BuildStream you will need to first install
# Python >= 3.5, bubblewrap and OSTree >= v2016.8 as stated above.

# Create some arbitrary directory, dont use ~/buildstream because
# that's currently used by buildstream unless you override the 
# configuration.
mkdir ~/testing
cd testing
git clone https://gitlab.com/BuildStream/buildstream

# There are a handful of ways to install a python setuptools
# package, we recommend for developer checkouts that you first
# install pip, and run the following command.
#
# This should install build-stream and its pythonic dependencies
# into your users local python environment without touching any
# system directories:
cd buildstream
pip install --user -e .

# Clone the demo project repository
cd ..
git clone https://gitlab.com/BuildStream/buildstream-tests
cd buildstream-tests

# Take a peek of the gedit.bst pipeline state (optional)
#
# This will tell us about all the dependencies in the pipeline,
# what their cache keys are and their local state (whether they
# are already cached or need to be built, or are waiting for a
# dependency to be built first).
build-stream show --deps all gedit.bst

# Now build gedit on top of a GNOME Platform & Sdk
build-stream build gedit.bst

#
# This will take some time and quite some disk space, building
# on SSD is highly recommended.
#
# Once the artifact cache sharing features are in place then this
# will take half the disk space it currently takes, in the majority
# of cases where you BuildStream already has an artifact for the
# GNOME Platform and SDK bases.
#

# Ok, the build may have taken some time but I'm pretty sure it
# succeeded.
#
# Now we can launch a sandbox shell in an environment with the
# built gedit:
build-stream shell --scope run gedit.bst

# And launch gedit. Use the --standalone option to be sure we are
# running the gedit we just built, not a new window in the gedit
# installed on your host
gedit --standalone

Getting Involved

As you can see we’re currently hosted from my user account on gitlab, so our next steps is to sort out a proper hosting for the project including mailing list, bug tracking and a place to publish our documentation.

For right now, the best place to reach us and talk about BuildStream is in the #buildstream channel on GNOME IRC.

If you’d like to play around with the source, a quick read into the HACKING file will provide some orientation for getting started, coding conventions, building documentation and running tests.

 

With that, I hope you’ve all enjoyed FOSDEM and the beer that it entails :)

11 thoughts on “Introducing BuildStream”

  1. Sounds interesting, but I’m kinda missing the big picture. What does BuildStream offer an app developer that flatpak-builder doesn’t? Is it possible to build any core GNOME module, like gnome-shell? How would you run that?

  2. Hi Allan,

    > Sounds interesting, but I’m kinda missing the big picture.
    > What does BuildStream offer an app developer that
    > flatpak-builder doesn’t?

    From the perspective of an app developer who wants to build a flatpak and only ever build a flatpak, there is not much gain (perhaps a quicker build if that flatpak includes orthogonal dependencies which we can parallelize).

    However one day that same app developer might want to deploy their app as an AppImage or a “snap” or something entirely different, BuildStream should allow that developer to do so without maintaining three separate sets of build instructions in different formats.

    > Is it possible to build any core GNOME module, like gnome-shell?
    > How would you run that?

    Yes this will certainly be possible, but you probably also want a system to run GNOME Shell on, and you probably want some things to run inside that shell (lets run a GNOME Shell on a system that has Nautilus at least ?)

    The user experience I expect for somebody developing GNOME Shell is that when running something like:

    build-stream build gnome.bst

    BuildStream will first download any already built software artifacts (from GNOME Continuous) which constitute an entire GNOME OS (as defined by the GNOME release modulesets) and only locally build anything which is not already up to date.

    The output of this build will be a bootable image which can be launched in qemu/kvm.

    Said developer can open a workspace in which they can modify the GNOME Shell sources to their hearts content and rebuild that same VM with their local copy of GNOME Shell sources in place of the active upstream version, and easily rebuild and run that same integrated VM many times in a row.

    There are other benefits of course, the deduplication of build metadata that we use to build the GNOME platform and it’s dependencies for various deployments (Flatpak SDK/Runtimes, GNOME OS live images and developer builds) means that we can assert that everything is always in sync.

    I.e. if the release team says we now use this fixed version of GNOME Shell in a given stable release; we just know that that security patch is reflected in our modulesets, in our Flatpak runtimes and in any live bootable images, all at once (not to mention the obvious reduction of maintenance burden).

    I hope this clarifies things a bit :)

  3. Hi Tristan,

    this sounds very interesting! Would this also be suitable for web applications where you would want to build packages for Django or PHP applications?

    -Jochen

  4. @brejoc,

    I had not seriously considered web applications and honestly have not spent a lot of time creating those. Thinking on this, I don’t see why this would not be useful for those cases.

    On the one hand, Python and PHP, not being compiled languages (or compiled at runtime), don’t seem to benefit much from the isolation and determinism of building in the sandbox, I could be wrong about that.

    On the other hand, it can be interesting just for bringing together multiple modules together and writing a plugin which could be used to bundle those together in a convenient way.

    Finally, the aspect of CI can be nice. For instance, I think it would be interesting to write Source plugins for BuildStream which allow one to stage a base runtime from popular package managers. This could be interesting for creating integrated VMs at will with your web applications pre-installed, providing some automation in how one might run tests of ones web application against different distros which might be available at various hostings.

  5. @Tristan You’d be surprised how much web applications benefit from a clean “build” – and even web applications have dependencies, which can be written in C, C++, Go, aso and need to be build. So having a sandbox is not only interesting for the non compiled web applications itself, but also for dependencies that are being used.

    Directly creating VMs sounds awesome! I don’t know how desktop applications are tested, but web applications are deployed all the time. Often people are deploying every branch of a repo, so that they are able to check every change as they are pushed in the whole team. Being able to compile this directly in to VM would be a huge benefit! Although containers would be even better.

    Right now I’m using something called basalt to build RPM or Deb packages. Yes, I’m the author of this very alpha glue code between invoke (a python make clone) and fpm (a Ruby tool to generate packages). basalt allows you to build parameterized packages. This is very handy, if you want to build several packages of one project for different customers, or if you need to build packages with small differences for several environments. E.g. testing, staging and production. To build a package, you need a YAML config[1] and a build file[2]. With this (somewhat older) demo the pkg generation would be invoked with `inv build –config configs/go.dajool.com.yaml`. At the end you would get a Debian package that could, in this example, be deployed standalone, because all configs are in place, dependencies are defined and services are reloaded. Usually I’m running the build in a Docker container[3], that includes all the build dependencies and does not poison the build. Creating relocatable and working virutalenv’s for Python can be a pita, if your dev box has an other libjpeg than the production system.

    So I guess my question would be this: Would it be possible to use BuildStream to accomplish at least the parameterized package generation, so that I can trash basalt? ;)

    Thanks for your answer and sorry for the long comment! :)

    [1] https://github.com/basalt/go-get-a-martini-bottle/blob/master/hello_martini/configs/go.dajool.com.yaml
    [2] https://github.com/basalt/go-get-a-martini-bottle/blob/master/hello_martini/tasks.py
    [3] http://hub.docker.com/u/basaltbuild

  6. Worth replying here too… @brejoc and I had a fruitful discussion on IRC about some very interesting use cases for BuildStream.

    The conclusion was that, with Source implementations which represent distro packages, one would be able to construct pipelines which allow one to:
    a.) Stage a “specific version of debian”
    b.) Build software on that base
    c.) Create / Deploy debian packages targetting that version of debian

    At the same time this could let us do something interesting with GNOME release modulesets, we could for instance run CI of the whole GNOME Desktop and see how it integrates on the latest development/stable versions of Debian, Fedora, FooDistro etc.

    The result is that we’re now looking into implementing a dpkg Source implementation, a full brain dump on that work can be found here: https://gitlab.com/BuildStream/buildstream/issues/10

Comments are closed.

Leave a Reply

Your email address will not be published. Required fields are marked *