Nvidia drivers in Fedora Silverblue

I really like how Fedora Silverblue combines the best of atomic, image-based updates and local tweaking with its package layering idea.

However, one major issue many people has had with it is support for the NVIDIA drivers. Given they ares not free software they can’t be shipped with the image, so one imagines using package layering to would be a good way to install it. In theory this works, but unfortunately it often runs into issues, because frequent kernel updates cause there to be no pre-built nvidia module for your particular kernel/driver version.

In a normal Fedora installation this is handled by something called akmods. This is a system where the kernel modules ship as sources which get automatically rebuilt on the target system itself when a new kernel is installed.

Unfortunately this doesn’t quite work on Silverblue, because the system image is immutable. So, I’ve been working recently on making akmods work in silverblue. The approach I’ve taken is having the modules being built during the rpm-ostree update command (in the %post script) and the output of that being integrated into the newly constructed image.

Last week the final work landed in the akmods and kmodtools packages (currently available in updates-testing), which means that anyone can easily experiment with akmods, including the nvidia drivers.

Preparing the system

First we need the latest of everything:

$ sudo rpm-ostree update

The required akmods packages are in updates-testing at the moment, so we’ll enable that for now:

$ sudo vi /etc/yum.repos.d/fedora-updates-testing.repo
... Change enabled to 1 ..

Then we add the rpmfusion repository:

$ sudo rpm-ostree install https://download1.rpmfusion.org/free/fedora/rpmfusion-free-release-29.noarch.rpm https://download1.rpmfusion.org/nonfree/fedora/rpmfusion-nonfree-release-29.noarch.rpm

At this point you need to reboot into the new ostree image to enable installation from the new repositories.

$ systemctl reboot

Installing the driver

The akmod-nvidia package in the current rpm-fusion is not built against the new kmodtools, so until it is rebuilt it will not work. This is a temporary issue, but  I built a new version we can use until it is fixed.

To install it, and the driver itself we do:

$ sudo rpm-ostree install http://people.redhat.com/alexl/akmod-nvidia-418.43-1.1rebuild.fc29.x86_64.rpm xorg-x11-drv-nvidia

Once the driver in rpm-fusion is rebuilt the custom rpm should not be necessary.

We also need to blacklist the built-in nouveau driver so to avoid driver conflicts:

$ sudo rpm-ostree kargs --append=rd.driver.blacklist=nouveau --append=modprobe.blacklist=nouveau --append=nvidia-drm.modeset=1

Now you’re ready to boot into your fancy new silverblue nvidia experience:

$ systemctl reboot

What about Fedora 30/Rawhide?

All the changes necessary for this to work have landed, but there is no Fedora 30 Silverblue image yet (only a rawhide one), and the rawhide kernel is built with mutex debugging which is not compatible with the nvidia driver.

However, the second we have a Fedora 30 Silverblue image with a non-debug kernel the above should work there too.

Changes in Flathub land

The last month or so we’ve been working in the background on a major update to the Flathub infrastructure. This has been available for testing for a while, but this week we finally enabled it on the live system. There are some pretty cool internal changes, including a new repo manager microservice written in rust. Later blog posts will talk about some of the technical details, but for now I’ll just talk about the user visible changes.

Power to the maintainers!

Flathub uses buildbot to to manage the builds, and we have updated and customized the UI a bit to be nicer for maintainers. For example, we now have a page listing all the apps ever built, with links to per-app pages showing builds of that app.

We also integrated GitHub authentication so that maintainers of individual applications automatically have authority to do operations on their own apps and builds. For example, the home and per-app pages have buttons that let you start builds, which anyone with write permissions to the corresponding GitHub repository can use. Also, similarly they can cancel or retry the builds of their own apps. Previously you had to ask a Flathub administrator to restart or cancel a build, but no more!

New publish workflow

There has also been a major change in the workflow for builds. It used to be the case that a successful build was immediately imported into the repository and was then available to users. Now instead, a successful build is available for installation in a test-repository. The build system will display a link to it so that you can easily install and test the build results. When you’re satisfied that the build is ok you can then manually push a button to export it to the public repository.

If you don’t manually publish the build, then it will be automatically published, by default after 24 hours, but this is configurable by the app maintainer. See the wiki for details.

Testing the test builds

Test builds used to only verify that the app built, but with the new system they get built into test repositories just like regular builds. This means you can actually install and test the builds, for example from a pull request against your application. Such test repos stay around for 5 days, or until you explicitly delete then in the build web UI.

Test builds are also more useful now due to the permission work, as developers can easily create or cancel them from the web ui, or by using the “bot, build” command in a GitHub issue, without needing help from the Flathub admins.

Also, test builds started from a GitHub issue gets nice comments pointing to the test build and the build result. Here is an example of a pull request with automatically built tests showing how this looks.

We now automatically queue test builds for all new PRs, although such builds are less prioritized than regular builds (for resource reasons) and can take a while to start.

Publish beta releases!

In addition to the existing stable repository Flathub added a repository for beta builds. Exactly if and how this is used is up to each individual application maintainer, but the goal of this is to have a way for developers to get early releases of new stable versions into the hand of regular users.

This isn’t meant to be used for nightly builds, but for releases that has some level of testing and are expected to mostly work and be usable to non-developer end-users.

The way this works is that each GitHub repository builds the master branch for the stable repository, which will have the flatpak branch name “stable”, and then the beta git branch will build into the beta repository with the flatpak branch “beta”.

As a user, the beta channel looks like a separate remote. First you configure it as a remote:

$ flatpak remote-add flathub-beta https://flathub.org/beta-repo/flathub-beta.flatpakrepo

And then you can install any apps from it:

$ flatpak install --user flathub-beta org.godotengine.Godot

Alternatively, you can use a flatpakref, which are generated for each app:

$ flatpak install https://flathub.org/beta-repo/appstream/org.godotengine.Godot.flatpakref

The above Godot example is the latest beta of Godot 3.1, whereas the stable repo still contains 3.0. You can see how this beta build is set up in GitHub.

If you install both the beta and the stable version of an app then they will be installed in parallel. However, only one will be showed in the menus. You can switch which one is currently showed like this:

$ flatpak make-current org.godotengine.Godot [beta|stable]

But from the command line you can always start any installed version explicitly, like this:

$ flatpak run --branch=beta org.godotengine.Godot
or
$ flatpak run org.godotengine.Godot//beta

Now, go build some betas!

Moving away from the 1.6 freedesktop runtime

A flatpak runtime contains the basic dependencies that an application needs. It is shared by applications so that application authors don’t have to bother with complicated low-level dependencies, but also so that these dependencies can be shared and get shared updates.

Most flatpaks these days use the freedesktop runtime or one of its derivates (like the Gnome and KDE runtimes). Historically, these have been using the 1.6 version of the freedesktop runtime which is based on Yocto.

The 1.6 runtime has served its place to kickstart flatpak and flathub well, but it is getting quite long in the tooth. We still fix security issues in it now and then, but it is not seeing a lot of maintenance recently. Additionally, not a lot of people know enough yocto to work on it, so we were never able to build a larger community around it.

However, earlier this summer a complete reimplementation, version 18.08, was announced, and starting with version 3.30 the Gnome runtime is now based on it as well, with a KDE version is in the works. This runtime is based on BuildStream, making it much easier to work with, which has resulted in a much larger team working on this runtime. Partly this is due to the awesome fact that Codethink has several people paid to work on this, but there are also lots of community support.

The result is a better supported, easier to maintain runtime with more modern content. What we need to do now is to phase out the old runtime and start using the new one in apps.

So, this is a call to action!

Anyone who maintains a flatpak application, especially on flathub, please try to move to a runtime based on 18.08. And if you have any problems, please report them to the upstream freedesktop-sdk project.

Flatpak on windows

As I teased about last week I recently played around with WSL, which lets you run Linux applications on Windows. This isn’t necessarily very useful, as there isn’t really a lack of native applications on Windows, but it is still interesting from a technical viewpoint.

I created a wip/WSL branch of flatpak that has some workarounds needed for flatpak to work, and wrote some simple docs on how to build and test it.

There are some really big problems with this port. For example, WSL doesn’t support seccomp or network namespaces which removes some of the utility of the sandbox. There is also a bad bug that makes read-only bind-mounts not work for flatpak, which is really unsafe as apps can modify themselves (or the runtime). There were also various other bugs that I reported. Additionally, some apps rely on things on the linux host that don’t exist in the WSL environment (such as pulseaudio, or various dbus services).

Still, its amazing that it works as well as it does. I was able to run various games, gnome and kde apps, and even the linux versions of telegram. Massive kudos to the Microsoft developers who worked on this!

I know you crave more screenshots, so here is one:

Kick-starting the revolution 1.0

Yesterday marked the day when we finally released Flatpak 1.0 (check out the release video!). I want to thank everyone who helped make this a reality, from writing code, to Flathub packaging, to just testing stuff and spreading the word. Large projects like this can’t be done by a single person, its all about the community.

With 1.0 out, I expect the rate of change in Flatpak itself to slow down. Going forward the focus will be more on the infrastructure around it. Things like getting 1.0 into all distributions, making portals work well, ensuring Flathub works smoothly and keeps growing, improving our test-suites and working on the runtimes.

Most of my blog posts are about technical details, but the reason for the existence of Flatpak is not technical. I created flatpak because the Linux application desktop ecosystem is fundamentally broken. As a app developer you have no sane way to distribute the result of your work to users.

Unless you have massive resources, the only realistic way is to wait for distributions to pick up your app. However, there are many problems with this. First of all, not all distros pick up all apps, and even if they do they often wait until the app is well known leading to a chicken-and-egg problem for new apps. And when they finally ship it you have no control of what version is shipped, or when it is updated.

This is a real problem! For instance, maybe some web service it uses changed API. Its a quick fix for your app, but it takes a long time before it propagates to distros. In fact many stable distros never ever update to new versions.

This model leads to a disconnect between the developer and the users. Users file bugs against older versions, about bugs that are already fixed, yet don’t get any fixes. Developers add new features that users can’t use, and get no feedback on them.

With Flatpak, the goal is for the upstream developer to have control of updates. If the developer fixes an important bug, a new stable version is released that users can immediately use. Any bugs filed will be against the latest stable version, so they are not stale, and once the bug report is closed the user will actually get the fix. That means reporting bugs is useful to the user. Similarly, any new feature development will get immediate feedback, and user feedback will be based on the current state of the app.

This kind of virtuous cycle helps improving both speed of development and software quality. My hope is that this in turn will increase the interest in writing native Linux applications and trigger a revolution, leading to the YEAR OF THE LINUX DESKTOP! (ahem)

The birth of a new runtime

Runtimes are a core part of the flatpak design. They are a way to make bundling feasible, while still fully isolating from the host system. Application authors can bundle the libraries specific to the application, but don’t have to care about the lowlevel dependencies that are uninteresting (yet important) for the application.

Many people think of runtimes primarily as a way to avoid duplication (and thus bloat). However, they play two other important roles. First of all they allow an independent stream of updates for core libraries, so even dead apps get fixes. And secondly, they allow the work of the bundling to be shared between all application authors.

There are some runtimes bases on pre-existing distribution packages, such as Fedora and Debian. These are very useful if you want to produce flatpaks of the packages from these distributions. However, most of the “native” Flatpaks these days are based on the Freedesktop 1.6 runtime or one of its derivates (like the Gnome and KDE runtimes).

Unfortunately this runtime is starting to show its age.

The freedesktop runtime is built in two steps. The first is based on Yocto, which is a cross-compilation system maintained by the Linux Foundation. An image is created from this build which is then further extended using flatpak-builder. This was a great way to get something going initially. However, Yocto focuses mainly on cross compilation and embedded which isn’t a great fit, and the weird 2 layer split and the complex yocto build files lead to very few people being able to build or do any work on the runtime. It also didn’t help that the build system was a bunch of crappy scripts that needed a lot of handholding by me.

Fortunately this is now getting much better, because today the new Freedesktop runtime, version 18.08, was released!

This runtime has the same name, and its content is very similar, but it is really a complete re-implementation. It is based on a new build system called BuildStream, which is much nicer and a great fit for flatpak. So, no more Yocto, no more buildbake, no multi-layer builds!

Additionally, it has an entire group of people working on it, including support from Codethink. Its already using gitlab, with automatic builds, CI, etc, etc. There is also a new release model (year.month) with a well-defined support time. Also, all the packages are much newer!

Gnome is also looking at using this as the basics for its releases, its CI system and eventually the Gnome runtime.

The old freedesktop runtime is dead, long live the freedesktop runtime!

Flatpak – a history

I’ve been working on Flatpak for almost 4 years now, and 1.0 is getting closer. I think it might be interesting at this point to take a retrospective look at the history of Flatpak.

Early history

Ancient Egyptian Flatpak

The earliest history goes back to the summer of 2007. I had played a bit with a application image system called Klik, which had some interesting ideas. However, I was not really satisfied with some technical details. One day at the beach I got an interesting ideas for a hack that could improve this.

Fast forward until August 2007 when I released Glick in the wild, based on these ideas. The name is sort of a pun on the old KDE/Gnome first-letter naming scheme, although neither Klik or Glick are really desktop-specific.

Glick was a a single-file-image system. It predates usable kernel container APIs, so it uses fuse and some weird hacks. It doesn’t integrate with the desktop in any way, and applications have to decide what to bundle, falling back to system-libraries for the non-bundled things. This means its not terribly robust., but it is completely stand-alone and need nothing installed on the host system.

Around 2011 the initial support for kernel namespaces had landed and started being useful. Using these I could avoid some of the hacks that my earlier experiment used. So, I got interested in bundling again and released Glick 2 based on this.

Glick 2 requires some software to be installed on the host, which allows it to integrate better with the system. For example, you can “install” bundles by putting the file in a known location, and doing this allows some level of desktop integration. Glick 2 also uses SHA1 checksums to try to automatically de-duplicate files shared between applicatins. Here we can see an early version of the ideas that make up OSTree.

Bundling using namespaces was lot more robust than the previous hacks, but it still relied on the system for the core libraries that the application doesn’t bundle. So an app would sometimes work on one distro, but not another.

Around this time I posted a blog  about how I thought application bundling combined with read-only OS images can make a really good model for an OS. This idea is very similar to what Project Atomic / SilverBlue  are doing now.

Containers, Portals and Runtimes

A few years later, around 2013 the kernel support for containers was starting to shape up, and Docker hit the market. I did a lot of work on the early docker, like porting it away from aufs in order to run on RHEL.

Around this time I also attended the Gnome Developer Experience hackfest  in Brussels where one of the topics was Application deployment and sandboxing. From the discussions there (and my previous experiences) a lot of the core ideas of Flatpak, like runtimes, sandboxing and portals originated.

In 2014 the first version (then called xdg-app) was released. The current Flatpak is a lot more polished, but the initial version of xdg-app is still very much recognizable today.

xdg-app used OSTree to download, store and de-duplicate applications. It uses kernel namespaces (via a helper called xdg-app-helper) to do unprivileged containers. It has a split between applications and runtimes which allow applications to be portable between distros in a very robust fashion, while still limiting the duplication between applications and allowing security updates. There is also integration with the desktop (icons, desktop files, mimetypes, etc), and some very early portal code can be seen.

The great renaming

Modern Flatpak

The name xdg-app was just something I picked for the first commit without much consideration, and it was not very good. However, names are hard, and we spent a lot of time trying to come up with another, eventually settling on “Flatpak” (with the above logo). The 0.6.0 release in may 2016 was the first with the new name.

The 0.6 release was also the first that split out the unprivileged container launcher (xdg-app-helper) into its own project, now known as BubbleWrap , hosted by Project Atomic.

Soon thereafter we had the first release of xdg-desktop-portal which is the host-side implementation of the portal idea, allowing sandboxed applications to safely break out of the sandbox in a controlled fashion.

Version 0.8.0, released december 2016 was the first long-term stable release, which was included in Debian Stretch and RHEL 7. Since then we have had another stable release series called 0.10.x.

We want apps!

Flatpak was always a decentralized system, in that anyone can host their own applications and be on equal terms with everyone else. However, while this is an important feature, it leads to a poor initial experience, both for users (hard to find apps) or developers (need to maintain their own repository).

To solve this we started the Flathub project, which is a single repository where you can find most apps. In the last year it has gone from a minimal viable product building its first app to something with more than 300 apps and a diverse group of developers.

Onwards and upwards!

Future Flatpak

No software is ever finished, or bug-free, but we have had a list of core things that we wanted to have before calling Flatpak 1.0, and that list is now empty. So, I’m planning to release a release-candidate (called 0.99.1) later this week.

Then 1.0 will be released later this summer.

Introducing the 1.8 freedesktop runtime in the gnome nightly builds

All the current Flatpak runtimes in wide use are based on the 1.6 Freedesktop runtime. This is a two-layered beast where the lower layer is built using Yocto and the upper layer is built using flatpak-builder.

Yocto let us get a basic system bootstrapped, but it is not really a great match for the needs of a Flatpak runtime. We’ve long wanted a base that targeted sandboxed builds which is closer to upstream and without the embedded cross-compiled legacy of Yocto. This is part of the reason why Tristan started working on BuildStream, and why the 1.8 Freedesktop runtime was created from scratch, using it.

After a herculean effort from the people at Codethink (who sponsor this effort) we now have a working Yocto-free version of the Platform and Sdk. You can download the unstable version from here and start playing with it. It is not yet frozen or API/ABI stable, but its getting there.

The next step in this effort is to test it as widely as possible to catch any issues before the release is frozen. In order to do this I rebased the Gnome nightly runtime builds on top of the new Freedesktop version this week. This is a good match for a test release, because it has no real ABI requirements (being rebuilt with the apps daily), yet gets a fair amount of testing.

WARNING: During the initial phase it is likely that there will be problems. Please test your apps extra carefully and report all the issues you find.

In the future, the goal is to also convert the Gnome runtimes to BuildStream. Work on this has started, but for now we want to focus on getting the base runtime stable.

YAMLing the flathub

The most common way to build flatpak is using a tool called flatpak-builder. This is a tool that takes a higher level description of the sources that go into an application and generate the build commands to build it. This description is called a manifest, and is traditionally a JSON file.

JSON is very common in the web world, and it is a well known format that have many implementations. However, it is not really great for humans to write.

For example, JSON has no support for comments, which you typically want in something as complicated as a build script. There are ways to work around this,  but it is not pretty.

The syntax, while easy for a machine to parse, is very verbose and noisy to read and picky to write. For example, all lists have to be comma separated, but you can’t have a trailing comma, which you often run into when deleting or moving items around. Also, both keys and values always have to be quoted, which feels pretty unnecessary for the simple one-word keys and values that are common case in manifest files.

This is what a JSON manifest look like:

{
  "id": "org.gnome.frogr",
  "runtime": "org.gnome.Platform",
  "sdk": "org.gnome.Sdk",
  "runtime-version": "3.26",
  "command": "frogr",
  "finish-args": [
    "--share=ipc", "--socket=x11",
    "--socket=wayland",
    "--share=network",
    "--filesystem=xdg-pictures"
  ],
  "build-options" : {
    "cflags": "-O2 -g",
    "env": {
      "V": "1"
    }
  },
  "cleanup": [ "/share/man" ],
  "modules": [
    {
      "name": "frogr",
      "buildsystem": "meson",
      "sources": [
        {
          "type": "git",
          "url": "git://git.gnome.org/frogr",
          "branch": "RELEASE_1.4",
          "commit": "e2322c8f99f9d3a3cdc020b79c6c7224ad1988d0"
        }
      ]
    }
  ]
}

Starting in flatpak-builder 0.10.10 you can now also use YAML for manifests. In term of the content model, YAML is a superset of JSON, but it is easier for humans to read and write. This makes it a good fit for flatpak-builder. In fact, the implementation internally just converts the YAML parser nodes to JSON parser nodes.

Yesterday I enabled support in flathub, so you can now convert existing apps, or submit new apps using YAML.

Here is how the above manifest looks in YAML:

id: org.gnome.frogr
runtime: org.gnome.Platform
sdk: org.gnome.Sdk
runtime-version: 3.26
command: frogr
finish-args:
  - --share=ipc
  - --socket=x11
  - --socket=wayland
  - --share=network
  # Grant app access to ~/Pictures
  - --filesystem=xdg-pictures
build-options:
  cflags: -O2 -g
  env: { V: '1' }
cleanup: [ /share/man ]
modules:
  - name: frogr
    buildsystem: meson
    sources:
      - type: git
        url: git://git.gnome.org/frogr
        branch: RELEASE_1.4
        commit: e2322c8f99f9d3a3cdc020b79c6c7224ad1988d0

This is clearly the same content as above (except it has a comment), but shorter and more readable.

For a longer example, see this conversion of a flathub app.

Of course, YAML is not perfect. Its a pretty complex format, it relies on indentation, and the specification has historically been a bit vague with implementations sometimes differing in interpretation. Going forward both JSON and YAML will be supported (and can even be mixed when you use includes), allowing you to chose what works best for you.

Flatpak inception

One interesting usecase of flatpak is as a compliment to the ideas of Fedora Atomic Workstation and similar projects. In other words, a read-only core image for the base operating system, and then using various types of containers and sandboxes for the applications on top of that.

One problem in such a setup is doing development, in that the basic core rarely contains development tools. This is helped a bit by flatpak using runtimes and SDKs, because the compiler used during the build is not from the host. However, flatpaks are typically built using flatpak-builder, which still has some dependencies on the host, such as git/bzr/svn and strip. These pull in a lot of packages that you don’t want on a minimal core OS image.

So, how do we solve this?

We must go deeper!

Introducing org.flatpak.Builder, a flatpak:ed version of flatpak-builder, which only needs flatpak to be installed on the system, thus making it possible to build apps on systems with zero development tools:

$ flatpak install flathub org.flatpak.Builder
...
$ flatpak run org.flatpak.Builder build-dir org.flatpak.Builder.json 
...

Additionally, with recent versions of flatpak, if you add /var/lib/flatpak/exports/bin to PATH you can use the shorter:

$  org.flatpak.Builder build-dir org.flatpak.Builder.json

The future is one step closer!