Flatpak – a history

I’ve been working on Flatpak for almost 4 years now, and 1.0 is getting closer. I think it might be interesting at this point to take a retrospective look at the history of Flatpak.

Early history

Ancient Egyptian Flatpak

The earliest history goes back to the summer of 2007. I had played a bit with a application image system called Klik, which had some interesting ideas. However, I was not really satisfied with some technical details. One day at the beach I got an interesting ideas for a hack that could improve this.

Fast forward until August 2007 when I released Glick in the wild, based on these ideas. The name is sort of a pun on the old KDE/Gnome first-letter naming scheme, although neither Klik or Glick are really desktop-specific.

Glick was a a single-file-image system. It predates usable kernel container APIs, so it uses fuse and some weird hacks. It doesn’t integrate with the desktop in any way, and applications have to decide what to bundle, falling back to system-libraries for the non-bundled things. This means its not terribly robust., but it is completely stand-alone and need nothing installed on the host system.

Around 2011 the initial support for kernel namespaces had landed and started being useful. Using these I could avoid some of the hacks that my earlier experiment used. So, I got interested in bundling again and released Glick 2 based on this.

Glick 2 requires some software to be installed on the host, which allows it to integrate better with the system. For example, you can “install” bundles by putting the file in a known location, and doing this allows some level of desktop integration. Glick 2 also uses SHA1 checksums to try to automatically de-duplicate files shared between applicatins. Here we can see an early version of the ideas that make up OSTree.

Bundling using namespaces was lot more robust than the previous hacks, but it still relied on the system for the core libraries that the application doesn’t bundle. So an app would sometimes work on one distro, but not another.

Around this time I posted a blog  about how I thought application bundling combined with read-only OS images can make a really good model for an OS. This idea is very similar to what Project Atomic / SilverBlue  are doing now.

Containers, Portals and Runtimes

A few years later, around 2013 the kernel support for containers was starting to shape up, and Docker hit the market. I did a lot of work on the early docker, like porting it away from aufs in order to run on RHEL.

Around this time I also attended the Gnome Developer Experience hackfest  in Brussels where one of the topics was Application deployment and sandboxing. From the discussions there (and my previous experiences) a lot of the core ideas of Flatpak, like runtimes, sandboxing and portals originated.

In 2014 the first version (then called xdg-app) was released. The current Flatpak is a lot more polished, but the initial version of xdg-app is still very much recognizable today.

xdg-app used OSTree to download, store and de-duplicate applications. It uses kernel namespaces (via a helper called xdg-app-helper) to do unprivileged containers. It has a split between applications and runtimes which allow applications to be portable between distros in a very robust fashion, while still limiting the duplication between applications and allowing security updates. There is also integration with the desktop (icons, desktop files, mimetypes, etc), and some very early portal code can be seen.

The great renaming

Modern Flatpak

The name xdg-app was just something I picked for the first commit without much consideration, and it was not very good. However, names are hard, and we spent a lot of time trying to come up with another, eventually settling on “Flatpak” (with the above logo). The 0.6.0 release in may 2016 was the first with the new name.

The 0.6 release was also the first that split out the unprivileged container launcher (xdg-app-helper) into its own project, now known as BubbleWrap , hosted by Project Atomic.

Soon thereafter we had the first release of xdg-desktop-portal which is the host-side implementation of the portal idea, allowing sandboxed applications to safely break out of the sandbox in a controlled fashion.

Version 0.8.0, released december 2016 was the first long-term stable release, which was included in Debian Stretch and RHEL 7. Since then we have had another stable release series called 0.10.x.

We want apps!

Flatpak was always a decentralized system, in that anyone can host their own applications and be on equal terms with everyone else. However, while this is an important feature, it leads to a poor initial experience, both for users (hard to find apps) or developers (need to maintain their own repository).

To solve this we started the Flathub project, which is a single repository where you can find most apps. In the last year it has gone from a minimal viable product building its first app to something with more than 300 apps and a diverse group of developers.

Onwards and upwards!

Future Flatpak

No software is ever finished, or bug-free, but we have had a list of core things that we wanted to have before calling Flatpak 1.0, and that list is now empty. So, I’m planning to release a release-candidate (called 0.99.1) later this week.

Then 1.0 will be released later this summer.

Introducing the 1.8 freedesktop runtime in the gnome nightly builds

All the current Flatpak runtimes in wide use are based on the 1.6 Freedesktop runtime. This is a two-layered beast where the lower layer is built using Yocto and the upper layer is built using flatpak-builder.

Yocto let us get a basic system bootstrapped, but it is not really a great match for the needs of a Flatpak runtime. We’ve long wanted a base that targeted sandboxed builds which is closer to upstream and without the embedded cross-compiled legacy of Yocto. This is part of the reason why Tristan started working on BuildStream, and why the 1.8 Freedesktop runtime was created from scratch, using it.

After a herculean effort from the people at Codethink (who sponsor this effort) we now have a working Yocto-free version of the Platform and Sdk. You can download the unstable version from here and start playing with it. It is not yet frozen or API/ABI stable, but its getting there.

The next step in this effort is to test it as widely as possible to catch any issues before the release is frozen. In order to do this I rebased the Gnome nightly runtime builds on top of the new Freedesktop version this week. This is a good match for a test release, because it has no real ABI requirements (being rebuilt with the apps daily), yet gets a fair amount of testing.

WARNING: During the initial phase it is likely that there will be problems. Please test your apps extra carefully and report all the issues you find.

In the future, the goal is to also convert the Gnome runtimes to BuildStream. Work on this has started, but for now we want to focus on getting the base runtime stable.

YAMLing the flathub

The most common way to build flatpak is using a tool called flatpak-builder. This is a tool that takes a higher level description of the sources that go into an application and generate the build commands to build it. This description is called a manifest, and is traditionally a JSON file.

JSON is very common in the web world, and it is a well known format that have many implementations. However, it is not really great for humans to write.

For example, JSON has no support for comments, which you typically want in something as complicated as a build script. There are ways to work around this,  but it is not pretty.

The syntax, while easy for a machine to parse, is very verbose and noisy to read and picky to write. For example, all lists have to be comma separated, but you can’t have a trailing comma, which you often run into when deleting or moving items around. Also, both keys and values always have to be quoted, which feels pretty unnecessary for the simple one-word keys and values that are common case in manifest files.

This is what a JSON manifest look like:

{
  "id": "org.gnome.frogr",
  "runtime": "org.gnome.Platform",
  "sdk": "org.gnome.Sdk",
  "runtime-version": "3.26",
  "command": "frogr",
  "finish-args": [
    "--share=ipc", "--socket=x11",
    "--socket=wayland",
    "--share=network",
    "--filesystem=xdg-pictures"
  ],
  "build-options" : {
    "cflags": "-O2 -g",
    "env": {
      "V": "1"
    }
  },
  "cleanup": [ "/share/man" ],
  "modules": [
    {
      "name": "frogr",
      "buildsystem": "meson",
      "sources": [
        {
          "type": "git",
          "url": "git://git.gnome.org/frogr",
          "branch": "RELEASE_1.4",
          "commit": "e2322c8f99f9d3a3cdc020b79c6c7224ad1988d0"
        }
      ]
    }
  ]
}

Starting in flatpak-builder 0.10.10 you can now also use YAML for manifests. In term of the content model, YAML is a superset of JSON, but it is easier for humans to read and write. This makes it a good fit for flatpak-builder. In fact, the implementation internally just converts the YAML parser nodes to JSON parser nodes.

Yesterday I enabled support in flathub, so you can now convert existing apps, or submit new apps using YAML.

Here is how the above manifest looks in YAML:

id: org.gnome.frogr
runtime: org.gnome.Platform
sdk: org.gnome.Sdk
runtime-version: 3.26
command: frogr
finish-args:
  - --share=ipc
  - --socket=x11
  - --socket=wayland
  - --share=network
  # Grant app access to ~/Pictures
  - --filesystem=xdg-pictures
build-options:
  cflags: -O2 -g
  env: { V: '1' }
cleanup: [ /share/man ]
modules:
  - name: frogr
    buildsystem: meson
    sources:
      - type: git
        url: git://git.gnome.org/frogr
        branch: RELEASE_1.4
        commit: e2322c8f99f9d3a3cdc020b79c6c7224ad1988d0

This is clearly the same content as above (except it has a comment), but shorter and more readable.

For a longer example, see this conversion of a flathub app.

Of course, YAML is not perfect. Its a pretty complex format, it relies on indentation, and the specification has historically been a bit vague with implementations sometimes differing in interpretation. Going forward both JSON and YAML will be supported (and can even be mixed when you use includes), allowing you to chose what works best for you.