Category: General

Earlier this week I released Flatpak 0.8.0. The version change is meant to signal the start of a new long-term supported branch. The 0.8.x series will be strictly bugfixes, and all new
features will happen in 0.9.x.

The release has a few changes, such as a streamlined command line interface and OCI support, but it also has several additions that make Flatpak more future-proof. For instance, we added versioning to all file formats, and a minimal-flatpak-version-required field for applications.

My goal is to get the 0.8 series into the Debian 9 release, and as many other distributions as possible, so that people who create flatpaks can consider the features it supports as a reliable baseline.

Sandboxing has always been one of the pillars of Flatpak, but even more important to me is cross-distro application distribution, even if not sandboxed. This is important because it gives upstream developers a way to directly interact with their users, without having an intermediate distributor. With 0.8 I think we have reached a level where the support for this is solid. So, if you ever thought about experimenting with Flatpak, now is the time!

I leave you with a small screencast showing the new streamlined way to install an application om the command line (on an otherwise empty system):

For information on how to get flatpak, see flatpak.org. Version 0.8.0 is already in the Ubuntu PPA and in  Fedora. Other distributions hopefully will get it soon.

One of the features in the recent flatpak relase is described as:

Applications can now list a set of URIs that will be downloaded with the application

This seems a bit weird. Downloading an application already means your loading data from a URI. What is the usecase for this?

Normally all the data that is needed for your application will be bundled with it and this feature is not needed. However, in some cases applications are freely downloadable, but not redistributable. This only affects non-free software, but Flatpak wants to support both free and non-free software for pragmatic reasons.

Common examples of this are Spotify and Skype. I hope that they eventually will be available as native flatpaks, but in order to bootstrap the flatpak ecosystem we want to make it possible to distribute these right now.

So, how does this work? Lets take Spotify as an example. It is available as a binary debian package. I’ve created a wrapper application for it which contains all the dependencies it needs. It also specifies the URL of the debian package and its sha256 checksum, plus a script to unpack it.

When the user downloads the application it also downloads the deb, and then (in a sandbox) it runs the unpack script which extracts it and puts the files in the right place. Only then is the installation considered done, and from thereon it is used read-only.

I put up a build of the spotify app on S3, so to install spotify all you need to do is:

flatpak install --from https://s3.amazonaws.com/alexlarsson/spotify-repo/spotify.flatpakref

(Note: This requires flatpak 0.6.13 and the remote with the freedesktop runtime configured)

Here is an example of installing spotify from scratch:

Today I released version 0.6.13 of flatpak which has a lot of nice new features. One that I’d like to talk a bit about is the new command line argument format.

The flatpak command line was always a bit low-level and hard to use. Partly this was because of lack of focus on this, and partly due to the fact that the expected UI for flatpak for most people would be a graphical user interface like gnome-software. However, with this new release this changed, and flatpak is now much nicer to use from the commandline.

So, what is new?

Flatpakrepo files

Before you can really use flatpak you have to configure it so it can find the applications and runtimes. This means setting up one or more remotes. Historically you did this by manually specifying all the options for the remote as arguments to the flatpak remote-add command. To make this easier we added a file format (.flatpakrepo) to describe a remote, and made it easy to use it.

The new canonical example to configure the gnome stable repositories is:

$ flatpak remote-add --from gnome \
   https://sdk.gnome.org/gnome.flatpakrepo
$ flatpak remote-add --from gnome-apps \
   https://sdk.gnome.org/gnome-apps.flatpakrepo

Alternatively you can just click on the above links and they should open in gnome-software (if you have new enough versions installed).

Multiple arguments to install/update/uninstall

Another weakness of the command line has been that commands like install, uninstall and update only accepted one application name argument (and optionally a branch name). This made it hard to install multiple apps in one go, and the separate branch name made it hard to cut-and-paste from output of e.g. flatpak list.

Instead of the separate branch name all the commands now take multiple “partial refs” as arguments. These are partial versions of the OSTree ref format that flatpak uses internally. So, for an internal reference like app/org.gnome.gedit/x86_64/stable, one can now specify one of these:

org.gnome.gedit
org.gnome.gedit//stable
org.gnome.gedit/x86_64
org.gnome.gedit/x86_64/stable

And flatpak will automatically fill in the missing part in a natural way, and give a detailed error if you need to specify more details:

$ flatpak install gnome org.gnome.Platform
error: Multiple branches available for org.gnome.Platform, you must specify one of: org.gnome.Platform//3.20, org.gnome.Platform//3.22, org.gnome.Platform//3.16, org.gnome.Platform//3.18

Automatic dependencies

The other problem with the CLI has been that it is not aware of runtime dependencies. To run an app you generally had to know what runtime it used and install that yourself. The idea here was that the commandline should be simple, non-interactive and safe. If you instead use the graphical frontend it will install dependencies interactively so you can control where things get installed from.

However, this just made the CLI a pain to use, and you could easily end up in situations where things didn’t work. For instance, if you updated gedit from 3.20 to 3.22 it suddenly depended on a new runtime version, and if you weren’t aware of this it probably just stopped working.

Of course, we still can’t just install dependencies from wherever we find them, because that would be a security issue (any configured remote can supply a runtime for any applications). So, the solution here is for flatpak to become interactive:

$ flatpak update org.gnome.gedit
Looking for updates...
Required runtime for org.gnome.gedit/x86_64/stable (org.gnome.Platform/x86_64/3.22) is not installed, searching...
Found in remote gnome, do you want to install it? [y/n]: y
Installing: org.gnome.Platform/x86_64/3.22 from gnome
Installing: org.gnome.Platform.Locale/x86_64/3.22 from gnome
Updating: org.gnome.gedit/x86_64/stable from gnome-apps
Updating: org.gnome.gedit.Locale/x86_64/stable from gnome-apps

If you have remotes you never want to install dependencies from, you can install them with --no-use-for-deps, and they will not be used. Flatpakrepo files for app-only repositories should set NoDeps=true.

Note that this is not a package-system-like dependency solver that can solve sudoku. It is still a very simple two-way split.

Flatpakref files

The primary way Flatpak is meant to be used is that you configure a few remotes that has most of the applications that you use, then you install from these either on the command line, or via a graphical installer. However, sometimes it is nice to have a single link you can put on a website to install your application. Flatpak now supports that via .flatpakref files. These are very similar to flatpakrepo files, in that they describe a repository, but they additionally contain a particular application in that repository which will be installed.

Such files can be installed by just clicking on them in your web-browser (which will open them for installation in gnome-software) or on the command line:

flatpak install --from https://sdk.gnome.org/gedit.flatpakref

This will try to install the required runtime, so you first need to add the remote with the runtimes.

Launching runtimes

During development and testing it is often common to launch commands in a sandbox to experiment with how the runtime works. This was always possible by running a custom command in an application that used the runtime, like so:

$ flatpak run --command=sh org.gnome.gedit
sh-4.3$ ls /app
bin lib libexec manifest.json share
sh-4.3$

You can even specify a custom runtime with --runtime. However, there really should be no need to have an application installed to do this, so the new version allows you to directly run a runtime:

$ flatpak run org.gnome.Platform//3.22
sh-4.3$ ls /app
sh-4.3$

At the core of xdg-app is a small helper binary that uses Linux features like namespaces to set up sandbox for the application. The main difference between this helper and a full-blown container system is that it runs entirely as the user. It does not require root privileges, and can never allow you to get access to things you would not otherwise have.

This is obviously very useful for desktop application sandboxing, but has all sort of other uses. For instance, you can sandbox your builds to avoid them being polluted from the host, or you can run a nonprivileged service with even less privileges.

The current helper was a bit too tied to xdg-app, so as part of Project Atomic we decided to create a separate project based on this code, but more generic and minimal. Thus Bubblewrap was born.

Bubblewrap is a wrapper tool, similar to sudo or chroot. You pass it an executable and its argument on the command line. However, the executable is run in a custom namespace, which starts out completely empty, with just a tmpfs mounted as the root, and nothing else. You can then use commandline arguments to build up the sandbox.

For example, a very simple use of bubblewrap to run a binary with access to everything, but readonly:

$ bwrap --ro-bind / / touch ~/foo
touch: cannot touch ‘/home/alex/foo’: Read-only file system

Or you can use bubblewrap as an regular chroot that doesn’t require you to be root to use it:

$ bwrap --bind /some/chroot / /bin/sh

Here is a more complicated example with a read-only host /usr (and no other host files), a private pid namespace, and no network access:

$ bwrap --ro-bind /usr /usr \
      --dir /tmp \
      --proc /proc \
      --dev /dev \
      --ro-bind /etc/resolv.conf /etc/resolv.conf \
      --symlink usr/lib /lib \
      --symlink usr/lib64 /lib64 \
      --symlink usr/bin /bin \
      --chdir / \
      --unshare-pid \
      --unshare-net \
      /bin/sh

See the bubblewrap manpage for more details.

Today I changed xdg-app to use bubblewrap instead of its own helper, which means if you start an app with xdg-app it is using bubblewrap underneath.

For now it is using its own copy of the code (using git submodules), but as bubblewrap starts to get deployed more widely we can start using the system installed version of it.

In sync with the new stable runtime release, there is a new xdg-app release, version 0.5.2.

This has some major changes in how we build language/locale extension, which makes it important to update if you’re building an application or runtime.

Of course, there are some new features, fixes and updates, so regular users should update to!

Finally there are now stable builds of the Gnome 3.20 runtime. I updated the Runtime wiki page  with details on how to install it.

This has been a long time coming, because this new runtime as well as the freedesktop.org runtime that it is based is built in a new way. It used to be built with a custom script, but now it uses the new xdg-app-builder tool. This makes it a lot cleaner and easier to maintain.

Also worth mentioning is the base freedesktop.org 1.4 runtime. This has been updated, modernized and cleaned up to make a great base both for applications that don’t need a lot of runtime support (such as games) and as a base for other runtimes. In fact, Alexei is using this as the base for the KDE runtimes too.

I have also set up a gnome-apps repository with stable builds of some Gnome 3.20 applications. This lets you test Gnome 3.20 applications on older distributions.

Documentation on how to install these can be found on Runtime wiki page , but as a teaser i give you the commands to test gedit if you have zero xdg-app setup:

$ wget https://people.gnome.org/~alexl/keys/gnome-sdk.gpg
$ xdg-app remote-add --user --gpg-import=gnome-sdk.gpg gnome http://sdk.gnome.org/repo/
$ xdg-app --user install gnome org.gnome.Platform 3.20
$ xdg-app --user remote-add --gpg-key=gnome-sdk.gpg gnome-apps http://sdk.gnome.org/repo-apps/
$ xdg-app --user install gnome-apps org.gnome.gedit stable
$ xdg-app run org.gnome.gedit

These runtimes will be a great base if you want to experiment with building xdg-apps, and going forward we will start looking at integrating xdg-app better into the Gnome development and release schedule.

Also, Endless Mobile are sponsoring Gnome with ARM hardware and people working on that, so I hope we can soon have official Gnome runtimes for ARM too.

In the previous parts (1,2,3,4) we created various applications and tested them locally. Now we will look into how this application can be distributed to your users.

First we need to talk a bit about OSTree. This is the core distribution and installation mechanism of xdg-app. OSTree is similar to git, but has been designed to handle trees of large binaries. Just like git it has the concept of repositories, commits and branches (refs) . Branches are names which points to a commit id, and an application stored in a repository is such a branch.

On your machine there is a local OSTree repository which has a bunch of “remotes” configured. When you install or update an application from one on these remotes, what happens is that we ostree pull the branch from the remote, then do a local checkout of the branch (with hard links back to the repository files, so this is cheap).

The OSTree repositories created by xdg-app build-export are of a type (called archive-z2) that are meant to be served by a dumb webserver. So, to distribute your application, all you have to do is copy the repository to a webserver and give the url to your users.

There are however some extra details you have to be aware off .

First of all, the format of the archive-z2 repositories is one file per file in the app. This means a pull operation will do a lot of http requests. Since new requests are slow, make sure you enable support for HTTP keep-alive in your webserver.

OSTree also supports something called static deltas. These are single files in the repo that contains all the data needed to go between two revisions (or from nothing to a revision). Creating such deltas will take a bit more space, but will make downloads much faster.

xdg-app has the build-update-repo command that is very useful to manage repositories. Starting at version 0.4.13 it supports --generate-static-deltas to generate the deltas. but it also has several other features that are useful. For instance, you can use:

xdg-app build-update-repo --title="Nice name" repo

To set a user readable name for the repository, which will be used by default in the UI when users see this repository.

It also lets you prune (--prune) unused objects and deltas from the repository, and even remove older revisions from your repository (--prune-depth) which is useful for things like automatic nightly build repositories.

Another thing build-update-repo does is appstream extraction. It scans all the branches in the repository and looks for an AppStream xml file. For instance, the dictionary app from part 3 contains such a file in files/share/app-info/xmls/org.gnome.Dictionary.xml.gz. All these files plus the icons they reference are collected and commited into a repo-wide appstream branch. xdg-app keeps a local copy of this branch for each remote, which you can manually update with:

xdg-app --user update --appstream nightly

This is used by graphical installation tools such as gnome-software, so having appstream data for your applications is important.

Note: xdg-app-builder automatically runs the appstream-compose command after the build. This will collect information from appdata files and desktop files and create the right xml.gz and icon files. This is where the xml in gnome-dictionary came from.

Another important part about app distribution is gpg signatures. By default OSTree refuses to pull anything from a remote repository that is not signed. This is only allowed if you use --no-gpg-verify when you add the remote (or you can change it with xdg-app remote-modify).

In OSTree, signatures are on each commit and on the summary file that lists all the branches in the repositories. These objects are created by the build-update-repo, and build-export commands, as well as indirectly by xdg-app-builder. So, when you call these, you should pass in the gpg keys to sign with, and optionally the gpg home directory to use. For example:

xdg-app build-export --gpg-sign=KEYID --gpg-homedir=/some/dir appdir repo

That concludes this tutorial series, and you should now know enough to package and distribute your own applications.

In part 1 we created a very small application. All it did was print to stdout. Such a program is very easy to sandbox. In fact, since we didn’t specify any permissions for it this application already runs in a very tight sandbox.

Here is what the sandbox does:

• No access to any host files except the runtime, the app, and ~/.var/app/org.test.Hello,  and only the last of these  is writable.
• /tmp is unique to the application instance.
• setuid functionallity is disabled.
• Processes are in a cgroup which they cannot get out of. This lets you know what application a process is running in, in an unfakable way. (This will be very important later for per-application preferences.)
• No access to the network.
• No access to any device nodes (apart from /dev/null, etc).
• Can’t see any processes outside the sandbox.
• A set of seccomp rules are loaded that limits what syscalls the app can do.  For instance, it can’t use nonstandard network socket types, or ptrace other processes.
• Limited access to the session dbus instance, is only allowed  to own its own name on the bus and can’t talk to anyone else.
• No access to host services like X, system dbus, or pulseaudio.

Of course, its very easy to secure a Hello world program, as it doesn’t need to do anything. There are designs (and experimental code) to allow applications to securely break out of the sandbox (using something called Portals). However, that is not stable for general use at this point. What is usable now though is punching minimal holes in the sandbox such that the applications can do what it needs.

We saw this already in part 2, where we used this command, granting X11 and network access:

xdg-app build-finish appdir2 --socket=x11 --share=network --command=gnome-dictionary

These arguments translate to these properties in the application metadata file:

[Application]
name=org.gnome.Dictionary
runtime=org.gnome.Platform/x86_64/3.20
sdk=org.gnome.Sdk/x86_64/3.20
command=gnome-dictionary

[Context]
shared=network;
sockets=x11;

As we gave no access to the filesystem in the above the resulting app can’t see your files. We can verify this with a command like:

$ xdg-app run --command=ls org.gnome.Dictionary ~/
<nothing>

The permissions in the metadata file are only the default values though. You can override these in different ways. First of all, xdg-app run takes the same command line arguments as build-finish, which can override permissions for that instance. For example, this will let it see your home directory:

$ xdg-app run --filesystem=home --command=ls org.gnome.Dictionary ~/
<homedir listing...>

You can also permanently override permissions for an application:

$ xdg-app --user override --filesystem=home org.gnome.Dictionary
$ xdg-app run --command=ls org.gnome.Dictionary ~/
<homedir listing...>

This can then again be overridden in a specific instance:

$ xdg-app run --nofilesystem=home --command=ls org.gnome.Dictionary ~/
<nothing>

There are quite a few options that control sandbox permissions. So, I’m going to list some useful examples of what can be done. (Note that all options have a negative version too, i.e. –unshare undoes –share.)

Grant access to some of your files:

  --filesystem=host    # All files
  --filesystem=home    # Your homedirectory
  --filesystem=home:ro # Your homedirectory, read-only
  --filesystem=/some/dir --filesystem=~/other/dir # paths
  --filesystem=xdg-download # The XDG Download directory
  --nofilesystem=...   # Undo some of the above

Allow the application to show windows using X11:

--socket=x11 --share=ipc

Note: –share=ipc means that the sandbox shares ipc namespace with the host. This is not necessarily required, but without it the X shared memory extension will not work, which is very bad for X performance.

Allow OpenGL rendering:

 --device=dri

Allow the application to show windows using wayland:

--socket=wayland

Let the application play sounds using pulseaudio:

--socket=pulseaudio

Let the application access the network:

 --share=network

Note: Giving network access also grants access to all host services listening on abstract unix sockets (due to how network namespaces work), and these have no permission checks. This unfortunately affects e.g. the X server and the session bus which listens to abstract sockets by default. A secure distribution should disable these and just use regular sockets.

Let the application talk to a named service on the session bus:

 --talk-name=org.freedesktop.secrets

Let the application talk to a named service on the system bus:

 --system-talk-name=org.freedesktop.GeoClue2

Give the application unlimited access to all of dbus:

 --socket=system-bus --socket=session-bus

That was it for today. In the next part we’ll take a look at how to distribute your application.