Gnome apps migrated to flathub

Last week I finally migrated the last app from the gnome stable application flatpak repo to flathub. The old repo is now deprecated and will not get any new builds.

In the future, all stable flatpak builds of gnome apps will be on flathub only, so make sure you add it as a remote.

There are a lot of apps on flathub now, have a look. And if your application is not there but you’re interested in adding it, please check out the flathub docs.

On application sizes and bloat in flatpak

Flatpak is fundamentally a bundling system, and as such there are worries about bloat compared to non-bundling systems. Obviously there is some truth to this, as bundling do generally increase size. It is my opinion that the advantages of bundling offset these costs, however that is a discussion for another time. Today we’re going to look into what flatpak does to minimize bloat.

On the highest level, flatpak avoids duplication by having a runtime/application split. This means that applications don’t have to bundle everything, and that common dependencies are shared between applications.

However, runtimes were not primarily added in order to minimize bloat. Runtimes were added to have a way to do shared ownership, maintainership and releases of core stuff. Lowlevel things that application authors are not interested in nor experienced with, letting the applications focus on the application specific stuff, while still getting security updates of the core.

In fact, flatpak uses a system called ostree for the application storage which means, even without runtime sharing, a lot of duplication automatically go away.

When I try to explain what ostree is I always start with “Its like git, but for directories with large binaries”. This helps because most linux users these days have some knowledge about git. However, most people only know how to use git, and not its underlying data model. These details are important to understand when discussing ostree, so I will try to give a short explanation.

Lets start by looking at a simple ostree repository, with a single branch called “master”, containing a single file “a-file.txt”.  (Note: I manually shortened all IDs to 8 characters below to make it more readable)

Here is how the repository looks on disk:

├── config
├── objects
│   ├── 03
│   │   └── 02f468.file
│   ├── 7a
│   │   └── f088f5.commit
│   ├── 3e
│   │   └── ca26e9.dirtree
│   └── d5
│       └── 85282a.dirmeta
└── refs
    └── heads
        └── master

Pretty weird eh? Lets explore it a bit. The adventure starts with looking up the ref for the branch master:

$ cat repo/refs/heads/master
7af088f5

This references the id of the commit object that the master branch currently points to. The commit object info can be found in objects/7a/f088f5.commit. This is a binary file, but the contents are something like:

Root metadata: d585282a
Root files: 3eca26e9
Committer: Alex
Commit date: ...

The root metadata is a reference to objects/d5/85282a.dirmeta, which contains uid/gid/permissions for the root directory.

The root files contains a list of file and directory names and their object ids. In the case of subdirectories that would be a .dirtree object, but in this case there is only a file, 0302f468, which contains the actual file data:

$ cat repo/objects/03/02f468.file
a-file contents

So, we understand how to interpret the repository data, but it still seems weird. What are all those long ids for, and whats the reason for the whole thing? It turns out that the ids are the main point. They are in fact sha256 checksums of the object contents.

Whenever ostree commits a file it computes the checksum of the file and uses that as the id of the object in the repository. This means that if for whatever reason two files with identical contents are committed to the same repo, then they will use the same object. And this continues to directories too, if all the files and directories in two dirtree objects are identical (and have the same names) then the same dirtree object will be used.

In the example repo above, suppose that another branch was added, which contained the same file, plus some other file. Only the commit, the new file and the new dirtree for the root would be added, all the other objects would be reused.

What about flatpak?

This ostree feature is very interesting, because in flatpak  each installed runtime and application is a branch in a single ostree repository (combined with a checked out version that has hard-links to the files in the repository). So, any files that are identical are automatically shared between applications.

It should be noted that this sharing does not only affect disk space use. Since shared files are hard-linked, any RAM used for disk cache and memory mappings are automatically shared too. Additionally, when applications are downloaded we don’t have to download any objects that are already locally available, so network bandwidth is shared too.

How well does this work in practice? One way to look at this is comparing the gnome and the freedesktop runtimes. The gnome runtime is based on the freedesktop runtime, starting with it a copy of it, removing some files and then adding some.

We can check the size of the individual runtimes:

$ du -sh org.gnome.Platform/x86_64/3.26
665M org.gnome.Platform/x86_64/3.26
$ du -sh org.freedesktop.Platform/x86_64/1.6
435M org.freedesktop.Platform/x86_64/1.6

But we can also see how much space they use together, which takes into account files that are just hard-linked copies of each other:

$ du -sh org.gnome.Platform/x86_64/3.26 \
        org.freedesktop.Platform/x86_64/1.6
665M org.gnome.Platform/x86_64/3.26
18M org.freedesktop.Platform/x86_64/1.6

Here we see that installing both runtimes just adds an extra 18M due to the shared files. Similar measurements are true for the KDE runtime, which is based on the same freedesktop runtime build. And since they use the same base the gnome and KDE runtimes share this space too.

The above example is the ideal case for sharing, but even in unrelated builds this helps a lot. For example, let us compare the x86-64 and i386 builds of the gnome runtime:

$ du -sh org.gnome.Platform/i386/3.26
682M org.gnome.Platform/i386/3.26
$ du -sh org.gnome.Platform/x86_64/3.26
665M org.gnome.Platform/x86_64/3.26

$ du -sh org.gnome.Platform/x86_64/3.26 \
         org.gnome.Platform/i386/3.26
665M org.gnome.Platform/x86_64/3.26
462M org.gnome.Platform/i386/3.26

Here we saved 220M in two completely unrelated builds. This is caused by things like fonts, icons, and app data being the identical in the two builds.

Thats all for todays deep dive into flatpak internals. I hope you found this interesting.

Spotify and Skype flatpaks moved to flathub

This is a public service announcement.

I used to maintain two custom repositories of flatpaks for spotify and skype. These are now at flathub (in addition to a lot of other apps), and if you were using the old repository you should switch to the new one to continue getting updates.

This is easiest done by removing the current version and then following the directions on the flathub site for installing.

Maintaining a flatpak repository

So, you’ve built a flatpak, using flatpak-builder, and now you have a directory called repo. How do you go from here to something that your users can install the  application from?

To start with, lets make a minimal sample application, so that we have a repo to look at:

$ echo $'#!/bin/sh\necho hello world' > hello.sh
$ flatpak build-init appdir com.example.App \
          org.freedesktop.Sdk \
          org.freedesktop.Platform 1.4
$ flatpak build appdir mkdir /app/bin
$ flatpak build appdir install --mode=750 hello.sh /app/bin
$ flatpak build-finish --command=hello.sh appdir

This produces the following appdir directory:

├── export
├── files/bin/hello.sh
└── metadata

We can now export this to the repo directory:

flatpak build-export repo appdir stable

If we copy this directory to a webserver, so that it is accessible via http, lets say as http://example.com/repo, then anyone can install your application like this:

flatpak remote-add --no-gpg-verify example \
        http://example.com/repo
flatpak install example com.example.App

Note: You can of cours add a local remote using the path of the repo instead of a uri if you just want to test your build.

The anatomy of a repo

So, how does this work? Lets look a bit deeper at the repository. It
looks something like this:

├── config
├── objects
│ ├── 44/6a0ef11b7cc167f3b...0b6c5488.dirmeta
│ ├── 6e/340b9cffb37a989ca...617afa01d.dirtree
│ ├── 74/cfc6e8dd69905a525...64fc0d5e5.dirtree
│ ├── 75/af15131ec214e5074...41858318a.commit
│ ├── c4/083227ca305c44cd6...ceeecd27c.dirtree
│ ├── d8/fdc0a16351fa8bcd8...ac90e63e2.filez
│ ├── e4/2aa65aae7c7b61539...5557cad40.filez
│ └── e9/c16c731270e0a8718...cc8be2601.dirtree
├── refs/heads/app/com.example.App/x86_64/stable
└── summary

If you’ve ever seen a git repository, this looks very similar. The application is referred to by what is called a ref, in this case app/com.example.App/x86_64/stable. If we look at the file with that name in the refs/heads directory we get latest commit id:

$ cat repo/refs/heads/app/com.example.App/x86_64/stable
75af15131ec214e5074...41858318a

When flatpak installs the app it starts by pulling from the repository, which means we download into a local repository the corresponding commit object (i.e. objects/75/af15131ec214e5074...41858318a.commit) and then follow that until all required objects are downloaded.

Another important feature is the summary file. This is a single file that contains information from all the refs files in the repository. It is regenerated from the refs files each time you export a new app or run flatpak build-update-repo. This file is needed because when the repository is accessed via http there is no standard way to list all the refs files, for instance when listing available apps.

This is basically all you have to do to get something running, just build it and copy the results to a webserver. However, there are some details that get important when you’re maintaining a production repository.

GPG signatures

For an official build you want to have a GPG key to sign the repository. In ostree (which is what flatpak uses) every commit is signed, as well as the summary file. When we pull from the repository the summary and the commit signatures are verified, to make sure nothing was modified. This is important because it allows us to use unencrypted http for faster download, and it allows secure use of mirrored repos.

The example repo above did not have signatures, so lets create some. If you have a GPG key already you can use it, although it is recommended that you create a custom key for the repository. That way you can transfer maintainership without having to give anyone else your private key.

So, lets create a custom key:

$ mkdir gpg
$ gpg2 --homedir gpg --quick-gen-key user@example.com

This eventually create a keyring in the gpg directory and print it out:

pub rsa2048 2017-02-09 [S]
 770B194227ED91BF6C9038B83F4BF10E09A00F3B
 uid [ultimate] user@example.com
 sub rsa2048 2017-02-09 []

We can then sign the repo after the fact (you can also sign it at build-export time):

$ flatpak build-sign repo \
  --gpg-sign=770B194227ED91BF6C9038B83F4BF10E09A00F3B \
  --gpg-homedir=gpg

This will create the signature of the commit:

repo/objects/75/af15131ec214e5074...41858318a.commitmeta

You also need to sign the summary file:

$ flatpak build-update-repo repo                     \
 --gpg-sign=770B194227ED91BF6C9038B83F4BF10E09A00F3B \
 --gpg-homedir=gpg

Which will recreate the summary file, now with a summary.sig file next to it.

If we export the public key for the repo we can give it to users:

$ gpg2 --homedir=gpg \
  --export 770B194227ED91BF6C9038B83F4BF10E09A00F3B > example.gpg

And you can then use this instead of --no-gpg-verify when adding the remote:

$ flatpak remote-add --gpg-import=example.gpg \
         example http://example.com/repo

Flatpakref files

Having to both add a remote (with the right gpg key file) and then install the application is not a great user experience. To make this easier for users, flatpak has something called flatpakref files. These contain both the information about the repository, including the GPG key, and the application id. The result is that you can just point flatpak at one file and it will install the application.

Lets create a example-app.flatpakref file for our app:

[Flatpak Ref]
Title=Example App
Name=com.example.App
Branch=stable
Url=http://example.com/repo
GPGKey=mQENBFicjMMBCAC...
IsRuntime=false
RuntimeRepo=https://sdk.gnome.org/gnome.flatpakrepo

The GPGKey field is just a base64 version of the key, which you can get like this:

$ base64 example.gpg | tr -d '\n'

Most of the fields are obvious, but the last one need a special explanation. It specifies a file that describes the repository containing the runtimes that the application uses. If this repository is not configured on the users system it will be automatically added and the runtimes will be installed from it as needed.

Once the flatpakref file is put on the webpage your application can be installed in a single command:

flatpak install http://example.com/example-app.flatpakref

The user can also click on the flatpakref link, and it will open in the default application for software installation (like gnome-software). There are also some additional fields that will be shown in gnome-software: Homepage, Comment, Description and Icon.

Static deltas

When downloading an application, flatpak will do individual requests for each file in the application. If you have a lot of small files this can be slow, so you want to make sure the HTTP keep-alive is enabled on the web server. But even then it can be slow. Additionally, as objects are whole files, we still have to download the entire file even if just one byte in the file changed.

In order to solve this flatpak supports something called static deltas. These are pre-generated delta files (using bsdiff) that contain all the data needed to go from one version to another (or from nothing to a version). If these files are available they are used instead of the individual objects, which allows pulls to be much more efficient.

You can generate static deltas for the latest versions of all apps by passing --generate-static-deltas to flatpak build-update-repo, and I recommend everyone do this for production repositories.

Configuration

There are a couple of configurations in a flatpak repo that you can set via build-update-repo. One is the title (--title=TITLE). This will be used by default as the title when listing the configured remotes.

Another is the default branch (--default-branch=BRANCH). If this is set, then this will be used as the default branch name (the last part of the ref) when it is not specified. This is useful when you have multiple versions of the same app (for instance a nightly build and a stable build) in the same repository.

AppStream branches

In order to support nice graphical frontends like gnome-software, flatpak uses a metadata format called AppStream. The way this works is that each application ships an AppData xml file, and an icon. Then when you run flatpak build-update-repo, each such xml file and icon are extracted and put in a separate per-arch appdata branch (called e.g. appstream/x86_64). Flatpak then mirrors this branch locally for each remote, and this data is used by gnome-software.

So, make sure your application ships an appdata file, and make sure you run build-update-repo to update the branches whenever an application changes.

Syncing updates

If you build a new version of your app on you will get an updated repository locally that has both the old build and the new build. You then need then to copy this updated repo to the webserver. Generally if you just copy the files over with e.g. rsync then things will work fine.

However, there are some race conditions if the new repo files are written at the same time as someone is pulling from the repository. For instance, if you write the summary file before all the objects are copied, then there is a short period where a new pull will not see a complete version of the commit. Another race is if you delete objects from an older version before the new commit is fully uploaded.

All these races are easily avoided by ordering the sync so that the summary file is uploaded after all the objects, and any deletes are done at the end. The ostree-releng-scripts repo has a script to do this.

Hosting

Many developers don’t run their own server to host things on, relying instead on services like github or gitlab to host releases. Unfortunately these are generally designed to host single-file releases (such as tarballs), rather than a multi-file repository.

Since flatpak repositories are just a bunch of static files using Amazon S3, or similar services is very effective. I personally use S3 for the Skype and Spotify flatpak repos, and the cost is about $0.01 a month so far. So, in practice this is essentially free. However, you still need to register with Amazon and have a credit card.

I haven’t found any perfect completely free alternative, but there are some workarounds:

On github, you can use the github pages feature. To do this you create a new git repo and then you commit the flatpak repository into the git repo. Then you enable github pages for the repo and point it at the master branch.

Here is an example of using this: https://github.com/alexlarsson/test-releases
With the flatpak repo itself available as: https://alexlarsson.github.io/test-releases/repo

The maximum size of a github pages size is 1GB, and there is a soft bandwidth limit at 100GB per month, so depending on the size of your application and how popular it is, this could be a useful approach.

On gitlab you can use the gitlab pages feature to do something similar.

The flatpak security model, part 3 – The long game

We saw previously that you can open up the Flatpak sandbox in order to run applications that are not built to be sandboxed. However, the long term plan is to make it possible for applications to work sandboxed, and then slowly make this the default for most applications (while losing little or no features). This is a work in progress on many levels, we need work on Flatpak itself, but also on the Linux desktop platform and the applications themselves.

So, how do we reach this long-term goal?

Some things were mentioned in earlier parts. For example, once we have a realistic shot at sandboxing we need to make the permissions more visible in the UI, and make it easier to override them. We also need to drop X11 and move to Wayland, as well as finish the work on PulseAudio permissions.

However, the really important part is being able to have dynamic, fine-grained permissions. This is achieved with something we call Portals.

A Portal is a service that runs on the host, yet is accessible from the sandbox. This is ok because the interface it exposes has been designed in order to be “safe”.

So, what makes a portal safe?

Lets start with a very simple portal, the Network Monitor portal. This service returns network connection status (online/offline) and signals when it changes. You can already get the basic status from the kernel even in the sandbox, but the portal can use Network Manager to get additional information like whether there is a captive portal active, and if the network is metered.

This portal looks at whether the calling app has network access, and if so allows it to read the current status, because this information could already be collected by the app manually (by replicating what network manager does). The portal is merely a more efficient and easy way to do this.

The next example is the Open URI portal. The application sends a request with a URI that it wants to be shown. For instance you could use this for links the user clicked on in the application, but also to show your application documentation.

We don’t want the sandbox to be able to start apps with caller-controlled URIs in the background, because that would be an easy way to do attack them. The way we make this safe is to make the operation interactive and cancellable. So, the portal shows a dialog, allowing the user to select the app to open the URI in, or (if the dialog was unexpected) close the dialog. All this happens outside the sandbox, which means that the user is always in control of what gets launched and when.

A similar example is the FileChooser portal. The sandbox cannot see the users files, but it can request the user to pick a file. Once a file is chosen outside the sandbox, the application is granted access to it (and only it). In this case too it is the interactivity and cancellability that makes the operation safe.

Another form of portal is geolocation. This is safe because the portal can reliably identify the calling application, and it keeps a list of which applications are currently allowed access. If the application is not allowed it replies that it doesn’t know the location. Then a UI can be created in the desktop to allow the user to configure these permissions. For instance, there can be an application list in the control center, or there could be a notification that lets you grant access.

To sum up, portals are considered safe for one of these reasons:

  • The portal doesn’t expose any info you wouldn’t already know, or which is deemed unsafe
  • The operation is interactive and cancellable
  • Portals can reliably identify applications and apply dynamic rules

Theoretically any kind of service designed in this way could be called a portal. For instance, one could call Wayland a UI portal. However, in practice portals are dbus services.  In fact by default Flatpak lets the sandbox talk to any service named org.freedesktop.portal.* on the session bus.

The portals mentioned above are part of the Desktop portal, xdg-desktop-portal.  It is a UI-less, desktop-independent service. But for all user-interaction or policy it defers to a desktop-specific backend. There are currently backends for Gtk and (work in progress) KDE. For sandboxing to work these need to be installed on the host system.

In addition the the previously listed portals, xdg-desktop-portal also contains:

  • Printing
  • User account information
  • Inhibiting suspend
  • Notifications
  • Proxy configuration
  • Screenshot request
  • Device access request

There is also a portal shipped with flatpak, the Document portal. It’s permissions based, and is what the FileChooser portal uses to grant access to files dynamincally on a file-by-file basis.

We are also planning to add more portals as needed. For instance we’d like to add a Share portal that lets you easily share content with registered handlers (for instance posting text to a Twitter or Facebook app).

The flatpak security model – part 2: Who needs sandboxing anyway?

The ability to run an application sandboxed is a very important  feature of flatpak. However, its is not the only reason you might want to use flatpak. In fact, since currently very few applications work in a fully sandboxed environment, most of the apps you’d run are not sandboxed.

In the previous part we learned that by default the application sandbox is very limiting. If we want to run a normal application we need to open things up a bit.

Every flatpak application contains a manifest, called metadata. This file describes the details of the application, like its identity (app-id) and what runtime it uses. It also lists the permissions that the application requires.

By default, once installed, an application gets all the permissions that it requested. However, you can override the permissions each time you call flatpak run or globally on a per-application basis by using flatpak override (see manpages for flatpak-run and flatpak-override for details). The handling of application permissions are currently somewhat hidden in the interface, but the long term plan is to show permissions during installation and make it easier to override them.

So, what kind of permissions are there?

First apps need to be able to produce output and get input. To do this we have permissions that allow access to PulseAudio for sound and X11 and/or Wayland for graphical output and input. The way this works is that we just mount the unix domain socket for the corresponding service into the sandbox.

It should be noted that X11 is not very safe when used like this, you can easily use the X11 protocol to do lots of malicious things. PulseAudio is also not very secure, but work is in progress on making it better. Wayland however was designed from the start to isolate clients from each other, so it is pretty secure in a sandbox.

But, secure or not, almost all Linux desktop applications currently in existence use X11, so it is important that we are able to use it.

Another way for application to integrate with the system is to use DBus. Flatpak has a filtering dbus proxy, which lets it define rules for what the application is allowed to do on the bus. By default an application is allowed to own its app-id and subnames of it (i.e. org.gnome.gedit and org.gnome.gedit.*) on the session bus. This means other clients can talk to the application, but it can only talk to the bus itself, not any other clients.

Its interesting to note this connection between the app-id and the dbus name. In fact, valid flatpak app-ids are defined to be the same form as valid dbus names, and when applications export files to the host (such as desktop files, icons and dbus service files), we only allow exporting files that start with the app-id. This ties very neatly into modern desktop app activation were the desktop and dbus service files also have to be named by the dbus name. This rule ensures that applications can’t accidentally conflict with each other, but also that applications can’t attack the system by exporting a file that would be triggered by the user outside the sandbox.

There are also permissions for filesystem access. Flatpak always uses a filesystem namespace, because /usr and /app are never from the host, but other directories from the host can be exposed to the sandbox. The permission here is quite fine grained, starting with access to all host files, to your home-directory only or to individual directories. The directories can also be exposed read-only.

The default sandbox only has a loopback network interface and thius has no connection to the network, but if you grant network access then the app will get full network access. There are no partial access for network access however. For instance one would like to be able to set up a per-application firewall configuration. Unfortunately, it is quite complex and risky to set up networking so we can’t expose it in a safe way for unprivileged use.

There are also a few more specialized permissions, like various levels of hardware device access and some other details. See man flatpak-metadata for the available settings.

All this lets us open up exactly what is needed for each application, which means we can run current Linux desktop applications without modifications. However, the long term goal is to introduce features so that applications can run without opening the sandbox. We’ll get to this plan in the next part.

Until then, happy flatpaking.

The flatpak security model – part 1: The basics

This is the first part of a series talking about the approach flatpak takes to security and sandboxing.

First of all, a lot of people think of container technology like docker, rkt or systemd-nspawn when they think of linux sandboxing. However, flatpak is fundamentally different to these in that it is unprivileged.

What I mean is that all the above run as root, and to use them you either have to be root, or your access to it is equivalent to root. For instance, if you have access to the docker socket then you can get a full root shell with a command like:

docker run -t -i --privileged -v /:/host fedora chroot /host

Flatpak instead runs everything as the regular user.  To do this it uses a project called bubblewrap which is like a super-powered version of chroot, only you don’t have to be root to run it.

Bubblewrap can do more than just change the root, it lets you construct a custom filesystem mount tree for the process. Additionally it lets you create namespaces to further isolate things from the host. For instance if use –unshare-pid then your process will not see any processes from outside the sandbox.

Now, chroot is a root-only operation. How can it be that bubblewrap lets you do the same thing but doesn’t require root privileges? The answer is that it uses unprivileged user namespaces.

Inside such a user namespace you get a lot of capabilities that you don’t have outside it, such as creating new bind mounts or calling chroot. However, in order to be allowed to use this you have to set up a few process limits. In particular you need to set a process flag called PR_SET_NO_NEW_PRIVS. This causes all forms of privilege escalation (like setuid) to be disabled, which means the normal ways to escape a chroot jail don’t work.

Actually, I lied a bit above. We do use unprivileged user namespaces if we can, but many distributions disable them. The reason is that user namespaces open up a whole new attack surface against the kernel, allowing an unprivileged user access to lots of things that may not be perfectly adapted user access. For instance CVE-2016-3135 was a local root exploit which used a memory corruption in an iptables call. This is normally only accessible by root, but user namespaces made it user exploitable.

If user namespaces are disabled, bubblewrap can be built as a setuid helper instead. This still only lets you use the same features as before, and in many ways it is actually safer this way, because only a limited subset of the full functionality is exposed. For instance you cannot use bubblewrap to exploit the iptable bug above because it doesn’t set up iptable (and if it did it wouldn’t pass untrusted data to it).

Long story short, flatpak uses bubblewrap to create a filesystem namespace for the sandbox. This starts out with a tmpfs as the root filesystem, and in this we bind-mount read-only copies of the runtime on /usr and the application data on /app. Then we mount various system things like a minimal /dev, our own instance of /proc and symlinks into /usr from /lib and /bin. We also enable all the available namespaces so that the sandbox cannot see other processes/users or access the network.

On top of this we use seccomp to filter out syscalls that are risky. For instance ptrace, perf, and recursive use of namespaces, as well as weird network families like DECnet.

In order for the application to be able to write data anywhere we bind mount $HOME/.var/app/$APPID/ into the sandbox, but this is the only persistent writable location.

In this sandbox we then spawn the application (after having dropped all increased permissions). This is a very limited environment, and there isn’t much the application can do. In the next part of this series we’ll start looking into how things can be opened up to allow the app to do more.

A stable base for Flatpak: 0.8

Earlier this week I released Flatpak 0.8.0. The version change is meant to signal the start of a new long-term supported branch. The 0.8.x series will be strictly bugfixes, and all new
features will happen in 0.9.x.

The release has a few changes, such as a streamlined command line interface and OCI support, but it also has several additions that make Flatpak more future-proof. For instance, we added versioning to all file formats, and a minimal-flatpak-version-required field for applications.

My goal is to get the 0.8 series into the Debian 9 release, and as many other distributions as possible, so that people who create flatpaks can consider the features it supports as a reliable baseline.

Sandboxing has always been one of the pillars of Flatpak, but even more important to me is cross-distro application distribution, even if not sandboxed. This is important because it gives upstream developers a way to directly interact with their users, without having an intermediate distributor. With 0.8 I think we have reached a level where the support for this is solid. So, if you ever thought about experimenting with Flatpak, now is the time!

I leave you with a small screencast showing the new streamlined way to install an application om the command line (on an otherwise empty system):

For information on how to get flatpak, see flatpak.org. Version 0.8.0 is already in the Ubuntu PPA and in  Fedora. Other distributions hopefully will get it soon.

Distributing spotify as a flatpak

One of the features in the recent flatpak relase is described as:

Applications can now list a set of URIs that will be downloaded with the application

This seems a bit weird. Downloading an application already means your loading data from a URI. What is the usecase for this?

Normally all the data that is needed for your application will be bundled with it and this feature is not needed. However, in some cases applications are freely downloadable, but not redistributable. This only affects non-free software, but Flatpak wants to support both free and non-free software for pragmatic reasons.

Common examples of this are Spotify and Skype. I hope that they eventually will be available as native flatpaks, but in order to bootstrap the flatpak ecosystem we want to make it possible to distribute these right now.

So, how does this work? Lets take Spotify as an example. It is available as a binary debian package. I’ve created a wrapper application for it which contains all the dependencies it needs. It also specifies the URL of the debian package and its sha256 checksum, plus a script to unpack it.

When the user downloads the application it also downloads the deb, and then (in a sandbox) it runs the unpack script which extracts it and puts the files in the right place. Only then is the installation considered done, and from thereon it is used read-only.

I put up a build of the spotify app on S3, so to install spotify all you need to do is:

flatpak install --from https://s3.amazonaws.com/alexlarsson/spotify-repo/spotify.flatpakref

(Note: This requires flatpak 0.6.13 and the remote with the freedesktop runtime configured)

Here is an example of installing spotify from scratch:

New flatpak command line

Today I released version 0.6.13 of flatpak which has a lot of nice new features. One that I’d like to talk a bit about is the new command line argument format.

The flatpak command line was always a bit low-level and hard to use. Partly this was because of lack of focus on this, and partly due to the fact that the expected UI for flatpak for most people would be a graphical user interface like gnome-software. However, with this new release this changed, and flatpak is now much nicer to use from the commandline.

So, what is new?

Flatpakrepo files

Before you can really use flatpak you have to configure it so it can find the applications and runtimes. This means setting up one or more remotes. Historically you did this by manually specifying all the options for the remote as arguments to the flatpak remote-add command. To make this easier we added a file format (.flatpakrepo) to describe a remote, and made it easy to use it.

The new canonical example to configure the gnome stable repositories is:

$ flatpak remote-add --from gnome \
   https://sdk.gnome.org/gnome.flatpakrepo
$ flatpak remote-add --from gnome-apps \
   https://sdk.gnome.org/gnome-apps.flatpakrepo

Alternatively you can just click on the above links and they should open in gnome-software (if you have new enough versions installed).

Multiple arguments to install/update/uninstall

Another weakness of the command line has been that commands like install, uninstall and update only accepted one application name argument (and optionally a branch name). This made it hard to install multiple apps in one go, and the separate branch name made it hard to cut-and-paste from output of e.g. flatpak list.

Instead of the separate branch name all the commands now take multiple “partial refs” as arguments. These are partial versions of the OSTree ref format that flatpak uses internally. So, for an internal reference like app/org.gnome.gedit/x86_64/stable, one can now specify one of these:

org.gnome.gedit
org.gnome.gedit//stable
org.gnome.gedit/x86_64
org.gnome.gedit/x86_64/stable

And flatpak will automatically fill in the missing part in a natural way, and give a detailed error if you need to specify more details:

$ flatpak install gnome org.gnome.Platform
error: Multiple branches available for org.gnome.Platform, you must specify one of: org.gnome.Platform//3.20, org.gnome.Platform//3.22, org.gnome.Platform//3.16, org.gnome.Platform//3.18

Automatic dependencies

The other problem with the CLI has been that it is not aware of runtime dependencies. To run an app you generally had to know what runtime it used and install that yourself. The idea here was that the commandline should be simple, non-interactive and safe. If you instead use the graphical frontend it will install dependencies interactively so you can control where things get installed from.

However, this just made the CLI a pain to use, and you could easily end up in situations where things didn’t work. For instance, if you updated gedit from 3.20 to 3.22 it suddenly depended on a new runtime version, and if you weren’t aware of this it probably just stopped working.

Of course, we still can’t just install dependencies from wherever we find them, because that would be a security issue (any configured remote can supply a runtime for any applications). So, the solution here is for flatpak to become interactive:

$ flatpak update org.gnome.gedit
Looking for updates...
Required runtime for org.gnome.gedit/x86_64/stable (org.gnome.Platform/x86_64/3.22) is not installed, searching...
Found in remote gnome, do you want to install it? [y/n]: y
Installing: org.gnome.Platform/x86_64/3.22 from gnome
Installing: org.gnome.Platform.Locale/x86_64/3.22 from gnome
Updating: org.gnome.gedit/x86_64/stable from gnome-apps
Updating: org.gnome.gedit.Locale/x86_64/stable from gnome-apps

If you have remotes you never want to install dependencies from, you can install them with --no-use-for-deps, and they will not be used. Flatpakrepo files for app-only repositories should set NoDeps=true.

Note that this is not a package-system-like dependency solver that can solve sudoku. It is still a very simple two-way split.

Flatpakref files

The primary way Flatpak is meant to be used is that you configure a few remotes that has most of the applications that you use, then you install from these either on the command line, or via a graphical installer. However, sometimes it is nice to have a single link you can put on a website to install your application. Flatpak now supports that via .flatpakref files. These are very similar to flatpakrepo files, in that they describe a repository, but they additionally contain a particular application in that repository which will be installed.

Such files can be installed by just clicking on them in your web-browser (which will open them for installation in gnome-software) or on the command line:

flatpak install --from https://sdk.gnome.org/gedit.flatpakref

This will try to install the required runtime, so you first need to add the remote with the runtimes.

Launching runtimes

During development and testing it is often common to launch commands in a sandbox to experiment with how the runtime works. This was always possible by running a custom command in an application that used the runtime, like so:

$ flatpak run --command=sh org.gnome.gedit
sh-4.3$ ls /app
bin lib libexec manifest.json share
sh-4.3$

You can even specify a custom runtime with --runtime. However, there really should be no need to have an application installed to do this, so the new version allows you to directly run a runtime:

$ flatpak run org.gnome.Platform//3.22
sh-4.3$ ls /app
sh-4.3$