Category Archives: Uncategorized

Flatpak, after 1.0

Flatpak 1.0 has happened a while ago, and we now have a stable base. We’re up to the 1.0.3 bug-fix release at this point, and hope to get 1.0.x adopted in all major distros before too long.

Does that mean that Flatpak is done ? Far from it! We have just created a stable branch and started landing some queued-up feature work on the master branch. This includes things like:

  • Better life-cycle control with ps and kill commands
  • Logging and history
  • File copy-paste and DND
  • A better testsuite, including coverage reports

Beyond these, we have a laundry list of things we want to work on in the near future, including

  • Using  host GL drivers (possibly with libcapsule)
  • Application renaming and end-of-life migration
  • A portal for dconf/gsettings (a stopgap measure until we have D-Bus container support)
  • A portal for webcam access
  • More tests!

We are also looking at improving the scalability of the flathub infrastructure. The repository has grown to more than 400 apps, and buildbot is not really meant for using it the way we do.

What about releases?

We have not set a strict schedule, but the consensus seems to be that we are aiming for roughly quarterly releases, with more frequent devel snapshots as needed. Looking at the calendar, that would mean we should expect a stable 1.2 release around the end of the year.

Open for contribution

One of the easiest ways to help Flatpak is to get your favorite applications on flathub, either by packaging it yourself, or by convincing the upstream to do it.

If you feel like contributing to Flatpak itself,  please do! Flatpak is still a young project, and there are plenty of small to medium-size features that can be added. The tests are also a nice place to stick your toe in and see if you can improve the coverage a bit and maybe find a bug or two.

Or, if that is more your thing, we have a nice design for improving the flatpak commandline user experience that is waiting to be implemented.

On Flatpak dependencies

Package managers have to deal with dependencies – too many of them. Over time things have gotten complicated: there are now soft dependencies, reverse dependencies and boolean conditions. So complicated that you can probably do general computation in the dependency solver now.

Thankfully flatpak is a lot simpler: there’s apps, and there’s runtimes, and every app depends on exactly one runtime. Simple and beautiful.

Of course, thats not the whole story.  Lets take a look.

Dependencies

The only hard dependencies in Flatpak are between an app and the runtime that it uses.

Every app uses exactly one runtime. When installing the app, Flatpak will automatically install the runtime it needs, and it will refuse to uninstall the runtime as long as there are apps using it. You can override this using the –force-remove option.

Related refs

The common term Flatpak uses for apps and runtimes is refs. Apart from dependencies, Flatpak has a notion of related refs. These are what other packaging systems might classify as soft dependencies, and they come in various forms and shapes.

The first group are standard pieces that get split off at build time, like .Locale and .Debug refs. The first contain translations and are similar to what other packaging systems call langpacks. The second contain debug information for binaries, and are the equivalent of debuginfo packages in other systems.

The next group are extensions. Both applications and runtimes can declare extension points,  and flatpak will look for refs that match those when setting up a sandbox, and mounts the ones it finds inside the sandbox.

Some extensions are a bit more special, in that they are conditional on some condition of the system. For example, they might only be relevant if the system has an Nvidia GPU, or apply for a certain GTK+ theme.

Another kind of relationship exists between a runtime and its matching sdk.

Whats used ?

The Flatpak uninstall command has a –unused option that makes an educated guess about what refs are no longer needed on your system. For each ref, it looks if it is an application, or the runtime of an installed application, or related to one of these. In each of these cases, it considers the ref used and skips it. Whatever is left afterwards get removed.

There are some heuristics involved when looking at related refs, and we are still fine-tuning these. One change that we’re recently made is to consider sdks used, to make –unused usable for developers.

There is still more fine-tuning to be done. For example, Flatpak will happily use a runtime from the system installation to run an application from the user installation. But the uninstall command works only on a single installation, so it does not see these dependencies, and might remove the runtime. Thankfully, it is easy to recover, should this happen to you: just install the runtime again.

About Flatpak installations

If you have tried Flatpak, you probably know that it can install apps per user or system-wide.  Installing an app system-wide has the advantage that all users on the system can use it. Installing per-user has the advantage that it doesn’t require privileges.

Most of the time, that’s more than enough choice. But there is more.

Standard locations

Before moving on, it is useful to briefly look at where Flatpak installs apps by default.

flatpak install --system flathub org.gnome.Todo

When you install an app like this. it ends up in in /var/lib/flatpak. If you instead use the –user option, it ends up in ~/.local/share/flatpak. To be 100% correct, I should say $XDG_DATA_HOME/flatpak, since Flatpak does respect the XDG basedir spec.

Anatomy of an installation

Flatpak calls the places where it installs apps installations. Every installation has a few subdirectories that are useful to know:

  • repo – this is the OSTree repository where the files for the installed apps and runtimes reside. It is a single repository, so all the apps and runtimes that are part of the same installation get the benefit of deduplication via content-addressing.
  • exports – when an app is installed, Flatpak extracts some files that need to be visible to the outside world, such a desktop files, icons, D-Bus services files, and this is where they end up.
  • appstream – a flatpak repository contains the appstream data for the apps it contains as a separate branch, and Flatpak extracts it on the client-side for consumers like KDE’s Discover or GNOME Software.
  • app, runtime – the deployed versions of apps and runtimes get checked out here. Diving deeper, you see the files of an app in app/org.gimp.GIMP/current/active/files. This directory is what gets mounted in the sandbox as /app if you run the GIMP.

Custom installations

So far, so good. But maybe you have a setup with dozens of machines, and have an existing setup where /opt is shared. Wouldn’t it be nice to have a flatpak installation there, instead of duplicating it in /var on every machine ? This is where custom installations come in.

You can tell Flatpak about another place to  install apps by dropping a file in /etc/flatpak/installations.d/. It can be as simple as the following:

[Installation "bigleaf"]
Path=/opt/flatpak
DisplayName=bigleaf

See the flatpak-installation man page for all the details about this file.

GNOME Software currently doesn’t know such custom installations, and you will have to adjust the shell glue in /etc/profile.d/flatpak.sh for GNOME shell to see apps from there, but that is easy enough.

A patch to make flatpak.sh pick up custom installations automatically would be a welcome contribution!

Apps on a stick

Flatpak has a few more tricks up its sleeve when it comes to sharing apps between machines. A pretty cool one is the recently added create-usb command. It lets you copy one (or more) apps on a usb stick, and install it from there on another machine. While trying this out, I hit a few hurdles, that I’ll briefly point out here.

To make this work, Flatpak relies on an extra piece of information about the remote, the collection ID. The collection ID is a property of the actual remote repository. Flathub has one, org.flathub.Stable.

To make use of it, we need to add it to the configuration for the remote, like this:

$ flatpak remote-modify --collection-id=org.flathub.Stable flathub
$ flatpak update

If you don’t add the collection ID to your remote configuration, you will be greeted by an error saying “Remote ‘flathub’ does not have a collection ID set”. If you omit the flatpak update, the error will say “No such branch (org.flathub.Stable, ostree-metadata) in repository”.

Another error you may hit is “fsetxattr: Operation not supported”.  I think the create-usb command is meant to work with FAT-formatted usb sticks, so this will hopefully be fixed soon. For now, just format your usb stick as EXT4.

After these preparations, we are ready for:

$ flatpak --verbose create-usb /run/media/mclasen/Flatpak org.gimp.GIMP

which will take some time to copy things to the usb stick (which happes to be mounted at /run/media/mclasen/Flatpak) . When this command is done, we can inspect the contents of the OSTree repository like this:

$ flatpak remote-ls file:///run/media/mclasen/Flatpak/.ostree/repo
Ref 
org.freedesktop.Platform.Icontheme.Adwaita
org.freedesktop.Platform.VAAPI.Intel 
org.freedesktop.Platform.ffmpeg 
org.gimp.GIMP 
org.gnome.Platform

Flatpak copied not just the GIMP itself, but also runtimes and extensions that it uses. This ensures that we can install the app from the usb stick even if some of these related refs are missing on the target system.

But of course, we still need to see it work! So I uninstalled the GIMP, disabled my network, plugged the usb stick back in, and:

$ flatpak install --user flathub org.gimp.GIMP
0 metadata, 0 content objects imported; 569 B transferred in 0 seconds

flatpak install: Error updating remote metadata for 'flathub': [6] Couldn't resolve host name
Installing in user:
org.gimp.GIMP/x86_64/stable flathub 1eb97e2d4cde
permissions: ipc, network, wayland, x11
file access: /tmp, host, xdg-config/GIMP, xdg-config/gtk-3.0
dbus access: org.gtk.vfs, org.gtk.vfs.*
Is this ok [y/n]: y
Installing for user: org.gimp.GIMP/x86_64/stable from flathub
[####################] 495 metadata, 4195 content objects imported; 569 B transferred in 1 seconds
Now at 1eb97e2d4cde.

Voilà, an offline installation of a Flatpak. I left the error message in there as proof that I was actually offline :) A nice detail of the collection ID approach is that Flatpak knows that it can still update the GIMP from flathub when I’m online.

Coming soon, peer-to-peer

This post is already too long, so I’ll leave peer-to-peer and advertising Flatpak repositories on the local network via avahi for another time.

Until then, happy Flatpaking! 💓📦💓📦

 

Flatpak portal experiments

One of the signs that a piece of software is reaching a mature state is its ability to serve  use cases that nobody had anticipated when it was started. I’ve recently had this experience with Flatpak.

We have been discussing some possible new directions for the GTK+ file chooser. And it occurred to me that it might be convenient to use the file chooser portal as a way to experiment with different file choosers without having to change either GTK+ itself or the applications.

To verify this idea, I wrote a quick portal implementation that uses the venerable GTK+ 2 file chooser.

Here is Corebird (a GTK+ 3 application) using the GTK+ 2 file chooser to select an image.

On Flatpak updates

Maybe you remember times when updating your system was risky business – your web browser might crash of start to behave funny because the update pulled data files or fonts out from underneath the running process, leading to fireworks or, more likely, crashes.

Flatpak updates on the other hand are 100% safe. You can call

 flatpak update

and the running instances of are not affected in any way. Flatpak keeps existing deployments around until the last user is gone.  If you quit the application and restart it, you will get the updated version, though.

This is very nice, and works just fine. But maybe we can do even better?

Improving the system

It would be great if the system was aware of the running instances, and offered me to restart them to take advantage of the new version that is now available. There is a good chance that GNOME Software will gain this feature before too long.

But for now, it does not have it.

Do it yourself

Many apps, in particular those that are not native to the Linux distro world, expect to update themselves, and we have had requests to enable this functionality in flatpak. We do think that updating software is a system responsibility that should be controlled by global policies and be under the users control, so we haven’t quite followed the request.

But Flatpak 1.0 does have an API that is useful in this context, the “Flatpak portal“. It has a Spawn method that allows applications to launch a process in a new sandbox.

Spawn (IN  ay    cwd_path,
       IN  aay   argv,
       IN  a{uh} fds,
       IN  a{ss} envs,
       IN  u     flags,
       IN  a{sv} options,
       OUT u     pid)

There are several use cases for this, from sandboxing thumbnailers (which create thumbnails for possibly untrusted content files) to sandboxing web browser tabs individually. The use case we are interested in here is restarting the latest version of the app itself.

One complication that I’ve run into when trying this out is the “unique application” pattern that is built into GApplication and similar application classes: Since there is already an owner for the application ID on the session bus, my newly spawned version will just back off and exit. Which is clearly not what I intended in this case.

Make it stop

The workaround I came up with is not very pretty, but functional. It requires several parts.

First, I need a “quit” action exported on the session bus. The newly spawned version will activate this action of the running instance to convince it to go away. Thankfully, my example app already had this action, for the Quit item in the app menu.

I don’t want this to happen unconditionally, but only if I am spawning a new version. To achieve this, I made my app only activate “quit” if the –replace option is present, and add that option to the commandline that I pass to the “Spawn” call.

The code for this part is less pretty than it could be, since GApplication gets in the way a bit. I have to manually check for the –replace option and do the “quit” D-Bus call by hand.

Doing the “quit” call synchronously is not quite enough to avoid a race condition between the running instance dropping the bus name and my new instance attempting to take it. Therefore, I explicitly wait for the bus name to become unowned before entering g_application_run().

But it all works fine. To test it, i exported a “restart” action and added it to the app menu.

Tell me about it

But who can remember to open the app menu and click “Restart”. That is just too cumbersome. Thankfully, flatpak has a solution for this: When you update an app that is running, it creates a marker file named

/app/.updated

inside the sandbox for each running instance.

That makes it very easy for the app to find out when it has been updated, by just monitoring this file. Once the file appears, it can pop up a dialog that offers the user to restart the newer version of the app. A good quality implementation of this will of course save and restore the state when doing this.

Voilá, updates made easy!

You can find the working example in the portal-test repository.

Flatpak – a look behind the portal

Flatpak allows sandboxed apps to interact with the rest of the system via portals. Portals are simply D-Bus services that are designed to be safe to expose to untrusted apps.

Principles

There are several principles that have guided the design of the existing portals.

 Keep the user in control

To achieve this, most portals  will show a dialog to let the user accept or deny the applications’ request. This is not a hard rule — in some cases, a dialog is just not practical.

Avoid yes/no questions

Direct questions about permissions tend to be dismissed without much thought, since they get in the way of the task at hand. Therefore, portals avoid this kind of question whenever possible and instead just let the user get on with the task.

For example, when an app is requesting to open a file on the host, we just present the user with a fille chooser. By selecting a file, the user implicitly grants the application access to the file. Or he can cancel the file selection and implicitly deny the applications’ request.

Don’t be annoying

Nothing is worse than having to answer the same question over and over. Portals make use of a database to record previous decisions and avoid asking repeatedly for the same thing.

Practice

The database used by portals is called the permission store. The permission store is organized in tables, with a table for each portal that needs one. It has a D-Bus api, but it is more convenient to explore it using the recently added flatpak commands:

flatpak permission-list
flatpak permission-list devices
flatpak permission-list desktop-used-apps video/webm

The first command will list all permissions in all tables, the second will show the content of the “devices” table, and the last one will show just the row for video/webm in the “desktop-used-apps” table.

There are also commands that deal with permissions on a per-application basis.

flatpak permission-show org.gnome.Epiphany
flatpak permission-reset org.gnome.Epiphany

The first command will show all the permissions that apply to the application, the second will remove all permissions for the application.

And more…

The most important table in the permission store is the “documents” table, where the documents portal stores information about host files that have been exported for applications. The documents portal makes the exported files available via a fuse filesystem at

/run/user/1000/doc

A useful subdirectory here is by-app, where the exported files are visible on a per-application bases (when setting up a sandbox, flatpak makes only this part of the document store available inside the sandbox).

It is instructive to browse this filesystem, but flatpak also has a dedicated set of commands for exploring the contents of the documents portal.

flatpak document-list
flatpak document-list org.gnome.Epiphany

The first command lists all exported files, the second shows only the files that are exported for an individual application.

flatpak document-info $HOME/example.pdf

This command shows information about a file that is exported in the document portal, such as which applications have access to it, and what they are allowed to do.

Lastly, there are document-export and document-unexport commands that allow to add or remove files from the document portal.

Summary

If you want to explore how portals work, or just need to double-check which files an app has access to, flatpak has tools that let you do so conveniently.

The Flatpak BoF at Guadec

Here is a quick summary of the Flatpak BoF that happened last week at Guadec.

1.0 approaching fast

We started by going over the list of outstanding 1.0 items. It is a very short list, and they should all be included in an upcoming 0.99.3 release.

  • Alex wants to add information about renaming to desktop files
  • Owen will hold his OCI appstream work for 1.1 at this point
  • James is interested in getting more information from snapd to portal backends, but this also does not need to block 1.0, and can be pulled into a 1.1 portals release
  • Matthias will review the open portal issues and make sure everything is in good shape for a 1.0 portal release

1.0 preparation

Alex will do a 0.99.3 release with all outstanding changes for 1.0 (Update: this release has happened by now). Matthias will work with  Allan and Bastien on the press release and other materials. Nick is very interested in having information about runtime availability, lifetime and stability easily available on the website for 1.0.

We agreed to remove the ‘beta’ label from the flathub website.

Post 1.0 plans

There was a suggestion that we should have an autostart portal. This request spawned a bigger discussion of application life-cycle control, background apps and services. We need to come up with a design for these intertwined topics before adding portals for it.

After 1.0, Alex wants to focus on tests and ci for a while. One idea in this area is to have a scriptable test app that can make portal requests.

Automatic migration on renames or EOL is on Endless’ wishlist.

Exporting repositories in local networks is a feature that Endless has, but it may end up upstream in ostree instead of flatpak.

Everybody agreed that GNOME Software should merge apps from different sources in a better way.

For runtimes, the GNOME release teams aims to have the GNOME runtime built using buildstream, on top of the freedesktop 1.8 runtime. This may or may not happen in time for GNOME 3.30.

Flatpak, making contribution easy

One vision that i’ve talked abut in the past is that moving to flatpak could make it much easier to contribute to applications.

Fast-forward 3 years, and the vision is (almost) here!

Every application on flathub has a Sources extension that you can install just like anything else from a flatpak repo:

flatpak install flathub org.seul.pingus.Sources

This extension contains a flatpak manifest which lists the exact revisions of all the sources that went into the build. This lets you reproduce the build — if you can find the manifest!

Assuming you install the sources per-user, the manifest is here (using org.seul.pingus as an example):

$HOME/.local/share/flatpak/runtime/org.seul.pingus.Sources/x86_64/stable/active/files/manifest/org.seul.pingus.json

And you can build it like this:

flatpak-builder build org.seul.pingus.json

I said the vision is almost, but not quite there. Here is why: gnome-builder also has a way to kickstart a project from a manifest:

gnome-builder --manifest org.seul.pingus.json

But sadly, this currently crashes. I filed an issue for it, and it will hopefully work very soon. Next step, flip-to-hack !

Flatpak in detail, part 3

The previous in this series looked at runtimes and filesystem organization. Here, we’ll take a look at the flatpak sandbox and explore how the world looks to a running flatpak app.

The sandbox, a view from the inside

Flatpak uses container technologies to set up a sandbox for the apps it runs. The name container brings to mind an image of a vessel with solid walls, which is a bit misleading.

In reality, a container is just a plain old process which talks directly to the kernel and other processes on the system. It is the job of the container management system, in this case flatpak, to launch the process with reduced privileges to limit the harm it can do to the rest of the system. To do this, flatpak uses the same kernel features that tools like docker use: namespaces, cgroups, seccomp.

Processes

One namespace that is used by flatpak is the pid namespace, which hides processes outside the sandbox from the sandboxed app.  If you run ps inside the sandbox, you’ll see something like this:

$ ps
PID TTY TIME CMD
1 ? 00:00:00 bwrap
2 ? 00:00:00 sh
3 ? 00:00:00 ps

One thing you’ll notice is that the app itself (the sh process, in this case) ends up with a PID of 2. Just something to keep in mind when getting your app ready for running in a sandbox: PIDs can’t be used to identify the app to the rest of the system anymore. This has caused minor problems, e.g with the _NET_WM_PID property that some window managers look at to decide about window decorations.

Filesystems

Another namespace that flatpak makes extensive use of is the mount namespace. It is used to create a customized view of the filesystem for the sandboxed app.

Running mount inside the sandbox will show you all the details, but its a bit much to show here, so we’ll just stick to the most important parts:

  • /app – this is where the apps files are mounted. It is a typical Unix filesystem with /app/share, /app/bin/, /app/lib and so on. When building an app for flatpak packaging, it typically gets configured with –prefix=/app for this reason.
  • /usr – the runtime files are placed here.
  • /run/host/ – various parts of the host filesystem that are safe to expose are mounted here, for example fonts (in /run/host/fonts) and icons (in /run/host/share/icons).
  • /usr/share/runtime/locale/ – the Locale extension of the runtime is mounted here.
  • Other extensions are mounted in their pre-determined places.
  • $HOME/.var/app/$APPID is made available to the app as well. This is the place in the home directory where flatpaked apps can store persistent data. The XDG_DATA_HOME, XDG_CONFIG_HOME and XDG_CACHE_HOME environment variables are set up to point to subdirectories in this place, and respecting these variables is a good way for an app to just work in a flatpak.

D-Bus

A common way for Linux apps to communicate with services and apps in the user session is via D-Bus, and most flatpak sandboxes allow the app to talk on the session bus (it can be turned off with a setting in the metadata). But we can’t just let the app talk to any other process on the bus, that would be a big risk. Therefore, flatpak sets up a filtering proxy that only allows the app to talk to certain names (this is again set up in the metadata).

The exception to this are portals – these are D-Bus interfaces specifically designed to be safe to expose, and apps are always allowed to talk to them. Portals offer APIs for a number of common needs, such as

  • org.freedestkop.portal.FileChooser  – Open a file
  • org.freedesktop.portal.OpenUri – Show a file in an app
  • org.freedesktop.portal.Print – Print a file
  • org.freedesktop.portal.Notification – Show a notification

Most of the time, portals will present a dialog to keep the user in control of what system access the sandboxed apps gets.

Toolkits like GTK+ can transparently use portals in some cases when they detect that the app is inside a sandbox, which greatly reduces the effort in using portals.

Metadata

But how does GTK+ find out that is being used inside a sandbox?

It looks for a file called .flatpak-info which flatpak places in the filesystem root of every sandbox. This file is not just a marker, it contains some useful information about the details of the sandbox setup, and is worth looking at. Some apps show information from here in their about dialog.

Summary

Flatpak sets up a customized sandbox for the apps it runs. It tries hard to preserve an environment that typical Linux desktop apps expect, including XDG locations and the session bus.

Trying out GTK+ 4

I was asked today if there is already a Flatpak runtime that includes GTK+ 3.94. A very natural question. GTK+ 4 and flatpak are both cool, so of course you want to try them together.

But the answer is: No, GTK+ 4 is not included in a runtime yet.

It is still a bit early for that – the master branch of the GNOME runtime includes development versions of libraries, but those are libraries that are following the GNOME release schedule, where we can expect them to have a stable release in time for GNOME 3.30 in September. That is not the case for GTK+ 4.

However, that does not mean you cannot take your first steps with GTK+ 3.94 and flatpak right now. Flatpak is a flexible system and the bundling capabilities are very well-suited for this kind of scenario.

Just add GTK+ to the list of bundled modules in your flatpak manifest:

...
 "modules": [
   {
     "name": "gtk+",
     "buildsystem": "meson",
     "builddir": true,
     "config-opts": [
       "--prefix="/app",
       "--libdir="lib",
       "-Ddemos=false",
       "-Dbuild-tests=false"
     ],
     "sources": [
       "type": "archive",
       "url": "https://download.gnome.org/sources/gtk+/3.94/gtk+-3.94.0.tar.xz"
     ]
   },
   {
     "name": "your-app",
     ...
   }
 ]
...

If you are more adventurous, you can also build the GTK+ master branch from git, to see our latest progress towards GTK+ 4 as it happens.

To see a complete example of a flatpak manifest bundling GTK+ and its dependencies, you can look at corebird.

We have the beginnings of a GTK+ 3 → 4 migration guide in the GTK+ documentation. If you give it a try, please let us know what works and what is missing.

One way to so so is to file an issue. But you also welcome to come by the GTK+ BoF at Guadec in Almeria and tell us in person.