Category Archives: Uncategorized

A PolicyKit refresher

PolicyKit has been around for a long time, and it is a mature system that basically does its job. I recently spent some time to improve the PolicyKit integration in Flatpak and thought it might be useful to write up some of the details.

Details, details

It is irritating when a dialog  pops up unexpectedly and asks questions without providing sufficient details to really know where it is coming from and what the context is. Like

Authentication is required to install software

It would be much better to say what software is being installed, and where it is coming from:

PolicyKit lets you do this by using variables in your message and providing the replacement for them via a PolkitDetails object:

Authentication is required to install $(ref) from $(origin)
polkit_details_insert (details, "origin", origin);
polkit_details_insert (details, "ref", ref);

No display – no problem

Its nice to get a  PolicyKit dialog when you are using a desktop app that needs to carry out a privileged operation. But PolicyKit is also used by commandline tools, such as flatpak. If you are running a command in a terminal, a dialog might still be ok. But what if you are using the flatpak on the console? A dialog is not available here, so privileged operations will fail.

PolicyKit provides the necessary plumbing to solve this situation, by letting apps register their own ‘agent’, which can handle authorization requests if no other agent is available. (In a graphical session, GNOME Shell provides an agent.)

Glancing over some details,  the code to do this looks roughly like this:

listener = polkit_agent_text_listener_new (NULL,
                                         &error);
polkit_agent_listener_register (listener,
            flags, subject, NULL, NULL, &error);

And it yields a result like this:If PolicyKits built-in text agent does not fit your needs, you can implement a PolkitAgentListener yourself. That is what I ended up doing for Flatpak.

Psst, don’t interrupt

As I said earlier, unexpected dialogs are annoying. Ideally, PolicyKit dialogs should only ever appear in response to a direct user action. For example, a dialog is ok if I am clicking the “Install” button in GNOME Software, but not if GNOME Software decides on it own that it is time to install some updates.

PolicyKit has a means to achieve this, by not passing the

POLKIT_CHECK_AUTHORIZATION_FLAGS_ALLOW_USER_INTERACTION

flag when checking for authorization. But this check happens in the system service (or mechanism, in PolicyKit lingo), not in your client. In order to take advantage of this flag, the client needs to pass information about  user interaction along whenever it calls a privileged method of the mechanism.

In the Flatpak case, I added a ‘no-interaction’ flag to all the flatpak-system-helper methods, and library API that GNOME Software can use to set this flag for background operations.

So there should be less unexpected PolicyKit dialogs in the future.

Update: A useful PolicyKit feature that I forgot to mention is implications. If the the permissions (or actions, in PolicyKit lingo) are ordered in some way (e.g. if you are allowed to install applications, you should also be allowed to update them), you can express these relations with “imply” annotations. This can further reduce the need for duplicate dialogs.`

An update on Flatpak updates

A while ago, I’ve described how to handle restarting Flatpak apps when they are updated.

While I showed working code back then, the example was a bit  contrived, since it had to work around GApplication’s built-in uniqueness features.

With the just-landed support for replacement in GApplication, this is now much more straightforward, and I’ve updated my example to show how it works.

Restart yourself

There are a few steps to this.

First, we need to opt into allowing replacement by passing G_APPLICATION_ALLOW_REPLACEMENT when creating our GApplication:

app = g_object_new (portal_test_app_get_type (),
        "application-id", "org.gnome.PortalTest",
        "flags", G_APPLICATION_ALLOW_REPLACEMENT,
        NULL);

This flag makes GApplication handle a –gapplication-replace commandline option, which we will see in action later.

Next, we monitor the /app/.updated file. Flatpak creates this file inside a running sandbox when a new version of the app is available, so this is our signal to restart ourselves:

file = g_file_new_for_path ("/app/.updated");
update_monitor =
       g_file_monitor_file (file, 0, NULL, NULL);
g_signal_connect (update_monitor, "changed",
       G_CALLBACK (update_monitor_changed), win);

The update_monitor_changed function presents a dialog that offers the user to restart the app:

 if (response == GTK_RESPONSE_OK)
      portal_test_app_restart (app);

The portal_test_app_restart function is where we take advantage of the new GApplication functionality:

const char *argv[3] = {
    "/app/bin/portal-test",
    "--gapplication-replace",
    NULL
};

xdp_flatpak_call_spawn (flatpak, cwd, argv, ...);

We call the Spawn method of the Flatpak portal, passing –gapplication-replace as an commandline option, and everything else is handled for us. Easy!

Update 1: If D-Bus is not your thing, there is a simple commandline tool called flatpak-spawn that can be used for the same purpose. It is available inside the sandbox.

Update 2: I forgot to mention that GApplication has also gained a name-lost signal. It can be used to save state in the old instance that can then be loaded in the new instance.

Flatpak, after 1.0

Flatpak 1.0 has happened a while ago, and we now have a stable base. We’re up to the 1.0.3 bug-fix release at this point, and hope to get 1.0.x adopted in all major distros before too long.

Does that mean that Flatpak is done ? Far from it! We have just created a stable branch and started landing some queued-up feature work on the master branch. This includes things like:

  • Better life-cycle control with ps and kill commands
  • Logging and history
  • File copy-paste and DND
  • A better testsuite, including coverage reports

Beyond these, we have a laundry list of things we want to work on in the near future, including

  • Using  host GL drivers (possibly with libcapsule)
  • Application renaming and end-of-life migration
  • A portal for dconf/gsettings (a stopgap measure until we have D-Bus container support)
  • A portal for webcam access
  • More tests!

We are also looking at improving the scalability of the flathub infrastructure. The repository has grown to more than 400 apps, and buildbot is not really meant for using it the way we do.

What about releases?

We have not set a strict schedule, but the consensus seems to be that we are aiming for roughly quarterly releases, with more frequent devel snapshots as needed. Looking at the calendar, that would mean we should expect a stable 1.2 release around the end of the year.

Open for contribution

One of the easiest ways to help Flatpak is to get your favorite applications on flathub, either by packaging it yourself, or by convincing the upstream to do it.

If you feel like contributing to Flatpak itself,  please do! Flatpak is still a young project, and there are plenty of small to medium-size features that can be added. The tests are also a nice place to stick your toe in and see if you can improve the coverage a bit and maybe find a bug or two.

Or, if that is more your thing, we have a nice design for improving the flatpak commandline user experience that is waiting to be implemented.

On Flatpak dependencies

Package managers have to deal with dependencies – too many of them. Over time things have gotten complicated: there are now soft dependencies, reverse dependencies and boolean conditions. So complicated that you can probably do general computation in the dependency solver now.

Thankfully flatpak is a lot simpler: there’s apps, and there’s runtimes, and every app depends on exactly one runtime. Simple and beautiful.

Of course, thats not the whole story.  Lets take a look.

Dependencies

The only hard dependencies in Flatpak are between an app and the runtime that it uses.

Every app uses exactly one runtime. When installing the app, Flatpak will automatically install the runtime it needs, and it will refuse to uninstall the runtime as long as there are apps using it. You can override this using the –force-remove option.

Related refs

The common term Flatpak uses for apps and runtimes is refs. Apart from dependencies, Flatpak has a notion of related refs. These are what other packaging systems might classify as soft dependencies, and they come in various forms and shapes.

The first group are standard pieces that get split off at build time, like .Locale and .Debug refs. The first contain translations and are similar to what other packaging systems call langpacks. The second contain debug information for binaries, and are the equivalent of debuginfo packages in other systems.

The next group are extensions. Both applications and runtimes can declare extension points,  and flatpak will look for refs that match those when setting up a sandbox, and mounts the ones it finds inside the sandbox.

Some extensions are a bit more special, in that they are conditional on some condition of the system. For example, they might only be relevant if the system has an Nvidia GPU, or apply for a certain GTK+ theme.

Another kind of relationship exists between a runtime and its matching sdk.

Whats used ?

The Flatpak uninstall command has a –unused option that makes an educated guess about what refs are no longer needed on your system. For each ref, it looks if it is an application, or the runtime of an installed application, or related to one of these. In each of these cases, it considers the ref used and skips it. Whatever is left afterwards get removed.

There are some heuristics involved when looking at related refs, and we are still fine-tuning these. One change that we’re recently made is to consider sdks used, to make –unused usable for developers.

There is still more fine-tuning to be done. For example, Flatpak will happily use a runtime from the system installation to run an application from the user installation. But the uninstall command works only on a single installation, so it does not see these dependencies, and might remove the runtime. Thankfully, it is easy to recover, should this happen to you: just install the runtime again.

About Flatpak installations

If you have tried Flatpak, you probably know that it can install apps per user or system-wide.  Installing an app system-wide has the advantage that all users on the system can use it. Installing per-user has the advantage that it doesn’t require privileges.

Most of the time, that’s more than enough choice. But there is more.

Standard locations

Before moving on, it is useful to briefly look at where Flatpak installs apps by default.

flatpak install --system flathub org.gnome.Todo

When you install an app like this. it ends up in in /var/lib/flatpak. If you instead use the –user option, it ends up in ~/.local/share/flatpak. To be 100% correct, I should say $XDG_DATA_HOME/flatpak, since Flatpak does respect the XDG basedir spec.

Anatomy of an installation

Flatpak calls the places where it installs apps installations. Every installation has a few subdirectories that are useful to know:

  • repo – this is the OSTree repository where the files for the installed apps and runtimes reside. It is a single repository, so all the apps and runtimes that are part of the same installation get the benefit of deduplication via content-addressing.
  • exports – when an app is installed, Flatpak extracts some files that need to be visible to the outside world, such a desktop files, icons, D-Bus services files, and this is where they end up.
  • appstream – a flatpak repository contains the appstream data for the apps it contains as a separate branch, and Flatpak extracts it on the client-side for consumers like KDE’s Discover or GNOME Software.
  • app, runtime – the deployed versions of apps and runtimes get checked out here. Diving deeper, you see the files of an app in app/org.gimp.GIMP/current/active/files. This directory is what gets mounted in the sandbox as /app if you run the GIMP.

Custom installations

So far, so good. But maybe you have a setup with dozens of machines, and have an existing setup where /opt is shared. Wouldn’t it be nice to have a flatpak installation there, instead of duplicating it in /var on every machine ? This is where custom installations come in.

You can tell Flatpak about another place to  install apps by dropping a file in /etc/flatpak/installations.d/. It can be as simple as the following:

[Installation "bigleaf"]
Path=/opt/flatpak
DisplayName=bigleaf

See the flatpak-installation man page for all the details about this file.

GNOME Software currently doesn’t know such custom installations, and you will have to adjust the shell glue in /etc/profile.d/flatpak.sh for GNOME shell to see apps from there, but that is easy enough.

A patch to make flatpak.sh pick up custom installations automatically would be a welcome contribution!

Apps on a stick

Flatpak has a few more tricks up its sleeve when it comes to sharing apps between machines. A pretty cool one is the recently added create-usb command. It lets you copy one (or more) apps on a usb stick, and install it from there on another machine. While trying this out, I hit a few hurdles, that I’ll briefly point out here.

To make this work, Flatpak relies on an extra piece of information about the remote, the collection ID. The collection ID is a property of the actual remote repository. Flathub has one, org.flathub.Stable.

To make use of it, we need to add it to the configuration for the remote, like this:

$ flatpak remote-modify --collection-id=org.flathub.Stable flathub
$ flatpak update

If you don’t add the collection ID to your remote configuration, you will be greeted by an error saying “Remote ‘flathub’ does not have a collection ID set”. If you omit the flatpak update, the error will say “No such branch (org.flathub.Stable, ostree-metadata) in repository”.

Another error you may hit is “fsetxattr: Operation not supported”.  I think the create-usb command is meant to work with FAT-formatted usb sticks, so this will hopefully be fixed soon. For now, just format your usb stick as EXT4.

After these preparations, we are ready for:

$ flatpak --verbose create-usb /run/media/mclasen/Flatpak org.gimp.GIMP

which will take some time to copy things to the usb stick (which happes to be mounted at /run/media/mclasen/Flatpak) . When this command is done, we can inspect the contents of the OSTree repository like this:

$ flatpak remote-ls file:///run/media/mclasen/Flatpak/.ostree/repo
Ref 
org.freedesktop.Platform.Icontheme.Adwaita
org.freedesktop.Platform.VAAPI.Intel 
org.freedesktop.Platform.ffmpeg 
org.gimp.GIMP 
org.gnome.Platform

Flatpak copied not just the GIMP itself, but also runtimes and extensions that it uses. This ensures that we can install the app from the usb stick even if some of these related refs are missing on the target system.

But of course, we still need to see it work! So I uninstalled the GIMP, disabled my network, plugged the usb stick back in, and:

$ flatpak install --user flathub org.gimp.GIMP
0 metadata, 0 content objects imported; 569 B transferred in 0 seconds

flatpak install: Error updating remote metadata for 'flathub': [6] Couldn't resolve host name
Installing in user:
org.gimp.GIMP/x86_64/stable flathub 1eb97e2d4cde
permissions: ipc, network, wayland, x11
file access: /tmp, host, xdg-config/GIMP, xdg-config/gtk-3.0
dbus access: org.gtk.vfs, org.gtk.vfs.*
Is this ok [y/n]: y
Installing for user: org.gimp.GIMP/x86_64/stable from flathub
[####################] 495 metadata, 4195 content objects imported; 569 B transferred in 1 seconds
Now at 1eb97e2d4cde.

Voilà, an offline installation of a Flatpak. I left the error message in there as proof that I was actually offline :) A nice detail of the collection ID approach is that Flatpak knows that it can still update the GIMP from flathub when I’m online.

Coming soon, peer-to-peer

This post is already too long, so I’ll leave peer-to-peer and advertising Flatpak repositories on the local network via avahi for another time.

Until then, happy Flatpaking! 💓📦💓📦

 

Flatpak portal experiments

One of the signs that a piece of software is reaching a mature state is its ability to serve  use cases that nobody had anticipated when it was started. I’ve recently had this experience with Flatpak.

We have been discussing some possible new directions for the GTK+ file chooser. And it occurred to me that it might be convenient to use the file chooser portal as a way to experiment with different file choosers without having to change either GTK+ itself or the applications.

To verify this idea, I wrote a quick portal implementation that uses the venerable GTK+ 2 file chooser.

Here is Corebird (a GTK+ 3 application) using the GTK+ 2 file chooser to select an image.

On Flatpak updates

Maybe you remember times when updating your system was risky business – your web browser might crash of start to behave funny because the update pulled data files or fonts out from underneath the running process, leading to fireworks or, more likely, crashes.

Flatpak updates on the other hand are 100% safe. You can call

 flatpak update

and the running instances of are not affected in any way. Flatpak keeps existing deployments around until the last user is gone.  If you quit the application and restart it, you will get the updated version, though.

This is very nice, and works just fine. But maybe we can do even better?

Improving the system

It would be great if the system was aware of the running instances, and offered me to restart them to take advantage of the new version that is now available. There is a good chance that GNOME Software will gain this feature before too long.

But for now, it does not have it.

Do it yourself

Many apps, in particular those that are not native to the Linux distro world, expect to update themselves, and we have had requests to enable this functionality in flatpak. We do think that updating software is a system responsibility that should be controlled by global policies and be under the users control, so we haven’t quite followed the request.

But Flatpak 1.0 does have an API that is useful in this context, the “Flatpak portal“. It has a Spawn method that allows applications to launch a process in a new sandbox.

Spawn (IN  ay    cwd_path,
       IN  aay   argv,
       IN  a{uh} fds,
       IN  a{ss} envs,
       IN  u     flags,
       IN  a{sv} options,
       OUT u     pid)

There are several use cases for this, from sandboxing thumbnailers (which create thumbnails for possibly untrusted content files) to sandboxing web browser tabs individually. The use case we are interested in here is restarting the latest version of the app itself.

One complication that I’ve run into when trying this out is the “unique application” pattern that is built into GApplication and similar application classes: Since there is already an owner for the application ID on the session bus, my newly spawned version will just back off and exit. Which is clearly not what I intended in this case.

Make it stop

The workaround I came up with is not very pretty, but functional. It requires several parts.

First, I need a “quit” action exported on the session bus. The newly spawned version will activate this action of the running instance to convince it to go away. Thankfully, my example app already had this action, for the Quit item in the app menu.

I don’t want this to happen unconditionally, but only if I am spawning a new version. To achieve this, I made my app only activate “quit” if the –replace option is present, and add that option to the commandline that I pass to the “Spawn” call.

The code for this part is less pretty than it could be, since GApplication gets in the way a bit. I have to manually check for the –replace option and do the “quit” D-Bus call by hand.

Doing the “quit” call synchronously is not quite enough to avoid a race condition between the running instance dropping the bus name and my new instance attempting to take it. Therefore, I explicitly wait for the bus name to become unowned before entering g_application_run().

But it all works fine. To test it, i exported a “restart” action and added it to the app menu.

Tell me about it

But who can remember to open the app menu and click “Restart”. That is just too cumbersome. Thankfully, flatpak has a solution for this: When you update an app that is running, it creates a marker file named

/app/.updated

inside the sandbox for each running instance.

That makes it very easy for the app to find out when it has been updated, by just monitoring this file. Once the file appears, it can pop up a dialog that offers the user to restart the newer version of the app. A good quality implementation of this will of course save and restore the state when doing this.

Voilá, updates made easy!

You can find the working example in the portal-test repository.

Flatpak – a look behind the portal

Flatpak allows sandboxed apps to interact with the rest of the system via portals. Portals are simply D-Bus services that are designed to be safe to expose to untrusted apps.

Principles

There are several principles that have guided the design of the existing portals.

 Keep the user in control

To achieve this, most portals  will show a dialog to let the user accept or deny the applications’ request. This is not a hard rule — in some cases, a dialog is just not practical.

Avoid yes/no questions

Direct questions about permissions tend to be dismissed without much thought, since they get in the way of the task at hand. Therefore, portals avoid this kind of question whenever possible and instead just let the user get on with the task.

For example, when an app is requesting to open a file on the host, we just present the user with a fille chooser. By selecting a file, the user implicitly grants the application access to the file. Or he can cancel the file selection and implicitly deny the applications’ request.

Don’t be annoying

Nothing is worse than having to answer the same question over and over. Portals make use of a database to record previous decisions and avoid asking repeatedly for the same thing.

Practice

The database used by portals is called the permission store. The permission store is organized in tables, with a table for each portal that needs one. It has a D-Bus api, but it is more convenient to explore it using the recently added flatpak commands:

flatpak permission-list
flatpak permission-list devices
flatpak permission-list desktop-used-apps video/webm

The first command will list all permissions in all tables, the second will show the content of the “devices” table, and the last one will show just the row for video/webm in the “desktop-used-apps” table.

There are also commands that deal with permissions on a per-application basis.

flatpak permission-show org.gnome.Epiphany
flatpak permission-reset org.gnome.Epiphany

The first command will show all the permissions that apply to the application, the second will remove all permissions for the application.

And more…

The most important table in the permission store is the “documents” table, where the documents portal stores information about host files that have been exported for applications. The documents portal makes the exported files available via a fuse filesystem at

/run/user/1000/doc

A useful subdirectory here is by-app, where the exported files are visible on a per-application bases (when setting up a sandbox, flatpak makes only this part of the document store available inside the sandbox).

It is instructive to browse this filesystem, but flatpak also has a dedicated set of commands for exploring the contents of the documents portal.

flatpak document-list
flatpak document-list org.gnome.Epiphany

The first command lists all exported files, the second shows only the files that are exported for an individual application.

flatpak document-info $HOME/example.pdf

This command shows information about a file that is exported in the document portal, such as which applications have access to it, and what they are allowed to do.

Lastly, there are document-export and document-unexport commands that allow to add or remove files from the document portal.

Summary

If you want to explore how portals work, or just need to double-check which files an app has access to, flatpak has tools that let you do so conveniently.

The Flatpak BoF at Guadec

Here is a quick summary of the Flatpak BoF that happened last week at Guadec.

1.0 approaching fast

We started by going over the list of outstanding 1.0 items. It is a very short list, and they should all be included in an upcoming 0.99.3 release.

  • Alex wants to add information about renaming to desktop files
  • Owen will hold his OCI appstream work for 1.1 at this point
  • James is interested in getting more information from snapd to portal backends, but this also does not need to block 1.0, and can be pulled into a 1.1 portals release
  • Matthias will review the open portal issues and make sure everything is in good shape for a 1.0 portal release

1.0 preparation

Alex will do a 0.99.3 release with all outstanding changes for 1.0 (Update: this release has happened by now). Matthias will work with  Allan and Bastien on the press release and other materials. Nick is very interested in having information about runtime availability, lifetime and stability easily available on the website for 1.0.

We agreed to remove the ‘beta’ label from the flathub website.

Post 1.0 plans

There was a suggestion that we should have an autostart portal. This request spawned a bigger discussion of application life-cycle control, background apps and services. We need to come up with a design for these intertwined topics before adding portals for it.

After 1.0, Alex wants to focus on tests and ci for a while. One idea in this area is to have a scriptable test app that can make portal requests.

Automatic migration on renames or EOL is on Endless’ wishlist.

Exporting repositories in local networks is a feature that Endless has, but it may end up upstream in ostree instead of flatpak.

Everybody agreed that GNOME Software should merge apps from different sources in a better way.

For runtimes, the GNOME release teams aims to have the GNOME runtime built using buildstream, on top of the freedesktop 1.8 runtime. This may or may not happen in time for GNOME 3.30.

Flatpak, making contribution easy

One vision that i’ve talked abut in the past is that moving to flatpak could make it much easier to contribute to applications.

Fast-forward 3 years, and the vision is (almost) here!

Every application on flathub has a Sources extension that you can install just like anything else from a flatpak repo:

flatpak install flathub org.seul.pingus.Sources

This extension contains a flatpak manifest which lists the exact revisions of all the sources that went into the build. This lets you reproduce the build — if you can find the manifest!

Assuming you install the sources per-user, the manifest is here (using org.seul.pingus as an example):

$HOME/.local/share/flatpak/runtime/org.seul.pingus.Sources/x86_64/stable/active/files/manifest/org.seul.pingus.json

And you can build it like this:

flatpak-builder build org.seul.pingus.json

I said the vision is almost, but not quite there. Here is why: gnome-builder also has a way to kickstart a project from a manifest:

gnome-builder --manifest org.seul.pingus.json

But sadly, this currently crashes. I filed an issue for it, and it will hopefully work very soon. Next step, flip-to-hack !