GNOME applications (and others) are commonly using the GSettings API for storing their application settings.

GSettings has many nice aspects:

• flexible data types, with GVariant
• schemas, so others can understand your settings (e.,g. dconf-editor)
• overrides, so distros can tweak defaults they don’t like

And it has different backends, so it can be adapted to work transparently in many situations. One example for where this comes in handy is when we use a memory backend to avoid persisting any settings while running tests.

The GSettings backend that is typically used for normal operation is the DConf one.

DConf

DConf features include profiles,  a stack of databases, a facility for locking down keys so they are not writable, and a single-writer design with a central service.

The DConf design is flexible and enterprisey – we have taken advantage of this when we created fleet commander to centrally manage application and desktop settings for large deployments.

But it is not a great fit for sandboxing, where we want to isolate applications from each other and from the host system.  In DConf, all settings are stored in a single database, and apps are free to read and write any keys, not just their own – plenty of potential for mischief and accidents.

Most of the apps that are available as flatpaks today are poking a ‘DConf hole’ into their sandbox to allow the GSettings code to keep talking to the dconf daemon on the session bus, and mmap the dconf database.

Here is how the DConf hole looks in the flatpak metadata file:

[Context]
filesystems=xdg-run/dconf;~/.config/dconf:ro;

[Session Bus Policy]
ca.desrt.dconf=talk

Sandboxes

Ideally, we want sandboxed apps to only have access to their own settings, and maybe readonly access to a limited set of shared settings (for things like the current font, or accessibility settings). It would also be nice if uninstalling a sandboxed app did not leave traces behind, like leftover settings  in some central database.

It might be possible to retrofit some of this into DConf. But when we looked, it did not seem easy, and would require reconsidering some of the central aspects of the DConf design. Instead of going down that road, we decided to take advantage of another GSettings backend that already exists, and stores settings in a keyfile.

Unsurprisingly, it is called the keyfile backend.

Keyfiles

The keyfile backend was originally created to facilitate the migration from GConf to GSettings, and has been a bit neglected, but we’ve given it some love and attention, and it can now function as the default GSettings backend inside sandboxes.

It provides many of the isolation aspects we want: Apps can only read and write their own settings, and the settings are in a single file, in the same place as all the application data:

~/.var/app/$APP/config/glib-2.0/settings/keyfile

One of the things we added to the keyfile backend is support for locks and overrides, so that fleet commander can keep working for apps that are in flatpaks.

For shared desktop-wide settings, there is a companion Settings portal, which provides readonly access to some global settings. It is used transparently by GTK and Qt for toolkit-level settings.

What does all this mean for flatpak apps?

If your application is not yet available as a flatpak, and you want to provide one, you don’t have to do anything in particular. Things will just work. Don’t poke a hole in your sandbox for DConf, and GSettings will use the keyfile backend without any extra work on your part.

If your flatpak is currently shipping with a DConf hole, you can keep doing that for now. When you are ready for it, you should

• Remove the DConf hole from your flatpak metadata
• Instruct flatpak to migrate existing DConf settings, by adding a migrate-path setting to the X-DConf section in your flatpak metadata. The value fo the migrate-path key is the DConf path prefix where your application’s settings are stored.

Note that this is a one-time migration; it will only happen if the keyfile does not exist. The existing settings will be left in the DConf database, so if you need to do the migration again for whatever reason, you can simply remove the the keyfile.

This is how the migrate-path key looks in the metadata file:

[X-DConf]
migrate-path=/org/gnome/builder/

Closing the DConf hole is what makes GSettings use the keyfile backend, and the migrate-path key tells flatpak to migrate settings from DConf – you need both parts for a seamless transition.

There were some recent fixes to the keyfile backend code, so you want to make sure that the runtime has GLib 2.60.6, for best results.

Happy flatpaking!

Update: One of the most recent fixes in the keyfile backend was to correct under what circumstances GSettings will choose it as the default backend. If you have problems where the wrong backend is chosen, as a short-term workaround, you can override the choice with the GSETTINGS_BACKEND environment variable.

Update 2: To add the migrate-path setting with flatpak-builder, use the following option:

--metadata=X-DConf=migrate-path=/your/path/




Pango is in clearly maintenance mode — Behdad and  I have a Pango work day once every few months, whenever we get together somewhere.

But thats about it. Things that we don’t get done in a day often sit unfinished for long times in git branches or issues. Examples for this are the color Emoji support that took many years to land, or the  subpixel positioning work that is still unfinished.

This doesn’t mean that text rendering is in decline. Far from it. In fact, Harfbuzz is more active than ever and has had unprecedented success: All major Web browsers, toolkits, and applications are using it.

We’ve discussed for a while how to best take advantage of Harfbuzz’ success for text rendering on the desktop. Our conclusion is that we have to keep Pango from getting in the way. It should be a thin and translucent layer, and not require us to plumb APIs for every new feature through several internal abstractions.

We have identified several steps that will let us make progress towards this goal.

Unicode APIs

Many libraries provide subsets of Unicode data and APIs: GLib has some. ICU has some, fribidi has some. Even Harfbuzz has some.

Pango really does not need to  provide its own wrappers for text direction or Unicode scripts. Not doing so means we don’t have to update Pango when there is a new version of Unicode.

Harfbuzz passthrough

New font features land regularly in Harfbuzz. By providing direct access to Harfbuzz objects, we can make these available to GTK and applications without adding APIs to Pango.

Stop using freetype

Freetypes FT_Face object has locking semantics that are broken and hard to work with; they are constantly getting in the way as we are juggling hb_fonts, FT_Face and cairo scaled font objects.

We’ve concluded that the best way forward is to stop using freetype for font loading or accessing font and glyph metrics. We can use Harfbuzz for all of these (a small gap will be closed soon).

Using Harfbuzz for font loading means that we will lose support for bitmap and type1 fonts. We think this is an acceptable trade-off, but others might disagree. Note that Harfbuzz does support loading bitmap-only OpenType fonts.

Unified shaping

Shaping is the process of turning a paragraph of text and fonts into a sequence of positioned glyphs for rendering. Historically,  Pango has used a different implementation on each platforms.

Going forward, we want to use Harfbuzz for shaping on all platforms. The web browsers already do this, and it works well. This will let us clean up a lot of old, unused shaping engine abstractions in Pango.

Unhinted rendering

As a general direction, we want to move Pango towards (horizontally) unhinted rendering, combined with subpixel positioning. Other platforms are already doing this. And it gives us resolution-independent layout that is better suited for scalable apis and OpenGL rendering.

 

A few weeks ago, I wrote a post outlining the major improvements in Silverblue that are coming in Fedora 30. Now that Fedora 30 is released, you should go and try them out!

Today, I’ll dive a bit deeper into one particular item from that post which is not Silverblue-specific: the toolbox.

The toolbox experience

I’ll be honest: my path towards using the toolbox has not been 100% smooth.

I’m using the Rawhide version of Silverblue on my laptop, and my ability to use the toolbox has been affected by a number of regressions in the underlying tooling (unprivileged containers, podman and buildah) – which is to be expected for such young technology.

Thankfully, we got it all to work in time for the release, so your experience in Fedora 30 should be much smoother.

First steps

I’m doing all of my GTK and Flatpak development in a toolbox now. A small, but maybe interesting detail: While my host system is bleeding edge rawhide, the toolbox I’m using runs Fedora 30. I created it using the command:

$ toolbox create --release 30

After creating a toolbox, you can enter it using

$ toolbox enter

You will find yourself in another shell. If you pay close attention, you’ll notice that we show a colorful glyph in the prompt as a hint that you are in a different environment.

A minimal base

Entering the toolbox for the first time and looking around, it does not look much like a development environment at all:

$ rpm -q gcc gdb make
package gcc is not installed
package gdb is not installed
package make is not installed

But thankfully, dnf is available, so we can take advantage of the Fedora package universe to install what we need.  Thus, the first steps in working in a toolbox are easy, but a bit repetitive – install things.

I went into my GTK checkout and tried to build it in this environment:

[gtk+] $ meson --prefix=/usr build
bash: meson: command not found...

[gtk+] $ sudo dnf install meson
...

After a few rounds of installing missing dependencies, I eventually got to the point where GTK buit successfully.

From then on, everything basically worked as normal. Running tests and examples that I’ve build works just fine since the toolbox takes care of setting up DISPLAY, etc. I still had to install the occasional missing tool, such as vim or gdb, but development works just as it did before.

Some things are different, though.

A different perspective

In the past, I’ve done GTK development on a Fedora Workstation system, which means that many things are always installed, whether I think of them or not: icon themes, cursor themes, 3rd party pixbuf loaders, mime handlers, and many other things that make up a full desktop.

Working in a toolbox gives a somewhat different perspective. As explained earlier, the toolbox is created from a pretty minimal base (the fedora-toolbox image). None of the things I listed above are in it. Since they are not build dependencies, I didn’t get around to installing them at first either.

And I got to see how gtk-widget-factory and other demos look when they are missing – that has let to several bug fixes and improvements in the way GTK handles such situations.

A maybe unexpected beneficiary: other platforms, such as Windows, where it is much more likely that GTK will be installed without some of these extras.

Overall, I’m quite happy with this way of doing development.

Trying it out

I hope you are now eager to try it out yourself. You can do that on a traditional Fedora 30 system just as easily as on a Silverblue installation.

One thing I will recommend is that you should install the 0-day updates for the toolbox and gnome-terminal – we’ve worked hard during the freeze to make the toolbox as polished and welcoming as we can, and the result is in these updates.

Enjoy! ❤️ ⬢

It has been a bit more than a year that we’ve set up the Atomic Workstation SIG. A little later,  we settled on the name Silverblue, and did a preview release with Fedora 29.

The recent F30 beta release is an good opportunity to look back. What have we achieved?

When we set out to turn Atomic Workstation into an every-day-usable desktop, we had a list of items that we knew needed to be addressed. As it turns out, we have solved most of them, or are very close to that.

Here is an unsorted list.

Full Flatpak support

GNOME Software already had support for installing Flatpaks, a year ago, so this is not 100% new. But the support has been greatly improved with the port to libflatpak – GNOME Software is now using the same code as the Flatpak commandline. And  more recently, it learned to display information about sandbox permissions, so that users can see what level of system access the installed applications have.

This information is now also available in the new Application Settings panel. The panel also offers some control over permissions and lets you clean up storage per application.

A Flatpak registry

Flathub is a great place to find desktop applications – there are over 500 now. But since we can’t enable Flathub by default, we have looked for an alternative, and started to provide Flatpak apps in the Fedora container registry. This is taking advantage of Flatpaks support for the OCI format, and uses the Fedora module-build-system.

GNOME Software support for rpm-ostree

GNOME Software was designed as an application installer, but it also provides the UI for OS updates and upgrades. On a Silverblue system, that means supporting rpm-ostree. GNOME Software has learned to do this.

Another bit of functionality for which GNOME Software was traditionally talking to PackageKit is Addons. These are things that could be classified as system extensions: fonts, language support, shell extensions,, etc.  On a Silverblue system, the direct replacement is to use the rpm-ostree layering capability to add such packages to the OS image. GNOME Software knows how to do this now. It is not ideal, since you probably don’t expect to have to reboot your system for installing a font. But it gets us the basic functionality back until we have better solutions for system extensions.

Nvidia driver support

One class of system extensions that I haven’t mention in the previous section is drivers.  If you have an Nvidia graphics card, you may want the Nvidia driver to make best use of your hardware.  The situation with the Nvidia drivers is a little more complicated than with plain rpms, since the rpm needs to match your kernel, and if you don’t have the right driver, your system may boot to a black screen.

These complications are not unique to Silverblue, and the traditional solution for this in Fedora is to use the akmod system to build drivers that match your kernel. With Fedora 30, we put the necessary changes in place in rpm-ostree and the OS image to make this work for Silverblue as well.

Third-party rpms

Fedora contains a lot of apps, but there’s always the odd one that you can’t find in the repositories. A popular app in this category is the Chrome browser. Thankfully, Google provides an rpm that works on Fedora. But, it installs its content into /opt. That is not technically wrong, but causes a problem on Silverblue, since rpm-ostree has so far insisted on keeping packaged content under its tight control in /usr.

Ultimatively, we  want to see apps shipped as Flatpaks, but for Fedora 30, we have managed to get rpm-ostree to handle this situation, so chrome and similar 3rd party rpms can now be installed via package layering on Silverblue.

A toolbox

An important target audience for Fedora Workstation is developers. Not being able to install toolchains and libraries (because the OS is immutable) is obviously not going to make this audience happy.

The short answer is: switch to container-based workflows. Its the future!

But that doesn’t excuse us  from making these workflows easy and convenient for people who are used to the power of the commandline. So, we had to come up with a better answer, and started to develop the toolbox. The toolbox is a commandline tool to take the pain out of working with ‘pet’ containers. With a single command,

toolbox enter

it gives you a ‘traditional’ Fedora environment with dnf,  where you can install the packages you need. The toolbox has the infrastructure to manage multiple named containers, so you can work on different projects in parallel without interference.

Whats missing?

There are many bigger and smaller things that can still be improved – software is never finished. To name just a few:

• Make IDEs work well with containers on an immutable OS
• Codec availability and installation
• Handle “difficult” applications such as virtualbox well
• Find better ways to handle system extensions

But we’ve come a long way in the one year since I’ve started using Atomic Workstation as my day-to-day OS.

If you want to see for yourself, download the F30 beta image and give it a try!

Time flies, Flatpak 1.0 was released four months ago.  Today we released the next major version, Flatpak 1.2. Lets take a look at what’s new.

New User Experience

A lot of effort since 1.0 has gone into improving the commandline user experience. This includes everything from better progress reporting to search and completion.

I’ve already written a detailed post about this aspect of the Flatpak 1.2 work, so I not going to say much more about it here.

Life-cycle control

1.2 includes new commands which make it easier to manage running Flatpaks.

Like other containers, Flatpaks are just regular processes, so traditional tools like ps and kill can be used with them. But there is often a bit more to a Flatpak sandbox than just a single process – there’s a babysitter, and D-Bus proxies,  and it can be a little daunting to identify the right process to kill, in a process listing.

To make this easy and obvious, we’ve added two new commands, flatpak ps and flatpak kill. For now, flatpak ps just lists basic information, but it is the natural place to show e.g. resource consumption in the future.

This functionality is available to other Flatpak front ends as well, in the form of the FlatpakInstance API in libflatpak.

Logging

Installing software is serious business, and it is well worth keeping a record of changes over time.

Flatpak now logs changes to installations and installed apps in the systemd journal. We take advantage of the capabilities offered by the journal, and write structured logs that contain useful fields like the affected remote, the commit IDs, and so on.

A particularly nice feature of the journal is that it keeps track of the originator of changes for us, so even if Flatpaks system-helper service is modifying the system installation, we still record the user who initiated the change.

You can of course use journalctl to study these logs. We also added a flatpak history command, which may be a bit more convenient, since it offers filtering and display that is more tailored to the specfics of Flatpak.

In the ecosystem

Portals are an important part of the Flatpak approach, and we’ve made releases of the portals to go along with the new Flatpak.  The 1.2 release of the portals brings a new location portal, a new portal for toolkit settings, respecting desktop lockdown settings, and better validation of input in all portals.

Flatpak itself also got some changes that will improve sandboxing in the future. One noteworthy change is that Flatpak will now bind dconf defaults and locks into the sandbox.  This will become useful with the next GLib release, when GSettings will no longer be using the dconf backend, allowing us to close an all-to-common sandbox hole.

In a similar vein, Flatpak is now creating a fontconfig configuration snipplet that will be used by a future fontconfig release to map font directories to their caches.

The common theme in these changes is that the libraries in the runtime need some changes to work well in a sandboxed setting. Getting these changes in place takes time, but we’re making steady progress.

Over the fence

On the desktop side, both GNOME Settings and GNOME Software will show Flatpak permissions in more detail in their upcoming releases.

Looking ahead

Now that Flatpak 1.2 is out, we are getting  ready to unveil a much-improved Flathub. Stay tuned!

Flatpak is made to run desktop apps, and there are apps like KDE Discover or GNOME Software that let you manage the Flatpak installations on your system.

Of course, we still need a way to handle Flatpak from the commandline. The flatpak commandline tool in 1.0 is powerful without being overwhelming (like git) and way friendlier than some other tools (for example, ostree).

But we can always do better. For Flatpak 1.2, we’ve gone back to the drawing board and did some designs for the commandline user experience (yes, that needs design too).

Powerful: Columns

Many Flatpak commands produce information in tabular form. You can list remotes, apps, documents, etc. And all these have a bunch information that can be displayed. It can be overwhelming when you only need a particular piece of information (and overflowing the available space in a terminal).

To help with that, we’ve introduced a –columns option which you select which information you want to see.

You can explore the available columns using –columns=help. In Flatpak 1.2, all the list-producing commands will support the –columns option: list, search, remotes, remote-ls, ps, history.

One nice side-effect of having –columns is that we can add more information to the output of these commands without overflowing the display, by adding optional columns that are not included in the default output.

Concise: Errors

It happens all the time, I want to type search, but my c key is sticky, and it comes out as searh. Flatpak used to react to unknown command by dumping out its –help option, which is long and a bit overwhelming.

Now we try to be more concise and more helpful at the same time, by making a guess at what was meant, and just pointing out the –help option.

Friendly: Search

In the same vein, the reverse-DNS style application ID that Flatpak relies on has been criticized as unwieldy and hard to handle. Nobody wants to type

flatpak install flathub org.gnome.meld

and commandline completion does not help too much if you have no idea what the application ID might be.

Thankfully, that is no longer necessary. With Flatpak 1.2, you can type

flatpak install meld

and Flatpak will ask you a few questions to confirm which remote to use and what exact application you meant. Much friendlier. This search also works for the uninstall command, and may be added to more commands in the future.

Informative: Descriptions

Flatpak repos contain appstream data describing the apps in detail. That is what e.g. GNOME Software uses on its detail page for an application.

So far, the Flatpak commandline has not used appstream data at all. But that is changing. In 1.2, a number of commands will show useful information from appstream data, such as the description shown here by the list and info commands:

If you pay close attention you may notice that column names can be abbreviated with –columns.

Fun: Prompts

Here’s the commandline version of theming. We now set a custom prompt. to let you know what context you are in when using a shell in a flatpak sandbox.

You can customize the prompt using

flatpak override --env=PS1="..."

The application ID for the sandbox is available in the FLATPAK_ID environment variable.

The beast: Progress

Updates and installs can take a long time – there are possibly big downloads and there may be dependencies that need to be updated as well, etc. For a good user experience it is  important to provide some feedback on the progress and the expected remaining time.

The Ostree library which Flatpak uses for downloads provides progress information, but it is very detailed and hard to digest.  To make this even more challenging, there is not that much space in a terminal window to display all the relevant information.

For 1.2, we’ve come up with  a combination of a table that gets updated to display overall status and a progress line for the current operation:

A similar, but simpler layout is used for uninstalls.

Coming Soon

All of these improvements will appear in Flatpak 1.2, hopefully very soon after the New Year. Something to look forward to.  📦📦📦

A while ago, I’ve described how to handle restarting Flatpak apps when they are updated.

While I showed working code back then, the example was a bit  contrived, since it had to work around GApplication’s built-in uniqueness features.

With the just-landed support for replacement in GApplication, this is now much more straightforward, and I’ve updated my example to show how it works.

Restart yourself

There are a few steps to this.

First, we need to opt into allowing replacement by passing G_APPLICATION_ALLOW_REPLACEMENT when creating our GApplication:

app = g_object_new (portal_test_app_get_type (),
        "application-id", "org.gnome.PortalTest",
        "flags", G_APPLICATION_ALLOW_REPLACEMENT,
        NULL);

This flag makes GApplication handle a –gapplication-replace commandline option, which we will see in action later.

Next, we monitor the /app/.updated file. Flatpak creates this file inside a running sandbox when a new version of the app is available, so this is our signal to restart ourselves:

file = g_file_new_for_path ("/app/.updated");
update_monitor =
       g_file_monitor_file (file, 0, NULL, NULL);
g_signal_connect (update_monitor, "changed",
       G_CALLBACK (update_monitor_changed), win);

The update_monitor_changed function presents a dialog that offers the user to restart the app:

 if (response == GTK_RESPONSE_OK)
      portal_test_app_restart (app);

The portal_test_app_restart function is where we take advantage of the new GApplication functionality:

const char *argv[3] = {
    "/app/bin/portal-test",
    "--gapplication-replace",
    NULL
};

xdp_flatpak_call_spawn (flatpak, cwd, argv, ...);

We call the Spawn method of the Flatpak portal, passing –gapplication-replace as an commandline option, and everything else is handled for us. Easy!

Update 1: If D-Bus is not your thing, there is a simple commandline tool called flatpak-spawn that can be used for the same purpose. It is available inside the sandbox.

Update 2: I forgot to mention that GApplication has also gained a name-lost signal. It can be used to save state in the old instance that can then be loaded in the new instance.

Flatpak 1.0 has happened a while ago, and we now have a stable base. We’re up to the 1.0.3 bug-fix release at this point, and hope to get 1.0.x adopted in all major distros before too long.

Does that mean that Flatpak is done ? Far from it! We have just created a stable branch and started landing some queued-up feature work on the master branch. This includes things like:

• Better life-cycle control with ps and kill commands
• Logging and history
• File copy-paste and DND
• A better testsuite, including coverage reports

Beyond these, we have a laundry list of things we want to work on in the near future, including

• Using  host GL drivers (possibly with libcapsule)
• Application renaming and end-of-life migration
• A portal for dconf/gsettings (a stopgap measure until we have D-Bus container support)
• A portal for webcam access
• More tests!

We are also looking at improving the scalability of the flathub infrastructure. The repository has grown to more than 400 apps, and buildbot is not really meant for using it the way we do.

What about releases?

We have not set a strict schedule, but the consensus seems to be that we are aiming for roughly quarterly releases, with more frequent devel snapshots as needed. Looking at the calendar, that would mean we should expect a stable 1.2 release around the end of the year.

Open for contribution

One of the easiest ways to help Flatpak is to get your favorite applications on flathub, either by packaging it yourself, or by convincing the upstream to do it.

If you feel like contributing to Flatpak itself,  please do! Flatpak is still a young project, and there are plenty of small to medium-size features that can be added. The tests are also a nice place to stick your toe in and see if you can improve the coverage a bit and maybe find a bug or two.

Or, if that is more your thing, we have a nice design for improving the flatpak commandline user experience that is waiting to be implemented.

Package managers have to deal with dependencies – too many of them. Over time things have gotten complicated: there are now soft dependencies, reverse dependencies and boolean conditions. So complicated that you can probably do general computation in the dependency solver now.

Thankfully flatpak is a lot simpler: there’s apps, and there’s runtimes, and every app depends on exactly one runtime. Simple and beautiful.

Of course, thats not the whole story.  Lets take a look.

Dependencies

The only hard dependencies in Flatpak are between an app and the runtime that it uses.

Every app uses exactly one runtime. When installing the app, Flatpak will automatically install the runtime it needs, and it will refuse to uninstall the runtime as long as there are apps using it. You can override this using the –force-remove option.

Related refs

The common term Flatpak uses for apps and runtimes is refs. Apart from dependencies, Flatpak has a notion of related refs. These are what other packaging systems might classify as soft dependencies, and they come in various forms and shapes.

The first group are standard pieces that get split off at build time, like .Locale and .Debug refs. The first contain translations and are similar to what other packaging systems call langpacks. The second contain debug information for binaries, and are the equivalent of debuginfo packages in other systems.

The next group are extensions. Both applications and runtimes can declare extension points,  and flatpak will look for refs that match those when setting up a sandbox, and mounts the ones it finds inside the sandbox.

Some extensions are a bit more special, in that they are conditional on some condition of the system. For example, they might only be relevant if the system has an Nvidia GPU, or apply for a certain GTK+ theme.

Another kind of relationship exists between a runtime and its matching sdk.

Whats used ?

The Flatpak uninstall command has a –unused option that makes an educated guess about what refs are no longer needed on your system. For each ref, it looks if it is an application, or the runtime of an installed application, or related to one of these. In each of these cases, it considers the ref used and skips it. Whatever is left afterwards get removed.

There are some heuristics involved when looking at related refs, and we are still fine-tuning these. One change that we’re recently made is to consider sdks used, to make –unused usable for developers.

There is still more fine-tuning to be done. For example, Flatpak will happily use a runtime from the system installation to run an application from the user installation. But the uninstall command works only on a single installation, so it does not see these dependencies, and might remove the runtime. Thankfully, it is easy to recover, should this happen to you: just install the runtime again.