Category Archives: Uncategorized

Fedora Atomic Workstation: Ruling the commandline

Note: Fedora Atomic Workstation has recently been renamed to Team Silverblue. Learn more here.

In my recent posts, I’ve mostly focused on finding my way around with GNOME Builder and using it to do development in Flatpak sandboxes. But I am not really the easiest target audience for an IDE like GNOME Builder, having spent most of my life on the commandline with tools like vim and make.

So, what about the commandline in an Atomic Workstation environment? There are many container tools, like buildah, atomic, oc, podman, and so on. I am not going to talk about these, since I don’t know them very well, and they are covered, e.g. on

But there are a few commands that are essential to life on the Atomic Workstation: rpm-ostree and flatpak.


First of all, there’s rpm-ostree, which is the commandline frontend to the rpm-ostreed daemon that manages the OS image(s) on the Atomic Workstation.

You can run

rpm-ostree status

to get some information about your OS image (and the other images that may be present on your system). And you can run

rpm-ostree upgrade

to get the latest update for your OS image (the terminology clash here is a bit unfortunate; rpm-ostree calls an upgrade what most Linux distros and packaging tools call an update).

You can run this command as normal user in a terminal, and rpm-ostreed will present you with a polkit dialog to do its privileged operations. Recently, rpm-ostreed has also gained the ability to check for and deploy upgrades automatically.

An important thing to keep in mind is that rpm-ostree never changes your running system. You have to reboot into the new image to see the changes, so

systemctl reboot

should be in your repertoire of commands as well. Alternatively, you can use the –reboot option to tell rpm-ostree to reboot when the upgrade command completes.


The other essential command is flatpak. Where rpm-ostree controls your OS image, flatpak rules the applications. flatpak has many commands that are worth exploring, I’ll only mention the most important ones here.

It is quite common to have more than one source for flatpaks enabled.

flatpak remotes

lists them all. If you want to find applications, then

flatpak search

will do that for you, and

flatpak install

will let you install what you found. An important detail to point out here is that applications can be installed in system-wide (in /var) or per-user (in ~/.local/share). You can choose the location with the –user and  –system options. If you choose to install system-wide, you will get a polkit prompt, since this is a privileged operation.

After installing applications, you should keep them up-to-date by installing updates. The most straightforward way to so is to just run

flatpak update

which will install available updates for all applications. To just check if updates are available, you can use

flatpak remote-ls --updates
Launching applications

Probably the most important thing you will want to do with flatpak is to run applications. Unsurprisingly, the command to do so is called run, and it expects you to specify the unique application ID:

flatpak run org.gnome.gitg

This is certainly a departure from the traditional commandline, and could be considered cumbersome (even though it has bash completion for the application ID).

Thankfully, flatpak has recently gained a way to recover the familiar interface. It now installs shell wrappers for the flatpak run command in ~/.local/share/flatpak/bin. After adding that directory to your PATH, you can run gitg like this:


If (like me), you are still not satisfied with this, you can add a shell alias to get the traditional command name back:

alias gitg=org.gnome.gitg

Now gitg works again, as it used to. Nice!


Fedora Atomic Workstation: Trying things out the easy way

Note: Fedora Atomic Workstation has recently been renamed to Team Silverblue. Learn more here.

If you’ve followed my posts about my first steps with Fedora Atomic Workstation, you may have seen that I’ve had to figure out how to make codecs work in the firefox that is included in the OS image. And then I had to work around some issues with the layered packages that I used for that when I rebased to a newer OS.

But maybe there is a better way to go about this?

Flatpak is the preferred way to install applications on the Atomic Workstation, so maybe that is the way to go. So far, all the flatpaks I’ve installed have come from Flathub, which is a convenient place to find all sorts of desktop apps. But firefox is not there (yet).

Thankfully, flatpak is fundamentally decentralized by design. There is nothing special about Flathub, flatpak can easily install apps from many different sources. A quick search yielded this unofficial firefox flatpak repository, with easy-to-follow instructions. Just clicking on this link will also work. If you open it with a sandboxed firefox, you will see the URI portal in action:

One nice aspect of this is that the nightly firefox is not just a one-off package that I’ve installed, it will receive updates just like any other app on my system. An important detail when it comes to web browsers!

Another nice aspect is that I did not have to remove the firefox that is already installed (I couldn’t anyway, since it is part of the immutable OS image). The two can peacefully coexist.

This is true not just for firefox, but in general: by isolating apps and their dependencies from each other, flatpak makes it very easy to try out different versions of apps without pain.

Want to try some bleeding-edge GNOME apps instead of the stable versions that are available on Flathub? No problem,  just enable the GNOME nightly repository like this:

flatpak remote-add --user --from gnome-apps-nightly

And then the nightly apps will be available for installation next to their stable counterparts. Note that GNOME software will indicate the source for each app in the search results.

Trying out new software has never been safer and easier!

Sorry for the wrong-colored screenshots in this post – there is still a price to pay for living on the bleeding edge of rawhide, even if Atomic Workstation makes it much safer.

Recipes hackfest, day 1

It has been a bit quiet around GNOME recipes recently, since most of us have other obligations. But this is about to change; we’re currently having a hackfest about GNOME recipes in Jogyakarta, Indonesia, and we’ve already made some interesting plans for for future work in this app.

The hackfest takes place at the AMIKOM university in Jogyakarta.


Before the event started, we had an outreach day where our group met some of the students and talked to them about GNOME and what we do. I was unfortunately not able to make it in time for that part, but I hear that it was a good day.


The hackfest itself started on Wednesday. We collected a list of goals and tasks, and started the day with a walkthrough/review of both GNOME recipes and the Endless cooking app.


A good part of the first day was spent discussing the storage layer of recipes. What we have now is very home-grown and primitive. I was hoping for somebody to show up and write a proper storage layer for me. That hasn’t happened …until now.

The outcome of our discussion is a plan to use the technologies from the Endless knowledge libs to store recipes and other data. Some adaptations will be necessary since most Endless knowledge apps are basically readonly, in contrast to GNOME recipes, which has a strong focus on creating and editing recipes. But we have a fairly concrete plan for what needs to happen.

And to not lose the momentum, we’ve set ourselves a goal to present on this at Guadec. No pressure!


The other big topic we tackled on the first day is encouraging more recipe contributions. We had a good start last year, when we collected around 60 recipes, but it would be good to have some more. We’ve come up with several ideas for this.

Local recipes

One idea is that we will start accepting recipes in languages other than English, since translating a recipe into English is a big extra hurdle that makes contributing harder.

There were several fascinating tangents in this discussion. One thing we discussed at some length is the challenges of translating recipes. To do a good job at that takes a lot more effort than just translating the words; you may have to substitute ingredients for what is available in other parts of the world, and find out about local variations of ingredients, and so on. And to add an extra twist to this, the translation here will go from a local language to English, which is the opposite of what our translation teams normally do.

The exact form that this will take is not quite decided yet. We may end up introducing a concept like “recipe packs”, say: “Indonesian recipes”, and those might only be available in a local language.

A contest

Another idea for contributions is to have a contest for contributing recipes. The details will have to be worked out, but the prizes could include a prominent spot in the ‘featured chefs’ section, or having your recipe featured on the front page.

Cook it!

We can do a contest online, but since this is about cooking and food, it might also be very nice to do this as a local get-together. This could not only include entering recipes together, but also cooking them, taking photos, and eating them. We will try this idea in one or two locations.

A hackfest for food – maybe that’s a cookfest ?

We ended this very productive first day with a very pleasant dinner.  On to day 2!


Fedora Atomic Workstation on the road

Note: Fedora Atomic Workstation has recently been renamed to Team Silverblue. Learn more here.

I am currently travelling with my laptop, after I’ve recently switched it to Fedora Atomic Workstation and then rebased to rawhide.

While I am sitting by the pool, I have some time to ponder: how has it been going ?

I have upgraded to a newer image a few times now, and I won’t lie: rawhide is still rawhide. Sometimes, the new image was a disappointment and would not give me a working system. In one case, it hung before getting to the login screen. The typical rawhide experience.

In the past, hitting such a bad compose meant a painful struggle to recover: Boot into single user-mode, do some live debugging, try to find a few problematic packages to downgrade, work around breakage, etc. You end up with a half-working system in a mixed state, and hope that the next update will get you back on track.

An ostree-based system like Fedora Atomic Workstation makes the recovery painless. In the case I mentioned, I simply rebooted, selected my previous, working image from the grub menu, and was back to a working system in 3 minutes. No time lost.

The tree that didn’t work is still present on my system. If I wanted, I could make changes to it in an attempt to understand and fix the problem and try booting it again. Having both trees on the system is useful even if I just want to report the problem, since rpm-ostree can tell me what exactly changed between the working and the broken tree:

$rpm-ostree db diff d15a65950972 21ac24694b6e
ostree diff commit old: d15a65950972
ostree diff commit new: 21ac24694b6e
 apr 1.6.3-5.fc28 -> 1.6.3-6.fc29
 atomic 1.21.1-1.fc28 -> 1.22.1-1.fc29

(I found the checksums to pass to rpm-ostree db diff by looking for the Commit fields in the output of rpm-ostree status)

Or I can simply wait for the next compose to see if rawhide ‘fixed itself’. Even if it takes more than one compose before things get back to working, this is a safe thing to do: ostree will never replace the current image that I am booted into. This makes it entirely safe to try newer composes until I find one that works for me.

It would of course be nice if broken composes never made it to my system. The current efforts at ‘gating’ updates in Fedora should get us to a place where we detect non-booting composes before they get sent out to users. But even in this (hopefully not too distant) future, the easy rollback with Atomic Workstation provides a useful safety net.

Automated QA will never be able to detect every problem that could affect my workflow. Maybe the system boots fine, but frequently loses network connections, or maybe it can’t handle my dual-monitor setup, or something else… I may only discover the problem hours later. But up until I run rpm-ostree upgrade again, I can just reboot back into my previous image.

ostree provides fearless upgrades.

Just right for the poolside.

Fedora Atomic Workstation: Works on the beach

Note: Fedora Atomic Workstation has recently been renamed to Team Silverblue. Learn more here.

My trip is getting really close, so I decided to upgrade my system to rawhide. Wait, what ? That is usually what everybody would tell you not to do. Rawhide has this reputation for frequent breakage, and who knows if my apps will work any given day. Not something you want to deal with while traveling.

Image result for project atomic logo

With rpm-ostree, installing a newer OS is very similar to updating your current OS: a new image is downloaded in the background, and when the download is complete, you boot into the new image. The previous image is still available to boot back into, as a safety net.  That is the reason that I felt confident enough to try this a day before a major trip:

rpm-ostree rebase \
systemctl reboot

I would love to say that things went perfectly and I was back to a working system in 10 minutes. But it was not quite as easy, and i did encounter a few (solvable) problems. It is worth pointing out that while I was solving these problems, rpm-ostree had already downloaded all of the rawhide image, but I was still safely running my F27 OS. At no point was there a mess of a half-upgraded system with a mix of old and new rpms. I was running my old system until I had solved all the problems and had a OS image that was ready, and then I booted into it. A safe, atomic switch.

Problem 1: The rpmfusion repo is not available for f28 yet. It is a common occurrence that 3rd party repositories lag behind the Fedora releases a bit, so this is not surprising. It is a bit unfortunate that i had to remove my layered rpms from the repository to work around this.

Problem 2: buildah now in the base image. This is a good thing, of course, but it caused rpm-ostree to complain about the conflict between the OS image and the layered package. In this case, I removed the layered rpm without any qualms.

Problem 3: Rawhide repositories had a bad day. For some reason,  they were missing the repomd.xml file today.

This is a good reminder that as long as you are using package layering, you haven’t really left the world of yum repositories and out-of-sync mirrors behind. rpm-ostree has to check the yum repositories for updates to the layered packages, which means that it can be hit by the same issues as dnf on a traditional Fedora workstation.

For my rebase to proceed,  I had to remove everything that was layered on top of the OS image. After I did that, rpm-ostree no longer needed to look at yum repositories, and switched my system to the already-downloaded rawhide image.

After the reboot, I’m now running rawhide… and all my applications are just the same as they were before. A nice aspect of the Atomic Workstation approach is that (flatpak) applications are decoupled from the OS. I can update the one without the other. We are not entirely there yet: as you can see in the screenshot below, a number of applications are still installed as part of the OS image.

More importantly, the screenshot shows that GNOME software will support updating the OS on the Atomic Workstation in Fedora 28. It does so by talking to the rpm-ostree daemon.

Switching from one Fedora release to the next ls already working pretty well in the last few releases. With the Atomic Workstation, it can become as undramatic as installing the latest updates.

One could almost do it on the beach.

Fedora Atomic Workstation for development

Note: Fedora Atomic Workstation has recently been renamed to Team Silverblue. Learn more here.

I’m frequently building GTK+.  Since I am using Fedora Atomic Workstation now, i have to figure out how to do GTK+ development in this new environment. GTK+ may be a good example for the big middle ground of things that are not desktop applications, but also not part of the OS itself.

Image result for project atomic logo

Last week I figured out how to use a buildah container to build release tarballs for GNOME modules, and I actually used that setup to produce a GTK+ release as well.

But for fixing bugs and other development, I generally need to run test cases and demo apps, like the venerable gtk-demo. Running these outside the container does not work, since the GTK+ libraries I built are linked against libraries that are installed inside the container and not present on the host, such as libvulkan. I could of course resort to package layering to install them on the host, but that would miss the point of using Atomic Workstation.

The alternative is running the demo apps inside the container, which should work – its the same filesystem that they were built in. But they can’t talk to the compositor, since the Wayland socket is on the outside: /run/user/1000/wayland-0. I tried to work around this by making the socket visible in the container, but my knowledge of container tools and buildah is too limited to make it work. My apps still complain about not being able to open a display connection.

What now ? I decided that while GTK+ is not a desktop application, I can treat my test apps like one and write a flatpak manifest for them. This way, I can use GNOME builders awesome flatpak support to build and run them, like I already did for  GNOME recipes.

Here is a minimal  flatpak manifest that works:

  "id" : "org.gtk.gtk-demo",
  "runtime" : "org.gnome.Sdk",
  "runtime-version" : "master",
  "sdk" : "org.gnome.Sdk",
  "command" : "gtk4-demo",
  "finish-args" : [
  "modules" : [
     "name" : "graphene",
     "buildsystem" : "meson",
     "builddir" : true,
     "sources" : [
         "type" : "git",
         "url" : ""
     "name" : "gtk+",
     "buildsystem" : "meson",
     "builddir" : true,
     "sources" : [
         "type" : "git",
         "url" : ""

After placing this json file into the toplevel directory of my  GTK+ checkout, it appears as a new build configuration in GNOME builder:

If you look closely, you’ll notice that I added another manifest, for gtk4-widget-factory. You can have multiple manifests in your tree, and GNOME builder will let you switch between them in the Build Preferences.

After all this preparation, I can now hit the play button and have my demo app run right from inside GNOME builder. Note that the application is running inside a flatpak sandbox, using the runtime that was specified in the Build Preferences, so it is cleanly separated from the OS. And I can easily build and run against different runtimes, to test compatibility with older GNOME releases.

This may be the final push that makes me switch to GNOME Builder for day-to-day development on Fedora Atomic Workstation: It just works!

Fedora Atomic Workstation: Building flatpaks

Note: Fedora Atomic Workstation has recently been renamed to Team Silverblue. Learn more here.

In order to use my new Atomic Workstation for real, I need to be able to build things locally,  including creating flatpaks.

Image result for project atomic logo

One of the best tools for the  job (building flatpaks) is GNOME builder. I had already installed the stable build from flathub, but Christian told me that the nightly build is way better for flatpak building, so I went to install it from here.

Getting GNOME Builder

This highlights one of the nice aspects of flatpak: it is fundamentally decentralized. While flathub serves as a convenient one-stop-shop for many apps, it is entirely possible to have other remotes. Flathub is not privileged at all.

It is also perfectly possible to have both the stable gnome-builder from flathub and the nightly installed at the same time.

The only limitation is that only one of them will get to be presented as ‘the’ GNOME Builder by the desktop, since they use the same app id.  You can change between the installed versions of an application using the flatpak cli:

flatpak make-current --user org.gnome.Builder master

Building flatpaks

Now on to building flatpaks! Naturally, my testcase is GNOME Recipes. I have a git checkout of it, so I proceeded to open it in GNOME Builder, started a build … and it failed, with a somewhat cryptic error message about chdir() failing :-(

After quite a bit of head-scratching and debugging, we determined that this happens because flatpak is doing builds in a sandbox as well, and it is replacing /var with its own bind mount to do so. This creates a bit of confusion with the /home -> /var/home symlink that is part of the Atomic Workstation image. We are still trying to determine the best fix for this, you can follow along in this issue.

Since I am going to travel soon, I can’t wait for the official fix, so I came up with a workaround: Remove the /home -> /var/home symlink, create a regular /home directory in its place, and change /etc/fstab to mount my home partition there instead of /var/home. One reason why this is ugly is that I am modifying the supposedly immutable OS image. How ? By removing the immutable attribute with chattr -i /.  Another reason why it is ugly is that this has to be repeated everytime a new image gets installed (regardless whether it is via an update or via package layering).

But, with this workaround in place, there is no longer a troublesome symlink to cause trouble for flatpak, and my build succeeds. Once it is built, I can run the recipes flatpak with one click on the play button in builder.

Neat! I am almost ready to take Atomic Workstation on the road.

Fedora Atomic Workstation: What about apps ?

Note: Fedora Atomic Workstation has recently been renamed to Team Silverblue. Learn more here.

I recently switched my main system to Fedora Atomic Workstation, and described my initial experience here. But I am going to travel soon, so I won’t have much time to fiddle with my laptop, and need to get things into working order.

Image result for project atomic logoConnections

One thing I needed to investigate is getting my VPN connections over from the old system. After a bit of consideration, I decided that it was easiest to just copy the relevant files from the old installation – /etc is not part of the immutable OS image, so this works just fine. I booted back into the old system and:

cp /etc/NetworkManager/system-connections/* /ostree/boot-1/.../etc/NetworkManager/system-connections

Note that you may also have to copy certificates over in a similar way.


But the bigger task for getting this system into working order is, of course, getting the applications back.

I could of course just use rpm-ostree’s layering and install fedora rpms for many apps. But, that would mean sliding back into the old world where applications are part of the OS, and dependencies force the OS and the apps to be updated together, etc. Since I want to explore the proper workflows and advantages of the Atomic model, I’ll instead try to install the apps separately from the OS, as flatpaks.

Currently, the best way to get flatpaks is to use flathub, so i went back there to see if I can find all I need. flathub has a bit more than 200 applications currently. That may not seem much, compared to the Android playstore, but lets see whats actually there:

With  Telegram, Spotify, Gimp, LibreOffice and some others, I find most of what I need frequently. And Skype, Slack, Inkscape, Blender and others are there too. Not bad.


But what about web browsers? firefox is included in the atomic workstation image. To make it play media, you have to do the same things as on the traditional workstation – find an ffmpeg package and use rpm layering to make it part of the image.

chrome is unfortunately hard to package as a flatpak, since its own sandboxing technology conflicts with the sandboxing that is applied by flatpak. There is of course a chrome rpm, but it installs into /opt, which is not currently supported by rpm-ostree. See this issue for more details and possible solutions.

Beyond the major browsers, there’s some other choices available in flathub, such as GNOME Web or Eolie. These browsers use gstreamer for multimedia support, so they will pick up codecs that are available on the host system via the gstreamer runtime extension.

Next steps

The trip I’m going on is for a hackfest that will focus on an application (GNOME Recipes, in fact), so I will need a well-working setup for building flatpaks locally.

I’ll try out how well GNOME Builder handles this task on an Atomic System.

First steps with Fedora Atomic Workstation

Note: Fedora Atomic Workstation has recently been renamed to Team Silverblue. Learn more here.

There’s been a lot of attention for the Fedora Atomic Workstation project recently, with several presentations at devconf (Kalev LemberColin Walters, Jonathan Lebon) and fosdem (Sanja Bonic), blog posts and other docs.

Image result for project atomic logoI’ve played with the Atomic Workstation before, but it was always in a VM. That is a low-risk way to try it out, but the downside is that you can jump back to your ‘normal’ system at the first problem… which, naturally,  I did. The recent attention inspired me to try again.

This time, I wanted to try it for real and get some actual work done on the Atomic side. So this morning, I set out to convert my main system to Atomic Workstation. The goal I’ve set myself for today was to create a gnome-font-viewer release tarball using a container-based workflow.

There are two ways to install Atomic Workstation. You can either download an .iso and install from scratch, or you can convert an existing system. I chose the second option, following these instructions.  By and large, the instructions were accurate and led me to a successful installation. A few notes:

  • You need ~10G of free space on your root filesystem
  • I got server connection errors several time – just restarting the ostree pull command will eventually let it complete
  • The instructions recommend copying grub.cfg from /boot/loader to /boot/grub2/, but that only works for the current tree – if you install updates or add a layer to your ostree image, you have to repeat it. An easier solution is to create a symlink instead.

After a moment of fear, I decided to reboot, and found myself inside the Atomic Workstation – it just worked. After opening a terminal and finding my git checkouts, I felt a little helpless – none of git, gitg, gcc (or many other of the developer tools I’m used to) are around. What now ?

Thankfully, firefox was available, so I went to and installed gitg as a flatpak, with a single click.

For the other developer tools, remember that my goal was to use a container-based workflow, so my next step was to install buildah, which is a tool to work with containers without the need for docker.   Installing the buildah rpm on Atomic Workstation feels a bit like a magic trick – after all, isn’t this an immutable image-based OS ?

What happens when you run

rpm-ostree install buildah

is that rpm-ostree is composing a new image by layering the rpm on top of the existing image. As expected, I had to reboot into the new image to see the newly installed tool.

Next, I tried to figure out some of the basics of working with buildah – here is a brief introduction to buildah that i found helpful. After creating and starting a Fedora-based container with

buildah from fedora
buildah run fedora-working-container bash

I could use dnf to install git, gcc and a few other things in the container. So far, so good.  But in order to make a gnome-font-viewer release, there is still one thing missing: I need access to my git checkout inside the container.  After some browsing around, I came up with this command:

buildah run -v /srv:/srv:rslave fedora-working-container bash

which should make /srv from the host system appear inside the container. And… i was stuck – trying to enumerate the contents of /src in the container was giving me permission errors, despite running as root.

Eventually, it dawned on me that selinux is to blame… The command

sudo chcon -R -h -t container_file_t /srv

is needed to make things work as expected. Alternatively, you could set selinux to be permissive.

From here on, things were pretty straightforward. I additionally needed to make my ssh keys available so I could push my commits from inside the container, and I needed a series of dnf commands to make enough build dependencies and tools available:

dnf install  git
dnf install meson
dnf install gtk3-devel

But eventually,

meson . build
ninja -Cbuild
ninja -Cbuild dist

worked and produced this tarball – success!

So, when you try gnome-font-viewer 3.27.90 remember: it was produced in a container.

The first steps are always the hardest. I expect things to get easier as I learn more about this way of working.


GTK+ hackfest, day 2

The second day of the GTK+ hackfest in Brussels started with an hour of patch review. We then went through scattered items from the agenda and collected answers to some questions.

We were lucky to have some dear friends join us for part of the day.  Allison came by for an extended GLib bug review session with Philip, and Adrien discussed strategies for dealing with small and changing form factors with us.  Allison, Adrien: thanks for coming!

The bulk of the day was taking up by a line-by-line review of our GTK+ 4 task list. Even though some new items appeared, it got a bit shorter, and many of the outstanding ones are much clearer now.

Our (jumbled and unedited) notes from the day are here.

The day ended with a nice dinner that was graciously sponsored by Purism. Thanks!

Decisions, decisions

We discussed too many things during these two days for a concise summary of every result, but here are some of the highlights:

  • gitlab migration: We want to migrate the GTK+ git repository as soon as possible. The bug migration needs preparation and will follow later
  • GTK+ 4 roadmap: We are aiming for an initial release in the fall of this year. We’ll reevaluate this target date at GUADEC