It has been a bit quiet around GNOME recipes recently, since most of us have other obligations. But this is about to change; we’re currently having a hackfest about GNOME recipes in Jogyakarta, Indonesia, and we’ve already made some interesting plans for for future work in this app.

The hackfest takes place at the AMIKOM university in Jogyakarta.

Outreach

Before the event started, we had an outreach day where our group met some of the students and talked to them about GNOME and what we do. I was unfortunately not able to make it in time for that part, but I hear that it was a good day.

Review

The hackfest itself started on Wednesday. We collected a list of goals and tasks, and started the day with a walkthrough/review of both GNOME recipes and the Endless cooking app.

Storage

A good part of the first day was spent discussing the storage layer of recipes. What we have now is very home-grown and primitive. I was hoping for somebody to show up and write a proper storage layer for me. That hasn’t happened …until now.

The outcome of our discussion is a plan to use the technologies from the Endless knowledge libs to store recipes and other data. Some adaptations will be necessary since most Endless knowledge apps are basically readonly, in contrast to GNOME recipes, which has a strong focus on creating and editing recipes. But we have a fairly concrete plan for what needs to happen.

And to not lose the momentum, we’ve set ourselves a goal to present on this at Guadec. No pressure!

Contributions

The other big topic we tackled on the first day is encouraging more recipe contributions. We had a good start last year, when we collected around 60 recipes, but it would be good to have some more. We’ve come up with several ideas for this.

Local recipes

One idea is that we will start accepting recipes in languages other than English, since translating a recipe into English is a big extra hurdle that makes contributing harder.

There were several fascinating tangents in this discussion. One thing we discussed at some length is the challenges of translating recipes. To do a good job at that takes a lot more effort than just translating the words; you may have to substitute ingredients for what is available in other parts of the world, and find out about local variations of ingredients, and so on. And to add an extra twist to this, the translation here will go from a local language to English, which is the opposite of what our translation teams normally do.

The exact form that this will take is not quite decided yet. We may end up introducing a concept like “recipe packs”, say: “Indonesian recipes”, and those might only be available in a local language.

A contest

Another idea for contributions is to have a contest for contributing recipes. The details will have to be worked out, but the prizes could include a prominent spot in the ‘featured chefs’ section, or having your recipe featured on the front page.

Cook it!

We can do a contest online, but since this is about cooking and food, it might also be very nice to do this as a local get-together. This could not only include entering recipes together, but also cooking them, taking photos, and eating them. We will try this idea in one or two locations.

A hackfest for food – maybe that’s a cookfest ?

We ended this very productive first day with a very pleasant dinner.  On to day 2!

Note: Fedora Atomic Workstation has recently been renamed to Team Silverblue. Learn more here.

I am currently travelling with my laptop, after I’ve recently switched it to Fedora Atomic Workstation and then rebased to rawhide.

While I am sitting by the pool, I have some time to ponder: how has it been going ?

I have upgraded to a newer image a few times now, and I won’t lie: rawhide is still rawhide. Sometimes, the new image was a disappointment and would not give me a working system. In one case, it hung before getting to the login screen. The typical rawhide experience.

In the past, hitting such a bad compose meant a painful struggle to recover: Boot into single user-mode, do some live debugging, try to find a few problematic packages to downgrade, work around breakage, etc. You end up with a half-working system in a mixed state, and hope that the next update will get you back on track.

An ostree-based system like Fedora Atomic Workstation makes the recovery painless. In the case I mentioned, I simply rebooted, selected my previous, working image from the grub menu, and was back to a working system in 3 minutes. No time lost.

The tree that didn’t work is still present on my system. If I wanted, I could make changes to it in an attempt to understand and fix the problem and try booting it again. Having both trees on the system is useful even if I just want to report the problem, since rpm-ostree can tell me what exactly changed between the working and the broken tree:

$rpm-ostree db diff d15a65950972 21ac24694b6e
ostree diff commit old: d15a65950972
ostree diff commit new: 21ac24694b6e
Upgraded:
 apr 1.6.3-5.fc28 -> 1.6.3-6.fc29
 atomic 1.21.1-1.fc28 -> 1.22.1-1.fc29
 ...
Removed:
 NetworkManager-glib-1:1.10.2-1.fc28.x86_64
 libnm-gtk-1.8.10-2.fc28.2.x86_64
 ...
Added:
 authselect-0.3-3.fc29.x86_64
 authselect-libs-0.3-3.fc29.x86_64
 ...

(I found the checksums to pass to rpm-ostree db diff by looking for the Commit fields in the output of rpm-ostree status)

Or I can simply wait for the next compose to see if rawhide ‘fixed itself’. Even if it takes more than one compose before things get back to working, this is a safe thing to do: ostree will never replace the current image that I am booted into. This makes it entirely safe to try newer composes until I find one that works for me.

It would of course be nice if broken composes never made it to my system. The current efforts at ‘gating’ updates in Fedora should get us to a place where we detect non-booting composes before they get sent out to users. But even in this (hopefully not too distant) future, the easy rollback with Atomic Workstation provides a useful safety net.

Automated QA will never be able to detect every problem that could affect my workflow. Maybe the system boots fine, but frequently loses network connections, or maybe it can’t handle my dual-monitor setup, or something else… I may only discover the problem hours later. But up until I run rpm-ostree upgrade again, I can just reboot back into my previous image.

ostree provides fearless upgrades.

Just right for the poolside.

Note: Fedora Atomic Workstation has recently been renamed to Team Silverblue. Learn more here.

My trip is getting really close, so I decided to upgrade my system to rawhide. Wait, what ? That is usually what everybody would tell you not to do. Rawhide has this reputation for frequent breakage, and who knows if my apps will work any given day. Not something you want to deal with while traveling.

With rpm-ostree, installing a newer OS is very similar to updating your current OS: a new image is downloaded in the background, and when the download is complete, you boot into the new image. The previous image is still available to boot back into, as a safety net.  That is the reason that I felt confident enough to try this a day before a major trip:

rpm-ostree rebase \
   fedora-ws-rawhide:fedora/rawhide/x86_64/workstation
systemctl reboot

I would love to say that things went perfectly and I was back to a working system in 10 minutes. But it was not quite as easy, and i did encounter a few (solvable) problems. It is worth pointing out that while I was solving these problems, rpm-ostree had already downloaded all of the rawhide image, but I was still safely running my F27 OS. At no point was there a mess of a half-upgraded system with a mix of old and new rpms. I was running my old system until I had solved all the problems and had a OS image that was ready, and then I booted into it. A safe, atomic switch.

Problem 1: The rpmfusion repo is not available for f28 yet. It is a common occurrence that 3rd party repositories lag behind the Fedora releases a bit, so this is not surprising. It is a bit unfortunate that i had to remove my layered rpms from the repository to work around this.

Problem 2: buildah now in the base image. This is a good thing, of course, but it caused rpm-ostree to complain about the conflict between the OS image and the layered package. In this case, I removed the layered rpm without any qualms.

Problem 3: Rawhide repositories had a bad day. For some reason,  they were missing the repomd.xml file today.

This is a good reminder that as long as you are using package layering, you haven’t really left the world of yum repositories and out-of-sync mirrors behind. rpm-ostree has to check the yum repositories for updates to the layered packages, which means that it can be hit by the same issues as dnf on a traditional Fedora workstation.

For my rebase to proceed,  I had to remove everything that was layered on top of the OS image. After I did that, rpm-ostree no longer needed to look at yum repositories, and switched my system to the already-downloaded rawhide image.

After the reboot, I’m now running rawhide… and all my applications are just the same as they were before. A nice aspect of the Atomic Workstation approach is that (flatpak) applications are decoupled from the OS. I can update the one without the other. We are not entirely there yet: as you can see in the screenshot below, a number of applications are still installed as part of the OS image.

More importantly, the screenshot shows that GNOME software will support updating the OS on the Atomic Workstation in Fedora 28. It does so by talking to the rpm-ostree daemon.

Switching from one Fedora release to the next ls already working pretty well in the last few releases. With the Atomic Workstation, it can become as undramatic as installing the latest updates.

One could almost do it on the beach.

Note: Fedora Atomic Workstation has recently been renamed to Team Silverblue. Learn more here.

I’m frequently building GTK+.  Since I am using Fedora Atomic Workstation now, i have to figure out how to do GTK+ development in this new environment. GTK+ may be a good example for the big middle ground of things that are not desktop applications, but also not part of the OS itself.

Last week I figured out how to use a buildah container to build release tarballs for GNOME modules, and I actually used that setup to produce a GTK+ release as well.

But for fixing bugs and other development, I generally need to run test cases and demo apps, like the venerable gtk-demo. Running these outside the container does not work, since the GTK+ libraries I built are linked against libraries that are installed inside the container and not present on the host, such as libvulkan. I could of course resort to package layering to install them on the host, but that would miss the point of using Atomic Workstation.

The alternative is running the demo apps inside the container, which should work – its the same filesystem that they were built in. But they can’t talk to the compositor, since the Wayland socket is on the outside: /run/user/1000/wayland-0. I tried to work around this by making the socket visible in the container, but my knowledge of container tools and buildah is too limited to make it work. My apps still complain about not being able to open a display connection.

What now ? I decided that while GTK+ is not a desktop application, I can treat my test apps like one and write a flatpak manifest for them. This way, I can use GNOME builders awesome flatpak support to build and run them, like I already did for  GNOME recipes.

Here is a minimal  flatpak manifest that works:

{
  "id" : "org.gtk.gtk-demo",
  "runtime" : "org.gnome.Sdk",
  "runtime-version" : "master",
  "sdk" : "org.gnome.Sdk",
  "command" : "gtk4-demo",
  "finish-args" : [
    "--socket=wayland"
  ],
  "modules" : [
   {
     "name" : "graphene",
     "buildsystem" : "meson",
     "builddir" : true,
     "sources" : [
       {
         "type" : "git",
         "url" : "https://github.com/ebassi/graphene.git"
       }
     ]
   },
   {
     "name" : "gtk+",
     "buildsystem" : "meson",
     "builddir" : true,
     "sources" : [
       {
         "type" : "git",
         "url" : "https://gitlab.gnome.org/GNOME/gtk.git"
       }
     ]
   }
 ]
}

After placing this json file into the toplevel directory of my  GTK+ checkout, it appears as a new build configuration in GNOME builder:

If you look closely, you’ll notice that I added another manifest, for gtk4-widget-factory. You can have multiple manifests in your tree, and GNOME builder will let you switch between them in the Build Preferences.

After all this preparation, I can now hit the play button and have my demo app run right from inside GNOME builder. Note that the application is running inside a flatpak sandbox, using the runtime that was specified in the Build Preferences, so it is cleanly separated from the OS. And I can easily build and run against different runtimes, to test compatibility with older GNOME releases.

This may be the final push that makes me switch to GNOME Builder for day-to-day development on Fedora Atomic Workstation: It just works!

Note: Fedora Atomic Workstation has recently been renamed to Team Silverblue. Learn more here.

In order to use my new Atomic Workstation for real, I need to be able to build things locally,  including creating flatpaks.

One of the best tools for the  job (building flatpaks) is GNOME builder. I had already installed the stable build from flathub, but Christian told me that the nightly build is way better for flatpak building, so I went to install it from here.

Getting GNOME Builder

This highlights one of the nice aspects of flatpak: it is fundamentally decentralized. While flathub serves as a convenient one-stop-shop for many apps, it is entirely possible to have other remotes. Flathub is not privileged at all.

It is also perfectly possible to have both the stable gnome-builder from flathub and the nightly installed at the same time.

The only limitation is that only one of them will get to be presented as ‘the’ GNOME Builder by the desktop, since they use the same app id.  You can change between the installed versions of an application using the flatpak cli:

flatpak make-current --user org.gnome.Builder master

Building flatpaks

Now on to building flatpaks! Naturally, my testcase is GNOME Recipes. I have a git checkout of it, so I proceeded to open it in GNOME Builder, started a build … and it failed, with a somewhat cryptic error message about chdir() failing 🙁

After quite a bit of head-scratching and debugging, we determined that this happens because flatpak is doing builds in a sandbox as well, and it is replacing /var with its own bind mount to do so. This creates a bit of confusion with the /home -> /var/home symlink that is part of the Atomic Workstation image. We are still trying to determine the best fix for this, you can follow along in this issue.

Since I am going to travel soon, I can’t wait for the official fix, so I came up with a workaround: Remove the /home -> /var/home symlink, create a regular /home directory in its place, and change /etc/fstab to mount my home partition there instead of /var/home. One reason why this is ugly is that I am modifying the supposedly immutable OS image. How ? By removing the immutable attribute with chattr -i /.  Another reason why it is ugly is that this has to be repeated everytime a new image gets installed (regardless whether it is via an update or via package layering).

But, with this workaround in place, there is no longer a troublesome symlink to cause trouble for flatpak, and my build succeeds. Once it is built, I can run the recipes flatpak with one click on the play button in builder.

Neat! I am almost ready to take Atomic Workstation on the road.

Note: Fedora Atomic Workstation has recently been renamed to Team Silverblue. Learn more here.

I recently switched my main system to Fedora Atomic Workstation, and described my initial experience here. But I am going to travel soon, so I won’t have much time to fiddle with my laptop, and need to get things into working order.

Connections

One thing I needed to investigate is getting my VPN connections over from the old system. After a bit of consideration, I decided that it was easiest to just copy the relevant files from the old installation – /etc is not part of the immutable OS image, so this works just fine. I booted back into the old system and:

cp /etc/NetworkManager/system-connections/* /ostree/boot-1/.../etc/NetworkManager/system-connections

Note that you may also have to copy certificates over in a similar way.

Applications

But the bigger task for getting this system into working order is, of course, getting the applications back.

I could of course just use rpm-ostree’s layering and install fedora rpms for many apps. But, that would mean sliding back into the old world where applications are part of the OS, and dependencies force the OS and the apps to be updated together, etc. Since I want to explore the proper workflows and advantages of the Atomic model, I’ll instead try to install the apps separately from the OS, as flatpaks.

Currently, the best way to get flatpaks is to use flathub, so i went back there to see if I can find all I need. flathub has a bit more than 200 applications currently. That may not seem much, compared to the Android playstore, but lets see whats actually there:

With  Telegram, Spotify, Gimp, LibreOffice and some others, I find most of what I need frequently. And Skype, Slack, Inkscape, Blender and others are there too. Not bad.

Browsers

But what about web browsers? firefox is included in the atomic workstation image. To make it play media, you have to do the same things as on the traditional workstation – find an ffmpeg package and use rpm layering to make it part of the image.

chrome is unfortunately hard to package as a flatpak, since its own sandboxing technology conflicts with the sandboxing that is applied by flatpak. There is of course a chrome rpm, but it installs into /opt, which is not currently supported by rpm-ostree. See this issue for more details and possible solutions.

Beyond the major browsers, there’s some other choices available in flathub, such as GNOME Web or Eolie. These browsers use gstreamer for multimedia support, so they will pick up codecs that are available on the host system via the gstreamer runtime extension.

Next steps

The trip I’m going on is for a hackfest that will focus on an application (GNOME Recipes, in fact), so I will need a well-working setup for building flatpaks locally.

I’ll try out how well GNOME Builder handles this task on an Atomic System.

Note: Fedora Atomic Workstation has recently been renamed to Team Silverblue. Learn more here.

There’s been a lot of attention for the Fedora Atomic Workstation project recently, with several presentations at devconf (Kalev Lember,  Colin Walters, Jonathan Lebon) and fosdem (Sanja Bonic), blog posts and other docs.

I’ve played with the Atomic Workstation before, but it was always in a VM. That is a low-risk way to try it out, but the downside is that you can jump back to your ‘normal’ system at the first problem… which, naturally,  I did. The recent attention inspired me to try again.

This time, I wanted to try it for real and get some actual work done on the Atomic side. So this morning, I set out to convert my main system to Atomic Workstation. The goal I’ve set myself for today was to create a gnome-font-viewer release tarball using a container-based workflow.

There are two ways to install Atomic Workstation. You can either download an .iso and install from scratch, or you can convert an existing system. I chose the second option, following these instructions.  By and large, the instructions were accurate and led me to a successful installation. A few notes:

• You need ~10G of free space on your root filesystem
• I got server connection errors several time – just restarting the ostree pull command will eventually let it complete
• The instructions recommend copying grub.cfg from /boot/loader to /boot/grub2/, but that only works for the current tree – if you install updates or add a layer to your ostree image, you have to repeat it. An easier solution is to create a symlink instead.

After a moment of fear, I decided to reboot, and found myself inside the Atomic Workstation – it just worked. After opening a terminal and finding my git checkouts, I felt a little helpless – none of git, gitg, gcc (or many other of the developer tools I’m used to) are around. What now ?

Thankfully, firefox was available, so I went to http://flathub.org and installed gitg as a flatpak, with a single click.

For the other developer tools, remember that my goal was to use a container-based workflow, so my next step was to install buildah, which is a tool to work with containers without the need for docker.   Installing the buildah rpm on Atomic Workstation feels a bit like a magic trick – after all, isn’t this an immutable image-based OS ?

What happens when you run

rpm-ostree install buildah

is that rpm-ostree is composing a new image by layering the rpm on top of the existing image. As expected, I had to reboot into the new image to see the newly installed tool.

Next, I tried to figure out some of the basics of working with buildah – here is a brief introduction to buildah that i found helpful. After creating and starting a Fedora-based container with

buildah from fedora
buildah run fedora-working-container bash

I could use dnf to install git, gcc and a few other things in the container. So far, so good.  But in order to make a gnome-font-viewer release, there is still one thing missing: I need access to my git checkout inside the container.  After some browsing around, I came up with this command:

buildah run -v /srv:/srv:rslave fedora-working-container bash

which should make /srv from the host system appear inside the container. And… i was stuck – trying to enumerate the contents of /src in the container was giving me permission errors, despite running as root.

Eventually, it dawned on me that selinux is to blame… The command

sudo chcon -R -h -t container_file_t /srv

is needed to make things work as expected. Alternatively, you could set selinux to be permissive.

From here on, things were pretty straightforward. I additionally needed to make my ssh keys available so I could push my commits from inside the container, and I needed a series of dnf commands to make enough build dependencies and tools available:

dnf install  git
dnf install meson
dnf install gtk3-devel
...

But eventually,

meson . build
ninja -Cbuild
ninja -Cbuild dist

worked and produced this tarball – success!

So, when you try gnome-font-viewer 3.27.90 remember: it was produced in a container.

The first steps are always the hardest. I expect things to get easier as I learn more about this way of working.

The second day of the GTK+ hackfest in Brussels started with an hour of patch review. We then went through scattered items from the agenda and collected answers to some questions.

We were lucky to have some dear friends join us for part of the day.  Allison came by for an extended GLib bug review session with Philip, and Adrien discussed strategies for dealing with small and changing form factors with us.  Allison, Adrien: thanks for coming!

The bulk of the day was taking up by a line-by-line review of our GTK+ 4 task list. Even though some new items appeared, it got a bit shorter, and many of the outstanding ones are much clearer now.

Our (jumbled and unedited) notes from the day are here.

The day ended with a nice dinner that was graciously sponsored by Purism. Thanks!

Decisions, decisions

We discussed too many things during these two days for a concise summary of every result, but here are some of the highlights:

• gitlab migration: We want to migrate the GTK+ git repository as soon as possible. The bug migration needs preparation and will follow later
• GTK+ 4 roadmap: We are aiming for an initial release in the fall of this year. We’ll reevaluate this target date at GUADEC

A number of GTK+ developers met today in Brussels for 2-day hackfest ahead of FOSDEM. Sadly, a few friends who we’d have loved to see couldn’t make it, but we still have enough of the core team together for a productive meeting.

We decided that it would be a good idea to start the day with ‘short overviews of rewritten subsystems’, to get everybody on the same page. The quick overviews turned out to take most of the day, but it was intermixed with a lot of very productive discussions and decisions.

My (jumbled and undedited) notes from the day are here.

At the end of the day, we’ve found a nice Vietnamese restaurant around the corner from the venue, and Shaun came by for food and beer.

I hope that day 2 of this short hackfest will be similarly productive!

Thanks to GNOME for sponsoring this event.