Fitting Endless OS images on small disks

Last week I read Jorge Castro’s article On “Wasting disk space” with interest, and not only because it cites one of my own articles 😉. Jorge encouraged me to write up the conversation we had off the back of it, so here we go!

People like to fixate on the disk space used by installing a calculator app as a Flatpak when you don’t have any other Flatpak apps installed. For example, on my system GNOME Calculator takes up 9.3 MB for itself, plus 803.1 MB for the GNOME 42 runtime it depends on. Regular readers will not be surprised when I say that that 803.1 MB figure looks rather different when you realise that Calculator is just one of 70 apps on my system that use that runtime; 11.5 MB of runtime per app feels a lot more reasonable.

But I do have one app installed which depends on the GNOME 3.34 runtime, which has been unsupported since August 2020, and the GNOME 3.34 runtime only shares 102 MB of its files with the GNOME 42 runtime, leaving 769 MB installed solely for this one 11 MB app. Not such a big deal on my laptop with a half-terabyte drive, but this gets to a point Jorge makes near the end of his article:

Yes, they take up more room, but it’s not by much, and unless you’re on an extremely size-restrained system (like say a 64GB Chromebook you are repurposing) then for most systems it’s a wash.

Part of the insight behind Endless OS is that it can be more practical and cost-effective to fill a large hard disk with apps and educational resources than to get access to high-bandwidth connectivity in remote or disadvantaged communities, and the devices we and our deployment partners use generally have plentiful storage. But inexpensive, lower-end devices with 64 GB storage are still quite common, and having runtimes installed for one or two apps consumes valuable space that could be used for more offline content. So this can be a real problem for us, if a small handful of apps are bloating an image to the point where it doesn’t fit, or leaves little space for documents & updates.

So, our OS image builder has a mode to tabulate the apps that will be preinstalled in a given image configuration, grouped by the runtime they use, with their approximate sizes. I added this mode when I was trying to make an Endless OS ISO fit on a 4.7 GB DVD a few years back, packing in as much content as possible. It’s a bit rough and ready, and tends to overestimate the disk space needed (because it does not take into account deduplication identical files between different apps and runtimes, or installing only a subset of translations for an app or runtime) but it is still useful to get a sense of where the space is going.

I ran this for the downloadable English configuration of Endless OS earlier today. I’ll spare you the full output here but if you are curious it’s in this gist. It shows that, like on my system, TurtleBlocks is the only reason the GNOME 3.34 runtime is preinstalled, and so removing that app or updating it to a more modern runtime would probably save somewhere between 1 GB and 2 GB in the image! Normally after seeing this I would go and see if I can take a few minutes to send an update to the app, but in this case someone has beaten me to it. (If you are a Python expert interested in block-based programming tools for kids, why not lend a hand getting this over the line?) It also shows that a couple of our unmaintained and sadly closed-source first-party apps, Resumé and My Budget, are stuck on an even more prehistoric runtime.

Running it on a deployment partner’s custom image that they reported is too big for the 64 GB target hardware showed that the Othman Quran browser is also using an older runtime; once more, someone else has already noticed in the last couple of days, and if you are an expert on GTK’s font selection your input would be welcome on that pull request.

This tool is how it came to pass that I updated the runtimes of gbrainy and Genius, two apps I have never used, last month, and Klavaro back in May, among others.

There tends to be a flurry of community activity every 6-12 months to update all the apps that depend on newly end-of-lifed runtimes, and many of these turn out not to be rocket science, though it is not (yet!) something that flatpak-external-data-checker can do automatically. Then we are left with a long tail of more lightly-maintained apps where the update is more cumbersome to do, as in the case of gbrainy, where it took a bit of staring at the unfamiliar error messages produced by the Mono C# compiler to figure out where the problem might lie. If you are good at debugging build failures, this kind of thing is a great way to contribute to Flathub; and if someone can remind me of the URL of the live-updating TODO list of apps on obsolete runtimes I once saw, I’ll add the link here.

While I’ve focused on one of the problems that apps depending on obsolete runtimes can cause, it’s not all bad news. If you really love, or need, an app that is abandoned or has not been updated in a while, the Flatpak model you can still install & use that app and its old runtime without your distribution having to keep around some obsolete version of the libraries, or you having to stay on an old version of the distro. As and when that app does get an update, the unused and end-of-lifed runtime on your system will be uninstalled automatically by modern versions of Flatpak.

How many Flathub apps reuse other package formats?

Today I read Comparison of Fedora Flatpaks and Flathub remotes by Hari Rana, who is an active and valued member of the Flatpak community. The article is a well-researched and well-written overview of how these two Flatpak ecosystems differ, and contains the following remark about one major difference (emphasis mine):

Flathub is open with what source a Flatpak application (re)uses, whereas Fedora Flatpaks strictly reuses the RPM format.

As such, Flathub has tons of applications that reuse other package formats.

When this article was discussed in the Flatpak Matrix channel, several people wondered whether “tons” is a fair assessment. Let’s find out!

The specific examples given in the article are of apps which reuse a .deb (to which I will add .rpm), AppImage, Snap package, or binary .tar.gz archive. It’s not so easy to distinguish a binary tarball from a source tarball, so as a substitute I will look for apps which use the extra-data to download external sources at install time rather than at build time.

I have cloned every repo from the Flathub GitHub organisation with this script I had lying around. There are 2,220 such repositories. This is a bigger number than the 1,518 apps cited in the blog post, because it includes many thing which are not apps, such as 258 GTK themes and 60 digital audio workstation plugins. I also believe that the 1,518 number does not include end-of-lifed apps, whereas my methodology does. This post will also ignore the existence of OBS Studio and Firefox, where those projects build the Flatpak from source on their own infrastructure and push the result into Flathub.

Now I’m just going to grep them all for the offending strings:

$ (for i in */
do
    if git -C $i grep --quiet -E '(\.(deb|rpm|AppImage|snap)\>)|(extra-data)'
    then
        echo $i
    fi
done) | wc -l
237

(Splitting apart the search terms, we have 141 repos matching .deb, 10 for .rpm, 23 for .AppImage, 6 for .snap, and 110 for extra-data. These numbers don’t sum to 237 because the same repo can use multiple formats, and these binary files are often used by extra-data apps.)

So by my back-of-an-envelope calculation, 237 out of 2220 repos on Flathub repackage other binary formats. This is a little under 11%. Of those 237, 51 are GTK themes, specifically variations of the Mint, Pop and Yaru themes. If we assume that all the other 186 are apps, and that none of them are EOLed, then 186 divided by 1,518 gives us a little more than 12% of apps on Flathub that are repackaged from other binary formats. (I believe this is a slight overestimate but I have run out of time this morning.)

Is that a big number? It’s roughly what I expected. Is it “ton[ne]s”? Compared to Fedora’s Flatpak repo, where everything is built from source, it certainly is: indeed, it’s more than the total number of apps in the Fedora Flatpak repo!

If it is valuable for Flathub to provide proprietary apps like Slack whose publishers do not currently wish to support Flatpak (which I believe it is) then it’s unavoidable that some apps repackage other binary formats. OK, time for one last bit of data: what if we exclude extra-data apps?

$ (for i in */
do
    if ! git -C $i grep --quiet extra-data && \
       git -C $i grep --quiet -E '\.(deb|rpm|AppImage|snap)\>'
    then
        echo $i
    fi
done )| wc -l
127

So (ignoring non-extra-data apps which use binary tarballs, if any such apps exist) that’s something like 76 apps and 51 GTK themes which probably could be built from source by Flathub, but aren’t. It may be hard to build some of these apps from source (perhaps the upstream build system requires network access) but the rewards would include support for aarch64 and any other architectures Flathub may add, and arguably greater transparency in how the app is built.

If you want to do your own research in this vein, you may be interested in gasinvein‘s Flatpak remote metadata fetcher, which would let you generate and analyse a 200 MiB JSON file rather than by cloning and grep-ing 4.6 GiB of Git repositories. His analysis using this data yields 174 apps, quite close to my 186 estimate above.

./flatpak-remote-metadata.py -u https://dl.flathub.org/repo flathub | \
    jq -r '.[] | select(
        .manifest | objects | .modules[] | recurse(.modules | arrays | .[]) |
        .sources | arrays | .[] | .url | strings | test(".*.(deb|rpm|snap|AppImage)$")
    ) | .metadata.Application.name // .metadata.Runtime.name' | \
    sort -u | wc -l

Evince, Flatpak, and GTK print previews

Endless OS is distributed as an immutable OSTree snapshot, with apps added & removed with Flatpak (and podman for power users & developers). Although the snapshot is assembled from Debian packages, it’s not really possible to install additional system packages locally, nor to remove them. Over time, we have tried to remove as many apps out of the immutable OS as possible: Flatpak apps are sandboxed and can be updated at a faster cadence than the OS itself, or removed if not needed.

Evince is one such app built into the OS at present. As a PDF viewer, it handles untrusted input in a complex format with libraries that have historically contained vulnerabilities, so is a good candidate for sandboxing and timely updates. While exploring removing it from the OS in favour of the Flatpak version from Flathub, I learned some things that were non-obvious to me about print preview, and which prevented making this change at the time.

Caveats: the notes below are a simplification, but I believe they are broadly accurate for GNOME on Linux. I’m sure people more familiar with GTK and/or printing already know everything here.

Printing from GTK apps

GTK provides API for applications to print documents. This presents the user with a print dialog, with many knobs to control how your document will be printed. That dialog has a Preview button; when you press it, the dialog vanishes and another one appears, showing a preview of your document. You can press Print on that dialog to print the document, or close it to cancel.

Why does the Preview button close the print dialog and open another one? Why does the preview dialog not have any of the knobs from the print dialog, or a way to return to the print dialog?

The answer lies in the documentation for GtkPrintOperation:

By default GtkPrintOperation uses an external application to do print preview.

The default application is Evince! More specifically, it is the following command:

evince --unlink-tempfile --preview --print-settings %s %f

Cribbing from the manpage:

--preview
Run evince as a previewer.
--unlink-temp-file
If evince is run in preview mode, this will unlink the temporary file created by GTK+.
--print-settings %s %f
This sends the full path of the PDF file, f, and the settings of the print dialog, s, to evince.

So when the user chooses to preview the document, GTK asks the application to render the document with the settings from the dialog, generates a PDF, and then invokes Evince to display that PDF. When you press Print in the preview dialog, it is Evince that sends the job to CUPS and thence to the printer.

What if evince is not present on the $PATH? The button is still displayed, but pressing it does nothing and the following is logged to stderr:

sh: 1: exec: evince: not found

There is code in GTK which attempts to handle this case by logging its own warning and then invoking the default PDF viewer on the generated PDF, but it doesn’t work because GLib actually spawns sh, not evince directly, and then returns success because ‘sh’ was successfully launched.

Printing from sandboxed apps

What happens if the application using the GtkPrintOperation API is a Flatpak app? evince is not part of any runtime, so GTK running in the application process cannot invoke it to preview the document? Well, in the general case the app can’t talk directly to CUPS either. So it uses the print portal’s PreparePrint method to prompt the user to choose a printer & settings, then renders the document and sends it to the portal with the Print method. The desktop portal service, which also uses GTK but is running outside the sandbox, presents the print dialog, and invokes evince if needed. All good, nothing too tricky here.

But notice that a sandboxed app is feeding a PDF to an unsandboxed PDF viewer. If the sandboxed app is malicious and can convince a user to print-preview a document, and there is some arbitrary code execution bug in Evince’s PDF library, then you’re in for a bad day.

What if Evince is a Flatpak?

The Flatpak version of Evince does not put an ‘evince’ command onto the $PATH, by design of Flatpak. So if you remove Evince from the OS and install the Flatpak, print preview stops working.

The evince executable inside the org.gnome.Evince Flatpak supports the --preview flag as normal. So you can put something like the following into ~/.config/gtk-4.0/settings.ini:

# https://docs.gtk.org/gtk4/property.Settings.gtk-print-preview-command.html
[Settings]
gtk-print-preview-command=flatpak run --file-forwarding org.gnome.Evince --unlink-tempfile --preview --print-settings @@ %s %f @@

--file-forwarding triggers special handling of the arguments bracketed by @@:

If this option is specified, the remaining arguments are scanned, and all arguments that are enclosed between a pair of ‘@@’ arguments are interpreted as file paths, exported in the document store, and passed to the command in the form of the resulting document path.

And this does indeed cause Evince to be spawned. However Evince can’t print the document. This is because its previewer tries to talk directly to CUPS, and its sandbox does not allow it to talk to CUPS. You might try punching some crude holes in the sandbox:

[Settings]
gtk-print-preview-command=flatpak run --file-forwarding --socket=system-bus --socket=cups org.gnome.Evince --unlink-tempfile --preview --print-settings @@ %s %f @@

and it seems to get a bit further, but by this point you’ve given up and turned your printer off because you want to go to bed.

What next?

I think it’s desirable for a PDF viewer to be sandboxed. I also think it’s desirable for the print previewer in particular to be sandboxed, or else a malicious-but-sandboxed application could trick the user into printing a PDF that exploits some vulnerability in the previewer and run stuff on the host system.

As I write this up, the gtk-print-preview-command override seems more viable than it did when I first looked into this last year. I think at the time, GTK in the GNOME runtime didn’t have the CUPS backend enabled so it couldn’t print even if you punched the relevant sandbox holes, but apparently it does now, so maybe we can make this change after all. It’s a shame I only realised this after spending hours writing this post!

You could also imagine extending the print portal API to allow an external app to be used for the preview without allowing that app to talk directly to CUPS.

(You could gracefully handle Evince not being installed by putting a wrapper script onto the $PATH which invokes Evince if installed or prompts you to install it if not.)

On Flatpak disk usage and deduplication

There is a blog post doing the rounds asserting that Flatpak Is Not The Future. The post is really long, and it seems unlikely that I and the author will ever agree on this topic, so I’m only going to talk about a couple of paragraphs about disk usage and sharing of runtimes between apps which caught my eye. This is highly relevant to my day job because all apps on Endless OS are Flatpaks—for example, the English downloadable version has 58 Flatpak apps pre-installed, and 13 runtimes—and I’ve had and answered some of the same questions discussed in the post.

They claim that they deduplicate runtimes. I question how much can really be shared between different branches when everything is recompiled.

This question is really easy to answer using du, which does not double-count files which are hardlinked together. Let’s compare the 20.08 and 21.08 versions of the freedesktop runtime:

wjt@camille:~$ du -sh /var/lib/flatpak/runtime/org.freedesktop.Platform/x86_64/20.08
674M	/var/lib/flatpak/runtime/org.freedesktop.Platform/x86_64/20.08
wjt@camille:~$ du -sh /var/lib/flatpak/runtime/org.freedesktop.Platform/x86_64/21.08
498M	/var/lib/flatpak/runtime/org.freedesktop.Platform/x86_64/21.08
wjt@camille:~$ du -sh /var/lib/flatpak/runtime/org.freedesktop.Platform/x86_64/20.08 /var/lib/flatpak/runtime/org.freedesktop.Platform/x86_64/21.08
674M	/var/lib/flatpak/runtime/org.freedesktop.Platform/x86_64/20.08
385M	/var/lib/flatpak/runtime/org.freedesktop.Platform/x86_64/21.08
wjt@camille:~$ echo $(( 498 - 385 ))                                                                                                                                                
113

113 MB (out of 498 MB for the smaller, more up-to-date 21.08 runtime) is shared between these two runtimes.

How about the GNOME 41 runtime, which is derived from the 21.08 freedesktop runtime?

wjt@camille:~$ du -sh /var/lib/flatpak/runtime/org.freedesktop.Platform/x86_64/21.08                                                                                                
498M	/var/lib/flatpak/runtime/org.freedesktop.Platform/x86_64/21.08
wjt@camille:~$ du -sh /var/lib/flatpak/runtime/org.gnome.Platform/x86_64/41                                                                                                
715M	/var/lib/flatpak/runtime/org.gnome.Platform/x86_64/41
wjt@camille:~$ du -sh /var/lib/flatpak/runtime/org.freedesktop.Platform/x86_64/21.08 /var/lib/flatpak/runtime/org.gnome.Platform/x86_64/41
498M	/var/lib/flatpak/runtime/org.freedesktop.Platform/x86_64/21.08
327M	/var/lib/flatpak/runtime/org.gnome.Platform/x86_64/41
wjt@camille:~$ echo $(( 715 - 327 ))                                                                                                                                                
388

388 MB (out of 715 MB) of the GNOME 41 runtime is shared with the 21.08 runtime.

I can’t imagine what system updates will be like in the future when you have a few dozen apps storing tens of gigabytes of runtimes that all want to be kept up to date.

There is no need to imagine! I have 163 Flatpak apps on my Endless OS system. Let’s see how many runtimes I have, how big they are, and how many apps use each one:

wjt@camille:~$ flatpak list --app --columns=runtime | sort | uniq -c | wc -l
18
wjt@camille:~$ flatpak list --app --columns=runtime | sort | uniq -c | sort -n
      1 com.endlessm.apps.Platform/x86_64/6
      1 org.freedesktop.Platform/x86_64/18.08
      1 org.gnome.Platform/x86_64/3.34
      1 org.gnome.Sdk/x86_64/41
      1 org.kde.Platform/x86_64/5.14
      2 com.endlessm.Platform/x86_64/eos3.2
      2 org.gnome.Platform/x86_64/3.28
      3 org.freedesktop.Platform/x86_64/19.08
      3 org.freedesktop.Sdk/x86_64/21.08
      3 org.gnome.Platform/x86_64/3.38
      3 org.kde.Platform/x86_64/5.15-21.08
      5 org.gnome.Platform/x86_64/3.36
      9 org.kde.Platform/x86_64/5.15
     10 org.freedesktop.Platform/x86_64/20.08
     24 com.endlessm.apps.Platform/x86_64/5
     28 org.freedesktop.Platform/x86_64/21.08
     30 org.gnome.Platform/x86_64/40
     36 org.gnome.Platform/x86_64/41
wjt@camille:~$ cd /var/lib/flatpak/runtime; flatpak list --app --columns=runtime | sort | uniq | xargs du -sh --total
918M	com.endlessm.apps.Platform/x86_64/5
907M	com.endlessm.apps.Platform/x86_64/6
2.0G	com.endlessm.Platform/x86_64/eos3.2
632M	org.freedesktop.Platform/x86_64/18.08
211M	org.freedesktop.Platform/x86_64/19.08
569M	org.freedesktop.Platform/x86_64/20.08
385M	org.freedesktop.Platform/x86_64/21.08
782M	org.freedesktop.Sdk/x86_64/21.08
26M	org.gnome.Platform/x86_64/3.28
329M	org.gnome.Platform/x86_64/3.34
264M	org.gnome.Platform/x86_64/3.36
263M	org.gnome.Platform/x86_64/3.38
198M	org.gnome.Platform/x86_64/40
277M	org.gnome.Platform/x86_64/41
264M	org.gnome.Sdk/x86_64/41
436M	org.kde.Platform/x86_64/5.14
231M	org.kde.Platform/x86_64/5.15
215M	org.kde.Platform/x86_64/5.15-21.08
8.7G	total

I have 18 runtimes, totalling 8.7 GB of storage (deduplicated), not “tens of gigabytes”. The top 5 most-used runtimes on my system cover 128 of the 163 apps. (I am ignoring the .Locale extensions of each runtime: the English and French translations of the 21.08 runtime total 17 MB, compared to 498 MB for the runtime itself, so I think this is a reasonable simplification for rough numbers.) As for updates? GNOME Software applies them automatically and silently. I don’t think about them at all.

People may disagree about whether the numbers above are large or small, compared to the upsides that Flatpak does or does not bring. But the numbers themselves are readily accessible, as is much of the past and ongoing work that has gone into making them as small/large as they are.

Personally, I think the trade-off is absolutely worth it for me and for Endless OS users, particularly since going all-in on Flatpak means that the base, immutable Endless OS install is just 4.2 GB. Of course there is room for improvement, and years ago I wrote a quick hack to help study exactly which files would ideally be shared between two runtimes but are not. At the time, the primary cause was non-reproducible builds. Since then, the Flatpak ecosystem has moved over to Buildstream which should help a lot, though I haven’t rerun the experiments except for what you see above. Automated statistics about apps using obsolete runtimes might be useful for the Flathub community, as might automated runtime updates, and some further work on understanding if and how the derived runtimes (GNOME & KDE) could share more with the freedesktop runtime version they are based on. And, would widespread use of filesystems that support block-level deduplication (like btrfs) help?

Chromium on Flathub

In December 2020, Chromium reached the Flathub stable channel. Assuming you have Flatpak 1.8.2 or newer, and your kernel is configured to allow unprivileged user namespaces, you can download it now.

Screenshot of Chromium showing the Chromium page on flathub.org

History

Endless OS is based on Debian, but rather than releasing as a bunch of .debs, it is released as an immutable OSTree snapshot, with apps added and removed using Flatpak.

For many years, we maintained a branch of Chromium as a traditional OS package which was built into the OS itself, and updated it together with our monthly OS releases. This did not match up well with Chromium, which has a new major version every 6 weeks and typically 2–4 patch versions in between. It’s a security-critical component, and those patch versions invariably fix some rather serious vulnerability. In some ways, web browsers are the best possible example of apps that should be updated independently of the OS. (In a nice parallel, it seems that the Chrome OS folks are also working on separating OS updates from browser updates on Chrome OS.)

Browsers are also the best possible example of apps which should use elaborate sandboxing techniques to limit the impact of security vulnerabilities, and Chromium is indeed a pioneer in this space. Flatpak applies much the same tools to sandbox applications, which ironically made it harder to ship Chromium as a Flatpak: when running in the Flatpak sandbox, it can’t use those same sandboxing APIs provided by the kernel to sandbox itself further.

Flatpak provides its own API for sandboxed applications to launch new instances of themselves with tighter sandboxing; what’s needed is a way to make Chromium use that…

Solution

Ryan Gonzalez has had a long-running project with us to enable Chromium-based apps to work well as Flatpaks. The first targets were apps built with Electron: his zypak project provides an LD_PRELOAD-able library that redirects Chromium’s sandbox to use Flatpak’s sub-sandboxing API. This avoids the need to modify the (often proprietary) apps themselves, and is now used by dozens of Electron apps on Flathub which would otherwise not be usable with Flatpak. There’s also a version of Chrome in the Flathub beta channel using this technique.

For Chromium, we can take a different approach. It’s open-source code, being compiled by Flathub, so Ryan prepared some patches to teach it to use the Flatpak sandboxing APIs directly, for better performance and robustness.

Once the sandbox integration was done, there was a long list of other changes needed to make the Chromium Flatpak work at least as well as our previous built-in version, which André Moreira Magalhães from Endless worked through with Ryan.

Some of these came from the old Endless OS package, such as using a royalty-free implementation of AAC, splitting encumbered codecs to a separate package so they can be excluded as needed for distribution, and discarding background tabs when the system is under memory pressure (which is useful on systems with limited RAM, but is disabled by default on desktop Linux builds).

Others were specific to Flatpak, such as dealing with udev not being available in the sandbox, restoring the ability to create app launchers for websites, integrating with Flatpak’s network proxy portal, and allowing Chromium policy files to be provided by the host system.

Over in Endless OS, we also needed to update users’ existing file associations and migrate their Chromium profiles to its new home.

Impact

Chart of 30 days of Chromium downloads, with three large spikes of around 20,000 daily downloads

The chart above is the Flathub download statistics for Chromium in the past 30 days. Counting the points between 14th March (when the most recent update was pushed) and 21st March, there have been nearly 60,000 downloads. The majority of these will be Endless OS users: our 3.9.2 release in January 2021 rolled this change out to all users, and Endless OS has automatic updates enabled by default. But Flathub has a broader reach than just Endless OS! I believe that users of System76’s Pop!_OS have been migrated from a .deb of Chromium to this Flatpak, and surely there are many users on other distributions, too. It’s also been used as the basis for other apps on Flathub, including ungoogled-chromium.

As an added bonus, the Flatpak is wired up to flatpak-external-data-checker, which now automatically opens a pull request when a new Chromium release is published. Typically, new major releases need manual intervention to refresh the Flatpak patches, but minor releases often build without issue: for these, one can just smoke-test the test build from the pull request, and then merge it, reducing what used to be days of effort rebasing the package in Endless OS to the work of minutes. I love it when a plan comes together.

A quick glance at the issues on the flathub/org.chromium.Chromium repo will show that there is always more work to be done. We would love to see other distributions getting involved, reducing the duplicated work of maintaining Chromium packages for each distro, and making it easier for users of long-term stable branches to get important browser updates quickly and easily.

Bustleman’s Holiday

I recently had a few weeks off work, and mostly did a good job of staying away from my computer. However, I couldn’t resist putting in a few hours to release Bustle 0.8.0, someone’s favourite D-Bus profiler. Get it on Flathub today!

Screenshot of Bustle 0.8.0's About dialog. “Someone's favourite D-Bus profiler”
I never have clarified whose favourite D-Bus profiler it is…

The main user-facing change is a great new icon, designed and drawn by Tobias Bernard. (Unfortunately the Flathub website still shows the old one for CDN cache reasons, but you can feast your eyes above or in your favourite app centre.)

I also rewrote some parts of the app from Haskell to C. In particular, it no longer uses the Haskell D-Bus implementation and libpcap binding, instead using GDBus and GVariant (via some minimal hand-written FFI bindings) and some mostly-existing C code, respectively. Me circa 2008 would be horrified to learn that I’ve done this, and me circa 2020 is sad to be introducing more unsafe code, but the net result is a smaller app with way fewer transitive dependencies.

Why can’t it use GDBus via GObject Introspection, you ask? GLib didn’t have a D-Bus implementation in 2008, and there was no GObject Introspection support for Haskell. These days, bindings for Gio etc. generated via GObject Introspection exist, but migrating Bustle to these would amount to a rewrite, and I don’t have the time or stomach for that. (These libraries all have my name on them because I made a small start at the generator back in 2011 with the express goal of using them in Bustle, but I’d be amazed if any of my work is still present.)

I’ve been fortunate to have had a dozen or so contributors to Bustle over the years, but it’s certainly true that my choice of implementation language has been a big barrier to entry for other contributors. I maintain that it was not a bad choice in and of itself for a weekend project: in 2008, a time before GObject Introspection was widely available, the Haskell GTK and Cairo bindings were unusually thorough and well-written and allowed me to write concise code and iterate quickly. But going against the grain of the community has a cost: the intersection of “Haskell programmer”, “D-Bus expert” and “GNOME enthusiast” is a very small set, and I know several people personally who were interested in contributing until they saw the language.

Why C rather than Rust, you ask? If you want to try to glue Cabal, Cargo and C together, be my guest.

(In case the joke in the title doesn’t translate outside the UK: busman’s holiday on Wiktionary.)

Vanilla is a complex and delicious flavour

Last week, Tobias Bernard published a thought-provoking article, There is no “Linux” Platform (Part 1), based on a talk at LAS 2019. (Unfortunately I couldn’t make it to LAS, and I haven’t found the time to watch a recording of the talk, so I’m going solely from the blog post here.) The article makes some interesting observations, and I found a fair few things to agree with. But I want to offer a counterpoint to this paragraph of the final section, “The Wrong Incentives”:

The Endless OS shell is a great example of this. They started out with vanilla GNOME Shell, but then added ever more downstream patches in order to address issues found in in-house usability tests. This means that they end up having to do huge rebases every release, which is a lot of work. At the same time, the issues that prompted the changes do not get fixed upstream (Endless have recently changed their strategy and are working upstream much more now, so hopefully this will get better in the future).

If we’re looking at the code shipping in Endless OS today, then yes, our desktop is vanilla GNOME Shell with a few hundred patches on top, and yes, as a result, rebasing onto new GNOME releases is a lot of work. But the starting point for Endless OS was not “what’s wrong with GNOME?” but “what would the ideal desktop look like for a new category of users?”.

When Endless began, the goal was to create a new desktop computing product, targeting new computer users in communities which were under-served by existing platforms and products. The company conducted extensive field research, and designed a desktop user interface for those users. Prototypes were made using various different components, including Openbox, but ultimately the decision was made to base the desktop on GNOME, because GNOME provided a collection of components closest to the desired user experience. The key point here is that basing the Endless desktop on GNOME was an implementation detail, made because the GNOME stack is a robust, feature-rich and flexible base for a desktop.

Over time, the strategy shifted away from being based solely around first-party hardware, towards distributing our software to a broader set of users using standard desktop and laptop hardware. Around the same time, Endless made the switch from first- and third-party apps packaged as a combination of Debian packages and an in-house system towards using Flatpak for apps, and contributed towards the establishment of Flathub. Part of the motivation for this switch was to get Endless out of the business of packaging other people’s applications, and instead to enable app developers to directly target desktop Linux distributions including, but not limited to, Endless OS.

A side-effect of this change is that our user experience has become somewhat less consistent because we have chosen not to theme apps distributed through Flathub, with the exception of minimize/maximize window controls and a different UI font; and, of course, Flathub offers apps built with many different toolkits. This is still a net positive: our users have access to many more applications than they would have done if we had continued distributing everything ourselves.

As the prototypal Endless OS user moved closer to the prototypal GNOME user, we have focused more on finding ways to converge with the GNOME user experience. In some cases, we’ve simply removed functionality which we don’t think is necessary for our current crop of users. For example, Endless OS used to target users whose display was a pre-digital TV screen, with a 720×480 resolution. I think persuading the upstream maintainers of GNOME applications to support this resolution would have been a hard sell in 2014, let alone in 2019!

Some other changes we’ve made can and have been simply be proposed upstream as they are, but the bulk of our downstream functionality forms a different product to GNOME, which we feel is still valuable to our users. We are keen to both improve GNOME, and reduce the significant maintenance burden which Tobias rightly refers to, so we’re incrementally working out which functionality could make sense in both Endless and GNOME in some form, working out what that form could be, and implementing it. This is a big project because engaging constructively with the GNOME community involves more thought and nuance than opening a hundred code-dump merge requests and sailing away into the sunset.

If you are building a product whose starting point is “GNOME, but better”, then I encourage you to seriously consider whether you can work upstream first. I don’t think this is a groundbreaking idea in our community! However, that was not the starting point for Endless OS, and even today, we are aiming for a slightly different product to GNOME.

Back out to the big picture that is the subject of Tobias’ article: I agree that desktop fragmentation is a problem for app developers. Flatpak and Flathub are, in my opinion, a major improvement on the status quo: app developers can target a common environment, and have a reasonable expectation of their apps working on all manner of distributions, while we as distro maintainers need not pretend that we know best how to package a Java IDE. As the maintainer of a niche app written using esoteric tools, Flathub allowed me – for the first time since I wrote the first version in 2008 – to distribute a fully-functional, easy-to-install application directly to users without burdening distribution developers with the chore of packaging bleeding-edge versions of Haskell libraries. It gave me a big incentive to spend some of my (now very limited) free time on some improvements to the app that I had been putting off until I had a way to get them to users (including myself on Endless OS) in a timely manner.

On the other hand, we shouldn’t underestimate the value of GNOME – and distros like Debian – being a great base for products that look very different to GNOME itself: it enables experimentation, exploration, and reaching a broader base of users than GNOME alone could do, while pooling the bulk of our resources. (On a personal level, I owe pretty much my entire career in free software to products based on Debian and the GNOME stack!)

Some caveats: I joined Endless in mid-2016, midway through the story above, so I am relying on my past and current colleagues’ recollections of the early days of the company. Although today I am our Director of Platform, I am still just one person in the team! We’re not a hive mind, and I’m sure you’ll find different opinions on some of these points if you ask around.

Flatpak External Data Checker

(This post is a slightly longer version of a lightning talk I gave at GUADEC 2019.)

Many non-free applications’ binaries cannot be redistributed (particularly not in modified form), so they cannot be included directly in a Flatpak. To work around this, Flatpak supports the concept of “extra data”: files which will be downloaded and unpacked from a third-party URI when the app is installed. The URI is accompanied by a checksum and a size, to provide some hope that the data unpacked on the user’s system is the same as what the packager tested. This is used by, for example, the Dropbox Flatpak.

Of course, the Flatpak needs to be kept up to date when new versions of the app are released. At best, the old URL will still point to the same file, so at least the old version of the app will continue to be installed; in some cases, however, vendors publish new versions of the app at the same URL, which means the Flatpak cannot be installed until it is updated.

Some time ago, Joaquim Rocha started work on Flatpak External Data Checker to periodically check a Flatpak manifest and report when it needs updating. As well as just checking that a URL is reachable and has the expected size and checksum, it also knows how to follow a redirect to a stable URI for the latest version (a helpful pattern some apps use), or to find the latest package in an apt repository. I subsequently taught it how to determine the new app’s version, update the AppData file, commit the necessary changes to Git, and send a pull request (like this one).

I tried moderately hard to preserve YAML and XML comments and formatting. For JSON, I gave up trying to preserve formatting (let alone json-glib’s non-standard extensions); the output is at least deterministic, so once it’s reformatted the JSON, the diffs will be smaller in future.

At Endless, we run this for a short list of apps on Flathub (and a few on Endless’s Flatpak repo). If you want to get PRs for an app you maintain, add the necessary metadata to your Flathub application’s manifest, then send a pull request to update the list of repos we check. I hope that in the medium term we could move this over to Flathub’s build infrastructure and run it on every repo (with some way to opt out).

There are a fair few open issues – PRs, suggestions and bug reports all very welcome!

Age rating data for Flathub apps

OARS (Open Age Ratings Service) defines a scheme to include content rating information in apps’ AppData/AppStream file. GNOME Software and similar tools use this metadata to show age ratings for applications. In Endless OS, we also support restricting which applications a given user can install based on this data – see this page, and the reports it links to, for a bit more information about this feature and its future.

Screenshot: “Age Rating: 7. The application was rated this way because it features: Characters in aggressive conflict easily distinguishable from reality”

Every new app on Flathub should include OARS metadata, but there are many existing apps which don’t have this data, so it’s not (yet) enforced at build time. Edit: Bartłomiej tells me that it has been enforced at build time for a little over a month. (See App Requirements and AppData Guidelines on the Flathub wiki for some more information on what’s required and recommended; this branch of appstream-glib is modified to enforce Flathub’s policies.) My colleague Andre Magalhaes crunched the data and opened a tracker task for Flathub apps without OARS metadata. ((We have a similar list for our in-house apps.)) This information is much more useful if it can be relied upon to be present.

If you’re familiar with an app on this list, generating the OARS data is a simple process: open this generator in your browser, answer some questions about the app, and receive some XML. The next step is to put that OARS data into the AppData. ((If you’re the upstream maintainer for the app, you probably already know how to do all this, and can stop reading here!)) Take a look at the app’s Flathub repo and check whether it has an .appdata.xml file.

If it doesn’t, then the app’s AppData must be maintained upstream. Great! Find the upstream repository for the project, and send a merge request there. (Here’s one I sent earlier, for D-Feet.) You can either add the same patch to the Flathub packaging (as I did for D-Feet) or wait for a new upstream release and then update the Flathub packaging to that version (as I also did for D-Feet, a couple of days later).

If the appdata is maintained in the Flathub repo, make the relevant changes there directly. (Here’s a PR I opened for Tux, of Math Command while writing this post.) Ideally, the appdata would make its way upstream, but there are a fair few apps on Flathub which do not have active upstreams.

You might well find that the appdata requirements have become more strict about things other than OARS since the app was last updated, and these will have to be fixed too. In both the example cases above, I had to add release information, which has become mandatory for Flathub apps since these were last updated.

Yes, it was something to do with controlling terminals

Back in June, I wrote about the difficulty of killing the dbus-monitor subprocess when Bustle is recording the system bus:

But sending SIGKILL from the unprivileged Bustle process to the privileged dbus-monitor process has no effect. Weirdly, if I run pkexec dbus-monitor --system in a terminal, I can hit Ctrl+C and kill it just fine. (Is this something to do with controlling terminals? Please tell me if you know how I could make this work.)

Greetings, past me! I’m just like you, only 6 months older, and I’m here to tell you how to make this work.

When you run a process in a terminal (emulator), its controlling TTY is a pseudoterminal (hereafter PTY) slave ((not my choice of terminology)) device. When you press Ctrl+C, the terminal doesn’t send SIGINT to the running process; it writes the ^C character '\x03' into the corresponding PTY master. This is intercepted by the kernel’s TTY layer, which eats this character and sends SIGINT to the (foreground) process (group) attached to the PTY slave. It doesn’t matter if that process (group) is owned by another user: being able to write to its controlling terminal is the capability you need.

Armed with this knowledge, and some further reading on how to set the controlling TTY, it was reasonably easy to wire this up in Bustle:

You may wonder how the second step works in the Flatpak case, where the pkexec dbus-monitor process is not spawned directly by Bustle, but by the Flatpak session helper daemon. Well, we don’t get a chance to run our own code between fork() and exec(), but that’s okay because the session helper does exactly the same thing for us. (I think this is why Ctrl+C works in a host shell running within Flatpak:d GNOME Builder.)