Creating Windows installation media on Linux

Every so often I need to install Windows, most recently for my GNOME on WSL experiments, and to do this I need to write the Windows installer ISO to a USB stick. Unlike most Linux distro ISOs, these are true, pure ISO 9660 images—not hybrid images that can also be treated as a DOS/MBR disk image—so they can’t just be written directly to the disk. Microsoft’s own tool is only available for Windows, of course.

I’m sure there are other ways but this is what I do. Edit: check the comments for an approach which involves 2 partitions and a little more careful copying, but no special tools. I’m writing it down so I can easily find the instructions next time!

The basic process is quite simple:

  • Download an ISO 9660 disk image from Microsoft
  • Partition the USB drive with a single basic data partition, formatted as FAT32
  • Mount the ISO image – on GNOME, you should just be able to double-click it to mount it with Disk Image Mounter
  • Copy all the files from the mounted ISO image to the USB drive

But there is a big catch with that last step: at least one of the .wim files in the ISO is too large for a FAT32 partition.

The trick is to first copy all the files to a writeable directory on internal storage, then use a tool called wimlib-imagex split from wimlib to split the large .wim file into a number of smaller .swm files before copying them to the FAT32 partition. I think I compiled it from source, in a toolbox container, but you could also use this OCI container image whose README helpfully provides these instructions:

find . -size +4294967000c -iname '*.wim' -print | while read -r wimpath; do
  wimbase="$(basename "$wimpath" '.wim')"
  wimdir="$(dirname "$wimpath")"
  echo "splitting ${wimpath}"
  docker run \
    --rm \
    --interactive \
    --tty \
    --volume "$(pwd):/work" \
    "backplane/wimlib-imagex" \
      split "$wimpath" "${wimdir}/${wimbase}.swm" 4000
done

Now you can copy all those files, minus the too-large .wim, onto the FAT32 drive, and then boot from it.

This all assumes that you only care about a modern system with EFI firmware. I have no idea about creating a BIOS-bootable Windows installer on Linux, and fortunately I have never needed to do this: to test stuff on a BIOS Windows installation, I have used the time-limited virtual machines that Microsoft publishes for testing stuff in old versions of Internet Explorer.

I was inspired to resurrect this old draft post by a tweet by Ross Burton.

How many Flathub apps reuse other package formats?

Today I read Comparison of Fedora Flatpaks and Flathub remotes by Hari Rana, who is an active and valued member of the Flatpak community. The article is a well-researched and well-written overview of how these two Flatpak ecosystems differ, and contains the following remark about one major difference (emphasis mine):

Flathub is open with what source a Flatpak application (re)uses, whereas Fedora Flatpaks strictly reuses the RPM format.

As such, Flathub has tons of applications that reuse other package formats.

When this article was discussed in the Flatpak Matrix channel, several people wondered whether “tons” is a fair assessment. Let’s find out!

The specific examples given in the article are of apps which reuse a .deb (to which I will add .rpm), AppImage, Snap package, or binary .tar.gz archive. It’s not so easy to distinguish a binary tarball from a source tarball, so as a substitute I will look for apps which use the extra-data to download external sources at install time rather than at build time.

I have cloned every repo from the Flathub GitHub organisation with this script I had lying around. There are 2,220 such repositories. This is a bigger number than the 1,518 apps cited in the blog post, because it includes many thing which are not apps, such as 258 GTK themes and 60 digital audio workstation plugins. I also believe that the 1,518 number does not include end-of-lifed apps, whereas my methodology does. This post will also ignore the existence of OBS Studio and Firefox, where those projects build the Flatpak from source on their own infrastructure and push the result into Flathub.

Now I’m just going to grep them all for the offending strings:

$ (for i in */
do
    if git -C $i grep --quiet -E '(\.(deb|rpm|AppImage|snap)\>)|(extra-data)'
    then
        echo $i
    fi
done) | wc -l
237

(Splitting apart the search terms, we have 141 repos matching .deb, 10 for .rpm, 23 for .AppImage, 6 for .snap, and 110 for extra-data. These numbers don’t sum to 237 because the same repo can use multiple formats, and these binary files are often used by extra-data apps.)

So by my back-of-an-envelope calculation, 237 out of 2220 repos on Flathub repackage other binary formats. This is a little under 11%. Of those 237, 51 are GTK themes, specifically variations of the Mint, Pop and Yaru themes. If we assume that all the other 186 are apps, and that none of them are EOLed, then 186 divided by 1,518 gives us a little more than 12% of apps on Flathub that are repackaged from other binary formats. (I believe this is a slight overestimate but I have run out of time this morning.)

Is that a big number? It’s roughly what I expected. Is it “ton[ne]s”? Compared to Fedora’s Flatpak repo, where everything is built from source, it certainly is: indeed, it’s more than the total number of apps in the Fedora Flatpak repo!

If it is valuable for Flathub to provide proprietary apps like Slack whose publishers do not currently wish to support Flatpak (which I believe it is) then it’s unavoidable that some apps repackage other binary formats. OK, time for one last bit of data: what if we exclude extra-data apps?

$ (for i in */
do
    if ! git -C $i grep --quiet extra-data && \
       git -C $i grep --quiet -E '\.(deb|rpm|AppImage|snap)\>'
    then
        echo $i
    fi
done )| wc -l
127

So (ignoring non-extra-data apps which use binary tarballs, if any such apps exist) that’s something like 76 apps and 51 GTK themes which probably could be built from source by Flathub, but aren’t. It may be hard to build some of these apps from source (perhaps the upstream build system requires network access) but the rewards would include support for aarch64 and any other architectures Flathub may add, and arguably greater transparency in how the app is built.

If you want to do your own research in this vein, you may be interested in gasinvein‘s Flatpak remote metadata fetcher, which would let you generate and analyse a 200 MiB JSON file rather than by cloning and grep-ing 4.6 GiB of Git repositories. His analysis using this data yields 174 apps, quite close to my 186 estimate above.

./flatpak-remote-metadata.py -u https://dl.flathub.org/repo flathub | \
    jq -r '.[] | select(
        .manifest | objects | .modules[] | recurse(.modules | arrays | .[]) |
        .sources | arrays | .[] | .url | strings | test(".*.(deb|rpm|snap|AppImage)$")
    ) | .metadata.Application.name // .metadata.Runtime.name' | \
    sort -u | wc -l

Release (semi-)automation

The time I have available to maintain GNOME Initial Setup is very limited, as anyone who has looked at the commit history will have noticed. I’d love more eyes & hands on this important but easy-to-overlook component, particularly to guide it kindly but firmly into the modern age of GTK 4 and the refreshed HIG.

I found that making a batch of 1–3 releases across different GNOME branches every few months was surprisingly time-consuming and error-prone, even with the pretty comprehensive release process checklist on the GNOME Wiki, so I’ve been periodically trying to automate bits of it away.

Philip Withnall’s gitlab-changelog script makes writing the NEWS file a lot quicker. I taught it to output the human-readable names of each updated translation (a nice additional contribution would be to also include the name of the human who updated the translation) and made it a little smarter about guessing the Git commit range to scan.

Beyond that, I added a Meson run target, maintainer-upload-release pointing at a script which performs some rudimentary coherence checks on the version number, tags the release (using git-evtag if available), atomically pushes the branch and that tag to GNOME GitLab, then copies the source tarball to master.gnome.org. (Apparently it has been almost 12 years since I did something similar in telepathy-gabble, building on the make maintainer-upload-release target that Simon McVittie added in 2008, which is where I borrowed the name.) Maybe other module maintainers may find this script useful too – it’s quite generic.

Putting these together, the release flow looks like this:

git switch gnome-42
git pull
../pwithnall/gitlab-changelog/gitlab-changelog.py GNOME/gnome-initial-setup
# Manually edit NEWS to incorporate the changelog, adjusted as needed
# Manually check the version in meson.build
git commit -am 'NEWS for 42.Y'
ninja -C _build dist maintainer-upload-release

Another release-related quality-of-life improvement is to make GitLab CI not only build and test the project (in the vain hope that there might actually be tests!) but also check that the install and gnome-initial-setup-pot targets both work. (At one point or another both have failed at or around release time; now they never will again, famous last words.)

I know none of this is rocket science, but I find it all makes the process quicker and less cumbersome, and it’s stopped me from repeating errors like uploading the wrong version on a few tired evenings. Obviously this could all be taken further: perhaps a manually-invoked CI pipeline that does all this stuff, more checks, etc. But while I’m on this train of thought:

Why do we release GNOME modules one-by-one at all?

The workflow we use to release Endless OS is a bit different to GNOME. Once we merge a change to some module’s Git repository, such as eos-updater or our shrinking branch of GNOME Software, that change embarks on a scenic automated journey that takes it to the next nightly build of the entire OS, both as an OSTree update and as fresh installation media. I use these nightly builds for my daily work, safe in the knowledge that I can roll back to the previous build if necessary.

We don’t make releases of individual modules: instead, when it comes time to release the OS, we trigger a pipeline that (among many other things) pushes the already-built OS update to the production repo, and creates Release_x.y.z tags on each Git repo.

This was quite an adjustment for me at first, compared to lovingly hand-crafting NEWS files and coming up with funny/esoteric release names, but now that I’m used to it it’s hard to go back. Why can’t GNOME do the same?

At this point in the post, we are straying into territory that I have limited first-hand knowledge of. Caveat lector! But here goes:

Thanks to GNOME OS, GNOME already has nightly builds of the entire desktop and apps: so rather than having to build everything yourself, or wait for a development release of GNOME, you can just update & reboot your GNOME OS VM and test the change right there. gnome-build-meta knows how to build every GNOME module; and if you can build the code, it seems a conceptually small step to run ninja dist and the stuff above to publish tags and tarballs for each module.

So you could well imagine on 43.beta release day, someone in the release team could boot the latest GNOME OS nightly, declare it to be Good, and push a button that tags every relevant GNOME module & builds and uploads all the tarballs, and then go back to their day, rather than having to chase down module owners who haven’t quite got around to making the release, fix random build breakages, and so on.

To make this work reliably, I think you’d need every module’s CI to be run through gnome-build-meta, building that MR against the rest of the project, so that g-b-m build failures would be caught before (not after) the offending change lands in the module in question. Seems doable – in Endless we have the equivalent thing managed by a jenkins-job-builder template, the GitHub Pull Request Builder plugin, and a gnarly script.

Continuous integration and deployment are becoming the norm throughout the software industry, for good reasons laid out quite well in articles like Shipping Fast Changes Your Life: the smaller the gap between making a change and it reaching a user, the faster the feedback, and the less costly it is to fix a bug or change course.

The free software movement has historically been ahead of the curve on this, with the “release early, release often” philosophy. And GNOME in particular has used a time-based release process for two decades, allowing major distros to align their schedules to GNOME and get updates into the hands of users quickly, which went some way towards overcoming the fact that GNOME does not own the full pipeline from source code to end users.

Havoc Pennington’s June 2002 email proposing this model has aged rather well, in my opinion, and places a heavy emphasis on the development branch being usable:

The unstable branch must always be dogfood-quality. If testers can’t test it by using it daily, they can’t make the jump. If the unstable branch becomes too unstable, we can’t release it on a reliable schedule, so we have to start breaking the stable branch as a stopgap.

Interestingly the time-based release schedule wiki page states that the schedule should contain:

Regular test release dates, approximately every 2 weeks.

These days, GNOME releases are closer to monthly. In the context of the broader industry where updates reach users multiple times a day, this is starting to look a little less forward-thinking! Of course, continuously deploying an entire OS to production is rather harder than continuously deploying web apps or apps in app stores, if only because the stakes are higher: you need a really robust automatic rollback mechanism to save your users’ plant-based bacon substitute if a new OS build fails to boot, or worse, contains an updater bug that prevents future updates being applied! Still, I believe that a bit of automation would go a long way in allowing module maintainers and the release team alike to spend their scarce mental energy on other things, and allow the project to increase the frequency of releases. What am I missing?

Evince, Flatpak, and GTK print previews

Endless OS is distributed as an immutable OSTree snapshot, with apps added & removed with Flatpak (and podman for power users & developers). Although the snapshot is assembled from Debian packages, it’s not really possible to install additional system packages locally, nor to remove them. Over time, we have tried to remove as many apps out of the immutable OS as possible: Flatpak apps are sandboxed and can be updated at a faster cadence than the OS itself, or removed if not needed.

Evince is one such app built into the OS at present. As a PDF viewer, it handles untrusted input in a complex format with libraries that have historically contained vulnerabilities, so is a good candidate for sandboxing and timely updates. While exploring removing it from the OS in favour of the Flatpak version from Flathub, I learned some things that were non-obvious to me about print preview, and which prevented making this change at the time.

Caveats: the notes below are a simplification, but I believe they are broadly accurate for GNOME on Linux. I’m sure people more familiar with GTK and/or printing already know everything here.

Printing from GTK apps

GTK provides API for applications to print documents. This presents the user with a print dialog, with many knobs to control how your document will be printed. That dialog has a Preview button; when you press it, the dialog vanishes and another one appears, showing a preview of your document. You can press Print on that dialog to print the document, or close it to cancel.

Why does the Preview button close the print dialog and open another one? Why does the preview dialog not have any of the knobs from the print dialog, or a way to return to the print dialog?

The answer lies in the documentation for GtkPrintOperation:

By default GtkPrintOperation uses an external application to do print preview.

The default application is Evince! More specifically, it is the following command:

evince --unlink-tempfile --preview --print-settings %s %f

Cribbing from the manpage:

--preview
Run evince as a previewer.
--unlink-temp-file
If evince is run in preview mode, this will unlink the temporary file created by GTK+.
--print-settings %s %f
This sends the full path of the PDF file, f, and the settings of the print dialog, s, to evince.

So when the user chooses to preview the document, GTK asks the application to render the document with the settings from the dialog, generates a PDF, and then invokes Evince to display that PDF. When you press Print in the preview dialog, it is Evince that sends the job to CUPS and thence to the printer.

What if evince is not present on the $PATH? The button is still displayed, but pressing it does nothing and the following is logged to stderr:

sh: 1: exec: evince: not found

There is code in GTK which attempts to handle this case by logging its own warning and then invoking the default PDF viewer on the generated PDF, but it doesn’t work because GLib actually spawns sh, not evince directly, and then returns success because ‘sh’ was successfully launched.

Printing from sandboxed apps

What happens if the application using the GtkPrintOperation API is a Flatpak app? evince is not part of any runtime, so GTK running in the application process cannot invoke it to preview the document? Well, in the general case the app can’t talk directly to CUPS either. So it uses the print portal’s PreparePrint method to prompt the user to choose a printer & settings, then renders the document and sends it to the portal with the Print method. The desktop portal service, which also uses GTK but is running outside the sandbox, presents the print dialog, and invokes evince if needed. All good, nothing too tricky here.

But notice that a sandboxed app is feeding a PDF to an unsandboxed PDF viewer. If the sandboxed app is malicious and can convince a user to print-preview a document, and there is some arbitrary code execution bug in Evince’s PDF library, then you’re in for a bad day.

What if Evince is a Flatpak?

The Flatpak version of Evince does not put an ‘evince’ command onto the $PATH, by design of Flatpak. So if you remove Evince from the OS and install the Flatpak, print preview stops working.

The evince executable inside the org.gnome.Evince Flatpak supports the --preview flag as normal. So you can put something like the following into ~/.config/gtk-4.0/settings.ini:

# https://docs.gtk.org/gtk4/property.Settings.gtk-print-preview-command.html
[Settings]
gtk-print-preview-command=flatpak run --file-forwarding org.gnome.Evince --unlink-tempfile --preview --print-settings @@ %s %f @@

--file-forwarding triggers special handling of the arguments bracketed by @@:

If this option is specified, the remaining arguments are scanned, and all arguments that are enclosed between a pair of ‘@@’ arguments are interpreted as file paths, exported in the document store, and passed to the command in the form of the resulting document path.

And this does indeed cause Evince to be spawned. However Evince can’t print the document. This is because its previewer tries to talk directly to CUPS, and its sandbox does not allow it to talk to CUPS. You might try punching some crude holes in the sandbox:

[Settings]
gtk-print-preview-command=flatpak run --file-forwarding --socket=system-bus --socket=cups org.gnome.Evince --unlink-tempfile --preview --print-settings @@ %s %f @@

and it seems to get a bit further, but by this point you’ve given up and turned your printer off because you want to go to bed.

What next?

I think it’s desirable for a PDF viewer to be sandboxed. I also think it’s desirable for the print previewer in particular to be sandboxed, or else a malicious-but-sandboxed application could trick the user into printing a PDF that exploits some vulnerability in the previewer and run stuff on the host system.

As I write this up, the gtk-print-preview-command override seems more viable than it did when I first looked into this last year. I think at the time, GTK in the GNOME runtime didn’t have the CUPS backend enabled so it couldn’t print even if you punched the relevant sandbox holes, but apparently it does now, so maybe we can make this change after all. It’s a shame I only realised this after spending hours writing this post!

You could also imagine extending the print portal API to allow an external app to be used for the preview without allowing that app to talk directly to CUPS.

(You could gracefully handle Evince not being installed by putting a wrapper script onto the $PATH which invokes Evince if installed or prompts you to install it if not.)

Small steps towards a GTK 4-based Initial Setup

Over the Christmas holidays, I was mostly occupied with the literal care and feeding of small humans, but I found a bit of time for the metaphorical care and feeding of Initial Setup for GNOME 42 as well. Besides a bit of review and build and CI housekeeping, I wrote some patches to update it for API changes in libgnome-desktop (merged) and libgweather (pending). The net result is an app which looks and works exactly the same, complete with a copy of the widget formerly known as GWeatherLocationEntry (RIP) with its serial numbers filed off.

Of course, my ultimate goal was to port Initial Setup to GTK 4. I made some other tiny steps in that direction, such as removing a redundant use of GtkFrame that becomes actively harmful with the removal of the shadow-type property in GTK 4, and now have a proof-of-concept port of just the final page which both compiles and runs!

Screenshots of "All done!" page of Initial Setup

But, I will not have time to complete this port in time for the GNOME 42 UI freeze on 12th February. If you are reading this and feel inspired to pick this up, even just a page or two, more hands would be much appreciated.

On Flatpak disk usage and deduplication

There is a blog post doing the rounds asserting that Flatpak Is Not The Future. The post is really long, and it seems unlikely that I and the author will ever agree on this topic, so I’m only going to talk about a couple of paragraphs about disk usage and sharing of runtimes between apps which caught my eye. This is highly relevant to my day job because all apps on Endless OS are Flatpaks—for example, the English downloadable version has 58 Flatpak apps pre-installed, and 13 runtimes—and I’ve had and answered some of the same questions discussed in the post.

They claim that they deduplicate runtimes. I question how much can really be shared between different branches when everything is recompiled.

This question is really easy to answer using du, which does not double-count files which are hardlinked together. Let’s compare the 20.08 and 21.08 versions of the freedesktop runtime:

wjt@camille:~$ du -sh /var/lib/flatpak/runtime/org.freedesktop.Platform/x86_64/20.08
674M	/var/lib/flatpak/runtime/org.freedesktop.Platform/x86_64/20.08
wjt@camille:~$ du -sh /var/lib/flatpak/runtime/org.freedesktop.Platform/x86_64/21.08
498M	/var/lib/flatpak/runtime/org.freedesktop.Platform/x86_64/21.08
wjt@camille:~$ du -sh /var/lib/flatpak/runtime/org.freedesktop.Platform/x86_64/20.08 /var/lib/flatpak/runtime/org.freedesktop.Platform/x86_64/21.08
674M	/var/lib/flatpak/runtime/org.freedesktop.Platform/x86_64/20.08
385M	/var/lib/flatpak/runtime/org.freedesktop.Platform/x86_64/21.08
wjt@camille:~$ echo $(( 498 - 385 ))                                                                                                                                                
113

113 MB (out of 498 MB for the smaller, more up-to-date 21.08 runtime) is shared between these two runtimes.

How about the GNOME 41 runtime, which is derived from the 21.08 freedesktop runtime?

wjt@camille:~$ du -sh /var/lib/flatpak/runtime/org.freedesktop.Platform/x86_64/21.08                                                                                                
498M	/var/lib/flatpak/runtime/org.freedesktop.Platform/x86_64/21.08
wjt@camille:~$ du -sh /var/lib/flatpak/runtime/org.gnome.Platform/x86_64/41                                                                                                
715M	/var/lib/flatpak/runtime/org.gnome.Platform/x86_64/41
wjt@camille:~$ du -sh /var/lib/flatpak/runtime/org.freedesktop.Platform/x86_64/21.08 /var/lib/flatpak/runtime/org.gnome.Platform/x86_64/41
498M	/var/lib/flatpak/runtime/org.freedesktop.Platform/x86_64/21.08
327M	/var/lib/flatpak/runtime/org.gnome.Platform/x86_64/41
wjt@camille:~$ echo $(( 715 - 327 ))                                                                                                                                                
388

388 MB (out of 715 MB) of the GNOME 41 runtime is shared with the 21.08 runtime.

I can’t imagine what system updates will be like in the future when you have a few dozen apps storing tens of gigabytes of runtimes that all want to be kept up to date.

There is no need to imagine! I have 163 Flatpak apps on my Endless OS system. Let’s see how many runtimes I have, how big they are, and how many apps use each one:

wjt@camille:~$ flatpak list --app --columns=runtime | sort | uniq -c | wc -l
18
wjt@camille:~$ flatpak list --app --columns=runtime | sort | uniq -c | sort -n
      1 com.endlessm.apps.Platform/x86_64/6
      1 org.freedesktop.Platform/x86_64/18.08
      1 org.gnome.Platform/x86_64/3.34
      1 org.gnome.Sdk/x86_64/41
      1 org.kde.Platform/x86_64/5.14
      2 com.endlessm.Platform/x86_64/eos3.2
      2 org.gnome.Platform/x86_64/3.28
      3 org.freedesktop.Platform/x86_64/19.08
      3 org.freedesktop.Sdk/x86_64/21.08
      3 org.gnome.Platform/x86_64/3.38
      3 org.kde.Platform/x86_64/5.15-21.08
      5 org.gnome.Platform/x86_64/3.36
      9 org.kde.Platform/x86_64/5.15
     10 org.freedesktop.Platform/x86_64/20.08
     24 com.endlessm.apps.Platform/x86_64/5
     28 org.freedesktop.Platform/x86_64/21.08
     30 org.gnome.Platform/x86_64/40
     36 org.gnome.Platform/x86_64/41
wjt@camille:~$ cd /var/lib/flatpak/runtime; flatpak list --app --columns=runtime | sort | uniq | xargs du -sh --total
918M	com.endlessm.apps.Platform/x86_64/5
907M	com.endlessm.apps.Platform/x86_64/6
2.0G	com.endlessm.Platform/x86_64/eos3.2
632M	org.freedesktop.Platform/x86_64/18.08
211M	org.freedesktop.Platform/x86_64/19.08
569M	org.freedesktop.Platform/x86_64/20.08
385M	org.freedesktop.Platform/x86_64/21.08
782M	org.freedesktop.Sdk/x86_64/21.08
26M	org.gnome.Platform/x86_64/3.28
329M	org.gnome.Platform/x86_64/3.34
264M	org.gnome.Platform/x86_64/3.36
263M	org.gnome.Platform/x86_64/3.38
198M	org.gnome.Platform/x86_64/40
277M	org.gnome.Platform/x86_64/41
264M	org.gnome.Sdk/x86_64/41
436M	org.kde.Platform/x86_64/5.14
231M	org.kde.Platform/x86_64/5.15
215M	org.kde.Platform/x86_64/5.15-21.08
8.7G	total

I have 18 runtimes, totalling 8.7 GB of storage (deduplicated), not “tens of gigabytes”. The top 5 most-used runtimes on my system cover 128 of the 163 apps. (I am ignoring the .Locale extensions of each runtime: the English and French translations of the 21.08 runtime total 17 MB, compared to 498 MB for the runtime itself, so I think this is a reasonable simplification for rough numbers.) As for updates? GNOME Software applies them automatically and silently. I don’t think about them at all.

People may disagree about whether the numbers above are large or small, compared to the upsides that Flatpak does or does not bring. But the numbers themselves are readily accessible, as is much of the past and ongoing work that has gone into making them as small/large as they are.

Personally, I think the trade-off is absolutely worth it for me and for Endless OS users, particularly since going all-in on Flatpak means that the base, immutable Endless OS install is just 4.2 GB. Of course there is room for improvement, and years ago I wrote a quick hack to help study exactly which files would ideally be shared between two runtimes but are not. At the time, the primary cause was non-reproducible builds. Since then, the Flatpak ecosystem has moved over to Buildstream which should help a lot, though I haven’t rerun the experiments except for what you see above. Automated statistics about apps using obsolete runtimes might be useful for the Flathub community, as might automated runtime updates, and some further work on understanding if and how the derived runtimes (GNOME & KDE) could share more with the freedesktop runtime version they are based on. And, would widespread use of filesystems that support block-level deduplication (like btrfs) help?

Endless Orange Week: GNOME on WSL

The week of 8th–12th November was Endless Orange Week, a program where the entire Endless OS Foundation team engaged in projects designed to grow our collective learning related to our skills, work and mission. My project was to explore running a complete GNOME desktop in a window on Windows, via Windows Subsystem for Linux.

Screenshot of Windows Remote Desktop Connection window, containing a GNOME desktop running Hack and GNOME Setttings, in turn showing Virtualization: wsl

Why?!

We’ve long faced the challenge of getting Endless OS into the hands of existing PC users, whether to use it themselves or to try it out with a view to a larger deployment. Most people don’t know what an OS is, and even if they have a spare PC find the process of replacing the OS technically challenging. Over the years, we’ve tried various approaches: live USBs, an ultra-simple standalone installer (as seen in GNOME OS), dual-booting with Windows (with a 3-click installer app), virtual machine images, and so on. These have been modestly successful – 5% of our download users are using a dual-boot system, for example – but there’s still room for improvement. (I have a personal interest in this problem space because it’s what I joined Endless to work on in 2016!)

In the last few years, it’s become possible to run Linux executables on Windows, using Windows Subsystem for Linux (WSL). Installing the Debian app from the Windows Store gives you a command-line environment which works pretty much like a normal Debian system, running atop a Microsoft-supplied Linux kernel. Most recently, unmodified applications can present X11 or Wayland windows and play audio through PulseAudio; behind the scenes, a Microsoft-supplied distribution with their branches of Weston and PulseAudio exports each Wayland or X11 window & its audio over RDP to the host system, where it appears as a free-standing window like any other. There is also support upstream in Mesa for using the host system’s GPU, with DirectX commands forwarded to the host via a miraculous kernel interface.

This raises an interesting question: rather than individual apps installed and launched from a command line, could the whole desktop be run as a window, packaged up into an easy-to-use launcher, and published in the Windows Store? If so, this would be a nice improvement on the other installation methods we’ve tried to date!

Proofs of concept

I spent last week researching this. Yes, you can indeed run a complete GNOME desktop under WSL. I tried two approaches, which each have strengths and weaknesses. I worked with Debian Bookworm, which has GNOME 41 and an up-to-date Mesa with the Direct3D backend. Imagine telling someone 20 years ago that Debian would one day include development headers for DirectX in the main repository!

I packaged up my collection of scripts and hacks into something which can build a suitable rootfs to import into WSL and launch either demo with a single command. There may be things that I got working in my “pet” container that don’t quite work in this replicable “cattle” container, but I did my best.

GNOME desktop as X11 app

GNOME Shell can be run as a so-called “nested session”, with the entire desktop appearing as a window inside your existing session. Thanks to my team-mate Georges Stavracas for his help understanding this mode, and particularly the surprising (to me) detail that the nested session can only be run as an X11 window, not a Wayland window, which explained some baffling errors I saw when I first tried to get this going.

Once you’ve got enough of the environment GNOME expects running (more on this below), you can indeed just run it with a carefully-crafted set of environment variables and command-line arguments, and it appears on your Windows system, resplendent with Weston’s window decorations:

GNOME desktop running a game, in a window on Windows

Apps can even emit sound over PulseAudio as normal, or at least they could once I fixed an edge case in Flatpak’s handling of the PulseAudio socket. It’s hard to show audio in a screenshot, but you can see the sounds being emitted by Sidetrack (the WebKitWebViewProcess) as well as Hack somewhere in the background (the python3 process), both of which are Flatpak apps, and the RDP sink as output.

So on the face of it this seems quite promising! But Shell’s nested mode is primarily intended for development, with the window size fixed at launch by an environment variable with DEBUG in its name.

I found WSLg to be quite fragile. WSLg’s Xwayland often fell over for unknown reasons. I had to go out of my way to install a newer Intel graphics driver than would automatically be used, to get the vGPU support needed for Mesa’s Direct3D backend to work. But having done this, on one of my machines, the driver would just crash with SIGILL – apparently the driver unconditionally uses AVX instructions even if the CPU doesn’t support them. On my higher-end machine, most apps worked fine, but Shell on this stack would display just a few frames and then hang. In both cases, I could work around the problem by forcing Mesa to use software rendering, but this means losing one of the key advantages of WSLg!

Another weird anecdote: using the D3D12 backend, the GTK demo app’s shadertoy demo works fine, but its gears demo doesn’t render anything – and nor does glxgears! Apparently D3D12 just doesn’t like gears⁈

Thanks to Daniel Stone at Collabora for his patient help navigating Mesa and D3D12 passthrough.

GNOME desktop exported over RDP

In the past few GNOME releases, it’s become possible to access the desktop remotely using RDP. This is Windows’ native remote-access protocol, and is also the mechanism used by WSLg to export windows and audio to the host.

So another approach is to launch GNOME Shell in its headless mode, and then explicitly connect to it with Windows’ RDP client. A nice touch in WSL is that, by the magic of binfmt_misc and an automatic mount of the host system’s drive, you can invoke Windows executables on the host from within the WSL environment, so a single launch script can bring up GNOME, then spawn the Windows RDP client on the host with the correct parameters. (This is how WSLg works too.)

Not pictured below are the ugly authentication and certificate warning dialogs during the connection flow:

GNOME desktop, accessed over RDP, showing GTK 4's Shadertoy GL demo

Here, GNOME Shell is (AFAICT) rendering using Mesa’s accelerated D3D12 backend, as are GL applications running on it (the GTK 4 Shadertoy demo). But in this model we lose WSLg’s PulseAudio forwarding, and its use of shared memory to send the pixel contents of the desktop to the client. Both of these are solvable problems, though. GNOME Remote Desktop uses the same RDP library, FreeRDP, as WSLg, and all the other supporting code on the Linux side is open-source. GNOME Remote Desktop uses Pipewire rather than PulseAudio, and Mutter rather than Weston, so WSLg’s RDP plugins for PulseAudio and Weston could not be used directly, but audio forwarding over RDP seems a desirable feature to support for normal remoting use-cases. Supporting the shared-memory transport for RDP in GNOME Remote Desktop is perhaps a harder sell, but in principle it could be done.

Just like the nested session, the dimensions of a headless GNOME Shell session are currently fixed on startup. But again I think this would be desirable to solve anyway: this already works well for regular virtual machines, and when connecting to Windows RDP servers.

Rough edges

When you start a WSL shell, PID 1 is an init process provided by WSL, and that’s pretty much all you have: no systemd, no D-Bus system or session bus, nothing. GNOME requires, at least, a functioning system and session bus with various services on them. So for this prototype I used genie, which launches systemd in its own PID namespace and gives you shells within. This works OK, once you change the default target to not try to bring up a full graphical session, disable features not supported by the WSL kernel, and deal with something trampling on WSLg’s X11 sockets. (I thought it is systemd-tmpfiles, but I tried masking the x11.conf file with no success, so I hacked around it for now.) It may be easier to manually launch the D-Bus system bus and session bus without systemd, and run gnome-session in its non-systemd mode, but I expect over time that running GNOME without a systemd user instance will be an increasingly obscure configuration.

Speaking of X11 sockets: both my demos launch GNOME Shell as a pure Wayland compositor, without X11 support. This is because Mutter requires the X11 socket directory to have the sticky bit set, for security reasons, and refuses to start Xwayland if this is not true. But on WSLg /tmp/.X11-unix is a symlink; it is not possible to set the sticky bits on symlinks, and Mutter uses lstat() to explicitly check the symlink’s permissions rather than its target. I think it would be safe to check the symlink’s target instead, provided that Mutter also checked that /tmp had the sticky bit (preventing the symlink from being replaced), but I haven’t fully thought this through.

WSL is non-trivial to set up. The first time you try to run a WSL distro installed from the Windows Store, you have to follow a link to a support article which tells you how to install WSL, which involves a trip deep into the Settings app followed by a download and reboot. As mentioned above, I also had issues with the vGPU support in Intel’s driver on both my systems, which I had to go out of my way to install in the first place, and WSLg’s Xwayland session was somewhat unstable. So I fear it may not be much easier for a non-technical user than our existing installation methods. Perhaps this will change over time.

GNOME Remote Desktop’s RDP backend needs some manual set-up at present. You have to manually generate a TLS key and certificate, set up a new username & password combo, and set the session to be read-write. You also have to arrange for the GNOME Keyring to be unlocked in your headless session, which is a nice chicken-and-egg problem. Once you’ve done all this (and remembered to install a PipeWire session manager in your minimal container) it works rather nicely, and I know that design and engineering work is ongoing to make the set-up easier.

Conclusions

Although I got the desktop running, and there are some obvious bits of follow-up work that could be done to make the experience better, I don’t think this is currently a viable approach for making it easier to try GNOME or Endless OS. There is too much manual set-up required; while it might be possible to bundle some of the Linux side of this into the WSL wrapper app, installing WSL itself is still quite cumbersome, if not exactly rocket science. The many moving parts are still rather new, and I hit various crashes and strange behaviour. Even with software rendering, the performance was fine on my relatively high-end developer laptop, but performance was pretty bad on my normal Windows machine, a lower-spec device that might be more representative of computers in general.

I do still think this general approach of running the desktop windowed in a container on a foreign OS is an interesting one to keep an eye on and re-evaluate periodically. Chrome OS is another potential target, since it also supports running arbitrary Linux containers with Wayland forwarding, though my understanding is that it also involves rather a lot of manual set-up and is not supported on managed devices or when parental controls are in use…

I was happy to experience first-hand the progress GNOME has made in supporting RDP. This kind of functionality may not be important to most GNOME developers and enthusiasts but it’s really important in some contexts. I used to work in an environment where I needed remote access to my desktop, and RDP was the only permitted protocol. Back in 2014, the tools to do this on my Linux system were truly dire, particularly if you want to access a normal desktop remotely rather than a virtualised desktop; by contrast, accessing my Windows system from a Windows, Linux or macOS client worked really well. GNOME Remote Desktop has made some big strides in the right direction, with better integration with the desktop and fewer fragile hacks. I’ll keep watching this space with interest.

WSL itself is also an impressive technical achievement, and the entire Linux side of it is free software. There is nice integration with the host system – for example, you can run Visual Studio Code on the host by running code in WSL, and everything works transparently.

On a personal level, I learnt many new things during the course of Endless Orange Week. Besides learning about WSL, I also learnt how to break on a syscall in gdb (catch syscall 16 for ioctl() on x86_64) and inspect the parameter registers; how Mesa chooses its backend (fun fact: most of the modules in /usr/lib/x86_64-linux-gnu/dri/ are hardlinks of one another); the importance of a PipeWire session manager; more about how PID and mount namespaces work; and so on. It was a nice change from my usual day-to-day work, and I think the research is valuable, even if it doesn’t immediately translate into a production project.

Chromium on Flathub

In December 2020, Chromium reached the Flathub stable channel. Assuming you have Flatpak 1.8.2 or newer, and your kernel is configured to allow unprivileged user namespaces, you can download it now.

Screenshot of Chromium showing the Chromium page on flathub.org

History

Endless OS is based on Debian, but rather than releasing as a bunch of .debs, it is released as an immutable OSTree snapshot, with apps added and removed using Flatpak.

For many years, we maintained a branch of Chromium as a traditional OS package which was built into the OS itself, and updated it together with our monthly OS releases. This did not match up well with Chromium, which has a new major version every 6 weeks and typically 2–4 patch versions in between. It’s a security-critical component, and those patch versions invariably fix some rather serious vulnerability. In some ways, web browsers are the best possible example of apps that should be updated independently of the OS. (In a nice parallel, it seems that the Chrome OS folks are also working on separating OS updates from browser updates on Chrome OS.)

Browsers are also the best possible example of apps which should use elaborate sandboxing techniques to limit the impact of security vulnerabilities, and Chromium is indeed a pioneer in this space. Flatpak applies much the same tools to sandbox applications, which ironically made it harder to ship Chromium as a Flatpak: when running in the Flatpak sandbox, it can’t use those same sandboxing APIs provided by the kernel to sandbox itself further.

Flatpak provides its own API for sandboxed applications to launch new instances of themselves with tighter sandboxing; what’s needed is a way to make Chromium use that…

Solution

Ryan Gonzalez has had a long-running project with us to enable Chromium-based apps to work well as Flatpaks. The first targets were apps built with Electron: his zypak project provides an LD_PRELOAD-able library that redirects Chromium’s sandbox to use Flatpak’s sub-sandboxing API. This avoids the need to modify the (often proprietary) apps themselves, and is now used by dozens of Electron apps on Flathub which would otherwise not be usable with Flatpak. There’s also a version of Chrome in the Flathub beta channel using this technique.

For Chromium, we can take a different approach. It’s open-source code, being compiled by Flathub, so Ryan prepared some patches to teach it to use the Flatpak sandboxing APIs directly, for better performance and robustness.

Once the sandbox integration was done, there was a long list of other changes needed to make the Chromium Flatpak work at least as well as our previous built-in version, which André Moreira Magalhães from Endless worked through with Ryan.

Some of these came from the old Endless OS package, such as using a royalty-free implementation of AAC, splitting encumbered codecs to a separate package so they can be excluded as needed for distribution, and discarding background tabs when the system is under memory pressure (which is useful on systems with limited RAM, but is disabled by default on desktop Linux builds).

Others were specific to Flatpak, such as dealing with udev not being available in the sandbox, restoring the ability to create app launchers for websites, integrating with Flatpak’s network proxy portal, and allowing Chromium policy files to be provided by the host system.

Over in Endless OS, we also needed to update users’ existing file associations and migrate their Chromium profiles to its new home.

Impact

Chart of 30 days of Chromium downloads, with three large spikes of around 20,000 daily downloads

The chart above is the Flathub download statistics for Chromium in the past 30 days. Counting the points between 14th March (when the most recent update was pushed) and 21st March, there have been nearly 60,000 downloads. The majority of these will be Endless OS users: our 3.9.2 release in January 2021 rolled this change out to all users, and Endless OS has automatic updates enabled by default. But Flathub has a broader reach than just Endless OS! I believe that users of System76’s Pop!_OS have been migrated from a .deb of Chromium to this Flatpak, and surely there are many users on other distributions, too. It’s also been used as the basis for other apps on Flathub, including ungoogled-chromium.

As an added bonus, the Flatpak is wired up to flatpak-external-data-checker, which now automatically opens a pull request when a new Chromium release is published. Typically, new major releases need manual intervention to refresh the Flatpak patches, but minor releases often build without issue: for these, one can just smoke-test the test build from the pull request, and then merge it, reducing what used to be days of effort rebasing the package in Endless OS to the work of minutes. I love it when a plan comes together.

A quick glance at the issues on the flathub/org.chromium.Chromium repo will show that there is always more work to be done. We would love to see other distributions getting involved, reducing the duplicated work of maintaining Chromium packages for each distro, and making it easier for users of long-term stable branches to get important browser updates quickly and easily.

Bustleman’s Holiday

I recently had a few weeks off work, and mostly did a good job of staying away from my computer. However, I couldn’t resist putting in a few hours to release Bustle 0.8.0, someone’s favourite D-Bus profiler. Get it on Flathub today!

Screenshot of Bustle 0.8.0's About dialog. “Someone's favourite D-Bus profiler”
I never have clarified whose favourite D-Bus profiler it is…

The main user-facing change is a great new icon, designed and drawn by Tobias Bernard. (Unfortunately the Flathub website still shows the old one for CDN cache reasons, but you can feast your eyes above or in your favourite app centre.)

I also rewrote some parts of the app from Haskell to C. In particular, it no longer uses the Haskell D-Bus implementation and libpcap binding, instead using GDBus and GVariant (via some minimal hand-written FFI bindings) and some mostly-existing C code, respectively. Me circa 2008 would be horrified to learn that I’ve done this, and me circa 2020 is sad to be introducing more unsafe code, but the net result is a smaller app with way fewer transitive dependencies.

Why can’t it use GDBus via GObject Introspection, you ask? GLib didn’t have a D-Bus implementation in 2008, and there was no GObject Introspection support for Haskell. These days, bindings for Gio etc. generated via GObject Introspection exist, but migrating Bustle to these would amount to a rewrite, and I don’t have the time or stomach for that. (These libraries all have my name on them because I made a small start at the generator back in 2011 with the express goal of using them in Bustle, but I’d be amazed if any of my work is still present.)

I’ve been fortunate to have had a dozen or so contributors to Bustle over the years, but it’s certainly true that my choice of implementation language has been a big barrier to entry for other contributors. I maintain that it was not a bad choice in and of itself for a weekend project: in 2008, a time before GObject Introspection was widely available, the Haskell GTK and Cairo bindings were unusually thorough and well-written and allowed me to write concise code and iterate quickly. But going against the grain of the community has a cost: the intersection of “Haskell programmer”, “D-Bus expert” and “GNOME enthusiast” is a very small set, and I know several people personally who were interested in contributing until they saw the language.

Why C rather than Rust, you ask? If you want to try to glue Cabal, Cargo and C together, be my guest.

(In case the joke in the title doesn’t translate outside the UK: busman’s holiday on Wiktionary.)

GNOME 3.36 / Endless OS 3.8

Endless OS 3.8.0 has just been released, which brings GNOME 3.36 to our users. There’s a particularly big overlap between “improvements in Endless OS” and “improvements in GNOME” this cycle, so I wanted to take a minute to look back over what the Endless team worked on in GNOME 3.36.

Login & Unlock Screen

Allan Day has already written about the improvements to the login and unlock experience in GNOME 3.36, so I won’t retread this ground in too much detail. As he (and Nick Richards, in his trip report for Endless OS 3.8.0) mentioned, this change has been anticipated for a long time, so I’m particularly glad that Georges Stavracas and Umang Jain (together with Florian Müllner from Red Hat) could make this happen for this release. The first thing I interact with when I sit down at my computer is the login screen or the lock screen, and the refreshed design is a pleasure to use. (My daughter is sad that Granny’s cat is no longer visible on the lock screen, though.)

GNOME unlock dialog, with Will Thompson's name and face, and password “Tremendousdangerouslookingyak” visible

Peek Password

One improvement that’s perhaps most visible in the redesigned lock screen is the inline “eye” icon to reveal the text in the password field, which was implemented by Umang Jain independently of his work on the lock screen itself. The motivation for this change was actually another system dialogue: the Wi-Fi password dialogue.

During the development of the Hack product – a game-like platform for self-directed learning built atop Endless OS – the team ran many playtesting sessions. While the emphasis of these sessions was on Hack itself, the test users – typically younger teens – would often run through initial setup on a freshly-installed OS. Within a few clicks of turning on the computer, you select your Wi-Fi network and enter its password, which turned out to be a big stumbling block for many users. Wi-Fi passwords are long strings of randomly-generated characters, and on many occasions users simply couldn’t enter the password correctly. The entry has always had a Show Text option in the right-click menu, but right-clicking is itself an unfamiliar operation for younger users more familiar with mobile devices.

Parental Controls, redux

For a year or so, Endless OS has included a parental controls feature, which operates along a couple of axes:

  • Specific installed apps can be disabled for particular users. As a special case, all general-purpose web browsers are controlled by a separate toggle.
  • Not-yet-installed apps visible in GNOME Software — which we rebrand as App Center — can be filtered based on their OARS content rating metadata.
  • Users can be prevented from installing apps at all.

In past releases, this feature was hard to discover and use. At a superficial level, the UI to control it was buried in Settings → Details → Users → (select a non-administrator user) → (scroll down) → (frame within frame within frame). But the real issue was that many Endless OS systems have the child as the primary, administrator user, created through Initial Setup when the machine is unboxed. To meaningfully use parental controls, you’d need to create a separate parent user, then downgrade the child’s account, neither of which is a particularly discoverable operation.

In autumn last year, we met with Allan Day, Richard Hughes and Matthias Clasen from Red Hat to talk through this problem space. Following that, Robin Tafel, Philip Withnall and Matthew Leeds designed and implemented a new flow for parental controls. The key changes are:

  1. Parental controls can be enabled during initial setup. Check a box, choose some options, and specify a parent password.
  2. Once initial setup is complete, there is a dedicated Parental Controls app.

Screenshot of “About You” page from GNOME Initial Setup, showing “Set up parental controls for this user” checkbox (checked)

Screenshot of Parental Controls page of GNOME Initial Setup, showing options to restrict which applications can be installed or used

Screenshot of GNOME Initial Setup “Set a Parent Password” page, with two password fields and one password hint field

Screenshot of Parental Controls application, showing options to restrict which apps a user can install or run

There are a few downstream bits and bobs outstanding, such as a cross-reference from GNOME Settings’ Users panel, but the bulk of this feature is available upstream in GNOME Initial Setup, Software, and Shell 3.36. Parental controls needs close integration with the application management infrastructure, and Flatpak upstream has the necessary hooks. On Endless OS, supporting Flatpak apps — plus Chromium as a special case — is good enough, since that is the sole mechanism for installing applications. It would be great to see support in Malcontent for other package and app managers.

Special thanks to Jakub Steiner for creating a great icon at very short notice.

Malcontent icon: Silhoutte of parent and child holding hands

Renaming Folders

One of the biggest differences between vanilla GNOME and Endless OS is the app grid, which in Endless is on the desktop and fully under the user’s control. Georges Stavracas has been incrementally chipping away at this, and support for renaming folders landed in GNOME 3.36.

Screenshot of renaming a folder titled “Jeux”

The Long Tail

Besides highly-visible new features and redesigns, much (perhaps even most?) of the work of maintaining a desktop is in the parts you don’t see: improving libraries and plumbing, incremental tweaks to user interfaces, and dealing with the wide variety of hardware, software and users that interact with GNOME. Spelunking through the commit histories of various projects, I see many names of colleagues present and past, including André Moreira Magalhães and Philip Chimento respectively. Jian-Hong Pan from the Endless kernel team makes an appearance in GNOME Settings, as does a feature from erstwhile Endless kernel hacker Carlo Caione dating back to 2018.

Umang Jain, Philip Withnall and Matthew Leeds have put a lot of work into improving the robustness of GNOME Software and Flatpak, and there’s more landing as we speak. I’m particularly glad that Matthew has been tracking down missing Flatpak app updates in GNOME Software – bugs which hide information can be the trickiest ones to spot. And Philip is solving the latest Mystery of the Missing Progress Bar when installing Flatpak apps in GNOME Software.

I’m certain I’ve missed many great contributions. Please forgive me, fellow Endlessers.

A Broad Church

Perhaps my favourite part of being involved in GNOME is collaborating with great people from organisations who, in a different world, might be bitter rivals. All of the work I’ve described was a joint effort with others from the GNOME community; and, just as other distributors share the fruits of our labour, we and our users share the fruits of theirs. This is the latest in a long line of great GNOME releases – long may this trend continue.