On Ubuntu Updates

I’d been planning to announce that Ubuntu has updated Epiphany appropriately in Ubuntu 18.04, 19.04, and its snap packaging, but it seems I took too long and Sebastien has beaten me to that. So thank you very much, Sebastien! And also to Ken and Marcus, for helping with the snap update. I believe I owe you, and also Iain, an apology. My last blog post was not very friendly, and writing unfriendly blog posts is not a good way to promote a healthy community. That wasn’t good for GNOME or for Ubuntu.

Still, I was rather surprised by some of the negative reaction to my last post. I took it for granted that readers would understand why I was frustrated, but apparently more explanation is required.

We’re Only Talking about Micro-point Updates!

Some readers complained that stable operating systems should not take updates from upstream, because they could  introduce new bugs. Well, I certainly don’t expect stable operating systems to upgrade to new major release versions. For instance, I wouldn’t expect Ubuntu 18.04, which released with GNOME 3.28, to upgrade packages to GNOME 3.32. That would indeed defeat the goal of providing a stable system to users. We are only talking about micro version updates here, from 3.28.0 to 3.28.1, or 3.28.2, or 3.28.3, etc. These updates generally contain only bugfixes, so the risk of regressions is relatively low. (In exceptional circumstances, new features may be added in point releases, but such occurrences are very rare and carefully-considered; the only one I can think of recently was Media Source Extensions.) That doesn’t mean there are never any regressions, but the number of regressions introduced relative to the number of other bugs fixed should be very small. Sometimes the bugs fixed are quite serious, so stable release updates are essential to providing a quality user experience. Epiphany stable releases usually contain (a) fixes for regressions introduced by the previous major release, and (b) fixes for crashes.

Other readers complained that it’s my fault for releasing software with  bugs in the first place, so I shouldn’t expect operating system updates to fix the bugs. Well, the first point is clearly true, but the second doesn’t follow at all. Expecting free software to be perfect and never make any bad releases is simply unreasonable. The only way to fix problems when they occur is with a software update. GNOME developers try to ensure stable branches remain stable and reliable, so operating systems packaging GNOME can have high confidence in our micro-point releases, even though we are not perfect and cannot expect to never make a mistake. This process works very well in other Linux-based operating systems, like Fedora Workstation.

How Did We Get Here?

The lack of stable release updates for GNOME in Ubuntu has been a serious ongoing problem for most of the past decade, across all packages, not just Epiphany. (Well, probably for much longer than a decade, but my first Ubuntu was 11.10, and I don’t claim to remember how it was before that time.) Look at this comment I wrote on an xscreensaver blog post in 2016, back when I had already been fed up for a long time:

Last week I got a bug report from a Mint user, complaining about a major, game-breaking bug in a little GNOME desktop game that was fixed two and a half years ago. The user only needed a bugfix-only point release upgrade (from the latest Mint version x.y.z to ancient version x.y.z+1) to get the fix. This upgrade would have fixed multiple major issues.

I would say the Mint developers are not even trying, but they actually just inherit this mess from Ubuntu.

So this isn’t just a problem for Ubuntu, but also for every OS based on Ubuntu, including Linux Mint and elementary OS. Now, the game in question way back when was Iagno. Going back to find that old bug, we see the user must have been using Iagno 3.8.2, the version packaged in Ubuntu 14.04 (and therefore the version available in Linux Mint at the time), even though 3.8.3, which fixed the bug, had been available for over two years at that point. We see that I left dissatisfied yet entirely-appropriate comments on Bugzilla, like “I hate to be a distro crusader, but if you use Linux Mint then you are gonna have to live with ancient bugs.”

So this has been a problem for a very long time.

Hello 2019!

But today is 2019. Ubuntu 14.04 is ancient history, and a little game like Iagno is hardly a particularly-important piece of desktop software anyway. Water under the bridge, right? It’d be more interesting to look at what’s going on today, rather than one specific example of a problem from years ago. So, checking the state of a few different packages in Ubuntu 19.04 as of Friday, June 7, I found:

• gnome-shell 3.32.1 update released to Ubuntu 19.04 users on June 3, while 3.32.2 was released upstream on May 14
• mutter 3.32.1 update released to Ubuntu 19.04 users on June 3, while 3.32.2 was released upstream on May 14 (same as gnome-shell)
• glib 2.60.0 never updated in Ubuntu 19.04, while 2.60.1 was released upstream on April 15, and 2.60.3 is the current stable version
• glib-networking 2.60.1 never updated in Ubuntu 19.04, while I released 2.60.2 on May 2
• libsoup 2.66.1 never updated in Ubuntu 19.04, while 2.66.2 was released upstream on May 15

(Update: Sebastien points out that Ubuntu 19.04 shipped with git snapshots of gnome-shell and mutter very close to 3.32.1 due to release schedule constraints, which was surely a reasonable approach given the tight schedule involved. Of course, 3.32.2 is still available now.)

I also checked gnome-settings-daemon, gnome-session, and gdm. All of these are up-to-date in 19.04, but it turns out that there have not been any releases for these components since 3.32.0. So 5/8 of the packages I checked are currently outdated, and the three that aren’t had no new versions released since the original 19.04 release date. Now, eight packages is a small and very unscientific review — I haven’t looked at any packages other than the few listed here — but I think you’ll agree this is not a good showing. I leave it as an exercise for the reader to check more packages and see if you find similar results. (You will.)

Of course, I don’t expect all packages to be updated immediately. It’s reasonable to delay updates by a couple weeks, to allow time for testing. But that’s clearly not what’s happening here. (Update #2: Marco points out that Ubuntu is not shipping gnome-shell and mutter 3.32.2 yet due to specific regressions. So actually, I’m wrong and allowing time for testing is exactly what’s happening here, in these particular cases. Surprise! So let’s not count outdated gnome-shell and mutter against Ubuntu, and say 3/8 of the packages are old instead of 5/8. Still not great results, though.)

Having outdated dependencies like GLib 2.60.0 instead of 2.60.3 can cause just as serious problems as outdated applications: in Epiphany’s case, there are multiple fixes for name resolution problems introduced since GLib 2.58 that are missing from the GLib 2.60.0 release. When you use an operating system that provides regular, comprehensive stable release updates, like Fedora Workstation, you can be highly confident that you will receive such fixes in a timely manner, but no such confidence is available for Ubuntu users, nor for users of operating systems derived from Ubuntu.

So Epiphany and Iagno are hardly isolated examples, and these are hardly recent problems. They’re widespread and longstanding issues with Ubuntu packaging.

Upstream Release Monitoring is Essential

Performing some one-time package updates is (usually) easy. Now that the Epiphany packages are updated, the question becomes: will they remain updated in Ubuntu going forward? Previously, I had every reason to believe they would not. But for the first time, I am now cautiously optimistic. Look at what Sebastien wrote in his recent post:

Also while we have tools to track available updates, our reports are currently only for the active distro and not stable series which is a gap and leads us sometime to miss some updates.
I’ve now hacked up a stable report and reviewed the current output and we will work on updating a few components that are currently outdated as a result.

It’s no wonder that you can’t reliably provide stable release updates without upstream release monitoring. How can you provide an update if you don’t know that the update is available? It’s too hard for humans to manually keep track of hundreds of packages, especially with limited developer resources, so quality operating systems have an automated process for upstream release monitoring to notify them when updates are available. In Fedora, we use https://release-monitoring.org/ for most packages, which is an easy solution available for other operating systems to use. Without appropriate tooling, offering updates in a timely manner is impractical.

So now that Sebastien has a tool to check for outdated GNOME packages, we can hope the situation might improve. Let’s hope it does. It would be nice to see a future where Ubuntu users receive quality, stable software updates.

Dare to Not Package?

Now, I have no complaints with well-maintained, updated OS packages. The current state of Epiphany updates in Ubuntu is (almost) satisfactory to me (with one major caveat, discussed below). But outdated OS packages are extremely harmful. My post two weeks ago was a sincere request to remove the Epiphany packages from Ubuntu, because they were doing much more harm than good, and, due to extreme lack of trust built up over the course of the past decade, I didn’t trust Ubuntu to fix the problem and keep it fixed. (I am still only “cautiously optimistic” that things might improve, after all: not at all confident.) Bugs that we fixed upstream long ago lingered in the Ubuntu packages, causing our project serious reputational harm. If I could choose between outdated packages and no packages at all, there’s no question that I would greatly prefer the later.

You know what I want more than distros not theming my app out from under me? Them actually keeping the packages up to date. My call to action is far more drastic, a call to not package.

— Christian Hergert 👣 (@hergertme) May 28, 2019

As long as operating system packages are kept up-to-date — with the latest micro-point release corresponding to the system’s minor GNOME version — then I don’t mind packages. Conscientiously-maintained operating system packages are fine by me. But only if they are conscientiously-maintained and kept up-to-date!

Not packaging would not be a horrible fate. It would be just fine. The future of Linux application distribution is Flatpak (or, less-likely, snap), and I really don’t mind if we get there sooner rather than later.

Regarding OpenJPEG

We have one more issue with Ubuntu’s packaging left unresolved: OpenJPEG. No amount of software updates will fix Epiphany in Ubuntu if it isn’t web-compatible, and to be web-compatible it needs to display JPEG 2000 images. As long as we have Safari without Chromium in our user agent, we have to display JPEG 2000 images, because, sadly, JPEG 2000 is no longer optional for web compatibility. And we cannot change our user agent because that, too, would break web compatibility. We attempted to use user agent quirks only for websites that served JPEG 2000 images, but quickly discovered it was entirely impractical. The only practical way to avoid the requirement to support JPEG 2000 is to give up on WebKit altogether and become yet another Chromium-based browser. Not today!

Some readers complained that we are at fault for releasing a web browser that depends on OpenJPEG, as if this makes us bad or irresponsible developers. Some of the comments were even surprisingly offensive. Reality is: we have no other options. Zero. The two JPEG 2000 rendering libraries are libjasper and OpenJPEG. libjasper has been removed from both Debian and Ubuntu because it is no longer maintained. That leaves OpenJPEG. Either we use OpenJPEG, or we write our own JPEG 2000 image decoder. We don’t have the resources to do that, so OpenJPEG it is. We also don’t have the resources to fix all the code quality bugs that exist in OpenJPEG. Firefox and Chrome are certainly not going to help us, because they are big enough that they don’t need to support JPEG 2000 at all. So instead, we’ve devoted resources to sandboxing WebKit with bubblewrap. This will mitigate the damage potential from OpenJPEG exploits. Once the sandbox is enabled — which we hope to be ready for WebKitGTK 2.26 — then an OpenJPEG exploit will be minimally-useful unless combined with a bubblewrap sandbox escape. bubblewrap is amazing technology, and I’m confident this was the best choice of where to devote our resources. (Update: To clarify, the bubblewrap sandbox is for the entire web process, not just the OpenJPEG decoder.)

Of course, it would be good to improve OpenJPEG. I repeat my previous call for assistance with the OpenJPEG code quality issues reported by Ubuntu, but as before, I only expect to hear crickets.

So unfortunately, we’re not yet at a point where I’m comfortable with Epiphany’s Ubuntu packaging. (Well, the problem is actually in the WebKit packaging. Details.) I insist: distributing Epiphany without support for JPEG 2000 images is harmful and makes Epiphany look bad. Please, Ubuntu, we need you to either build WebKit with OpenJPEG enabled, or else just drop your Epiphany packages entirely, one or the other. Whichever you choose will make me happy. Please don’t accept the status quo!