Michael Catanzaro's Blog

Category: GNOME

  • Sandboxing WebKitGTK Apps

    When you connect to a Wi-Fi network, that network might block your access to the wider internet until you’ve signed into the network’s captive portal page. An untrusted network can disrupt your connection at any time by blocking secure requests and replacing the content of insecure requests with its login page. (Of course this can be done on wired networks as well, but in practice it mainly happens on Wi-Fi.) To detect a captive portal, NetworkManager sends a request to a special test address (e.g. http://fedoraproject.org/static/hotspot.txt) and checks to see whether it the content has been replaced. If so, GNOME Shell will open a little WebKitGTK browser window to display http://nmcheck.gnome.org, which, due to the captive portal, will be hijacked by your hotel or airport or whatever to display the portal login page. Rephrased in security lingo: an untrusted network may cause GNOME Shell to load arbitrary web content whenever it wants. If that doesn’t immediately sound dangerous to you, let’s ask me from four years ago why that might be bad:

    Web engines are full of security vulnerabilities, like buffer overflows and use-after-frees. The details don’t matter; what’s important is that skilled attackers can turn these vulnerabilities into exploits, using carefully-crafted HTML to gain total control of your user account on your computer (or your phone). They can then install malware, read all the files in your home directory, use your computer in a botnet to attack websites, and do basically whatever they want with it.

    If the web engine is sandboxed, then a second type of attack, called a sandbox escape, is needed. This makes it dramatically more difficult to exploit vulnerabilities.

    The captive portal helper will pop up and load arbitrary web content without user interaction, so there’s nothing you as a user could possibly do about it. This makes it a tempting target for attackers, so we want to ensure that users are safe in the absence of a sandbox escape. Accordingly, beginning with GNOME 3.36, the captive portal helper is now sandboxed.

    How did we do it? With basically one line of code (plus a check to ensure the WebKitGTK version is new enough). To sandbox any WebKitGTK app, just call webkit_web_context_set_sandbox_enabled(). Ta-da, your application is now magically secure!

    No, really, that’s all you need to do. So if it’s that simple, why isn’t the sandbox enabled by default? It can break applications that use WebKitWebExtension to run custom code in the sandboxed web process, so you’ll need to test to ensure that your application still works properly after enabling the sandbox. (The WebKitGTK sandbox will become mandatory in the future when porting applications to GTK 4. That’s thinking far ahead, though, because GTK 4 isn’t supported yet at all.) You may need to use webkit_web_context_add_path_to_sandbox() to give your web extension access to directories that would otherwise be blocked by the sandbox.

    The sandbox is critically important for web browsers and email clients, which are constantly displaying untrusted web content. But really, every app should enable it. Fix your apps! Then thank Patrick Griffis from Igalia for developing WebKitGTK’s sandbox, and the bubblewrap, Flatpak, and xdg-desktop-portal developers for providing the groundwork that makes it all possible.

  • Epiphany 3.36 and WebKitGTK 2.28

    So, what’s new in Epiphany 3.36?

    PDF.js

    Once upon a time, beginning with GNOME 3.14, Epiphany had supported displaying PDF documents via the Evince NPAPI browser plugin developed by Carlos Garcia Campos. Unfortunately, because NPAPI plugins have to use X11-specific APIs to draw web content, this didn’t  suffice for very long. When GNOME switched to Wayland by default in GNOME 3.24 (yes, that was three years ago!), this functionality was left behind. Using an NPAPI plugin also meant the code was inherently unsandboxable and tied to a deprecated technology. Epiphany disabled support for NPAPI plugins by default in Epiphany 3.30, hiding the functionality behind a hidden setting, which has now finally been removed for Epiphany 3.36, killing off NPAPI for good.

    Jan-Michael Brummer, who comaintains Epiphany with me, tried bringing back PDF support for Epiphany 3.34 using libevince, but eventually we decided to give up on this approach due to difficulty solving some user experience issues. Also, the rendering occurred in the unsandboxed UI process, which was again not good for security.

    But PDF support is now back in Epiphany 3.36, and much better than before! Thanks to Jan-Michael, Epiphany now supports displaying PDFs using the amazing PDF.js. We are thankful for Mozilla’s work in developing PDF.js and open sourcing it for us to use. Viewing PDFs in Epiphany using PDF.js is more convenient than downloading them and opening them in Evince, and because the PDF is rendered in the sandboxed web process, using web technologies rather than poppler, it’s also approximately one bazillion times more secure.

    Screenshot of Epiphany displaying a PDF document
    Look, it’s a PDF!

    One limitation of PDF.js is that it does not support forms. If you need to fill out PDF forms, you’ll need to download the PDF and open it in Evince, just as you would if using Firefox.

    Dark Mode

    Thanks to Carlos Garcia, it should finally be possible to use Epiphany with dark GTK themes. WebKitGTK has historically rendered HTML elements using the GTK theme, which has not been good for users of dark themes, which broke badly on many websites, usually due to dark text being drawn on dark backgrounds or various other problems with unexpected dark widgets. Since WebKitGTK 2.28, WebKit will try to manually change to a light GTK theme when it thinks a dark theme is in use, then use the light theme to render web content. (This work has actually been backported to WebKitGTK 2.26.4, so you don’t need to upgrade to WebKitGTK 2.28 to benefit, but the work landed very recently and we haven’t blogged about it yet.) Thanks to Cassidy James from elementary for providing example pages for testing dark mode behavior.

    Screenshot demonstrating broken dark mode support
    Broken dark mode support prior to WebKitGTK 2.26.4. Notice that the first two pages use dark color schemes when light color schemes are expected, and the dark blue links are hard to read over the dark gray background. Also notice that the text in the second image is unreadable.
    Screenshot demonstrating fixed dark mode support in WebKitGTK 2.26.4
    Since WebKitGTK 2.26.4, dark mode works as it does in most other browsers. Websites that don’t support dark mode are light, and websites that do support dark mode are dark. Widgets themed using GTK are always light.

    Since Carlos had already added support for the prefers-color-scheme media query last year, this now gets us up to dark mode parity with most browsers, except, notably, Safari. Unlike other browsers, Safari allows websites to opt-in to rendering dark system widgets, like WebKitGTK used to do before these changes. Whether to support this in WebKitGTK remains to-be-determined.

    Process Swap on Navigation (PSON)

    PSON, which debuted in Safari 13, is a major change in WebKit’s process model. PSON is the first component of site isolation, which Chrome has supported for some time, and which Firefox is currently working towards. If you care about web security, you should care a lot about site isolation, because the web browser community has arrived at a consensus that this is the best way to mitigate speculative execution attacks.

    Nowadays, all modern web browsers use separate, sandboxed helper processes to render web content, ensuring that the main user interface process, which is unsandboxed, does not touch untrusted web content. Prior to 3.36, Epiphany already used a separate web process to display each browser tab (except for “related views,” where one tab opens another and gains scripting ability over the opened tab, subject to the Same Origin Policy). But in Epiphany 3.36, we now also have a separate web process per website. Each tab will swap between different web processes when navigating between different websites, to prevent any one web process from loading content from different websites.

    To make these process swap navigations fast, a pool of prewarmed processes is used to hide the startup cost of launching a new process by ensuring the new process exists before it’s needed; otherwise, the overhead of launching a new web process to perform the navigation would become noticeable. And suspended processes live on after they’re no longer in use because they may be needed for back/forward navigations, which use WebKit’s page cache when possible. (In the page cache, pages are kept in memory indefinitely, to make back/forward navigations fast.)

    Due to internal refactoring, PSON previously necessitated some API breakage in WebKitGTK 2.26 that affected Evolution and Geary: WebKitGTK 2.26 deprecated WebKit’s single web process model and required that all applications use one web process per web view, which Evolution and Geary were not, at the time, prepared to handle. We tried hard to avoid this, because we hate to make behavioral changes that break applications, but in this case we decided it was unavoidable. That was the status quo in 2.26, without PSON, which we disabled just before releasing 2.26 in order to limit application breakage to just Evolution and Geary. Now, in WebKitGTK 2.28, PSON is finally available for applications to use on an opt-in basis. (It will become mandatory in the future, for GTK 4 applications.) Epiphany 3.36 opts in. To make this work, Carlos Garcia designed new WebKitGTK APIs for cross-process communication, and used them to replace the private D-Bus server that Epiphany previously used for this purpose.

    WebKit still has a long way to go to fully implement site isolation, but PSON is a major step down that road. Thanks to Brady Eidson and Chris Dumez from Apple for making this work, and to Carlos Garcia for handling most of the breakage (there was a lot). As with any major intrusive change of such magnitude, regressions are inevitable, so don’t hesitate to report issues on WebKit Bugzilla.

    highlight.js

    Once upon a time, WebKit had its own implementation for viewing page source, but this was removed from WebKit way back in 2014, in WebKitGTK 2.6. Ever since, Epiphany would open your default text editor, usually gedit, to display page source. Suffice to say that this was not a very satisfactory solution.

    I finally managed to implement view source mode at the Epiphany level for Epiphany 3.30, but I had trouble making syntax highlighting work. I tried using various open source syntax highlighting libraries, but most are designed to highlight small amounts of code, not large web pages. The libraries I tried were not fast enough, so I gave up on syntax highlighting at the time.

    Thanks to Jan-Michael, Epiphany 3.36 supports syntax highlighting using highlight.js, so we finally have view source mode working fully properly once again. It works much better than my failed attempts with different JS libraries. Please thank the highlight.js developers for maintaining this library, and for making it open source.

    Screenshot displaying Epiphany's view source mode
    Colors!

    Service Workers

    Service workers are now available in WebKitGTK 2.28. Our friends at Apple had already implemented service worker support a couple years ago for Safari 11, but we were pretty slow in bringing this functionality to Linux. Finally, WebKitGTK should now be up to par with Safari in this regard.

    Cookies!

    Patrick Griffis has updated libsoup and WebKitGTK to support SameSite cookies. He’s also tightened up our cookie policy by implementing strict secure cookies, which prevents http:// pages from setting secure cookies (as they could overwrite secure cookies set by https:// pages).

    Adaptive Design

    As usual, there are more adaptive design improvements throughout the browser, to provide a better user experience on the Librem 5. There’s still more work to be done here, but Epiphany continues to provide the best user experience of any Linux browser at small screen sizes. Thanks to Adrien Plazas and Jan-Michael for their continued work on this.

    Screenshot showing Epiphany running in mobile mode at small window size.
    As before, simply resize your browser window to see Epiphany dynamically transition between desktop mode and mobile mode.

    elementary OS

    With help from Alexander Mikhaylenko, we’ve also upstreamed many elementary OS design changes, which will be used when running under the Pantheon desktop (and not impact users on other desktops), so that the elementary developers don’t need to maintain their customizations as separate patches anymore. This will eliminate a few elementary-specific bugs, including some keyboard shortcuts that were previously broken only in elementary, and some odd tab bar behavior. Although Epiphany still doesn’t feel quite as native as an app designed just for elementary OS, it’s getting closer.

    Epiphany 3.34

    I failed to blog about Epiphany 3.34 when I released it last September. Hopefully you have updated to 3.34 already, and are already enjoying the two big features from this release: the new adblocker, and the bubblewrap sandbox.

    The new adblocker is based on WebKit Content Blockers, which was developed by Apple several years ago. Adrian Perez developed new WebKitGTK API to expose this functionality, changed Epiphany to use it, and deleted Epiphany’s older resource-hungry adblocker that was originally copied from Midori. Previously, Epiphany kept a large GHashMap of compiled regexes in every web process, consuming a very significant amount of RAM for each process. It also took time to compile these regexes when launching each new web process. Now, the adblock filters are instead compiled into an efficient bytecode format that gets mmapped between all web processes to avoid excessive resource use. The bytecode is interpreted by WebKit itself, rather than by Epiphany’s web process extension (which Epiphany uses to execute custom code in WebKit’s web process), for greatly improved performance.

    Lastly, Epiphany 3.34 enabled Patrick’s bubblewrap sandbox, which was added in WebKitGTK 2.26. Bubblewrap is an amazing sandboxing tool, already used effectively by flatpak and rpm-ostree, and I’m very pleased with Patrick’s decision to use it for WebKit as well. Because enabling the sandbox can break applications, it is currently opt-in for GTK 3 apps (but will become mandatory for GTK 4 apps). If your application uses WebKitGTK, you really need to take some time to enable this sandbox using webkit_web_context_set_sandbox_enabled(). The sandbox has introduced a couple regressions that we didn’t notice until too late; notably,  printing no longer works, which, half a year later, we still haven’t managed to fix yet. (I’ll try to get to it soon.)

    OK, this concludes your 3.36 and 3.34 updates. Onward to 3.38!

  • Enable Git Commit Message Syntax Highlighting in Vim on Fedora

    Were you looking forward to reading an exciting blog post about substantive technical issues affecting GNOME or the Linux desktop community? Sorry, not today.

    When setting up new machines, I’m often frustrated by lack of syntax highlighting for git commit messages in vim. On my main workstation, vim uses comforting yellow letters for the first line of my commit message to let me know I’m good on line length, or red background to let me know my first line is too long, and after the first line it automatically inserts a new line break whenever I’ve typed past 72 characters. It’s pretty nice. I can never remember how I get it working in the end, and I spent too long today trying to figure it out yet again. Eventually I realized there was another difference besides the missing syntax highlighting: I couldn’t see the current line or column number, and I couldn’t see the mode indicator either. Now you might be able to guess my mistake: git was not using /usr/bin/vim at all! Because Fedora doesn’t have a default $EDITOR, git defaults to using /usr/bin/vi, which is basically sad trap vim. Solution:

    $ git config --global core.editor vim

    You also have to install the vim-enhanced package to get /usr/bin/vim, but that’s a lot harder to forget to do.

    You’re welcome, Internet!

  • Let’s Learn Spelling!

    Were you looking forward to reading an exciting blog post about substantive technical issues affecting GNOME or the Linux desktop community? Sorry, not today.

    GNOME

    It used to be an acronym, so it’s all uppercase. Write “GNOME,” never “Gnome.” Please stop writing “Gnome.”

    Would it help if you imagine an adorable little garden gnome dying each time you get it wrong?

    If you’re lazy and hate capital letters, or for technical contexts like package or project names, then all-lowercase “gnome” might be appropriate, but “Gnome” certainly never is.

    Red Hat

    This one’s not that hard. Why are some people writing “RedHat” without any space? It doesn’t make sense. Red Hat. Easy!

    SUSE and openSUSE

    S.u.S.E. and SuSE are both older spellings for the company currently called SUSE. Apparently at some point in the past they realized that the lowercase u was stupid and causes readers’ eyes to bleed.  Can we please let it die?

    Similarly, openSUSE is spelled “openSUSE,” not “OpenSUSE.” Do not capitalize the o, even if it’s the first word in a sentence. Do not write “openSuSE” or “OpenSuSE” (which people somehow manage to do even when they’re not trolling) or anything at all other than “openSUSE.” I know this is probably too much to ask, but once you get the hang of it, it’s not so hard.

    elementary OS

    I don’t often see this one messed up. If you can write elementary OS, you can probably write openSUSE properly too! They’re basically the same structure, right? All lowercase, then all caps. I have faith in you, dear reader! Don’t let me down!

    GTK and WebKitGTK

    We removed the + from the end of both of these, because it was awful. You’re welcome!

    Again, all lowercase is probably OK in technical contexts. “gtk-webkit” is not. WebKitGTK.

  • Epiphany Technology Preview Users: Action Required

    Epiphany Technology Preview has moved from https://sdk.gnome.org to https://nightly.gnome.org. The old Epiphany Technology Preview is now end-of-life. Action is required to update. If you installed Epiphany Technology Preview prior to a couple minutes ago, uninstall it using GNOME Software and then reinstall using this new flatpakref.

    Apologies for this disruption.

    The main benefit to end users is that you’ll no longer need separate remotes for nightly runtimes and nightly applications, because everything is now hosted in one repo. See Abderrahim’s announcement for full details on why this transition is occurring.

  • WebKit Vulnerabilities Facilitate Human Rights Abuses

    Chinese state actors have recently abused vulnerabilities in the JavaScriptCore component of WebKit to hack the personal computing devices of Uighur Muslims in the Xinjiang region of China. Mass digital surveillance is a key component of China’s ongoing brutal human rights crackdown in the region.

    This has resulted in a public relations drama that is largely a distraction to the issue at hand. Whatever big-company PR departments have to say on the matter, I have no doubt that the developers working on WebKit recognize the severity of this incident and are grateful to Project Zero, which reported these vulnerabilities and has previously provided numerous other high-quality private vulnerability reports. (Many other organizations deserve credit for similar reports, especially Trend Micro’s Zero Day Initiative.)

    WebKit as a project will need to reassess certain software development practices that may have facilitated the abuse of these vulnerabilities. The practice of committing security fixes to open source long in advance of corresponding Safari releases may need to be reconsidered.

    Sadly, Uighurs should assume their personal computing devices have been compromised by state-sponsored attackers, and that their private communications are not private. Even if not compromised in this particular incident, similar successful attacks are overwhelmingly likely in the future.

  • On Version Numbers

    I’m excited to announce that Epiphany Tech Preview has reached version 3.33.3-33, as computed by git describe. That is 33 commits after 3.33.3:

    Epiphany about dialog displaying the version number

    I’m afraid 3.33.4 will arrive long before we  make it to 3.33.3-333, so this is probably the last cool version number Epiphany will ever have.

    I might be guilty of using an empty commit to claim the -33 commit.

    I might also apologize for wasting your time with a useless blog post, except this was rather fun. I await the controversy of your choice in the comments.

  • An OpenJPEG Surprise

    My previous blog post seems to have resolved most concerns about my requests for Ubuntu stable release updates, but I again received rather a lot of criticism for the choice to make WebKit depend on OpenJPEG, even though my previous post explained clearly why there are are not any good alternatives.

    I was surprised to receive a pointer to ffmpeg, which has its own JPEG 2000 decoder that I did not know about. However, we can immediately dismiss this option due to legal problems with depending on ffmpeg. I also received a pointer to a resurrected libjasper, which is interesting, but since libjasper was removed from Ubuntu, its status is not currently better than OpenJPEG.

    But there is some good news! I have looked through Ubuntu’s security review of the OpenJPEG code and found some surprising results. Half the reported issues affect the library’s companion tools, not the library itself. And the other half of the issues affect the libmj2 library, a component of OpenJPEG that is not built by Ubuntu and not used by WebKit. So while these are real security issues that raise concerns about the quality of the OpenJPEG codebase, none of them actually affect OpenJPEG as used by WebKit. Yay!

    The remaining concern is that huge input sizes might cause problems within the library that we don’t yet know about. We don’t know because OpenJPEG’s fuzzer discards huge images instead of testing them. Ubuntu’s security team thinks there’s a good chance that fixing the fuzzer could uncover currently-unknown multiplication overflow issues, for instance, a class of vulnerability that OpenJPEG has clearly had trouble with in the past. It would be good to see improvement on this front. I don’t think this qualifies as a security vulnerability, but it is certainly a security problem that would facilitate discovering currently-unknown vulnerabilities if fixed.

    Still, on the whole, the situation is not anywhere near as bad as I’d thought. Let’s hope OpenJPEG can be included in Ubuntu main sooner rather than later!

  • On Ubuntu Updates

    I’d been planning to announce that Ubuntu has updated Epiphany appropriately in Ubuntu 18.04, 19.04, and its snap packaging, but it seems I took too long and Sebastien has beaten me to that. So thank you very much, Sebastien! And also to Ken and Marcus, for helping with the snap update. I believe I owe you, and also Iain, an apology. My last blog post was not very friendly, and writing unfriendly blog posts is not a good way to promote a healthy community. That wasn’t good for GNOME or for Ubuntu.

    Still, I was rather surprised by some of the negative reaction to my last post. I took it for granted that readers would understand why I was frustrated, but apparently more explanation is required.

    We’re Only Talking about Micro-point Updates!

    Some readers complained that stable operating systems should not take updates from upstream, because they could  introduce new bugs. Well, I certainly don’t expect stable operating systems to upgrade to new major release versions. For instance, I wouldn’t expect Ubuntu 18.04, which released with GNOME 3.28, to upgrade packages to GNOME 3.32. That would indeed defeat the goal of providing a stable system to users. We are only talking about micro version updates here, from 3.28.0 to 3.28.1, or 3.28.2, or 3.28.3, etc. These updates generally contain only bugfixes, so the risk of regressions is relatively low. (In exceptional circumstances, new features may be added in point releases, but such occurrences are very rare and carefully-considered; the only one I can think of recently was Media Source Extensions.) That doesn’t mean there are never any regressions, but the number of regressions introduced relative to the number of other bugs fixed should be very small. Sometimes the bugs fixed are quite serious, so stable release updates are essential to providing a quality user experience. Epiphany stable releases usually contain (a) fixes for regressions introduced by the previous major release, and (b) fixes for crashes.

    Other readers complained that it’s my fault for releasing software with  bugs in the first place, so I shouldn’t expect operating system updates to fix the bugs. Well, the first point is clearly true, but the second doesn’t follow at all. Expecting free software to be perfect and never make any bad releases is simply unreasonable. The only way to fix problems when they occur is with a software update. GNOME developers try to ensure stable branches remain stable and reliable, so operating systems packaging GNOME can have high confidence in our micro-point releases, even though we are not perfect and cannot expect to never make a mistake. This process works very well in other Linux-based operating systems, like Fedora Workstation.

    How Did We Get Here?

    The lack of stable release updates for GNOME in Ubuntu has been a serious ongoing problem for most of the past decade, across all packages, not just Epiphany. (Well, probably for much longer than a decade, but my first Ubuntu was 11.10, and I don’t claim to remember how it was before that time.) Look at this comment I wrote on an xscreensaver blog post in 2016, back when I had already been fed up for a long time:

    Last week I got a bug report from a Mint user, complaining about a major, game-breaking bug in a little GNOME desktop game that was fixed two and a half years ago. The user only needed a bugfix-only point release upgrade (from the latest Mint version x.y.z to ancient version x.y.z+1) to get the fix. This upgrade would have fixed multiple major issues.

    I would say the Mint developers are not even trying, but they actually just inherit this mess from Ubuntu.

    So this isn’t just a problem for Ubuntu, but also for every OS based on Ubuntu, including Linux Mint and elementary OS. Now, the game in question way back when was Iagno. Going back to find that old bug, we see the user must have been using Iagno 3.8.2, the version packaged in Ubuntu 14.04 (and therefore the version available in Linux Mint at the time), even though 3.8.3, which fixed the bug, had been available for over two years at that point. We see that I left dissatisfied yet entirely-appropriate comments on Bugzilla, like “I hate to be a distro crusader, but if you use Linux Mint then you are gonna have to live with ancient bugs.”

    So this has been a problem for a very long time.

    Hello 2019!

    But today is 2019. Ubuntu 14.04 is ancient history, and a little game like Iagno is hardly a particularly-important piece of desktop software anyway. Water under the bridge, right? It’d be more interesting to look at what’s going on today, rather than one specific example of a problem from years ago. So, checking the state of a few different packages in Ubuntu 19.04 as of Friday, June 7, I found:

    • gnome-shell 3.32.1 update released to Ubuntu 19.04 users on June 3, while 3.32.2 was released upstream on May 14
    • mutter 3.32.1 update released to Ubuntu 19.04 users on June 3, while 3.32.2 was released upstream on May 14 (same as gnome-shell)
    • glib 2.60.0 never updated in Ubuntu 19.04, while 2.60.1 was released upstream on April 15, and 2.60.3 is the current stable version
    • glib-networking 2.60.1 never updated in Ubuntu 19.04, while I released 2.60.2 on May 2
    • libsoup 2.66.1 never updated in Ubuntu 19.04, while 2.66.2 was released upstream on May 15

    (Update: Sebastien points out that Ubuntu 19.04 shipped with git snapshots of gnome-shell and mutter very close to 3.32.1 due to release schedule constraints, which was surely a reasonable approach given the tight schedule involved. Of course, 3.32.2 is still available now.)

    I also checked gnome-settings-daemon, gnome-session, and gdm. All of these are up-to-date in 19.04, but it turns out that there have not been any releases for these components since 3.32.0. So 5/8 of the packages I checked are currently outdated, and the three that aren’t had no new versions released since the original 19.04 release date. Now, eight packages is a small and very unscientific review — I haven’t looked at any packages other than the few listed here — but I think you’ll agree this is not a good showing. I leave it as an exercise for the reader to check more packages and see if you find similar results. (You will.)

    Of course, I don’t expect all packages to be updated immediately. It’s reasonable to delay updates by a couple weeks, to allow time for testing. But that’s clearly not what’s happening here. (Update #2: Marco points out that Ubuntu is not shipping gnome-shell and mutter 3.32.2 yet due to specific regressions. So actually, I’m wrong and allowing time for testing is exactly what’s happening here, in these particular cases. Surprise! So let’s not count outdated gnome-shell and mutter against Ubuntu, and say 3/8 of the packages are old instead of 5/8. Still not great results, though.)

    Having outdated dependencies like GLib 2.60.0 instead of 2.60.3 can cause just as serious problems as outdated applications: in Epiphany’s case, there are multiple fixes for name resolution problems introduced since GLib 2.58 that are missing from the GLib 2.60.0 release. When you use an operating system that provides regular, comprehensive stable release updates, like Fedora Workstation, you can be highly confident that you will receive such fixes in a timely manner, but no such confidence is available for Ubuntu users, nor for users of operating systems derived from Ubuntu.

    So Epiphany and Iagno are hardly isolated examples, and these are hardly recent problems. They’re widespread and longstanding issues with Ubuntu packaging.

    Upstream Release Monitoring is Essential

    Performing some one-time package updates is (usually) easy. Now that the Epiphany packages are updated, the question becomes: will they remain updated in Ubuntu going forward? Previously, I had every reason to believe they would not. But for the first time, I am now cautiously optimistic. Look at what Sebastien wrote in his recent post:

    Also while we have tools to track available updates, our reports are currently only for the active distro and not stable series which is a gap and leads us sometime to miss some updates.
    I’ve now hacked up a stable report and reviewed the current output and we will work on updating a few components that are currently outdated as a result.

    It’s no wonder that you can’t reliably provide stable release updates without upstream release monitoring. How can you provide an update if you don’t know that the update is available? It’s too hard for humans to manually keep track of hundreds of packages, especially with limited developer resources, so quality operating systems have an automated process for upstream release monitoring to notify them when updates are available. In Fedora, we use https://release-monitoring.org/ for most packages, which is an easy solution available for other operating systems to use. Without appropriate tooling, offering updates in a timely manner is impractical.

    So now that Sebastien has a tool to check for outdated GNOME packages, we can hope the situation might improve. Let’s hope it does. It would be nice to see a future where Ubuntu users receive quality, stable software updates.

    Dare to Not Package?

    Now, I have no complaints with well-maintained, updated OS packages. The current state of Epiphany updates in Ubuntu is (almost) satisfactory to me (with one major caveat, discussed below). But outdated OS packages are extremely harmful. My post two weeks ago was a sincere request to remove the Epiphany packages from Ubuntu, because they were doing much more harm than good, and, due to extreme lack of trust built up over the course of the past decade, I didn’t trust Ubuntu to fix the problem and keep it fixed. (I am still only “cautiously optimistic” that things might improve, after all: not at all confident.) Bugs that we fixed upstream long ago lingered in the Ubuntu packages, causing our project serious reputational harm. If I could choose between outdated packages and no packages at all, there’s no question that I would greatly prefer the later.

    As long as operating system packages are kept up-to-date — with the latest micro-point release corresponding to the system’s minor GNOME version — then I don’t mind packages. Conscientiously-maintained operating system packages are fine by me. But only if they are conscientiously-maintained and kept up-to-date!

    Not packaging would not be a horrible fate. It would be just fine. The future of Linux application distribution is Flatpak (or, less-likely, snap), and I really don’t mind if we get there sooner rather than later.

    Regarding OpenJPEG

    We have one more issue with Ubuntu’s packaging left unresolved: OpenJPEG. No amount of software updates will fix Epiphany in Ubuntu if it isn’t web-compatible, and to be web-compatible it needs to display JPEG 2000 images. As long as we have Safari without Chromium in our user agent, we have to display JPEG 2000 images, because, sadly, JPEG 2000 is no longer optional for web compatibility. And we cannot change our user agent because that, too, would break web compatibility. We attempted to use user agent quirks only for websites that served JPEG 2000 images, but quickly discovered it was entirely impractical. The only practical way to avoid the requirement to support JPEG 2000 is to give up on WebKit altogether and become yet another Chromium-based browser. Not today!

    Some readers complained that we are at fault for releasing a web browser that depends on OpenJPEG, as if this makes us bad or irresponsible developers. Some of the comments were even surprisingly offensive. Reality is: we have no other options. Zero. The two JPEG 2000 rendering libraries are libjasper and OpenJPEG. libjasper has been removed from both Debian and Ubuntu because it is no longer maintained. That leaves OpenJPEG. Either we use OpenJPEG, or we write our own JPEG 2000 image decoder. We don’t have the resources to do that, so OpenJPEG it is. We also don’t have the resources to fix all the code quality bugs that exist in OpenJPEG. Firefox and Chrome are certainly not going to help us, because they are big enough that they don’t need to support JPEG 2000 at all. So instead, we’ve devoted resources to sandboxing WebKit with bubblewrap. This will mitigate the damage potential from OpenJPEG exploits. Once the sandbox is enabled — which we hope to be ready for WebKitGTK 2.26 — then an OpenJPEG exploit will be minimally-useful unless combined with a bubblewrap sandbox escape. bubblewrap is amazing technology, and I’m confident this was the best choice of where to devote our resources. (Update: To clarify, the bubblewrap sandbox is for the entire web process, not just the OpenJPEG decoder.)

    Of course, it would be good to improve OpenJPEG. I repeat my previous call for assistance with the OpenJPEG code quality issues reported by Ubuntu, but as before, I only expect to hear crickets.

    So unfortunately, we’re not yet at a point where I’m comfortable with Epiphany’s Ubuntu packaging. (Well, the problem is actually in the WebKit packaging. Details.) I insist: distributing Epiphany without support for JPEG 2000 images is harmful and makes Epiphany look bad. Please, Ubuntu, we need you to either build WebKit with OpenJPEG enabled, or else just drop your Epiphany packages entirely, one or the other. Whichever you choose will make me happy. Please don’t accept the status quo!

  • Dear Ubuntu: Please Stop Packaging Epiphany If You Won’t Do It Properly

    Dear Ubuntu,

    When users try Epiphany on Ubuntu, they receive a sub-par, broken browser. If you’re not willing to do this right, please just remove Epiphany from your repositories. We’d all be happier this way. You are the  most popular distributor of Epiphany by far, and your poor packaging is making the browser look bad.

    Epiphany 3.28.1 Is Stupid Old

    Currently you’ve packaged Epiphany 3.28.1 for Ubuntu 18.04, your current LTS release. 3.28.1 is a seriously broken version with an overaggressive adblock filters subscription that blocks legitimate resources on a wide variety of websites, resulting in broken page rendering. We obviously don’t want users to ever use 3.28.1. There is a 3.28.2, released on May 22, 2018, which fixes this problem, but after one year you have still not yet updated. Ideally you would update to 3.28.5, which has been available since September 21, 2018. It’s not like I’m expecting you to upgrade to 3.30 or to 3.32 (the current stable series). I’d be happy to release a 3.28.6, except I know that it’s pointless: you would not upgrade to it if I did.

    In Ubuntu 19.04, you have packaged Epiphany 3.32.0. The current version is 3.32.2. There are a lot of bugs fixed in between. (Update: Exalm has pointed out that your snap package takes precedence over the Debian package in GNOME Software, so most users will actually receive the snap instead. The snap is still using 3.30.4, because Epiphany 3.32 depends on GTK 3.24, and that is not available in snaps yet. All app menu items are unavailable because Ubuntu’s GNOME Shell 3.32 does not display Epiphany 3.30’s app menu, so there’s no way to edit preferences, view history, import/export bookmarks, etc. This is not good.)

    Because Epiphany is in your universe repository, rather than main, I understand that Canonical does not provide updates. But this is really not OK. Do I really need to add an xscreensaver-style time bomb to protect the reputation of Epiphany?

    You’ve Disabled the JPEG 2000 Support

    WebKitGTK is in main and you have been updating it regularly and in a timely manner, which is good. Thanks for this!

    But we also need WebKitGTK to be built with OpenJPEG support, so that it can display JPEG 2000 images. Because you build WebKitGTK without OpenJPEG support, lots of popular websites are broken, including all websites using Akamai Image Manager. Because we have “Safari” but not “Chromium” in our user agent, these websites send us lots of JPEG 2000 images, and we need to be prepared to handle them properly to avoid broken websites. (Changing our user agent to avoid receiving JPEG 2000 images is, unfortunately, not practical.)

    Here we have a really difficult issue, because you admittedly have a good reason for disabling OpenJPEG use. OpenJPEG has failed your security review for inclusion in main. Seth Arnold from the Ubuntu Security Team has reported 24 issues in OpenJPEG, of which 21 still remain unfixed. (It’s probably too much to ask, but if any readers want to help tackle a couple of these, that would be really great.) WebKitGTK is only as secure as its least-secure image decoder, and it seems likely that that would be OpenJPEG. Exposing the low-quality OpenJPEG library to the entire web is risky.

    And yet, a web browser that doesn’t display websites properly is simply not worth delivering to users. We need this image decoder for web compatibility. WebKitGTK 2.26 will (hopefully) ship with a sandbox to mitigate security risks. Perhaps future versions of Epiphany should refuse to start if OpenJPEG support is unavailable?