Patching Vendored Rust Dependencies

Recently I had a difficult time trying to patch a CVE in librsvg. The issue itself was simple to patch because Federico kindly backported the series of commits required to fix it to the branch we are using downstream. Problem was, one of the vendored deps in the old librsvg tarball did not build with our modern rustc, because the code contained a borrow error that was not caught by older versions of rustc. After finding the appropriate upstream fix, I tried naively patching the vendored dep, but that failed because cargo tries very hard to prevent you from patching its dependencies, and complains if the dependency does not match its checksum in Cargo.lock. I tried modifying the checksum in Cargo.lock, but then it complains that you modified the Cargo.lock. It seems cargo is designed to make patching dependencies as difficult as possible, and that not much thought was put into how cargo would be used from rpmbuild with no network access.

Anyway, it seems the kosher way to patch Rust dependencies is to add a [patch] section to librsvg’s Cargo.toml, but I could not figure out how to make that work. Eventually, I got some help: you can edit the .cargo-checksum.json of the vendored dependency and change “files” to an empty array, like so:

diff --git a/vendor/cssparser/.cargo-checksum.json b/vendor/cssparser/.cargo-checksum.json
index 246bb70..713372d 100644
--- a/vendor/cssparser/.cargo-checksum.json
+++ b/vendor/cssparser/.cargo-checksum.json
@@ -1 +1 @@
-{"files":{".cargo-ok":"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",".travis.yml":"f1fb4b65964c81bc1240544267ea334f554ca38ae7a74d57066f4d47d2b5d568","Cargo.toml":"7807f16d417eb1a6ede56cd4ba2da6c5c63e4530289b3f0848f4b154e18eba02","LICENSE":"fab3dd6bdab226f1c08630b1dd917e11fcb4ec5e1e020e2c16f83a0a13863e85","README.md":"c5781e673335f37ed3d7acb119f8ed33efdf6eb75a7094b7da2abe0c3230adb8","build.rs":"b29fc57747f79914d1c2fb541e2bb15a003028bb62751dcb901081ccc174b119","build/match_byte.rs":"2c84b8ca5884347d2007f49aecbd85b4c7582085526e2704399817249996e19b","docs/.nojekyll":"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855","docs/404.html":"025861f76f8d1f6d67c20ab624c6e418f4f824385e2dd8ad8732c4ea563c6a2e","docs/index.html":"025861f76f8d1f6d67c20ab624c6e418f4f824385e2dd8ad8732c4ea563c6a2e","src/color.rs":"c60f1b0ab7a2a6213e434604ee33f78e7ef74347f325d86d0b9192d8225ae1cc","src/cow_rc_str.rs":"541216f8ef74ee3cc5cbbc1347e5f32ed66588c401851c9a7d68b867aede1de0","src/from_bytes.rs":"331fe63af2123ae3675b61928a69461b5ac77799fff3ce9978c55cf2c558f4ff","src/lib.rs":"46c377e0c9a75780d5cb0bcf4dfb960f0fb2a996a13e7349bb111b9082252233","src/macros.rs":"adb9773c157890381556ea83d7942dcc676f99eea71abbb6afeffee1e3f28960","src/nth.rs":"5c70fb542d1376cddab69922eeb4c05e4fcf8f413f27563a2af50f72a47c8f8c","src/parser.rs":"9ed4aec998221eb2d2ba99db2f9f82a02399fb0c3b8500627f68f5aab872adde","src/rules_and_declarations.rs":"be2c4f3f3bb673d866575b6cb6084f1879dff07356d583ca9a3595f63b7f916f","src/serializer.rs":"4ccfc9b4fe994aab3803662bbf31cc25052a6a39531073a867b14b224afe42dd","src/size_of_tests.rs":"e5f63c8c18721cc3ff7a5407e84f9889ffa10e66da96e8510a696c3e00ad72d5","src/tests.rs":"80b02c80ab0fd580dad9206615c918e0db7dff63dfed0feeedb66f317d24b24b","src/tokenizer.rs":"429b2cba419cf8b923fbcc32d3bd34c0b39284ebfcb9fc29b8eb8643d8d5f312","src/unicode_range.rs":"c1c4ed2493e09d248c526ce1ef8575a5f8258da3962b64ffc814ef3bdf9780d0"},"package":"8a807ac3ab7a217829c2a3b65732b926b2befe6a35f33b4bf8b503692430f223"}
\ No newline at end of file
+{"files":{},"package":"8a807ac3ab7a217829c2a3b65732b926b2befe6a35f33b4bf8b503692430f223"}

Then cargo will stop complaining and you can patch the dependency. Success!

Enable Git Commit Message Syntax Highlighting in Vim on Fedora

Were you looking forward to reading an exciting blog post about substantive technical issues affecting GNOME or the Linux desktop community? Sorry, not today.

When setting up new machines, I’m often frustrated by lack of syntax highlighting for git commit messages in vim. On my main workstation, vim uses comforting yellow letters for the first line of my commit message to let me know I’m good on line length, or red background to let me know my first line is too long, and after the first line it automatically inserts a new line break whenever I’ve typed past 72 characters. It’s pretty nice. I can never remember how I get it working in the end, and I spent too long today trying to figure it out yet again. Eventually I realized there was another difference besides the missing syntax highlighting: I couldn’t see the current line or column number, and I couldn’t see the mode indicator either. Now you might be able to guess my mistake: git was not using /usr/bin/vim at all! Because Fedora doesn’t have a default $EDITOR, git defaults to using /usr/bin/vi, which is basically sad trap vim. Solution:

$ git config --global core.editor vim

You also have to install the vim-enhanced package to get /usr/bin/vim, but that’s a lot harder to forget to do.

You’re welcome, Internet!

Let’s Learn Spelling!

Were you looking forward to reading an exciting blog post about substantive technical issues affecting GNOME or the Linux desktop community? Sorry, not today.

GNOME

It used to be an acronym, so it’s all uppercase. Write “GNOME,” never “Gnome.” Please stop writing “Gnome.”

Would it help if you imagine an adorable little garden gnome dying each time you get it wrong?

If you’re lazy and hate capital letters, or for technical contexts like package or project names, then all-lowercase “gnome” might be appropriate, but “Gnome” certainly never is.

Red Hat

This one’s not that hard. Why are some people writing “RedHat” without any space? It doesn’t make sense. Red Hat. Easy!

SUSE and openSUSE

S.u.S.E. and SuSE are both older spellings for the company currently called SUSE. Apparently at some point in the past they realized that the lowercase u was stupid and causes readers’ eyes to bleed.  Can we please let it die?

Similarly, openSUSE is spelled “openSUSE,” not “OpenSUSE.” Do not capitalize the o, even if it’s the first word in a sentence. Do not write “openSuSE” or “OpenSuSE” (which people somehow manage to do even when they’re not trolling) or anything at all other than “openSUSE.” I know this is probably too much to ask, but once you get the hang of it, it’s not so hard.

elementary OS

I don’t often see this one messed up. If you can write elementary OS, you can probably write openSUSE properly too! They’re basically the same structure, right? All lowercase, then all caps. I have faith in you, dear reader! Don’t let me down!

GTK and WebKitGTK

We removed the + from the end of both of these, because it was awful. You’re welcome!

Again, all lowercase is probably OK in technical contexts. “gtk-webkit” is not. WebKitGTK.

Mesa Update Breaks WebKitGTK+ in Fedora 29

If you’re using Fedora and discovered that WebKitGTK+ is displaying blank pages, the cause is a bad mesa update, mesa-18.2.3-1.fc29. This in turn was caused by a GCC bug that resulted in miscompilation of mesa.

To avoid this bug, downgrade to mesa-18.2.2-1.fc29:

$ sudo dnf downgrade mesa*

You can also update to mesa-18.2.4-2.fc29, but this build has not yet reached updates-testing, let alone stable, so downgrading is easier for now. Another workaround is to run your application with accelerated compositing mode disabled, to avoid OpenGL usage:

$ WEBKIT_DISABLE_COMPOSITING_MODE=1 epiphany

On the bright side of things, from all the bug reports I’ve received over the past two days I’ve discovered that lots of people use Epiphany and notice when it’s broken. That’s nice!

Huge thanks to Dave Airlie for quickly preparing the fixed mesa update, and to Jakub Jelenik for handling the same for GCC.

On Python Shebangs

So, how do you write a shebang for a Python program? Let’s first set aside the python2/python3 issue and focus on whether to use env. Which of the following is correct?

#!/usr/bin/env python
#!/usr/bin/python

The first option seems to work in all environments, but it is banned in popular distros like Fedora (and I believe also Debian, but I can’t find a reference for this). Using env in shebangs is dangerous because it can result in system packages using non-system versions of python. python is used in so many places throughout modern systems, it’s not hard to see how using #!/usr/bin/env in an important package could badly bork users’ operating systems if they install a custom version of python in /usr/local. Don’t do this.

The second option is broken too, because it doesn’t work in BSD environments. E.g. in FreeBSD, python is installed in /usr/local/bin. So FreeBSD contributors have been upstreaming patches to convert #!/usr/bin/python shebangs to #!/usr/bin/env python. Meanwhile, Fedora has begun automatically rewriting #!/usr/bin/env python to #!/usr/bin/python, but with a warning that this is temporary and that use of #!/usr/bin/env python will eventually become a fatal error causing package builds to fail.

So obviously there’s no way to write a shebang that will work for both major Linux distros and major BSDs. #!/usr/bin/env python seems to work today, but it’s subtly very dangerous. Lovely. I don’t even know what to recommend to upstream projects.

Next problem: python2 versus python3. By now, we should all be well-aware of PEP 394. PEP 394 says you should never write a shebang like this:

#!/usr/bin/env python
#!/usr/bin/python

unless your python script is compatible with both python2 and python3, because you don’t know what version you’re getting. Your python script is almost certainly not compatible with both python2 and python3 (and if you think it is, it’s probably somehow broken, because I doubt you regularly test it with both). Instead, you should write the shebang like this:

#!/usr/bin/env python2
#!/usr/bin/python2
#!/usr/bin/env python3
#!/usr/bin/python3

This works as long as you only care about Linux and BSDs. It doesn’t work on macOS, which provides /usr/bin/python and /usr/bin/python2.7, but still no /usr/bin/python2 symlink, even though it’s now been six years since PEP 394. It’s hard to understate how frustrating this is.

So let’s say you are WebKit, and need to write a python script that will be truly cross-platform. How do you do it? WebKit’s scripts are only needed (a) during the build process or (b) by developers, so we get a pass on the first problem: using /usr/bin/env should be OK, because the scripts should never be installed as part of the OS. Using #!/usr/bin/env python — which is actually what we currently do — is unacceptable, because our scripts are python2 and that’s broken on Arch, and some of our developers use that. Using #!/usr/bin/env python2 would be dead on arrival, because that doesn’t work on macOS. Seems like the option that works for everyone is #!/usr/bin/env python2.7. Then we just have to hope that the Python community sticks to its promise to never release a python2.8 (which seems likely).

…yay?

Endgame for WebKit Woes

In my original blog post On WebKit Security Updates, I identified three separate problems affecting WebKit users on Linux:

  • Distributions were not providing updates for WebKitGTK+. This was the main focus of that post.
  • Distributions were shipping a insecure compatibility package for old, unmaintained WebKitGTK+ 2.4 (“WebKit1”).
  • Distributions were shipping QtWebKit, which was also unmaintained and insecure.

Let’s review these problems one at a time.

Distributions Are Updating WebKitGTK+

Nowadays, most major community distributions are providing regular WebKitGTK+ updates, so this is no longer a problem for the vast majority of Linux users. If you’re using a supported version of Ubuntu (except Ubuntu 14.04), Fedora, or most other mainstream distributions, then you are good to go.

My main concern here is still Debian, but there are reasons to be optimistic. It’s too soon to say what Debian’s policy will be going forward, but I am encouraged that it broke freeze just before the Stretch release to update from WebKitGTK+ 2.14 to 2.16.3. Debian is slow and conservative and so has not yet updated to 2.16.6, which is sad because 2.16.3 is affected by a bug that causes crashes on a huge number of websites, but my understanding is it is likely to be updated in the near future. I’m not sure if Debian will update to 2.18 or not. We’ll have to wait and see.

openSUSE is another holdout. The latest stable version of openSUSE Leap, 42.3, is currently shipping WebKitGTK+ 2.12.5. That is disappointing.

Most other major distributions seem to be current.

Distributions Are Removing WebKitGTK+ 2.4

WebKitGTK+ 2.4 (often informally referred to as “WebKit1”) was the next problem. Tons of desktop applications depended on this old, insecure version of WebKitGTK+, and due to large API changes, upgrading applications was not going to be easy. But this transition is going much smoother and much faster than I expected. Several distributions, including Debian, Fedora, and Arch, have recently removed their compatibility packages. There will be no WebKitGTK+ 2.4 in Debian 10 (Buster) or Fedora 27 (scheduled for release this October). Most noteworthy applications have either ported to modern WebKitGTK+, or have configure flags to disable use of WebKitGTK+. In some cases, such as GnuCash in Fedora, WebKitGTK+ 2.4 is being bundled as part of the application build process. But more often, applications that have not yet ported simply no longer work or have been removed from these distributions.

Soon, users will no longer need to worry that a huge amount of WebKitGTK+ applications are not receiving security updates. That leaves one more problem….

QtWebKit is Back

Upstream QtWebKit has not been receiving security updates for the past four years or thereabouts, since it was abandoned by the Qt project. That is still the status quo for most distributions, but Arch and Fedora have recently switched to Konstantin Tokarev’s fork of QtWebKit, which is based on WebKitGTK+ 2.12. (Thank you Konstantin!) If you are using any supported version of Fedora, you should already have been switched to this fork. I am hopeful that the fork will be rebased on WebKitGTK+ 2.16 or 2.18 in the near future, to bring it current on security updates, but in the meantime, being a year and a half behind is an awful lot better than being four years behind. Now that Arch and Fedora have led the way, other distributions should find little trouble in making the switch to Konstantin’s QtWebKit. It would be a disservice to users to continue shipping the upstream version.

So That’s Cool

Things are better. Some distributions, notably Arch and Fedora, have resolved all of the above problems (or will in the very near future). Yay!

More On Private Internet Access

A few quick follow-up thoughts from my original review. First, problems I haven’t solved yet:

  • I forgot an important problem in my first blog: email. Evolution is borderline unusable with PIA. My personal GMail account usually works reliably, but my Google Apps school GMail account (which you’d think would function the same) and my Igalia email both time out with the error “Source doesn’t support prompt for credentials”. That’s Evolution’s generic error that it throws up whenever the mail server is taking too long to respond. So what’s going on here? I can check my email via webmail as a workaround in the meantime, but this is really terrible.
  • Still no solution for the first attempt to connect always failing. That’s really annoying! I was expecting some insight (or at least guesses) as to what might be going wrong here, but nobody has suggested anything about this yet. Update: The problem is that I had selected “Make available to other users” but “Store the password only for this user”, which results in the first attempt to connect always failing, because it’s performed by the gdm user. The fix is to store the password for all users.

Some solutions and answers to problems from my original post:

  • Jonh Wendell suggested using TCP instead of UDP to connect to PIA. I’ve been trying this and so far have not noticed a single instance of connection loss. So I think my biggest problem has been solved. Yay!
  • Dan LaManna posted a link to vpnfailsafe. I’m probably not going to use this since it’s a long shell script that I don’t understand, and since my connection drop problems seem to be solved now that I’ve switched to TCP, but it looks like it’d probably be a good solution to its problem. Real shame this is not built in to NetworkManager already.
  • Christel Dahlskjaer has confirmed that freenode requires NickServ/SASL authentication to use via PIA. This isn’t acceptable for me, since Empathy can’t handle it well, so I’m probably just going to stop using freenode for the most part. The only room I was ever really active in was #webkitgtk+, but in practice our use of that room is basically redundant with #epiphany on GIMPNet (where you’ll still find me, and which would be a better location for a WebKitGTK+ channel anyway), so I don’t think I’ll miss it. I’ve been looking to reduce the number of IRC rooms I join for a long time anyway. The only thing I really need freenode for is Fedora Workstation meetings, which I can attend via a web gateway. (Update: I realized that I am going to miss #webkit as well. Hmm, this could be a problem….)

So my biggest issue now is that I can’t use my email. That’s pretty surprising, as I wouldn’t think using a VPN would make any difference for that. I don’t actually care about my Google Apps account, but I need to be able to read my Igalia mail in Evolution. (Note: My actual IP seems to leak in my email headers, but I don’t care. My name is on my emails anyway. I just care that it works.)

On Private Internet Access

I’m soon going to be moving to Charter Communications territory, but I don’t trust Charter and don’t want it to keep records of all the websites that I visit.  The natural solution is to use a VPN, and the natural first choice is Private Internet Access, since it’s a huge financial supporter of GNOME, and I haven’t heard anybody complain about problems with using it. This will be a short review of my experience.

The service is not free. That’s actually good: it means I’m the customer, not the product. Cost is $40 per year if you pay a year in advance, but you should probably start with the $7/month plan until you’re sure you’re happy with the service and will be keeping it long-term. Anyway, this is a pretty reasonable price that I’m happy to pay.

The website is fairly good. It makes it easy to buy or discontinue service, so there are no pricing surprises, and there’s a pretty good library of support documentation. Unfortunately some of the claims on the website seem to be — arguably — borderline deceptive. A VPN service provides excellent anonymity against your ISP, but relying on a VPN would be a pretty bad idea if your adversary is the government (it can perform a traffic correlation attack) or advertising companies (they know your screen resolution, the performance characteristics of your graphics card, and until recently the rate your battery drains…). But my adversary is going to be Charter Communications, so a VPN is the perfect solution for me. If you need real anonymity, you absolutely must use the Tor Browser Bundle, but that’s going to make your life harder, and I don’t want my life to be harder, so I’ll stick with a VPN.

Private Internet Access provides an Ubuntu app, but I’m going to ignore that because (a) I use Fedora, not Ubuntu, and (b) why on Earth would you want a separate desktop app for your VPN when OpenVPN integration is already built-in on Ubuntu and all modern Linux desktops? Unfortunately the documentation provided by Private Internet Access is not really sufficient — they have a script to set it up automatically, but it’s really designed for Ubuntu and doesn’t work on Fedora — so configuration was slightly challenging.  I wound up following instructions on some third-party website, which I have long since forgotten. There are many third-party resources for how to configure PIA on Linux, which you might think is good but actually indicates a problem with the official documentation in my opinion. So there is some room for improvement here. PIA should ditch the pointless desktop app and improve its documentation for configuring OpenVPN via NetworkManager. (Update: After publishing this post, I discovered this article. Seems the installation script now supports for Fedora/RHEL and Arch Linux. So my claim that it only works on Ubuntu is outdated.) But anyway, once you get it configured properly with NetworkManager, it works: no need to install anything (besides the OpenVPN certificate, of course).

Well, it mostly works. Now, I have two main requirements to ensure that Charter can’t keep records of the websites I’m visiting:

  • NetworkManager must autoconnect to the VPN, so I don’t have to do it manually.
  • NetworkManager must reconnect to the VPN service if connection drops, and must never send any data if the VPN is off.

The first requirement was hard to solve, and I still don’t have it working perfectly. There is no GUI configuration option for this in gnome-control-center, but I eventually found it in nm-connection-editor: you have to edit your normal non-VPN connection, which has a preference to select a VPN to connect to automatically. So we should improve that in gnome-control-center. Unfortunately, it doesn’t work at all the first time your computer connects to the internet after it’s booted. Each time I boot my computer, I’m greeted with a Connection Failed notification on the login screen. This is probably a NetworkManager bug. Anyway, after logging in, I just have to manually connect once, then it works.

As for the next requirement, I’ve given up. My PIA connection is routinely lost about once every 30-45 minutes, usually when watching YouTube or otherwise using a lot of data. This is most likely a problem with PIA’s service, but I don’t know that: it could just as well be my current ISP cutting the connection, or maybe even some client-side NetworkManager bug. Anyway, I could live with brief connection interruptions, but when this happens, I lose connection entirely for about a minute — too long — and then the VPN times out and NetworkManager switches back to sending all the data outside the VPN. That’s totally unacceptable. To be clear, sending data outside the VPN is surely a NetworkManager problem, not a PIA problem, but it needs to be fixed for me to be comfortable using PIA. I see some discussion about that on this third-party GitHub issue, but the “solution” there is to stop using NetworkManager, which I’m not going to do. This is probably one of the reasons why PIA provides a desktop app — I think the PIA app doesn’t suffer from this issue? — but like I said, I’m not going to use a third-party OpenVPN app instead of the undoubtedly-nicer support that’s built in to GNOME.

Another problem is that I can’t connect to Freenode when I’m using the VPN. GIMPNet works fine, so it’s not a problem with IRC in general: Freenode is specifically blocking Private Internet Access users. This seems very strange, since Freenode has a bunch of prominent advertising for PIA all over its website. I could understand blocking PIA if there are too many users abusing it, but not if you’re going to simultaneously advertise it.

I also cannot access Igalia’s SIP service when using PIA. I need that too, but that’s probably something we have to fix on our end.

So I’m not sure what to do now. We have two NetworkManager bugs and a problem with Freenode. Eventually I’ll drop Empathy in favor of Matrix or some other IRC client where registering with NickServ is not a terrible mistake (presumably they’re only blocking unregistered users?), so the Freenode issue seems less-important. I think I’d be willing to just stop visiting Freenode if required to use PIA, anyway. But those NetworkManager issues are blockers to me. With those unfixed, I’m not sure if I’m going to renew my PIA subscription or not. I would definitely renew if someone were to fix those two issues. The ideal solution would be for PIA to adopt NetworkManager’s OpenVPN plugin and ensure it gets cared for, but if not, maybe someone else will fix it?

Update: See part two for how to solve some of these problems.

On “Insights On Companies/Developers Behind Wayland”

I recently read a peer-reviewed academic paper from a couple years ago that analyzed the contributions of different companies to WebKit. The authors didn’t bother to account for individuals using non-corporate email addresses, since that’s hard work, and did not realize that most Google developers contribute to the project using @chromium.org email addresses, resulting in Google’s contributions being massively undercounted. There were other serious mistakes in the paper too, but this is the one that came to mind when reading The FOSS Post’s article Insights On Companies/Developers Behind Wayland.

The FOSS Post didn’t bother to account for where some big developers work, incorrectly trusting that all employees use corporate emails when contributing to open source projects. It contains some interesting claims, like “Clearly, Samsung and the individual ‘Bryce Harrington’ are almost doing the same work [on Wayland build tools]” and “75% of the code [in libinput] is written by Peter Hutterer. Followed by 10% for a group of individuals and 5% by Red Hat.” I have only very passing familiarity with the Wayland project, but I do know that Bryce works for Samsung, and that Peter works for Red Hat. Suggesting that Red Hat contributed only 5% of the code to libinput, when the real number looks more like 80%, does not speak well of the quality of The FOSS Post’s insights. Also notably, Kristian Høgsberg’s massive contributions to the  project were not classed as contributions from Intel, where he was working at the time.

You don’t have to be an expert on the community to take the time to account for people not using corporate emails before publishing an analysis. This is why it’s important to understand the community you are analyzing at least somewhat before publishing such “insights.”

Update: The FOSS Post’s article was completely updated with new charts to address this issue.

Update #2: Jonas reports in the comments below that the charts are still completely wrong.

An Update on WebKit Security Updates

One year ago, I wrote a blog post about WebKit security updates that attracted a fair amount of attention at the time. For a full understanding of the situation, you really have to read the whole thing, but the most important point was that, while WebKitGTK+ — one of the two WebKit ports present in Linux distributions — was regularly releasing upstream security updates, most Linux distributions were ignoring the updates, leaving users vulnerable to various security bugs, mainly of the remote code execution variety. At the time of that blog post, only Arch Linux and Fedora were regularly releasing WebKitGTK+ updates, and Fedora had only very recently begun doing so comprehensively.

Progress report!

So how have things changed in the past year? The best way to see this is to look at the versions of WebKitGTK+ in currently-supported distributions. The latest version of WebKitGTK+ is 2.14.3, which fixes 13 known security issues present in 2.14.2. Do users of the most popular Linux operating systems have the fixes?

  • Fedora users are good. Both Fedora 24 and Fedora 25 have the latest version, 2.14.3.
  • If you use Arch, you know you always have the latest stuff.
  • Ubuntu users rejoice: 2.14.3 updates have been released to users of both Ubuntu 16.04 and 16.10. I’m very  pleased that Ubuntu has decided to take my advice and make an exception to its usual stable release update policy to ensure its users have a secure version of WebKit. I can’t give Ubuntu an A grade here because the updates tend to lag behind upstream by several months, but slow updates are much better than no updates, so this is undoubtedly a huge improvement. (Anyway, it’s hardly a bad idea to be cautious when releasing a big update with high regression potential, as is unfortunately the case with even stable WebKit updates.) But if you use the still-supported Ubuntu 14.04 or 12.04, be aware that these versions of Ubuntu cannot ever update WebKit, as it would require a switch to WebKit2, a major API change.
  • Debian does not update WebKit as a matter of policy. The latest release, Debian 8.7, is still shipping WebKitGTK+ 2.6.2. I count 184 known vulnerabilities affecting it, though that’s an overcount as we did not exclude some Mac-specific security issues from the 2015 security advisories. (Shipping ancient WebKit is not just a security problem, but a user experience problem too. Actually attempting to browse the web with WebKitGTK+ 2.6.2 is quite painful due to bugs that were fixed years ago, so please don’t try to pretend it’s “stable.”) Note that a secure version of WebKitGTK+ is available for those in the know via the backports repository, but this does no good for users who trust Debian to provide them with security updates by default without requiring difficult configuration. Debian testing users also currently have the latest 2.14.3, but you will need to switch to Debian unstable to get security updates for the foreseeable future, as testing is about to freeze.
  • For openSUSE users, only Tumbleweed has the latest version of WebKit. The current stable release, Leap 42.2, ships with WebKitGTK+ 2.12.5, which is coincidentally affected by exactly 42 known vulnerabilities. (I swear I am not making this up.) The previous stable release, Leap 42.1, originally released with WebKitGTK+ 2.8.5 and later updated to 2.10.7, but never past that. It is affected by 65 known vulnerabilities. (Note: I have to disclose that I told openSUSE I’d try to help out with that update, but never actually did. Sorry!) openSUSE has it a bit harder than other distros because it has decided to use SUSE Linux Enterprise as the source for its GCC package, meaning it’s stuck on GCC 4.8 for the foreseeable future, while WebKit requires GCC 4.9. Still, this is only a build-time requirement; it’s not as if it would be impossible to build with Clang instead, or a custom version of GCC. I would expect WebKit updates to be provided to both currently-supported Leap releases.
  • Gentoo has the latest version of WebKitGTK+, but only in testing. The latest version marked stable is 2.12.5, so this is a serious problem if you’re following Gentoo’s stable channel.
  • Mageia has been updating WebKit and released a couple security advisories for Mageia 5, but it seems to be stuck on 2.12.4, which is disappointing, especially since 2.12.5 is a fairly small update. The problem here does not seem to be lack of upstream release monitoring, but rather lack of manpower to prepare the updates, which is a typical problem for small distros.
  • The enterprise distros from Red Hat, Oracle, and SUSE do not provide any WebKit security updates. They suffer from the same problem as Ubuntu’s old LTS releases: the WebKit2 API change  makes updating impossible. See my previous blog post if you want to learn more about that. (SUSE actually does have WebKitGTK+ 2.12.5 as well, but… yeah, 42.)

So results are clearly mixed. Some distros are clearly doing well, and others are struggling, and Debian is Debian. Still, the situation on the whole seems to be much better than it was one year ago. Most importantly, Ubuntu’s decision to start updating WebKitGTK+ means the vast majority of Linux users are now receiving updates. Thanks Ubuntu!

To arrive at the above vulnerability totals, I just counted up the CVEs listed in WebKitGTK+ Security Advisories, so please do double-check my counting if you want. The upstream security advisories themselves are worth mentioning, as we have only been releasing these for two years now, and the first year was pretty rough when we lost our original security contact at Apple shortly after releasing the first advisory: you can see there were only two advisories in all of 2015, and the second one was huge as a result of that. But 2016 seems to have gone decently well. WebKitGTK+ has normally been releasing most security fixes even before Apple does, though the actual advisories and a few remaining fixes normally lag behind Apple by roughly a month or so. Big thanks to my colleagues at Igalia who handle this work.

Challenges ahead

There are still some pretty big problems remaining!

First of all, the distributions that still aren’t releasing regular WebKit updates should start doing so.

Next, we have to do something about QtWebKit, the other big WebKit port for Linux, which stopped receiving security updates in 2013 after the Qt developers decided to abandon the project. The good news is that Konstantin Tokarev has been working on a QtWebKit fork based on WebKitGTK+ 2.12, which is almost (but not quite yet) ready for use in distributions. I hope we are able to switch to use his project as the new upstream for QtWebKit in Fedora 26, and I’d encourage other distros to follow along. WebKitGTK+ 2.12 does still suffer from those 42 vulnerabilities, but this will be a big improvement nevertheless and an important stepping stone for a subsequent release based on the latest version of WebKitGTK+. (Yes, QtWebKit will be a downstream of WebKitGTK+. No, it will not use GTK+. It will work out fine!)

It’s also time to get rid of the old WebKitGTK+ 2.4 (“WebKit1”), which all distributions currently parallel-install alongside modern WebKitGTK+ (“WebKit2”). It’s very unfortunate that a large number of applications still depend on WebKitGTK+ 2.4 — I count 41 such packages in Fedora — but this old version of WebKit is affected by over 200 known vulnerabilities and really has to go sooner rather than later. We’ve agreed to remove WebKitGTK+ 2.4 and its dependencies from Fedora rawhide right after Fedora 26 is branched next month, so they will no longer be present in Fedora 27 (targeted for release in November). That’s bad for you if you use any of the affected applications, but fortunately most of the remaining unported applications are not very important or well-known; the most notable ones that are unlikely to be ported in time are GnuCash (which won’t make our deadline) and Empathy (which is ported in git master, but is not currently in a  releasable state; help wanted!). I encourage other distributions to follow our lead here in setting a deadline for removal. The alternative is to leave WebKitGTK+ 2.4 around until no more applications are using it. Distros that opt for this approach should be prepared to be stuck with it for the next 10 years or so, as the remaining applications are realistically not likely to be ported so long as zombie WebKitGTK+ 2.4 remains available.

These are surmountable problems, but they require action by downstream distributions. No doubt some distributions will be more successful than others, but hopefully many distributions will be able to fix these problems in 2017. We shall see!