Posted on by Michael Catanzaro

On WebKit Security Updates

Linux distributions have a problem with WebKit security.

Major desktop browsers push automatic security updates directly to users on a regular basis, so most users don’t have to worry about security updates. But Linux users are dependent on their distributions to release updates. Apple fixed over 100 vulnerabilities in WebKit last year, so getting updates out to users is critical.

This is the story of how that process has gone wrong for WebKit.

Before we get started, a few disclaimers. I want to be crystal clear about these points:

  1. This post does not apply to WebKit as used in Apple products. Apple products receive regular security updates.
  2. WebKitGTK+ releases regular security updates upstream. It is safe to use so long as you apply the updates.
  3. The opinions expressed in this post are my own, not my employer’s, and not the WebKit project’s.

Browser Security in a Nutshell

Web engines are full of security vulnerabilities, like buffer overflows and use-after-frees. The details don’t matter; what’s important is that skilled attackers can turn these vulnerabilities into exploits, using carefully-crafted HTML to gain total control of your user account on your computer (or your phone). They can then install malware, read all the files in your home directory, use your computer in a botnet to attack websites, and do basically whatever they want with it.

If the web engine is sandboxed, then a second type of attack, called a sandbox escape, is needed. This makes it dramatically more difficult to exploit vulnerabilities. Chromium has a top-class Linux sandbox. WebKit does have a Linux sandbox, but it’s not any good, so it’s (rightly) disabled by default. Firefox does not have a sandbox due to major architectural limitations (which Mozilla is working on).

For this blog post, it’s enough to know that attackers use crafted input to exploit vulnerabilities to gain control of your computer. This is why it’s not a good idea to browse to dodgy web pages. It also explains how a malicious email can gain control of your computer. Modern email clients render HTML mail using web engines, so malicious emails exploit many of the same vulnerabilities that a malicious web page might. This is one reason why good email clients block all images by default: image rendering, like HTML rendering, is full of security vulnerabilities. (Another reason is that images hosted remotely can be used to determine when you read the email, violating your privacy.)

WebKit Ports

To understand WebKit security, you have to understand the concept of WebKit ports, because different ports handle security updates differently.

While most code in WebKit is cross-platform, there’s a large amount of platform-specific code as well, to improve the user and developer experience in different environments. Different “ports” run different platform-specific code. This is why two WebKit-based browsers, say, Safari and Epiphany (GNOME Web), can display the same page slightly differently: they’re using different WebKit ports.

Currently, the WebKit project consists of six different ports: one for Mac, one for iOS, two for Windows (Apple Windows and WinCairo), and two for Linux (WebKitGTK+ and WebKitEFL). There are some downstream ports as well; unlike the aforementioned ports, downstream ports are, well, downstream, and not part of the WebKit project. The only one that matters for Linux users is QtWebKit.

If you use Safari, you’re using the Mac or iOS port. These ports get frequent security updates from Apple to plug vulnerabilities, which users receive via regular updates.

Everything else is broken.

Since WebKit is not a system library on Windows, Windows applications must bundle WebKit, so each application using WebKit must be updated individually, and updates are completely dependent on the application developers. iTunes, which uses the Apple Windows port, does get regular updates from Apple, but beyond that, I suspect most applications never get any security updates. This is a predictable result, the natural consequence of environments that require bundling libraries.

(This explains why iOS developers are required to use the system WebKit rather than bundling their own: Apple knows that app developers will not provide security updates on their own, so this policy ensures every iOS application rendering HTML gets regular WebKit security updates. Even Firefox and Chrome on iOS are required to use the system WebKit; they’re hardly really Firefox or Chrome at all.)

The same scenario applies to the WinCairo port, except this port does not have releases or security updates. Whereas the Apple ports have stable branches with security updates, with WinCairo, companies take a snapshot of WebKit trunk, make their own changes, and ship products with that. Who’s using WinCairo? Probably lots of companies; the biggest one I’m aware of uses a WinCairo-based port in its AAA video games. It’s safe to assume few to no companies are handling security backports for their downstream WinCairo branches.

Now, on to the Linux ports. WebKitEFL is the WebKit port for the Enlightenment Foundation Libraries. It’s not going to be found in mainstream Linux distributions; it’s mostly used in embedded devices produced by one major vendor. If you know anything at all about the internet of things, you know these devices never get security updates, or if they do, the updates are superficial (updating only some vulnerable components and not others), or end a couple months after the product is purchased. WebKitEFL does not bother with pretense here: like WinCairo, it has never had security updates. And again, it’s safe to assume few to no companies are handling security backports for their downstream branches.

None of the above ports matter for most Linux users. The ports available on mainstream Linux distributions are QtWebKit and WebKitGTK+. Most of this blog will focus on WebKitGTK+, since that’s the port I work on, and the port that matters most to most of the people who are reading this blog, but QtWebKit is widely-used and deserves some attention first.

It’s broken, too.

QtWebKit

QtWebKit is the WebKit port used by Qt software, most notably KDE. Some cherry-picked examples of popular applications using QtWebKit are Amarok, Calligra, KDevelop, KMail, Kontact, KTorrent, Quassel, Rekonq, and Tomahawk. QtWebKit provides an excellent Qt API, so in the past it’s been the clear best web engine to use for Qt applications.

After Google forked WebKit, the QtWebKit developers announced they were switching to work on QtWebEngine, which is based on Chromium, instead. This quickly led to the removal of QtWebKit from the WebKit project. This was good for the developers of other WebKit ports, since lots of Qt-specific code was removed, but it was terrible for KDE and other QtWebKit users. QtWebKit is still maintained in Qt and is getting some backports, but from a quick check of their git repository it’s obvious that it’s not receiving many security updates. This is hardly unexpected; QtWebKit is now years behind upstream, so providing security updates would be very difficult. There’s not much hope left for QtWebKit; these applications have hundreds of known vulnerabilities that will never be fixed. Applications should port to QtWebEngine, but for many applications this may not be easy or even possible.

Update: As pointed out in the comments, there is some effort to update QtWebKit. I was aware of this and in retrospect should have mentioned this in the original version of this article, because it is relevant. Keep an eye out for this; I am not confident it will make its way into upstream Qt, but if it does, this problem could be solved.

WebKitGTK+

WebKitGTK+ is the port used by GTK+ software. It’s most strongly associated with its flagship browser, Epiphany, but it’s also used in other places. Some of the more notable users include Anjuta, Banshee, Bijiben (GNOME Notes), Devhelp, Empathy, Evolution, Geany, Geary, GIMP, gitg, GNOME Builder, GNOME Documents, GNOME Initial Setup, GNOME Online Accounts, GnuCash, gThumb, Liferea, Midori, Rhythmbox, Shotwell, Sushi, and Yelp (GNOME Help). In short, it’s kind of important, not only for GNOME but also for Ubuntu and Elementary. Just as QtWebKit used to be the web engine for choice for Qt applications, WebKitGTK+ is the clear choice for GTK+ applications due to its nice GObject APIs.

Historically, WebKitGTK+ has not had security updates. Of course, we released updates with security fixes, but not with CVE identifiers, which is how software developers track security issues; as far as distributors are concerned, without a CVE identifier, there is no security issue, and so, with a few exceptions, distributions did not release our updates to users. For many applications, this is not so bad, but for high-risk applications like web browsers and email clients, it’s a huge problem.

So, we’re trying to improve. Early last year, my colleagues put together our first real security advisory with CVE identifiers; the hope was that this would encourage distributors to take our updates. This required data provided by Apple to WebKit security team members on which bugs correspond to which CVEs, allowing the correlation of Bugzilla IDs to Subversion revisions to determine in which WebKitGTK+ release an issue has been fixed. That data is critical, because without it, there’s no way to know if an issue has been fixed in a particular release or not. After we released this first advisory, Apple stopped providing the data; this was probably just a coincidence due to some unrelated internal changes at Apple, but it certainly threw a wrench in our plans for further security advisories.

This changed in November, when I had the pleasure of attending the WebKit Contributors Meeting at Apple’s headquarters, where I was finally able meet many of the developers I had interacted with online. At the event, I gave a presentation on our predicament, and asked Apple to give us information on which Bugzilla bugs correspond to which CVEs. Apple kindly provided the necessary data a few weeks later.

During the Web Engines Hackfest, a yearly event that occurs at Igalia’s office in A Coruña, my colleagues used this data to put together WebKitGTK+ Security Advisory WSA-2015-0002, a list of over 130 vulnerabilities disclosed since the first advisory. (The Web Engines Hackfest was sponsored by Igalia, my employer, and by our friends at Collabora. I’m supposed to include their logos here to advertise how cool it is that they support the hackfest, but given all the doom and gloom in this post, I decided perhaps they would perhaps prefer not to have their logos attached to it.)

Note that 130 vulnerabilities is an overcount, as it includes some issues that are specific to the Apple ports. (In the future, we’ll try to filter these out.) Only one of the issues — a serious error in the networking backend shared by WebKitGTK+ and WebKitEFL — resided in platform-specific code; the rest of the issues affecting WebKitGTK+ were all cross-platform issues. This is probably partly because the trickiest code is cross-platform code, and partly because security researchers focus on Apple’s ports.

Anyway, we posted WSA-2015-0002 to the oss-security mailing list to make sure distributors would notice, crossed our fingers, and hoped that distributors would take the advisory seriously. That was one month ago.

Distribution Updates

There are basically three different approaches distributions can take to software updates. The first approach is to update to the latest stable upstream version as soon as, or shortly after, it’s released. This is the strategy employed by Arch Linux. Arch does not provide any security support per se; it’s not necessary, so long as upstream projects release real updates for security problems and not simply patches. Accordingly, Arch almost always has the latest version of WebKitGTK+.

The second main approach, used by Fedora, is to provide only stable release updates. This is more cautious, reflecting that big updates can break things, so they should only occur when upgrading to a new version of the operating system. For instance, Fedora 22 shipped with WebKitGTK+ 2.8, so it would release updates to new 2.8.x versions, but not to WebKitGTK+ 2.10.x versions.

The third approach, followed by most distributions, is to take version upgrades only rarely, or not at all. For smaller distributions this may be an issue of manpower, but for major distributions it’s a matter of avoiding regressions in stable releases. Holding back on version updates actually works well for most software. When security problems arise, distribution maintainers for major distributions backport fixes and release updates. The problem is that this not feasible for web engines; due to the huge volume of vulnerabilities that need fixed, security issues can only practically be handled upstream.

So what’s happened since WSA-2015-0002 was released? Did it convince distributions to take WebKitGTK+ security seriously? Hardly. Fedora is the only distribution that has made any changes in response to WSA-2015-0002, and that’s because I’m one of the Fedora maintainers. (I’m pleased to announce that we have a 2.10.7 update headed to both Fedora 23 and Fedora 22 right now. In the future, we plan to release the latest stable version of WebKitGTK+ as an update to all supported versions of Fedora shortly after it’s released upstream.)

Ubuntu

Ubuntu releases WebKitGTK+ updates somewhat inconsistently. For instance, Ubuntu 14.04 came with WebKitGTK+ 2.4.0. 2.4.8 is available via updates, but even though 2.4.9 was released upstream over eight months ago, it has not yet been released as an update for Ubuntu 14.04.

By comparison, Ubuntu 15.10 (the latest release) shipped with WebKitGTK+ 2.8.5, which has never been updated; it’s affected by about 40 vulnerabilities fixed in the latest upstream release. Ubuntu organizes its software into various repositories, and provides security support only to software in the main repository. This version of WebKitGTK+ is in Ubuntu’s “universe” repository, not in main, so it is excluded from security support. Ubuntu users might be surprised to learn that a large portion of Ubuntu software is in universe and therefore excluded from security support; this is in contrast to almost all other distributions, which typically provide security updates for all the software they ship.

I’m calling out Ubuntu here not because it is specially-negligent, but simply because it is our biggest distributor. It’s not doing any worse than most of our other distributors.

Debian

Debian provides WebKit updates to users running unstable, and to testing except during freeze periods, but not to released version of Debian. Debian is unique in that it has a formal policy on WebKit updates. Here it is, reproduced in full:

Debian 8 includes several browser engines which are affected by a steady stream of security vulnerabilities. The high rate of vulnerabilities and partial lack of upstream support in the form of long term branches make it very difficult to support these browsers with backported security fixes. Additionally, library interdependencies make it impossible to update to newer upstream releases. Therefore, browsers built upon the webkit, qtwebkit and khtml engines are included in Jessie, but not covered by security support. These browsers should not be used against untrusted websites.

For general web browser use we recommend Iceweasel or Chromium.

Chromium – while built upon the Webkit codebase – is a leaf package, which will be kept up-to-date by rebuilding the current Chromium releases for stable. Iceweasel and Icedove will also be kept up-to-date by rebuilding the current ESR releases for stable.

(Iceweasel and Icedove are Debian’s de-branded versions of Firefox and Thunderbird, the product of an old trademark spat with Mozilla.)

Debian is correct that we do not provide long term support branches, as it would be very difficult to backport security fixes. But it is not correct that “library interdependencies make it impossible to update to newer upstream releases.” This might have been true in the past, but for several years now, we have avoided requiring new versions of libraries whenever it would cause problems for distributions, and — with one big exception that I will discuss below — we ensure that each release maintains both API and ABI compatibility. (Distribution maintainers should feel free to get in touch if we accidentally introduce some compatibility issue for your distribution; if you’re having trouble taking our updates, we want to help. I recently worked with openSUSE to make sure WebKitGTK+ can still be compiled with GCC 4.8, for example.)

The risk in releasing updates is that WebKitGTK+ is not a leaf package: a bad update could break some application. This seems to me like a good reason for application maintainers to carefully test the updates, rather than a reason to withhold security updates from users, but it’s true there is some risk here. One possible solution would be to have two different WebKitGTK+ packages, say, webkitgtk-secure, which would receive updates and be used by high-risk software like web browsers and email clients, and a second webkitgtk-stable package that would not receive updates to reduce regression potential.

Recommended Distributions

We regularly receive bug reports from users with very old versions of WebKit, who trust their distributors to handle security for them and might not even realize they are running ancient, unsafe versions of WebKit. I strongly recommend using a distribution that releases WebKitGTK+ updates shortly after they’re released upstream. That is currently only Arch and Fedora. (You can also safely use WebKitGTK+ in Debian testing — except during its long freeze periods — and Debian unstable, and maybe also in openSUSE Tumbleweed, and (update) also in Gentoo testing. Just be aware that the stable releases of these distributions are currently not receiving our security updates.) I would like to add more distributions to this list, but I’m currently not aware of any more that qualify.

The Great API Break

So, if only distributions would ship the latest release of WebKitGTK+, then everything would be good, right? Nope, because of a large API change that occurred two and a half years ago, called WebKit2.

WebKit (an API layer within the WebKit project) and WebKit2 are two separate APIs around WebCore. WebCore is the portion of the WebKit project that Google forked into Blink; it’s too low-level to be used directly by applications, so it’s wrapped by the nicer WebKit and WebKit2 APIs. The difference between the WebKit and WebKit2 APIs is that WebKit2 splits work into multiple secondary processes. Asides from the UI process, an application will have one or many separate web processes (for the actual page rendering), possibly a separate network process, and possibly a database process for IndexedDB. This is good for security, because it allows the secondary processes to be sandboxed: the web process is the one that’s likely to be compromised first, so it should not have the ability to access the filesystem or the network. (Remember, though, that there is no Linux sandbox yet, so this is currently only a theoretical benefit.) The other main benefit is robustness. If a web site crashes the renderer, only a single web process crashes (corresponding to one tab in Epiphany), not the entire browser. UI process crashes are comparatively rare.

Intermission: Certificate Verification

Another advantage provided by the API change is the opportunity to handle HTTPS connections more securely. In the original WebKitGTK+ API, applications must handle certificate verification on their own. This was a serious mistake; predictably, applications performed no verification at all, or did so improperly. For instance, take this Shotwell bug which is not fixed in any released version of Shotwell, or this Banshee bug which is still open. Probably many more applications are affected, because I have not done a comprehensive check. The new API is secure by default; applications can ignore verification errors, but only if they go out of their way to do so.

Remember that even though WebKitGTK+ 2.4.9 was released upstream over eight months ago, Ubuntu 14.04 is still on 2.4.8? It’s worth mentioning that 2.4.9 contains the fix for that serious networking backend issue I mentioned earlier (CVE-2015-2330). The bug is that TLS certificate verification was not performed until an HTTP response was received from the server; it’s supposed to be performed before sending an HTTP request, to prevent secure cookies from leaking. This is a disaster, as attackers can easily use it to get your session cookie and then control your user account on most websites. (Credit to Ross Lagerwall for reporting that issue.) We reported this separately to oss-security due to its severity, but that was not enough to convince distributions to update. But most applications in Ubuntu 14.04, including Epiphany and Midori, would not even benefit from this fix, because the change only affects WebKit2; remember, there’s no certificate verification in the original WebKitGTK+ API. (Modern versions of Epiphany do use WebKit2, but not the old version included in Ubuntu 14.04.) Old versions of Epiphany and Midori load pages even if certificate verification fails; the verification result is only used to change the status of a security indicator, basically giving up your session cookies to attackers.

Removing WebKit1

WebKit2 has been around for Mac and iOS for longer, but the first stable release for WebKitGTK+ was the appropriately-versioned WebKitGTK+ 2.0, in March 2013. This release actually contained three different APIs: webkitgtk-1.0, webkitgtk-3.0, and webkit2gtk-3.0. webkitgtk-1.0 was the original API, used by GTK+ 2 applications. webkitgtk-3.0 was the same thing for GTK+ 3 applications, and webkit2gtk-3.0 was the new WebKit2 API, available only for GTK+ 3 applications.

Maybe it should have remained that way.

But, since the original API was a maintenance burden and not as stable or robust as WebKit2, it was deleted after the WebKitGTK+ 2.4 release in March 2014. Applications had had a full year to upgrade; surely that was long enough, right? The original WebKit API layer is still maintained for the Mac, iOS, and Windows ports, but the GTK+ API for it is long gone. WebKitGTK+ 2.6 (September 2014) was released with only one API, webkit2gtk-4.0, which was basically the same as webkit2gtk-3.0 except for a couple small fixes; most applications were able to upgrade by simply changing the version number. Since then, we have maintained API and ABI compatibility for webkit2gtk-4.0, and intend to do so indefinitely, hopefully until GTK+ 4.0.

A lot of good that does for applications using the API that was removed.

WebKit2 Adoption

While upgrading to the WebKit2 API will be easy for most applications (it took me ten minutes to upgrade GNOME Initial Setup), for many others it will be a significant challenge. Since rendering occurs out of process in WebKit2, the DOM API can only be accessed by means of a shared object injected into the web process. For applications that perform only a small amount of DOM manipulation, this is a minor inconvenience compared to the old API. For applications that use extensive DOM manipulation — the email clients Evolution and Geary, for instance — it’s not just an inconvenience, but a major undertaking to upgrade to the new API. Worse, some applications (including both Geary and Evolution) placed GTK+ widgets inside the web view; this is no longer possible, so such widgets need to be rewritten using HTML5. Say nothing of applications like GIMP and Geany that are stuck on GTK+ 2. They first have to upgrade to GTK+ 3 before they can consider upgrading to modern WebKitGTK+. GIMP is working on a GTK+ 3 port anyway (GIMP uses WebKitGTK+ for its help browser), but many applications like Geany (the IDE, not to be confused with Geary) are content to remain on GTK+ 2 forever. Such applications are out of luck.

As you might expect, most applications are still using the old API. How does this work if it was already deleted? Distributions maintain separate packages, one for old WebKitGTK+ 2.4, and one for modern WebKitGTK+. WebKitGTK+ 2.4 has not had any updates since last May, and the last real comprehensive security update was over one year ago. Since then, almost 130 vulnerabilities have been fixed in newer versions of WebKitGTK+. But since distributions continue to ship the old version, few applications are even thinking about upgrading. In the case of the email clients, the Evolution developers are hoping to upgrade later this year, but Geary is completely dead upstream and probably will never be upgraded. How comfortable are you with using an email client that has now had no security updates for a year?

(It’s possible there might be a further 2.4 release, because WebKitGTK+ 2.4 is incompatible with GTK+ 3.20, but maybe not, and if there is, it certainly will not include many security fixes.)

Fixing Things

How do we fix this? Well, for applications using modern WebKitGTK+, it’s a simple problem: distributions simply have to start taking our security updates.

For applications stuck on WebKitGTK+ 2.4, I see a few different options:

  1. We could attempt to provide security backports to WebKitGTK+ 2.4. This would be very time consuming and therefore very expensive, so count this out.
  2. We could resurrect the original webkitgtk-1.0 and webkitgtk-3.0 APIs. Again, this is not likely to happen; it would be a lot of work to restore them, and they were removed to reduce maintenance burden in the first place. (I can’t help but feel that removing them may have been a mistake, but my colleagues reasonably disagree.)
  3. Major distributions could remove the old WebKitGTK+ compatibility packages. That will force applications to upgrade, but many will not have the manpower to do so: good applications will be lost. This is probably the only realistic way to fix the security problem, but it’s a very unfortunate one. (But don’t forget about QtWebKit. QtWebKit is based on an even older version of WebKit than WebKitGTK+ 2.4. It doesn’t make much sense to allow one insecure version of WebKit but not another.)

Or, a far more likely possibility: we could do nothing, and keep using insecure software.

37 Replies to “On WebKit Security Updates”

  1. This is fascinating. Do you know how this manifests in RHEL and derivatives? Do they act more like the snapshot distributions like Debian, or are security updates more regularly backported in?

    1. Judging from https://git.centos.org/summary/?r=rpms/webkitgtk3.git the answer is: kind of in between. They clearly took a big update, but it’s to a very old insecure version. They’re kind of out of luck here though; due to the WebKit2 API change, they can’t get rid of this old compatibility package, and I guess don’t have anything that needs modern WebKit.

  2. Thanks for the very informative post. It helped me to see how application are affected by an upstream software such as WebKit. Moreover, I can pick up my mail-client, distribution and web-based software more wisely.

  3. A few observations from me, as a longtime Linux/GNOME/Fedora user and a longtime Mozilla contributor. (Tho obviously not, I suppose I should dutifully clarify, in any capacity as employee or project representative.) Also: not meaning or intending to troll, although I’m certain some of these opinions are naturally trollish. :-) I’ll try to minimize (if not eliminate) comment interaction here to counteract devolutionary tendencies this comment might engender.

    A small part of me feels a slight bit of schadenfreude that you’re going through a similar pain to what Firefox and its addons are now going through, adapting to our multiprocess (and, eventually, sandboxed, as you allude) Electrolysis project. (Of course you arguably have it worse because mismatching means a complete failure to build, not just runtime failures of JS code, and that strongly incents remaining woefully insecure over doing massively-involved updating.) This is a bad part of me that should (and does) feel bad. Nobody’s any better off when the bad dudes complicate life on the Internet, for anyone, and make the entire public resource that much less valuable to everybody else via network effects.

    Getting distros to ship vendor security updates is one of the main reasons I’ve always been extraordinarily negative on the ironic “cathedral” model of having open source distros separately ship most software that runs on them, effectively boxing out almost all high-value proprietary software whose presence, I believe, is (like it or not) necessary for Linux desktop success. (And on each distro having its own packaging system, or at least on having multiple packaging systems.) Security-conscious projects can always be more attentive to their own security, than every separate distro having to care about hundreds of separate applications can be. This is practically harder for “library”-like things like browser rendering engines with separate users, but I don’t see any good answer there at all, and certainly not if depending on binary interfaces/ABIs. (Other than to pull an Apple and promptly and religiously ship API-stable security updates as part of the OS. Or to pull a Mozilla and, for better and worse, give embedding the shaft, which decision I have frequently grumbled about and that has long since come back to bite us, IMO. :-\ ) Distro software-shipping fragmentation is surely a big factor in Linux desktop never taking off. (Although even supposing that problem solved, it’s probably still doubtful it could take off without a single entity to advocate for it, as Microsoft and Apple can be.)

    Debian might ship ESRs all it wants, and in theory we do support them. In practice, mainline code changes a lot from the time an ESR is first released til its EOL. Not all security patches against trunk are backportable. ESRs might satisfy a (small) constituency, but we’d really prefer if no one used them, instead just sticking to our latest non-ESR release all the time. Unfortunately, but perhaps unsurprisingly, most people don’t care about security like browser vendors care about security, and they’ll often happily give up ideal security for something inferior along that line but marginally better along another.

    Putting so much WebKit infrastructure into port-specific code never made much sense to me. Especially not for security-critical stuff. For all its embedding demerits, I much prefer shoving basically everything into the the browser engine, even if it might then integrate somewhat less better with the platform. Security and having actually-working features are more important than utmost customizability.

  4. Pingback: Catanzaro: On WebKit security updates | Linux Press

  5. When you say “distributions simply have to start taking our security updates” you mean “distributions simply have to start taking our unstable feature and api updates”, because you don’t release security updates for gtkwebkit 2.4 or even 2.8!

    I mean, I understand, I do. The web platform is a huge clusterfuck, and we’re all stuck on this train (myself included) because it’s moving too fast to jump off.

    Unfortunate situation. But don’t blame stable distributions for it, it’s the whole messed up industry/ecosystem which isn’t any one person’s fault really.

  6. (I should clarify that, as a developer involved in the web, you probably have a different definition of “stable” than developers maintaining “stable distributions”)

  7. @Jeff: Whilst Mozilla does provide a convenient service with updates to the users on windows & make the latest or Aurora builds available for download under Linux. Most of the software that goes to make a usable system is NOT so actively supported, the distro’s do add value, and things like the Firefox integration into KDE that OpenSuSE produced have been a benefit to me. The distro’s are definitely not Cathedrals, the bar to entry is much lower than say GNU and the awkward differences with respect to package management are a consequence of the Bazaar approach. Yes, it drives me crazy when I see Fedora, re-implement a SAT solver, rather than “simply” use say zypper developed for OpenSuSE/SLES. But having installed a lot of FLOSS direct in the 90’s, it was clear the upstream developers did not generally understand the whole of the problem, so Linux end-users going end-to-end with development projects would lead to an incoherent mess.

    1. I think Fedora is actually using SUSE’s SAT solver. (hawkey depends on libsolv.) Your point remains if you consider everything above the SAT solver, though.

  8. This is a very interesting article, shining a light on the mess that is WebKit.

    Unfortunately, it seems to me that it shifts a little too much blame on the distributions.
    They have a very tough job balancing between the desire of the user to have the software they want available – and actually have it working – as well as keeping up with upstream which makes their job very hard.
    This leads to the situation that distributions will have to keep multiple versions of each WebKit-port around, even more work than the many ports, bindings and more are already.
    That is only made worse by the fact, that most distributions (not even Debian) don’t have the sheer manpower that Fedora and GNOME have available, because they rely much more on pure volunteers and have no full time employees available.

    KDE actually did the right thing, moving away from WebKit, because using WebKit is simply not feasible, it is just a way too large and complex project which can move in totally unexpected directions depending on what Apple does (and in the past, what Google has done). Building upon Chromium can only improve the situation.

    For the company behind WebKit-EFL – let’s just call them by their name, they won’t be mad: Samsung – it is a similar situation. Even though Samsung actually pays some people to keep the port working, it is still a lot of work and I think even they realized, that it is not financially feasible. Samsung actually does go a similar direction as Qt/KDE, they seem to plan to abandon WebKit as well in the long term, and decided to not spend the money on more manpower for the EFL port, but instead invest in Servo. Pure speculation on my part, but I’m fairly certain, that Samsung will move to Servo as soon as they possibly can.

    So while it is great, that GNOME does invest resources in cleaning up the mess that is WebKit security updates, I don’t think this will generally improve things much.
    As you rightly state, it is unlikely that all of the very popular applications will be updated, it is just too much work, so distributions will still need to have a couple of versions they “support”.

    All in all, I would assume that rather sooner than later, WebKit-gtk will be the only port remaining and the only party investing in actual WebKit will be Apple. Since Apple has spend far less on WebKit in the past couple of years, and the overwhelming WebKit dominance on mobile will, due to Google moving on, very soon be a thing of the past.
    This means even GNOME is solely dependent on Apple, because even GNOME would never be able to actually keep WebKit up to date with web-standards and security once Apple decides to move on as well – at some point they will.

    Servo will support the Chromium Embedded Framework, which means there is a strong argument to be made that sooner or later every party without active involvement in WebKit will move on to CEF, which is a much easier API to support – albeit not as powerful at this time. I would go as far to predict, that GNOME will have to follow, maybe sooner than expected.

    1. I don’t think it’s accurate to say KDE is “moving away from WebKit.” I’ve seen no evidence to support this, and that is a big problem, because QtWebKit is insecure and KDE legitimately needs to move away from WebKit as quickly as possible for that reason alone. Unfortunately, it’s not clear that such a move will be practical; I’d be interested it see if something like KMail could be ported to QtWebEngine.

      I find it interesting that you’re concerned about WebKit moving in “totally unexpected directions.” I have exactly the opposite concern. With WebKit, we have a good relationship with Apple and know they will not make changes that would cause significant problems for us, and that they also accommodate our changes even when it is somewhat inconvenient for them. In contrast, the needs of a hypothetical ChromiumGTK+ would have zero influence in the Chromium community: we have no illusions about this. We’ve certainly evaluated a switch to Chromium, but have decided to stick with WebKit indefinitely as it seems like a safer long-term choice. Who can say what the future will bring, but don’t expect to see this change anytime soon. (I am aware that Chromium has many advantages; no need to point this out.)

      I’m very interested in Servo, but I think you are underestimating how young that project is. I expect it will be many years before it will be competitive with WebKit. Let’s see what the future brings.

      1. I take it you’ve spent as much time building a relationship with Google as you have Apple, and can thus say for certain that Apple is more accommodating of your needs (whatever those might be that seem to so strongly clash with blink’s)?
        What features are missing from chromium that GNOME requires? Where are the bug reports that Google employees closed as wontfix?
        Please, don’t take this as me being aggressive, but at this point I’ve seen you make these claims several times without providing evidence for them so, at some point, my curiousity overcomes my laziness and pushes me to ask these questions.
        I’m particularly curious about what is missing from CEF that GNOME requires.

        Again, please don’t read this post as having an aggressive tone, as that wasn’t my intent at all.

        1. Hi, thanks for your question. I hesitate a bit to reply to this, since this is a difficult topic to discuss, and certainly I don’t want to get into a debate here, but here’s where we’re coming from. Probably the most important considerations are (a) it would take years of effort and therefore $$$$$ we don’t have to switch to Chromium, and (b) everyone currently involved prefers to work on WebKit. Nothing else really matters compared to those considerations. And no, I have no relationship with Google (though some of my colleagues do). But laying that aside, I still think WebKit is a safer bet, even if it might not turn out to be best; let me try to respond, you can tell me how convincing a case I make. For us, the goal is to have a great GObject API for making it as easy as possible to write browsers and to put web views in other applications. Strike the word GObject from that sentence, and you find our goals align almost exactly with the WebKit project’s. Chromium has little interest in that: their goal is to make Chrome a great web browser, on mobile devices specifically. They are doing a very good job at that, but that’s not our goal. That’s why CEF is an external project, not part of Chromium. It’s also why everything downstream of Chromium, like CEF, needs to be updated with each new release: the Chromium content API, which CEF is based on, is not stable. This is a significant maintenance burden for embedders, and it would be a significant maintenance burden for us. Currently we have all of our code in the upstream WebKit repository; sometimes Apple accidentally breaks it, but normally they fix it when this happens. It still requires lots of maintenance, but it would be a lot harder if we were a downstream project, like we would be with Chromium. APIs we rely on would change or disappear without notice; usually we would be able to make do, but occasionally we would be forced to break our API. With WebKit we get a big say in things, and we have a stable API and ABI as long as we want to (with one exception, we do deprecate portions of the DOM API when the DOM spec changes, and those functions sometimes stop working; but the DOM API is huge and I’ve never heard of this affecting any application in practice). Now, if we have to break API due to an upstream change… well, you read the blog, it means no distro security updates ever again for the lifetime of the distro. This is a hypothetical situation of course, but it’s a real risk with Chromium, in contrast to WebKit. It’s really hard to understate how huge of an advantage it is to be able to work upstream. Ports make the code harder to maintain; it would be bad for Chrome to allow them. But without ports, there are some things we just cannot accomplish. My favorite example is video playback. We can’t have ffmpeg in Fedora for legal reasons. There is a Chromium-GStreamer project, but I’ve heard conflicting accounts of whether it might be upstreamed or not; Google does not care about GStreamer, there is a bug for that closed WONTFIX; it’s not needed for Chrome, so why would they want it upstream? So, want to use a browser without any video playback? In WebKit, we get our GStreamer backend. Now, say Google does add GStreamer support; it might happen. Next requirement: when starting video playback, call out to GNOME Session via D-Bus to take a suspend inhibitor, so GNOME doesn’t blank the screen while you’re watching your video. It would probably be possible to figure out a way to do that with Chromium, but we would not be able to modify any code in Chromium itself; I think we could hack a solution together, but it would be harder than with WebKit we just add the code to HTMLMediaElement and move on. That is just the very first example that comes to mind. Next example: when playing a video for which the appropriate codecs are not installed, allow applications to inform the user, run codec installation via GStreamer (which goes through PackageKit on Fedora), then restart playback when the codec is installed. That’s not going to ever be possible with Chromium, I don’t think. How about Wayland support? In Chromium that has to be maintained out of tree, since Google doesn’t care about Wayland, and Chromium has to be patched a little to make that work. Then we have to bundle Chromium, because we patched it, which is nuts, because Chromium dwarfs WebKit and WebKit is probably bigger than the rest of your OS combined, we can’t bundle Chromium separately in everything that needs it (like your Atom text editor, like your Visual Studio, etc.), you wouldn’t have room for anything else on your disk. In WebKit we bundle only ANGLE (not counting code used only for tools and tests that never gets installed). The Wayland support is Chromium is also, frankly, very immature compared to WebKit’s (which is admittedly not quite ready for prime time either, but is close), and again maintained out of tree so that would also force us to bundle Chromium, because Google doesn’t care about Wayland. How about drawing buttons and checkboxes and progressbars and such things with the GTK+ theme? This feature is not necessary, but it’s quite nice to see GNOME checkboxes in particular on web sites — and there’s no way we could get that with Chromium. Those are, again, just a few things that immediately come to mind…. I haven’t looked much at CEF much myself, but my colleagues have evaluated it and concluded that we would be better off using the Chromium content API directly. That conclusion was for a different project, not WebKitGTK+, but I think it probably applies here too. It’s an extra layer to go through, bus factor 1, and I’ve heard the API is not very mature (I hesitate just a bit to repeat this claim, having not worked with it myself). From a very quick look, it seems the DOM API is only available from the browser process; if that’s correct (I took only a very quick look), then that is arguably a serious design error, as while convenient for applications, that’s going to have terrible performance. So one an example of a requirement would be a web extensions API, like we have with WebKit. Anyway, I’m pretty sure we could get that using the content API instead of CEF. “Extra layer to go through” is a really significant consideration; say we want to add some new API, it would suck to always need to add it in CEF first as well, right? In WebKit, we routinely change code at all levels of the stack. In Chromium, we get Chromium as-is; if we want to change anything, got to convince Google why it’s good for them to do so. We would be switching from a project where we have a decent amount of influence (though to be clear, Apple is running the show), to one where we have none. Try generating a GObject DOM API using Chromium or CEF. Maybe it’s possible, but I doubt it. So we would have to curate a DOM API manually, which would be very expensive, or half our applications would not be able to exist…. Well, more likely, we would patch Chromium to make it possible, but it would be very nice to not need to bundle Chromium. Next, portability. Distros like Debian and Fedora support tons of CPU architectures (on Fedora they are the secondary architectures) that will never be able to run Chromium, because of V8. With WebKit we get JSC, probably the most advanced JS engine right now (though I’m not one to debate that), and which runs anywhere (albeit slowly) if you turn off the JIT. I don’t care about these arches very much, but it would kind of suck to have to remove a huge amount of GNOME apps from them, right? Next, memory use. One big complaint about Epiphany is that it’s a memory hog, but WebKit is significantly better with memory than Chromium. Sometimes I think people are jumping to Chromium just because it’s trendy, without thinking these things through. Now, there would be one major benefit to switching to Chromium: we’d render pages just like Chromium. That’s a really, really big benefit! It is a tempting benefit. It’s the benefit you want. Sometimes I think Epiphany might be better off to use Chromium. But I’m pretty sure all the rest of GNOME is much better off with WebKit. Actually there’s another benefit too: security. Chromium has lots of HTTPS features I want for WebKit, and it has a top-class sandbox. (WebKit’s sandbox is so bad, it is disabled by default. My job was to fix that but I failed.) These are things we can fix in WebKit, but it requires investment and I haven’t made a successful case for that. Now, that all is ramble-y and not really a coherent argument, but there you go, but it’s not meant to be an argument per se, more of a “here is where we’re coming from, this is why WebKit seems good to us.” And again, most importantly, I rather like WebKit. :) Certainly, the troll comment down below about WebKit being a “toy engine” is ludicrous.

          1. Just how secure is JSC? I checked the webkit-gtk installations on my debian box and each included their own matching, presumably insecure, javascript implementation.

            i.e. It would be nice if JSC could be decoupled from the webkit implementation.

          2. I don’t have stats to back this up, but my impression is that JSC has significantly fewer vulnerabilities than the rest of WebKit. I would worry much more about HTML rendering than about JSC.

            Do you want to use WebKit with some other JavaScript engine? It used to be possible, for V8, but when Google left that was removed. I’m not aware of any compelling advantages of other engines over JSC nowadays so dunno why you would want this.

            Or do you want to use JSC without WebKit? I think you can do that by running Tools/Scripts build-jsc --gtk. There is work underway to make it possible without the GTK+ port dependencies.

      2. From the outside the would seem like the opposite would be true. Chromium runs on Linux supported by Google. WebKit does not run on Linux supported by Apple. Meaning WebKit running on Linux has no support from Apple themselves. On top of that some large double digit percentage of Googlers use Linux. I doubt 1% of Applers use Linux. In other words, Google needs Chromium to run on Linux. Apple does not need WebKit to run on Linux. On top of that; Chromium runs on ChromeOS, a Linux variant? And Android, another Linux variant?

        So it would seem Chromium is far more aligned with Linux than WebKit but like I said, this observation is only from the outside. I have no idea what the actual responses and support are like from the WebKit team or the Chromium team with respect to Linux or KDE.

    2. Thank you for this very informative post Michael, finally I understand something about the webkit situation.

  9. option one gets ruled out due to work, but upgrading applications probably requires more total manpower.

    application developers should be careful when matching their available manpower to upstream’s willingness to provide long term security support. similarly, as you mention with the Debian iceweasel example, distributions should carefully consider how well their update process matches up with upstream support. perhaps the Ubuntu process is not simply negligent, but rather pragmatic when considering finite manpower. (crazy Ubuntu custom application development aside…)

  10. > QtWebKit

    Was removed from Qt. See https://wiki.qt.io/New_Features_in_Qt_5.6

    > Calligra

    Is not using Web(Kit|Engine) at all. Kexi, the msaccess for Linux, used to have an optional reports component using QtWebKit but it was removed some years ago. The Words HTML-import filter is using (the DOM-part) of KHTML to convert to ODT.

    1. From the linked wiki page:

      “With Qt 5.6 the following modules are no longer part of the release packages, but users can still build them from source:”

      Guess what. Most distributors of KDE likely builds from source so this doesn’t affect them. In short, it’s not really been removed.

  12. Great post, thanks.

    I would like to mention that there’s also Oracle WebKit port.
    It’s part of JavaFX and ships with Oracle’s JRE and JDK for Linux, OS X and Windows.
    I’m afraid it’s in bad shape regarding security, the last time I checked there wasn’t anyone from Oracle on the WebKit security mailing list.

    Also I think Gentoo has at least some support for the latest WebKitGTK+ release:
    https://packages.gentoo.org/packages/net-libs/webkit-gtk

  13. For applications that embedded GTK+ widgets into the web view, could the GTK+ Broadway backend help?

    1. No, the backend used does not make any difference. (Well, it matters a bit in that WebKit doesn’t support Broadway. :)

      1. I wasn’t suggesting running the WebKit backend using Broadway; I was suggesting that if an application wants to embed GTK+ widgets in WebKit, it could render those widgets using Broadway and embed the result.

  14. The fix is simple — people should stop using toy browser engines for security sensitive applications.

    1. That is easier said than done. On Gnome, at least Evolution and Empathy still use Webkit1. Evolution is one of the selling points of the Gnome project. People can’t simply stop using it.

  15. In your blog post, you acknowledge the importance of having CVE to bug/commit mapping data available. You only mention it’s available to WebKit security team members, is the data available publicly? Can it be made available if it’s not? WSAs don’t provide such info, so they are not actionable for those who can’t move to the latest upstream versions immediately.

    1. You would have to ask Apple for this. I’m not sure if they would be willing to release it publicly. The data would not really be actionable for you regardless; due to the difficulty of backporting large numbers of fixes, it does not make any sense to attempt to do this downstream, especially if you’re not familiar with the code.

      If you want to try maintaining the 2.4 branch — it would be quite silly to try maintaining any other branch — I would start by backporting all patches with changelog links to private Bugzilla entries. That will take care of about half the vulnerabilities. You might be surprised at the effort that will require, and the risk of introducing new bugs in the process.

  17. A big issue is that many application maintainers do not want to switch to GTK3 for a variety of reasons–the API is not very stable, zany things keep happening like the application being expected to draw its own titlebars, etc.

    1. GTK+ 3 applications aren’t expected to draw their own titlebars, that’s completely false…. GNOME applications are, but they have to go out of their way to do so, by creating a GtkHeaderBar and setting it as the title bar. As long as you don’t do that, your window manager will draw the title bar as always. There was one release where this wasn’t the case, I think 3.10, where all apps drew their own titlebars… it got reverted due to complaints. That was kind of zany.

      Other than that, you have a good point. Nowadays, I don’t think anything zany is going on, but API stability is a problem with GTK+ 3 indeed, and even worse is theme stability. You can’t migrate to GTK+ 3 unless you’re willing to test your app with development versions of GTK+, because something is almost guaranteed to break with every minor release. (GTK+ 3.20 is breaking many somethings.) My recommendation for GTK+ 2 apps that don’t want to put up with that is to stick with GTK+ 2 and try migrating away from WebKitGTK+ instead, perhaps to Chromium.

  18. 2 things:

    1. Gentoo also offers the latest webkit-gtk package available adjacently to its release.

    2. I think what Debian guys mean when they say “library interdependencies make it impossible to update to newer upstream releases” is that there are still some irreplaceable applications that rely on the old API. I’ve encountered a couple myself – liferea and claws-mail: fancy addon.

    These are just what I’ve encountered there might be more and that’s makes it hard for distro maintainers to keep up. Unfortunately that’s the way it is…

    1. 1. The latest stable version of the net-libs/webkit-gtk Gentoo package is 2.8.5 (and only on amd64 and x86 arches), see: https://packages.gentoo.org/packages/net-libs/webkit-gtk. I suppose that’s equivalent to Debian testing, though. I guess I’d better give it credit; it’s been mentioned twice now.

      2. No, the policy was written many years before WebKit2, back when we really did bump our dependencies too often.

    2. OK, updated:

      (You can also safely use WebKitGTK+ in Debian testing — except during its long freeze periods — and Debian unstable, and maybe also in openSUSE Tumbleweed, and (update) also in Gentoo if you compile with the “test” USE flag. Just be aware that the stable releases of these distributions are currently not receiving our security updates.)

      1. Eh, I guess it’s not a USE flag, but a separate repo? Gentoo is confusing! I will edit this again….

Comments are closed.