redhat.corpmerchandise.com is no longer broken. That’s a relatively good reaction time to the problem. Unfortunately, I’ve seen no talk of change in NSS to prevent sites from making similar mistakes in the future, and they are out of medium Fedora T-shirts.
First of all, I should probably admit that, despite the title of this post, no, the redhat.corpmerchandise.com fiasco is not Mozilla’s fault: it’s Red Hat’s, because obviously Mozilla has no control over that domain. But that wouldn’t make for a very interesting title for a blog post, and Mozilla set the stage for this to happen, so let’s go with “Mozilla’s fault.” Also, it’s not really Red Hat’s fault; Staples is really to blame, since corpmerchandise.com is their domain, but I really shouldn’t be pointing that out when the point of this blog post is to blame Mozilla. And gosh now I’m off on a tangent, but it’s not really a fiasco either: it’s a significant screw-up, but not that big a deal; but words like “fiasco” make for good clickbait headlines, so let’s go with that. FIASCO.
One last note before I begin. I hold Mozilla to a higher standard than other software development companies. Sometimes it makes mistakes, like the one I’m about to present, and it’s important to call them out when this happens, but it’s because of good choices at Mozilla that Firefox still (mostly) respects your freedom, unlike other major browsers. It’s a good company.
OK, so you’ve read this far in suspense, I should probably explain the redhat.corpmerchandise.com fiasco before you reach the end of your three-paragraph Internet-length attention span. Yesterday the Fedora Store went live, where you can buy low-cost Fedora-branded items: a T-shirt, water bottle, pub glass, or baseball cap. I want a T-shirt. OK, that’s great, so what is the fiasco? Well click on this link (quick! before it gets fixed!) to find out: https://redhat.corpmerchandise.com/ProductList.aspx?did=20588
Now, depending on your browser, you may or may not have discovered the problem. When I load that site in Firefox, I see Fedora merchandise. When I load it in Epiphany, I see something noticeably less friendly:
“Legitimate banks, stores, and other public sites will not ask you to do this.” Ouch. (I actually took that language from Firefox when I designed that interstitial for Epiphany.) Ah, well, clearly there is some bug in Epiphany, because Firefox is a major browser and Firefox doesn’t get stuff like this wrong, right? Well, no, Epiphany is not wrong. Then Firefox is wrong? Well… from a certain point of view… (like mine)….
Firefox and Epiphany use different cryptography libraries to determine if the certificate is valid, and they sometimes differ in what certificates they will accept. Firefox uses NSS, a library maintained by Mozilla primarily for use by Firefox (it’s also used by Chrome on Linux), while Epiphany (indirectly) uses GnuTLS, originally a GNU project that is now de-facto maintained by Red Hat. So is NSS just better than GnuTLS at determining whether a certificate is valid? Actually, NSS really is more permissive than GnuTLS, and this does sometimes lead Firefox to approve of sites that Epiphany will not, but that’s not the case here. Let’s try a little experiment to see what’s happening. Firefox has a weird feature that feels like it was designed in the 90s for the era when computers had one user account apiece: it lets you create multiple profiles for bookmarks, history, and other settings. So let’s give this a whirl:
$ firefox -ProfileManager
Create a new profile, launch Firefox with it, then load https://redhat.corpmerchandise.com/ProductList.aspx?did=20588 to try this experiment again. Or just keep reading and trust me when I say that you’ll see this:
Oooh, that’s not good, now Firefox thinks we’re being attacked. We’re not. So what’s going on here? Why is Firefox so inconsistent?
First off, let’s get one thing straight: this site is totally and hopelessly broken. To see why, let’s use the super-handly tool
$ gnutls-cli redhat.corpmerchandise.com
Processed 182 CA certificate(s).
Connecting to '220.127.116.11:443'...
- Certificate type: X.509
- Got a certificate list of 1 certificates.
- Certificate info:
- subject `C=US,ST=Kansas,L=Overland Park,O=STAPLES CONTRACT & COMMERCIAL\, INC.,OU=Information Techology,CN=*.corpmerchandise.com', issuer `C=US,O=DigiCert Inc,OU=www.digicert.com,CN=DigiCert SHA2 High Assurance Server CA', RSA key 2048 bits, signed using RSA-SHA256, activated `2014-11-12 00:00:00 UTC', expires `2015-12-09 12:00:00 UTC', SHA-1 fingerprint `50cfb26c680434d132dc64e80db54de51a5a07a6'
Public Key ID:
Public key's random art:
+--[ RSA 2048]----+
| . |
| = S |
| .* O o o |
|o .+.X E o . |
| =.. =.+.+ . |
|..o oo+oo |
- Status: The certificate is NOT trusted. The certificate issuer is unknown.
*** PKI verification of server certificate failed...
*** Fatal error: Error in the certificate.
*** Handshake has failed
GnuTLS error: Error in the certificate.
If you’re familiar with digital certificates, it’s pretty obvious what’s wrong here. When you connect securely to a web site, it sends a chain of certificates: the first certificate is owned by the web site, then it sends some number of additional certificates, usually one or two, that belong to certificate authorities. Each certificate is signed by the next certificate in the chain (not quite, but it’s almost true, so let’s go with that for this post), up until you get to the last one in the chain, which must be signed by a certificate in your browser’s (or operating system’s) root trust store. The certificates in your root trust store are super valuable, and if one were to be compromised by an attacker the devastation to the Web would be terrible, so certificate authorities must keep their roots safe at all costs, and they do this by almost never using them. Legitimate certificate authorities never sign web sites’ certificates with their root certificate; instead, they create a few other certificates, sign them with the root, and use only those to sign web sites’ certificates. So if you ever visit a site and it sends you only one certificate, you know that the site is broken for sure. And here we have a site that has sent only one certificate (there’s a Certificate but no Certificate), a classic case of server misconfiguration (aka fiasco).
So why did Firefox allow the site at first, even though it has no chain of trust, but not allow it with a fresh profile? Well, even though the site has presented no chain of trust, NSS goes far, far out of its way to find one. Whenever you visit a web site, NSS saves each intermediate certificate it sees, makes sure it’s signed by a trusted root, and caches it for future use. Then, whenever you visit a site that sends a broken chain of trust, NSS will effectively treat all those intermediates as roots, and use them to complete the chain of trust. This is completely safe, since it has already verified them. Those intermediate certs are saved in your Firefox profile, so by switching to a fresh profile they are no longer used, and you can’t access the broken site anymore.
If you were able to access redhat.corpmerchanise.com in Firefox, you can verify this for yourself: open Preferences -> Advanced -> Certificates -> View Certificates -> Authorities. Anything listed as Default Trust or System Trust is a root, and anything listed as Software Security Device is a cached intermediate cert. Don’t touch those root certs, but feel free to Delete or Distrust any Software Security Device — it will just be cached again the next time you visit a web site that uses it. Scroll down to DigiCert SHA2 High Assurance Server CA. That’s the cached intermediate cert that is allowing you to visit redhat.corpmerchandise.com — it’s not shipped with Firefox, and new Firefox users won’t have it. Delete it, restart the browser, then try reloading https://redhat.corpmerchandise.com. Oh no, it’s untrusted! Now visit https://stackoverflow.com, which sends a certificate signed by DigiCert SHA2 High Assurance Server CA, which will cause NSS to cache it once again. Now back to https://redhat.corpmerchandise.com, and Firefox knows it’s safe again. And that, folks, is how you screw up your web site so that it only works if you first visit Stack Overflow.
So why does NSS do this? Well, once upon a time (ten years ago), browsers were less strict about verifying chains of trust, and on an untrusted connection would let you proceed with maybe just a pop-up warning, and maybe not even that. So sites were less diligent about making sure they had valid chains of trust than they are today, in the era of nasty interstitial warnings that discourage the user from visiting the site. Since there were a lot of sites with broken chains, NSS chose to cache intermediate certificates to reduce the number of unnecessary validation errors for Firefox users. At the time, this might have been an OK choice.
Today, if your online store is missing a chain of trust, the browser makes clear in no uncertain terms that this site is not to be trusted, and sites lose visitors/customers, so they try pretty hard to get this right. (How many lost visitors depends on the browser — a large majority of Chrome and probably Epiphany users will click through the warnings, but a large majority of Firefox users will not, because Firefox’s UI for this is much scarier.) When setting up a new site, you check it in a couple of browsers to make sure it works properly, and you trust that if it works in Firefox on your machine, surely it will work in Firefox for everyone else, right? Well, no, it won’t. When setting up a secure web site, you must always test it with a fresh Firefox profile to make sure that you got the chain of trust correct. Of course, nobody knows to do this, which is how we wind up with broken sites like redhat.corpmerchandise.com.
I suspect this breakage would happen far less often if NSS did not cache intermediate certs, tricking site admins into thinking their sites are set up properly. Sure, cached certs don’t hurt the user who has them cached, but they’re bad for all other users of Firefox, as well as users of browsers that do no certificate caching. And there’s no good reason for this, because browsers don’t need to cache intermediate certificates in 2015, because almost all sites that redirect from HTTP to HTTPS get this right nowadays, and those that get it wrong are probably getting it wrong because they tested with a browser that had the right cached intermediate. Chicken and egg much? There’s only one way to fix this problem, and that brings me to my request: Mozilla, do the Web a favor and stop caching intermediate certificates.
P.S. Astute readers would note that there’s absolutely no point in deleting an intermediate certificate with the Firefox certificate manager, except to test things like this. It’s just going to come back the next time you see it.
For the past couple of weeks, https://www.amazon.com and https://amazon.com have redirected me to http://www.amazon.com. Region-specific sites like https://www.amazon.co.uk/ still work fine. There is probably no MITM attacker, since the secure page is performing the redirect, so a MITM would have to have a valid certificate for www.amazon.com, and if so he would presumably not add a redirect.
Questions for Amazon:
- What the hell?
- Why does your site work at all without HTTPS?
- How am I going to buy things now?
It’s 2014, and this is unacceptable for an e-commerce site, plain and simple. Repent by implementing HSTS.
It’s already been a few weeks since the release of GNOME Web 3.14, so it’s a bit late to blog about the changes it brings, but better late than never. Unlike 3.12, this release doesn’t contain big user interface changes, so the value of the upgrade may not be as immediately clear as it was for 3.12. But this release is still a big step forward: the focus this cycle has just been on polish and safety instead of UI changes. Let’s take a look at some of the improvements since 3.12.1.
The most important changes help keep you safer when browsing the web.
Safer Handling of TLS Authentication Failures
When you try to connect securely to a web site (via HTTPS), the site presents identification in the form of a chain of digital certificates to prove that your connection has not been maliciously intercepted. If the last certificate in the chain is not signed by one of the certificates your browser has on file, the browser decides that the connection is not secure: this could be a harmless server configuration error, but it could also be an attacker intercepting your connection. (To be precise, your connection would be secure, but it would be a secure connection to an attacker.) Previously, Web would bet on the former, displaying an insecure lock icon next to the address in the header bar, but loading the page anyway. The problem with this approach is that if there really is an attacker, simply loading the page gives the attacker access to secure cookies, most notably the session cookies used to keep you logged in to a particular web site. Once the attacker controls your session, he can trick the web site into thinking he’s you, change your settings, perhaps make purchases with your account if you’re on a shopping site, for example. Moreover, the lock icon is hardly noticeable enough to warn the user of danger. And let’s face it, we all ignore those warnings anyway, right? Web 3.14 is much stricter: once it decides that an attacker may be in control of a secure connection, it blocks access to the page, like all major browsers already do:
(The white text on the button is probably a recent compatibility issue with GTK+ master: it’s fine in 3.14.)
Safety team members will note that this will obviously break sites with self-signed certificates, and is incompatible with a trust-on-first-use approach to certificate validation. As much as I agree that the certificate authority system is broken and provides only a moderate security guarantee, I’m also very skeptical of trust-on-first-use. We can certainly discuss this further, but it seemed best to start off with an approach similar to what major browsers already do.
The Load Anyway button is non-ideal, since many users will just click it, but this provides good protection for anyone who doesn’t. So, why don’t we get rid of that Load Anyway button? Well, different browsers have different strategies for validating TLS certificates (a good topic for a future blog post), which is why Web sometimes claims a connection is insecure even though Firefox loads the page fine. If you think this may be the case, and you don’t care about the security of your connection (including any passwords you use on the site), then go ahead and click the button. Needless to say, don’t do this if you’re using somebody else’s Wi-Fi access point, or on an email or banking or shopping site… when you use this button, the address in the address bar does not matter: there’s no telling who you’re really connected to.
But all of the above only applies to your main connection to a web site. When you load a web page, your browser actually creates very many connections to grab subresources (like images, CSS, or trackers) needed to display the page. Prior to 3.14, Web would completely ignore TLS errors for subresources. This means that the secure lock icon was basically worthless, since an attacker could control the page by modifying any script loaded by the page without being detected. (Fortunately, this attack is somewhat unlikely, since major browsers would all block this.) Web 3.14 will verify all TLS certificates used to encrypt subresources, and will block those resources if verification fails. This can cause web pages to break unexpectedly, but it’s how all major browsers I’ve tested behave, and it’s certainly the right thing to do. (We may want to experiment with displaying a warning, though, so that it’s clear what’s gone wrong.)
And if you’re a distributor, please read this mail to learn how not to break TLS verification in your distribution. I’m looking at you, Debian and derivatives.
Fewer TLS Authentication Failures
With glib-networking 2.42, corresponding to GNOME 3.14, Web will now accept certificate chains when the certificates are sent out of order. Sites that do this are basically broken, but all major browsers nevertheless support unordered chains. Sending certificates out of order is a harmless configuration mistake, not a security flaw, so the only harm in accepting unordered certificates is that this makes sites even less likely than before to notice their configuration mistake, harming TLS clients that don’t permute the certificates.
This change should greatly reduce the number of TLS verification failures you experience when using Web. Unfortunately, there are still too many differences in how certificate verification is performed for me to be comfortable with removing the Load Anyway button, but that is definitely the long-term goal.
WebKitGTK+ 2.6.1 plugs a serious HTTP authentication vulnerability. Previously, when a secure web page would require a password before the user could load the page, Web would not validate the page’s TLS certificate until after prompting the user for a password and sending it to the server.
Mixed Content Detection
If a secure (HTTPS) web page displays insecure content (usually an image or video) or executes an insecure script, Web now displays a warning icon instead of a lock icon. This means that the lock icon now indicates that your connection is completely private, with the exception that a passive adversary can always know the web site that you are visiting (but not which page you are visiting on the site). If the warning icon is displayed, then an adversary can compromise some (and possibly all) of the page, and has also learned something that might reveal which page of the site you are visiting, or the contents of the page.
If you’re curious where the insecure content is coming from and don’t mind leaving behind Web’s normally simple user interface, you can check using the web inspector:
The focus on safety will continue to drive the development of Web 3.16. Most major browsers, with the notable exception of Safari, take mixed content detection one step further by actively blocking some more dangerous forms of insecure content, such as scripts, on secure pages, and we certainly intend to do so as well. We’re also looking into support for strict transport security (HSTS), to ensure that your connection to HSTS-enabled sites is safe even if you tried to connect via HTTP instead of HTTPS. This is what you normally do when you type an address into the address bar. Many sites will redirect you from an HTTP URL to an HTTPS URL, but an attacker isn’t going to do this kindness for you. Since all HTTP pages are insecure, you get no security warning. This problem is thwarted by strict transport security. We’re currently hoping to have both mixed content blocking and strict transport security complete in time for 3.16.
Of course, security hasn’t been the only thing we’ve been working on.
- The most noticeable user experience change is not actually a change in Web at all, but in GNOME Document Viewer 3.14. The new Document Viewer browser plugin allows you to read PDFs in Web without having to download the file and open it in Document Viewer. (This is similar to the proprietary Adobe Reader browser plugin.) This is made possible by new support in WebKitGTK+ 2.6 for GTK+ 3 browser plugins.
- The refresh button has been moved from the address bar and is now next to the new tab button, where it’s always accessible. Previously, you would need to click to show the address bar before the refresh button would appear.
- The lock icon now opens a little popover to display the security status of the web page, instead of directly presenting the confusing certificate dialog. You can also now click the lock when the title of the page is displayed, without needing to switch to the address bar.
3.14 also contains some notable bugfixes that will improve your browsing experience.
- We fixed a race condition that caused the ad blocker to accidentally delete its list of filters, so ad block will no longer randomly stop working when enabled (it’s off by default). (We still need one additional fix in order to clean this up automatically if it’s already broken, but in the meantime you can reset your filters by deleting ~/.config/epiphany/adblock if you’re affected.)
- We (probably!) fixed a bug that caused pages to disappear from history after the browser was closed.
- We fixed a bug in Web’s aggressive removal of tracking parameters from URLs when the do not track preference is enabled (it’s off by default), which caused compatibility issues with some web sites.
- We fixed a bug that caused Web to sometimes not offer to remember passwords.
These issues have all been backported to our 3.12 branch, but were never released. We’ll need to consider making more frequent stable releases, to ensure that bugfixes reach users more quickly in the future.
- There are new context menu entries when right-clicking on an HTML video. Notably, this adds the ability to easily download a copy of the video for watching it offline.
- Better web app support. Recent changes in 3.14.1 make it much harder for a web app to escape application mode, and ensure that links to other sites open in the default browser when in application mode.
- Plus a host of smaller improvements: The subtitle of the header bar now changes at the same time as the title, and the URL in the address bar will now always match the current page when you switch to address bar mode. Opening multiple links in quick succession from an external application is now completely reliable (with WebKitGTK+ 2.6.1); previously, some pages would load twice or not at all. The search provider now exits one minute after you search for something in the GNOME Shell overview, rather than staying alive forever. The new history dialog that was added in 3.12 now allows you to sort history by title and URL, not just date. The image buttons in the new cookies, history, and passwords dialogs now have explanitory tooltips. Saving an image, video, or web page over top of an existing file now works properly (with Web 3.14.1). And of course there are also a few memory usage and crash fixes.
As always, the best place to send feedback is <email@example.com>, or Bugzilla if you’ve found a bug. Comments on this post work too. Happy browsing!
So, what new things happened to our games in GNOME 3.14?
GNOME Hitori has actually been around for a while, but it wasn’t until this cycle that I discovered it. After chatting with Philip Withnall, we agreed that with a minor redesign, the result would be appropriate for GNOME 3. And here it is:
The gameplay is similar to Sudoku, but much faster-paced. The goal is to paint squares such that the same digit appears in each row and column no more than once, without ever painting two horizontally- or vertically-adjacent squares and without ever creating a set of unpainted squares that is disconnected both horizontally and vertically from the rest of the unpainted squares. (This sounds a lot more complicated than it is: think about it for a bit and it’s really quite intuitive.) You can usually win each game in a minute or two, depending on the selected board size.
For Mines, the screenshots speak for themselves. The new design is by Allan Day, and was implemented by Robert Roth.
There is only one gameplay change: you can no longer get a hint to help you out of a tough spot at the cost of a small time penalty. You’ll have to actually guess which squares have mines now.
Right now, the buttons on the right disappear when the game is in progress. This may have been a mistake, which we’ll revisit in 3.16. You can comment in Bug #729250 if you want to join our healthy debate on whether or not to use colored numbers.
Sudoku has been rewritten in Vala with the typical GNOME emphasis on simplicity and ease of use. The design is again by Allan Day. Christopher Baines started work on the port for a Google Summer of Code project in 2012, and Parin Porecha completed the work this summer for his own Google Summer of Code project.
We’re also using a new Sudoku generator, QQwing, for puzzle generation. This allows us to avoid reimplementing bugs in our old Sudoku generator (which is documented to have generated at least one impossible puzzle, and sometimes did a very poor job of determining difficulty), and instead rely on a project completely focused on correct Sudoku generation. Stephen Ostermiller is the author of QQwing, and he worked with us to make sure QQwing met our needs by implementing symmetric puzzle generation and merging changes to make it a shared library. QQwing is fairly fast at generating puzzles, so we’ve dropped the store of pregenerated puzzles that Sudoku 3.12 used and now generate puzzles on the fly instead. This means a small (1-10 second) wait if you’re printing dozens of puzzles at once, but it ensures that you no longer get the same puzzle twice, as sometimes occurred in 3.12.
If you noticed from the screenshot, QQwing often uses more interesting symmetries than our old generator did. For the most part, I think this is exciting — symmetric puzzles are intended to be artistic — but I’m interested in comments from players on whether we should disable some of the symmetry options QQwing provides if they’re too flashy. We also need feedback on whether the difficulty levels are set appropriately; I have a suspicion that QQwing’s difficulty rating may not be as refined as our old one (when it was working properly), but I could be completely wrong: we really need player feedback to be sure.
A few features from Sudoku 3.12 did not survive the redesign, or changed significantly. Highlighter mode is now always active and uses a subtle gray instead of rainbow colors. I’m considering making it a preference in 3.16 and turning it off by default, since it’s primarily helpful for keyboard users and seems to get in the way when playing with a mouse. The old notes are now restricted to digits in the top row of the cell, and you set them by right-clicking in a square. (The Ctrl+digit shortcuts will still work.) This feels a lot better, but we need to figure out how to make notes more discoverable to users. Most notably, the Track Additions feature is completely gone, the victim of our desire to actually ship this update. If you used Track Additions and want it back, we’d really appreciate comments in Bug #731640. Implementation help would be even better. We’d also like to bring back the hint feature, which we removed because the hints in 3.12 were only useful when an easy move exists, and not very helpful in a tough position. Needless to say, we’re definitely open to feedback on all of these changes.
We received a Lights Off bug report that the seven-segment display at the bottom of the screen was difficult to read, and didn’t clearly indicate that it corresponded to the current level. With the magic of GtkHeaderBar, we were able to remove it. The result:
Robots was our final game (from the historical gnome-games package, so discounting Aisleriot) with a GNOME 2 menu bar. No longer:
It doesn’t look as slick as Mines or Sudoku, but it’s still a nice modernization.
I think that’s enough screenshots for one blog post, but I’ll also mention that Swell Foop has switched to using the dark theme (which blends in better with its background), Klotski grew a header bar (so now all of the historical gnome-games have a header bar as well), and Chess will now prompt the player at the first opportunity to claim a draw, allowing us to remove the confusing Claim Draw menu item and also the gear menu with it. (It’s been replaced by a Resign button.)
Easier Easy Modes
The computer player in Four-in-a-row used to be pratically impossible to defeat, even on level one. Nikhar Agrawal wrote a new artificial intelligence for this game as part of his Google Summer of Code project, so now it’s actually possible to win at connect four. And beginning with Iagno 3.14.1, the computer player is much easier to beat when set to level one (the default). Our games are supposed to be simple to play, and it’s not fun when the easiest difficulty level is overwhelming.
There have also been plenty of smaller improvements to other games. In particular, Arnaud Bonatti has fixed several Iagno and Sudoku bugs, and improved the window layouts for several of our games. He also wrote a new game that will appear in GNOME 3.16. But that has nothing to do with 3.14, so I can’t show you that just yet, now can I? For now, I will just say that it will prominently feature the Internet’s favorite animal.
Oh my, it’s a new GNOME blog. I’ll initially be using this space to show off some of the new features we have in GNOME 3.14. Stay tuned…