First of all, I should probably admit that, despite the title of this post, no, the redhat.corpmerchandise.com fiasco is not Mozilla’s fault: it’s Red Hat’s, because obviously Mozilla has no control over that domain. But that wouldn’t make for a very interesting title for a blog post, and Mozilla set the stage for this to happen, so let’s go with “Mozilla’s fault.” Also, it’s not really Red Hat’s fault; Staples is really to blame, since corpmerchandise.com is their domain, but I really shouldn’t be pointing that out when the point of this blog post is to blame Mozilla. And gosh now I’m off on a tangent, but it’s not really a fiasco either: it’s a significant screw-up, but not that big a deal; but words like “fiasco” make for good clickbait headlines, so let’s go with that. FIASCO.
One last note before I begin. I hold Mozilla to a higher standard than other software development companies. Sometimes it makes mistakes, like the one I’m about to present, and it’s important to call them out when this happens, but it’s because of good choices at Mozilla that Firefox still (mostly) respects your freedom, unlike other major browsers. It’s a good company.
OK, so you’ve read this far in suspense, I should probably explain the redhat.corpmerchandise.com fiasco before you reach the end of your three-paragraph Internet-length attention span. Yesterday the Fedora Store went live, where you can buy low-cost Fedora-branded items: a T-shirt, water bottle, pub glass, or baseball cap. I want a T-shirt. OK, that’s great, so what is the fiasco? Well click on this link (quick! before it gets fixed!) to find out: https://redhat.corpmerchandise.com/ProductList.aspx?did=20588
Now, depending on your browser, you may or may not have discovered the problem. When I load that site in Firefox, I see Fedora merchandise. When I load it in Epiphany, I see something noticeably less friendly:
“Legitimate banks, stores, and other public sites will not ask you to do this.” Ouch. (I actually took that language from Firefox when I designed that interstitial for Epiphany.) Ah, well, clearly there is some bug in Epiphany, because Firefox is a major browser and Firefox doesn’t get stuff like this wrong, right? Well, no, Epiphany is not wrong. Then Firefox is wrong? Well… from a certain point of view… (like mine)….
Firefox and Epiphany use different cryptography libraries to determine if the certificate is valid, and they sometimes differ in what certificates they will accept. Firefox uses NSS, a library maintained by Mozilla primarily for use by Firefox (it’s also used by Chrome on Linux), while Epiphany (indirectly) uses GnuTLS, originally a GNU project that is now de-facto maintained by Red Hat. So is NSS just better than GnuTLS at determining whether a certificate is valid? Actually, NSS really is more permissive than GnuTLS, and this does sometimes lead Firefox to approve of sites that Epiphany will not, but that’s not the case here. Let’s try a little experiment to see what’s happening. Firefox has a weird feature that feels like it was designed in the 90s for the era when computers had one user account apiece: it lets you create multiple profiles for bookmarks, history, and other settings. So let’s give this a whirl:
$ firefox -ProfileManager
Create a new profile, launch Firefox with it, then load https://redhat.corpmerchandise.com/ProductList.aspx?did=20588 to try this experiment again. Or just keep reading and trust me when I say that you’ll see this:
Oooh, that’s not good, now Firefox thinks we’re being attacked. We’re not. So what’s going on here? Why is Firefox so inconsistent?
First off, let’s get one thing straight: this site is totally and hopelessly broken. To see why, let’s use the super-handly tool gnutls-cli:
$ gnutls-cli redhat.corpmerchandise.com
Processed 182 CA certificate(s).
Resolving 'redhat.corpmerchandise.com'...
Connecting to '174.47.191.32:443'...
- Certificate type: X.509
- Got a certificate list of 1 certificates.
- Certificate[0] info:
- subject `C=US,ST=Kansas,L=Overland Park,O=STAPLES CONTRACT & COMMERCIAL\, INC.,OU=Information Techology,CN=*.corpmerchandise.com', issuer `C=US,O=DigiCert Inc,OU=www.digicert.com,CN=DigiCert SHA2 High Assurance Server CA', RSA key 2048 bits, signed using RSA-SHA256, activated `2014-11-12 00:00:00 UTC', expires `2015-12-09 12:00:00 UTC', SHA-1 fingerprint `50cfb26c680434d132dc64e80db54de51a5a07a6'
Public Key ID:
c273ca58bfdb2902ea30dbf5946c27178affd588
Public key's random art:
+--[ RSA 2048]----+
| |
| |
| |
| . |
| = S |
| .* O o o |
|o .+.X E o . |
| =.. =.+.+ . |
|..o oo+oo |
+-----------------+
- Status: The certificate is NOT trusted. The certificate issuer is unknown.
*** PKI verification of server certificate failed...
*** Fatal error: Error in the certificate.
*** Handshake has failed
GnuTLS error: Error in the certificate.
If you’re familiar with digital certificates, it’s pretty obvious what’s wrong here. When you connect securely to a web site, it sends a chain of certificates: the first certificate is owned by the web site, then it sends some number of additional certificates, usually one or two, that belong to certificate authorities. Each certificate is signed by the next certificate in the chain (not quite, but it’s almost true, so let’s go with that for this post), up until you get to the last one in the chain, which must be signed by a certificate in your browser’s (or operating system’s) root trust store. The certificates in your root trust store are super valuable, and if one were to be compromised by an attacker the devastation to the Web would be terrible, so certificate authorities must keep their roots safe at all costs, and they do this by almost never using them. Legitimate certificate authorities never sign web sites’ certificates with their root certificate; instead, they create a few other certificates, sign them with the root, and use only those to sign web sites’ certificates. So if you ever visit a site and it sends you only one certificate, you know that the site is broken for sure. And here we have a site that has sent only one certificate (there’s a Certificate[0] but no Certificate[1]), a classic case of server misconfiguration (aka fiasco).
So why did Firefox allow the site at first, even though it has no chain of trust, but not allow it with a fresh profile? Well, even though the site has presented no chain of trust, NSS goes far, far out of its way to find one. Whenever you visit a web site, NSS saves each intermediate certificate it sees, makes sure it’s signed by a trusted root, and caches it for future use. Then, whenever you visit a site that sends a broken chain of trust, NSS will effectively treat all those intermediates as roots, and use them to complete the chain of trust. This is completely safe, since it has already verified them. Those intermediate certs are saved in your Firefox profile, so by switching to a fresh profile they are no longer used, and you can’t access the broken site anymore.
If you were able to access redhat.corpmerchanise.com in Firefox, you can verify this for yourself: open Preferences -> Advanced -> Certificates -> View Certificates -> Authorities. Anything listed as Default Trust or System Trust is a root, and anything listed as Software Security Device is a cached intermediate cert. Don’t touch those root certs, but feel free to Delete or Distrust any Software Security Device — it will just be cached again the next time you visit a web site that uses it. Scroll down to DigiCert SHA2 High Assurance Server CA. That’s the cached intermediate cert that is allowing you to visit redhat.corpmerchandise.com — it’s not shipped with Firefox, and new Firefox users won’t have it. Delete it, restart the browser, then try reloading https://redhat.corpmerchandise.com. Oh no, it’s untrusted! Now visit https://stackoverflow.com, which sends a certificate signed by DigiCert SHA2 High Assurance Server CA, which will cause NSS to cache it once again. Now back to https://redhat.corpmerchandise.com, and Firefox knows it’s safe again. And that, folks, is how you screw up your web site so that it only works if you first visit Stack Overflow.
So why does NSS do this? Well, once upon a time (ten years ago), browsers were less strict about verifying chains of trust, and on an untrusted connection would let you proceed with maybe just a pop-up warning, and maybe not even that. So sites were less diligent about making sure they had valid chains of trust than they are today, in the era of nasty interstitial warnings that discourage the user from visiting the site. Since there were a lot of sites with broken chains, NSS chose to cache intermediate certificates to reduce the number of unnecessary validation errors for Firefox users. At the time, this might have been an OK choice.
Today, if your online store is missing a chain of trust, the browser makes clear in no uncertain terms that this site is not to be trusted, and sites lose visitors/customers, so they try pretty hard to get this right. (How many lost visitors depends on the browser — a large majority of Chrome and probably Epiphany users will click through the warnings, but a large majority of Firefox users will not, because Firefox’s UI for this is much scarier.) When setting up a new site, you check it in a couple of browsers to make sure it works properly, and you trust that if it works in Firefox on your machine, surely it will work in Firefox for everyone else, right? Well, no, it won’t. When setting up a secure web site, you must always test it with a fresh Firefox profile to make sure that you got the chain of trust correct. Of course, nobody knows to do this, which is how we wind up with broken sites like redhat.corpmerchandise.com.
I suspect this breakage would happen far less often if NSS did not cache intermediate certs, tricking site admins into thinking their sites are set up properly. Sure, cached certs don’t hurt the user who has them cached, but they’re bad for all other users of Firefox, as well as users of browsers that do no certificate caching. And there’s no good reason for this, because browsers don’t need to cache intermediate certificates in 2015, because almost all sites that redirect from HTTP to HTTPS get this right nowadays, and those that get it wrong are probably getting it wrong because they tested with a browser that had the right cached intermediate. Chicken and egg much? There’s only one way to fix this problem, and that brings me to my request: Mozilla, do the Web a favor and stop caching intermediate certificates.
P.S. Astute readers would note that there’s absolutely no point in deleting an intermediate certificate with the Firefox certificate manager, except to test things like this. It’s just going to come back the next time you see it.
Everything old is new again – Chrome is now pushing “browser profiles” as a major feature: https://support.google.com/chrome/answer/2364824?hl=en .
Profiles are actually really useful… as you’ve just demonstrated.
I’ve got several setup for various duties… and one that I always called “blank” (saves/accepts nothing) for testing.
Talk to some web developers around you. I’d be surprised if quite a few weren’t using them.
I don’t think the article at all justifies your topic or conclusion. It’s a complex area and no one entity is ‘to blame’. Firefox is a web browser for people to use to view web sites, it is not an SSL certificate verification tool; there are plenty of those.
The parties who clearly did something wrong here are:
Staples (by coming up with a bad configuration and not testing it properly)
Red Hat (by not verifying the Staples configuration properly when adopting this service)
I don’t think you presented a strong enough argument that Mozilla did anything at all *wrong*, per se; they made a questionable judgement call that it’s more important to help their users view sites that they are confident can be trusted than it is to act as the SSL Cops (which is something Mozilla does a very good job of in less prominent ways, btw, so it’s really a shame to be so eager to get the boot into them).
When you’ve got two parties who unambiguously did something one and one which maybe did something debatable, prominently saying the third party is “responsible” is not justified.
Well, you’re not wrong. To my credit, I did point out in the very first sentence that Mozilla is not responsible. And that they’re a good company, and that I hold them to a higher standard. I then said they “set the stage for this to happen” and they did. This isn’t just an issue with one particular site, it happens with other sites too, which neither Staples nor Red Hat can do anything about. E.g. my university ran a ludicrously important web site for several years that only worked if you first visited another university site. The tech support people noticed the problem only occurred for new students or on their regularly-wiped testing computers, and thought it was some sort of feature of the university network that was magically capable of automatically installing certificates on your operating system….
This problem only occurred because Mozilla went to extreme lengths to make it a problem; it’s not really reasonable, I think, to expect site admins to perform magic incantations with Firefox profiles (or even know about Firefox profiles) to make sure their site is deployed correctly. I don’t care too much about Staples, but I do about the larger issue, and a fix can only come from Mozilla, so they’re the ones who deserve a little pressure. (Or, well, another major browser vendor could change too — anyone know if Internet Explorer and Chrome on Windows cache intermediates too? — but there’s a faintly realistic chance of convincing Mozilla to change, so let’s focus on them.) Caching certificates is not just good for Firefox at the expense of other browsers, it’s good for certain Firefox users at the expense of other Firefox users, and that is not how a responsible browser vendor should operate.
I don’t agree that Mozilla doing a “very good” job of acting as SSL cops. It is probably doing better than any other player in this space, that is true, but that’s not a very high bar, and this was just one particular complaint I could make out of many. It is still supporting RC4, for instance (after an initial handshake attempt without it fails). The dithering over whether to drop insecure initial negotiation is quite disheartening; they’re probably going to adopt a whitelist of broken sites and treat them differently, not to mention completely undo this when TLS 1.3 is implemented, which discourages other browsers from improving. (This in particular is a wonderful mistake to talk about — a very small minority of servers just crap out when a browsers says it supports a higher TLS version than the server knows about. At the time, tolerance of these servers at the expense of security really was important to keep the Web working, but nowadays it just makes us all less secure. These servers will mostly all be fixed once a major browser vendor puts a foot down, but nobody wants to be first.) Mozilla accepts certificate chains sent completely out of order, when that is one of the only actual standardized constraints that browsers are not supposed to permit. Their laudable effort to stamp out 1024-bit RSA roots was done with no consideration of ca-certificates users other than NSS; we’re walking into a future where Epiphany will only work properly on Fedora since the NSS guys reverted their changes downstream. And more and more.
Mozilla is not just more conservative about breaking broken sites than it should be, it is dramatically more conservative than I find reasonable, to the detriment not only of the security of Firefox users, but also of users of other browsers.
But still, to be very clear, these criticisms only apply to their terrible (you can say “debatable”) choices regarding HTTPS. Mozilla is a good company. They try to do good things. They mostly succeed. Firefox is not just a good browser, it’s the only major cross-platform browser worth using, because it respects your freedom. Yay for Firefox.
One thing everyone should do when deploying a web site with SSL is to check it with a service like https://www.ssllabs.com/ssltest/ – there’s a few others too.
It’ll tell you about misconfiguration, software bugs, and (of course) missing intermediate certificates, without having to futz with browser profiles and stuff.
SSL Labs is great. They have a good server test and a good (lesser-known) client test that I use regularly. Ivan’s book “Bulletproof SSL and TLS” is excellent.
But beware that the letter grades for servers are not very good. For instance, with redhat.corpmerchandise.com:
“This server is vulnerable to the POODLE attack against TLS servers. Patching required. Grade set to F.” OK that’s bad and that grade makes sense.
“This server is vulnerable to the POODLE attack. If possible, disable SSL 3 to mitigate. Grade capped to C. ” Fair enough, though I don’t think this is a big deal since browsers don’t allow SSLv3 anymore.
“This server’s certificate chain is incomplete. Grade capped to B.” Erm, this should be an F. The chain issues under Additional Certificates and Certification Path should show up in red, not orange.
TLS 1.3 intolerance should get it capped to a D (or thereabouts) but does not affect its score at all.
https://www.ssllabs.com/ssltest/analyze.html?d=redhat.corpmerchandise.com
It is actually much worse than just a missing certificate in the chain :(
Yeah – no. Firefox is doing the correct thing here, GNU TLS is broken… The client end needs to store intermediate certificates for cross-signing purposes.
TLS only allows for a single chain to be transmitted, which may or may not be the one that the client is expecting. In a B2B situation, there is no guarantee that server and client will agree on certificate chain.
So, e.g. if I need my web-browser needs to connect to another company’s web-server, their root certificate will be cross-signed by mine. The chain and intermediate sent by the other company’s web-server will not be the correct one for me to use.
I’m not sure if you have a point or not, because I don’t understand what you’re saying. Why is caching related to cross-signing? Why would you ever want to transmit a chain if you don’t intend the client to use it? In particular, what do you mean when you say “the server and client will agree on a certificate chain” — there is no negotiation involved here, the server sends a chain and it’s either good enough for the client or it’s not.
If Company A’s root cert is signed by Company B, and Company B’s signing cert is in the root store of Company B’s computers, then the connection will work. If not, then it should not work: and why would you expect it to work? and why would you send more than one cert in this case?
I like what Mozilla is doing here. Their purpose is to serve the user. The important issue for the end user is to verify the identity of the website. Mozilla’s working over-time to do that, so that the user can carry on with his web-browsing, and purchasing.
The proposal seems to be that Mozilla should serve the server admin instead of serving the end user, if the server admin is too lazy to visit https://www.ssllabs.com/ssltest/analyze.html?d=redhat.corpmerchandise.com (which, apparently, he is: still vulnerable to POODLE). I disagree!
I also agree that the caching that Firefox does is good. Internet Explorer/Microsoft does even more and either ships all/most intermediate certificates or uses the “Authority Information Access” extension from the server certificate to get missing intermediate certificates. So there also websites that were tested with IE and worked, but do not always work in Firefox. Here the answer should not be to make sure that these websites never work in Firefox.
Also caching allows to speed up connecting to websites, if the browsers remembers that the intermediate certificat is valid, it does not always need to check the complete chain.