Watching #heartbleed (aka CVE-2014-0160) fly by in my twitter stream this week, I keep wishing we could all just pause time for a couple of weeks and properly reflect on all the angles here.
Some of the things I’d love to have more time to dig into:
- Obviously, the bug itself and its fix
- Why countermeasures in the kernel, libc or the compiler didn’t prevent this, or how static analysis tools didn’t catch it. (Update: see here and here)
- The protocol design – some wonder why a seemingly application-level feature is included in such a critical security protocol and why the feature is available before authentication has completed. Interesting to note that one of the stated purposes of the feature is PMTU discovery for DTLS yet it’s enabled for plain TLS too
- That it’s not only servers which are vulnerable – clients equally vulnerable
- The fact that attacks can’t be detected because there is no logging of this failure condition
- The health of the OpenSSL project, talk of its developers being irresponsible when it comes to their development practice
- How the bug was introduced, how much code review or testing was involved. (Update: comments from the original author of the code)
- The realization that few in our industry are actively investing in this project which so many place so much trust in
- The question of whether OpenSSL is the library everyone should be relying on, whether it should be e.g. NSS or GnuTLS
- How something like OpenSSL can be a critical part of a web service’s architecture even where e.g. the service is implemented with technologies like .NET
- The question of whether we’re putting all our eggs in one basket if an SSL vulnerability can have such far reaching implications
- Just how much was exposed by the vulnerability – e.g. private keys, usernames and passwords, other sensitive data – and how much data could have been compromised by a determined, well-resourced attacker in possession of this knowledge before it was patched
- The efficacy of certificate revocation measures in protecting users from potentially compromised certificates
- How Perfect Forward Security is an important mitigation measure for vulnerabilities like this
- The claim by one site that they could detect attacks in their logs, that there was someone exploiting the vulnerability two weeks before it was disclosed publicly. (Thierry Carrez points out the update to this post which says “other tools [..] can produce the same pattern in the SeaCat server log”)
- How our perception of a vulnerability like this has changed now that we know just how aggressive intelligence agencies (the NSA and others) are in their approach to population surveillance and monitoring. This FOSDEM talk is rather prescient in describing OpenSSL as the “crown jewels” for the NSA. (Update: claims that the NSA knew of the bug soon after its introduction and exploited it)
- The approach that was taken to disclosing the vulnerability, how responsible a disclosure process it was and the pros/cons of delaying disclosure further. Apparently CloudFlare was disclosed before the public disclosure, whereas vendors like Red Hat weren’t. (Update: more details on the disclosure timeline from Kurt Seifried) (Another update: OpenSSL’s official view of the timeline) (Yet another update: interview with Codenomicon’s David Chartier)
- How effective entities like Linux distros were in helping getting the vulnerability quickly patched in the wild (see e.g. that Stripe went and built their own patched package rather than wait for an Ubuntu package)
- How much more difficult patching this was due to applications bundling OpenSSL and how much worse it could have been if the practice of bundling was more widespread
- The impact that the impressive marketing – i.e. a cool name vs simply a CVE number, a logo, clear technical writing which speaks to the business impact, etc. – had on the speed with which the patch was deployed
> * The health of the OpenSSL project, talk of its developers being irresponsible when it
> comes to their development practice
> * The realization that few in our industry are actively investing in this project which so
> many place so much trust in
> * The question of whether OpenSSL is the library everyone should be relying on,
> whether it should be e.g. NSS or GnuTLS
> * The question of whether we’re putting all our eggs in one basket if an SSL
> vulnerability can have such far reaching implications
I feel these 4 are all different facets of the same point. The last one in particular though poses quite a difficult conundrum. Conventional wisdom is that having all your eggs in one basket is a bad thing because of the implications when something goes wrong. With cryptography though, I’m not sure that conventional wisdom works out well in practice. It is pretty clear that writing secure crypto libraries is a very hard job, for which the community/industry has a pretty limited pool of skilled & interested developers. If we were to spread our eggs by having 10 widely used crypto libraries instead of just the 3 main open sources impls (OpenSSL, NSS, GNUTLS) we’d be spreading that limited pool of dev resources even thinner. The end result would likely be that the libraries would be lower quality overall, thus actually increasing our overall security risks. People auditing crypto libraries would also have much more work todo if there were more libraries to audit. Application developers would have harder decisions to make – do they pick just one, or do they try to support a whole bunch of different libraries which increases the chance they’ll screw up their integration causing more security flaws. Is 3 the right number of eggs to have in our basket – IMHO it probably is for now, given the other problems inherent in diversifying further.
I think the second point is the key one to think about here. IMHO too many companies are shipping & relying on our three main open source TLS libraries without contributing enough resources to their development & testing – its a collective failure of investment since it isn’t a “sexy” headline grabbing project to throw resources at like, say, OpenStack. Count how many developers companies have thrown at OpenStack in the past 2 years and then count the same for the GNUTLS/OpenSSL/NSS projects…
Serious as this, and the other 2 TLS impl vulnerabilities this year, are though, we have to bear in mind that (the NSA aside) TLS/SSL has actually done a remarkable job at protecting internet traffic over the past 10-15 years. It seems that the internet is going through a period of attitude readjustment due to the Snowden revelations, with greater interest/activity in auditing security sensitive code, including our TLS impls, leading to a significant uptick in vulnerabilities identified. Hopefully this will lead to better development and testing practices, and a greater long term investment in security work from everyone involved & relying on them.
Here’s a cool post about using systemd cgroup to help fixing this problem in the wild https://www.getpantheon.com/heartbleed-fix
> Why countermeasures in the kernel, libc or the compiler didn’t prevent this, or how static analysis tools didn’t catch it
Because OpenSSL has exploit mitigation countermeasures to make sure it’s exploitable.
Yes, is not a joke:
http://article.gmane.org/gmane.os.openbsd.misc/211963
The seacat logs are probably a false positive but there are older smoking guns it seems https://www.eff.org/deeplinks/2014/04/wild-heart-were-intelligence-agencies-using-heartbleed-november-2013
Yeah, thanks clopez – I’d already linked to that post in the “talk of its developers being irresponsible” point
Re static analysis, I see that openssl is registered here and the last run was a year ago, though results aren’t publicly available: https://scan.coverity.com/projects/294?tab=Overview
Though I’d be surprised if static checking pointed any issue here, and checking a coverity scan I do have for d1_both.c doesn’t show anything.
I guess the security agencies would have more focused tools. clients searching for server stack/heap tagged values would be a cracking 101 tool I expect.
This is a great list, Mark. I’d also add:
* The prevalence of openssl in embedded devices that aren’t going to get updated, but will appear secure from here on out.
Probably you are also interested in the OpenSSL developers comments on the exploit mitigation countermeasures issue raised by OpenBSD developers: http://thread.gmane.org/gmane.comp.encryption.openssl.devel/24208
With my team, at TrustInSoft, we have just performed a mathematical validation of an open source SSL stack. This SSL stack is now immune to security flaws similar to heartbleed. I encourage you to have a look at formal methods and see how they can help to build a more reliable internet: http://trust-in-soft.com/no-more-heartbleed
Following on the static analysis topic I see that coverity have written about the “taint analyis” required here: http://blog.regehr.org/archives/1128