Heartbleed Update v3

https://blogs.akamai.com/2014/04/heartbleed-update-v3.html

by mycodebreaks • 12 years ago

262 46 12 years ago

blogs.akamai.com

zoowar• 12 years ago

I don't know akamai's relationship with open source projects, but it sounds like they had local changes and tried to karma whore them as, "trust us, our customers were protected, and we contribute to open source." I hope I'm wrong, or they deserve the public shaming they are receiving. Ultimately, we need more engagement from commercial organizations benefiting from open source projects.

Artemis2• 12 years ago

Good politics: they had custom code for parts of OpenSSL, they published it when it seemed needed, open-source community told them what was wrong, they fixed it and apologized. No bullshit.

glimcat• 12 years ago

Also: the way this played out makes an awesome story for the next time Akamai experiences internal pushback against participating in open source. "Remember that one time it saved our bacon" is a political goldmine.

notatoad• 12 years ago

Them and everybody else. "Remember that time akamai ran vulnerable code in production for 13 years and the bug got patched two days after they open-sourced it" should be able to drive open source contributions at all kinds of companies.

sobkas2• 12 years ago

They are only in problem now, because they have released the patch. If no one outside the company saw that patch, they could spot the problem and fix it without anyone noticing and creating a PR disaster. They probably wouldn't, but they could. That is one argument someone could use against "participating in open source".

It's a bit to late for saving bacon, patch should be publicly reviewed before it went into production. So no bacon saved, but there is a hope for a smoke detector.

madaxe_again• 12 years ago

Nope. People will remember it as "remember all that work as a result of OpenSSL?"

roel_v• 12 years ago

Look, I like open source as much as the next guy, but this is myopic - one might as well say 'remember that one time we used open source stuff and we had to spend weeks scrambling to fix things?'. Please let's not fool ourselves into believing things that simply aren't true - it detracts from the things we can leverage.

sjtgraham• 12 years ago

I'm so grateful we have people like Willem Pinckaers uncovering these things.

hrrsn• 12 years ago

Good response from Akamai. No beating around the bush there.

mycodebreaks• 12 years ago

Yes, they've been clear and honest in their communication.

daeken3• 12 years ago

No, this was not a good response. It was well written and took responsibility for bad code, but it failed to address the elephant in the room: why were keys not immediately rotated?

I've said this a number of times lately, but it bears repeating that the golden rule of security is: If a compromise occurs, assume that everything is compromised. Even if their approach was tested to be perfect, they shouldn't have made the assumption that that actually reflected reality.

They were extraordinarily irresponsible. There's simply no other way to slice it.

blahedo• 12 years ago

No, that goes too far the other way. If I have legitimate reason to believe that an exploit does not affect me (especially if it's because I acknowledged its possibility and programmed defensively against it), why would I automatically act as if I were breached too?

Your argument is totally correct if I am simply unsure---if I don't know whether an exploit affects me but I suspect it might, I assume the worst and act accordingly. But in this case Akamai had good reason to believe they were unaffected. (And, when someone demonstrated otherwise, they responded responsibly.)

Narkov2• 12 years ago

They believed that they were covered and that no compromise could/did occur. It's a mistake they now acknowledge but they have owned up to it and are taking action - just as you suggest.

I believe their actions are honest and completely reasonable. If they had known they still had an issue I seriously doubt they would have publicised the patch and risk the issue they now have.

sobkas2• 12 years ago

I would like to know how many security related patches they are keeping for themselves? And I think not only Akamai did this. I hope that this situation will prove that security needs a bit more patch reviews and a bit less of secret sauce.

sweettea• 12 years ago

I suspect their patches are oriented toward paranoia. Their particular patch (keeping what they believed to be all critical values in a special region of memory used only for critical things) had no obvious necessity -- it didn't obviously provide more security -- but was just paranoia. Paranoia isn't always necessary, or even desirable; hence, the remainder of the patches have no obvious external utility, so they presumably haven't released them. <p> They may also have performance improvements, conceivably, and those have a more obvious rationale for keeping secret.

kevinr2• 12 years ago

As Andy tweeted, we built that particular patch to keep unencrypted secrets off disk -- it actually was intended to provide tangible security benefit, it wasn't just paranoia: https://twitter.com/csoandy/status/455307255895060480

dsl• 12 years ago

Right before this all came out our Akamai rep came on site and swore up and down they were not vulnerable. They need to quickly communicate internally as well that a vulnerability existed and send account managers back out to work on high priority cert rollovers.

anonymouse01• 12 years ago

Our rep said the same thing even when we insisted they rotate our certs on day 1.

Of course we're still waiting. I don't know who's decision it was to not reissue all certs on day 1, but this is an epically bad decision bordering on gross negligence.

Heads should roll.

Yver• 12 years ago

The part I still don't get because I haven't seen it discussed so far is that their original patch mentionned they had been using a variant of the patch "for a decade" but I never got why they didn't seek to get it merged upstream a decade ago. Or why they released "a variant" of their patch instead of the real thing to begin with.

chubot• 12 years ago

What's interesting is that I just watched a video that strongly relates to this problem last night (from DJB at CCC):

https://www.youtube.com/watch?v=IuSnY_O8DqQ

http://facthacks.cr.yp.to/

It is only part of the video, but they talk about searching Google and finding partial private keys, e.g. ---- BEGIN PRIVATE KEY ---.

Even with text missing, it is trivial to recover the entire key, because of redundancy that Akamai neglected to take into account. I never really thought about how those ASCII strings are encoded, but it's actually the 6 or so values encoded. If some are missing, you can recover the others.

In retrospect, why shouldn't the private keyh have RFC 822-like key value format so this is obvious? I don't see why it has to be encoded at all; the format is unnecessarily obscure.

The rest of this talk is also great. Most of it is accessible to non-cryptographers (like me), as long as you understand basic arithmetic and have an undergrad-level overview of public key crypto (which they give). They provide very readable and runnable Sage scripts.

I really like the batch GCD attacks. Very elegant and effective.

jonhohle• 12 years ago

Great talk, thanks for the link. I've used DJB's software for years, but this was the first talk I've seen.

beambot• 12 years ago

Willem's letter to Akamai addressed three areas where their patch was likely insufficient:

(1) First question is then, are 'p', 'q' and 'd' the only sensitive parameters? NO. (The Chinese Remainder Theorem parameters are included in OpenSSL private keys by default and they are enough to recover the private key).

(2) Second question: Does OpenSSL ever copy the private key, or parts, after the key has been read? YES.

(3) Third question: Does OpenSSL allocate temporary variables for intermediates when performing operations with a private key. Can they be used to recover the private key? YES and YES.

(Willem's post submitted to HN: https://news.ycombinator.com/item?id=7582990 )

This update from Akamai only addresses (1), AFAICT.

mynameisvlad• 12 years ago

Why would their update need to address more than just (1)? That's the only part of the problem that they have to deal with. The OpenSSL team should be the one answering (2) and (3), since it's OpenSSL that copies the PK/parts after it's been read, or allocate temp vars for intermediates. If Akamai chooses to patch those two problems and then releases it to open source like their initial patch, then they can answer them, but otherwise, what's the point?

Danieru• 12 years ago

(2) & (3) are not bugs for OpenSSL because only Akamai is trying to segregate the key data.

In upstream OpenSSL the intermediates are stored in the same memory pool as the key data. Thus the intermediates do not expose privileged data anywhere it is not already present.

Without addressing issues 2 & 3 Akamai's patch is not a line of defense. It does not guard against dirty memory exploits like Heartbleed, it only increases the difficulty of exploitation.

schneidmaster• 12 years ago

They also said

In parallel, we are evaluating the other claims made by the researcher, to understand what actions we can take to improve our customer protection.

which seems to imply that they're at least looking into the other questions.

csoandy• 12 years ago

We are, in fact, evaluating those other claims, and testing them in our labs. We are also evaluating claims made by other researchers, both publicly and privately; and we hope to end up with both a better library out of this, as well as a better understanding of the hazards we do and don't provide safeties against.

Validating his first claim was sufficient to undermine our belief in key safety; the others are therefore only relevant for protection against future attacks. In that light, it was important to advise our customers and the community at large.

sitkack• 12 years ago

I think it is important to remember when validating flaws, that we show that attacks are possible not they are necessarily demonstrated. Just because one cannot construct an exploit doesn't mean that something is secure.

Sami_Lehtinen• 12 years ago

Why people are only talking about SSL keys? Because Heartbleed allowed to fetch any data, that could have also included SSH keys and or any other authentication credentials / information the server got, for itself or to other services. As example if your site got Facebook API keys it's possible that those have been leaking too.

jbg_• 12 years ago

Unless your web server makes SSH connections from some kind of in-process application, why would this bug allow SSH keys to leak?

jevinskie• 12 years ago

Is memory wiped with zeros by malloc/free in libc or in the kernel? If not, you're at risk.

tptacek• 12 years ago

No, your web server process memory does not include SSH keys.

forrestthewoods• 12 years ago

This does not inspire confidence in akamai for me. These days people on the internet seem to get super excited when various companies write post mortems on how they screwed up. Yay, they screwed up and were honest about it!!! I find those to be wildly overrated. I'd rather there no be a screw-up to begin with. And I don't particularly care if you come clean or not.

I know bugs are written and bugs happen. It's an unfortunate fact of life. And anyone who posts their code will likely have holes poked in it by the community. But the akamai thing really sets of warning flags. The fact they thought it would work and it doesn't, at all, does not sit right.

wdr1• 12 years ago

Interesting to note that if Akamai hadn't open sourced their modifications, they never would have realized their certs actually had been compromised.

joeybaker• 12 years ago

This is the perfect example to point people to when they ask why the company shouldn't just maintain an internal fork or write its own version of an open source project.

Actually contributing the code you write is helps _everyone_.

Heartbleed Update v3

Related Stories