Willem Pinckaers on Akamai's flawed OpenSSL allocator patch

by tptacek • 12 years ago

248 71 12 years ago

tptacek7• 12 years ago

(I worked with Willem at Matasano for a couple years, and he knows what he's talking about.)

The tone of his post is rather unfortunate. Still, he raises good points, even if they are put in an unnecessarily aggressive manner.

gojomo• 12 years ago

How would a less-aggressive tone be better at delivering the necessary info to Akamai and its customers?

davidmat• 12 years ago

I don't find it aggressive, it seems more like Dutch directness to me. As an inhabitant of a neighbouring country, I must say it's an acquired taste ;)

tptacek7• 12 years ago

I disagree. The post is written bluntly, but it argues with factual assertions, not with emotional appeals. I don't think criticism of its tone is warranted or really all that appropriate.

rocky_ee2• 12 years ago

Pls elaborate

gus_massa• 12 years ago

Tptacek has a software security company, and he is one of the unofficially designated security experts here (with cperciva). He is the #1 in the karma ranking and has even more karma that the site founder (pg)! If he says that “[Willem Pinckaers] knows what he's talking about”, for me it’s very strong reference for Willem Pinckaers.

Also, we generally don’t like oneliners here, try writing a longer comment. Definitively avoid Pls!!!! (And just in case, also try to avoid unoriginal jokes and memes.)

You are quite new here and your last 2 comments were heavily downvoted. I hope that someone upvote them until they are dark gray again. (The second comment, were you tried to explain your intentions, is not a bad comment.) [You’ve just deleted your second comment.]

Alternative redaction:

“I searched for Willem Pinckaers in Google and there is a lot of information about him. Can you recommend a few pages with his main projects to understand his background and credentials. Also, I’d love to know more about the projects in which you worked together in Matasano.”

[Note: perhaps the internal projects are under a NDA/top-secret/paranoid-client, and they can’t be explained until 2050.]

cryptbe• 12 years ago

Ha. This is probably something that we mountain view folks would enjoy working on :-).

I've updated my blog on how to recover the private key from the CRT parameters stored in the private key: http://vnhacker.blogspot.com/2014/04/idea-to-solve-cloudflar.... Trivial math, but still interesting to see how it actually works.

__alexs• 12 years ago

OpenSSL has pretty good support for HSMs in the form of their ENGINE API. It seems like it would be possible to use this layer to move all key handling and crypto operations out of the process that was dealing with TLS. Process isolation seems like a much better way of getting this kind of security than weird allocator tricks.

keithgabryelski• 12 years ago

gosh, i see this "weird allocator trick" to be exceptionally insightful as to the possible attacks that and likely vulnerabilities that may be introduced into code.

One of the biggest issues for a development team isn't how to do something, but how to do something so that someone (who has taken over the project) won't screw it up in 10 years when you and all your cohorts have left and are island hoping and coconut drink doing.

yukichan4• 12 years ago

I'd like to see Akamai create a CloudFlare style challenge using their patch and a compromised version of OpenSSL and see if anyone can get the private keys from it.

abus2• 12 years ago

It would be better if someone independent set it up.

brians7• 12 years ago

We're talking about it. The patch is dependent on details of our unreleasable (GPL+OpenSSL+all sorts of other things) server.

ironghost• 12 years ago

The funny thing is the fact that everybody is ignoring that the patch wasn't a patch. It was a POC (read demonstration) with the notes:

'...This patch is a variant of what we've been using to help protect customer keys for a decade.

This should really be considered more of a proof of concept than something that you want to put directly into production. It slides into the ASN1 code rather than adding a new API (OPENSSL_secure_allocate et al), the overall code isn't portable, and so on. If there is community interest, we would be happy to help work on addressing those issues. Let me restate that: do not just take this patch and put it into production without careful review.'

nolok3• 12 years ago

Oh but if you do that how are you going to make a pretentious critic to shoot it down ? The akamai guy straight up said "that's not actually our code, and that should not and cannot be used as-is, this is a POC", rendering this entire "answer" irrelevant especially given its mocking tone.

hueving2• 12 years ago

Isn't a POC supposed to prove a concept? i.e. shouldn't a POC defend against the attack it is stated to defend against?

fancy_betta• 12 years ago

I'm guessing the patch was broken because they didn't have an actual patch laying around, they had to do a diff against the upstream and try to pick out the relevant parts. That doesn't excuse some of the other errors though, like integer overflows and not checking return codes. That also doesn't excuse the fact that they didn't actually try the patch before sending it out.

There could be other pieces that we're missing, and this patch alone doesn't prove that you can obtain private keys from Akamai's servers with the Heartbleed bug. It just proves that there could be key parts outside of their protected storage area and they suck at creating patches of their modifications. That being said, I'd love to see someone apply Akamai's OpenSSL patch and still pull the key with Heartbleed.

kalkin• 12 years ago

Yes. It's not really fair of the OP to say "this is broken in totally obvious ways and would clearly not actually run" and yet critique it as if it was production code. It's fairly obviously (at this point) a hand-constructed pseudo-diff.

delsarto3• 12 years ago

A review by a security engineer would have prevented a false sense of security

Everyone involved in this, from the people who wrote the heartbeat code, the people who committed the code, the people at Akami who wrote this patch, this poster ... all would describe themselves as security engineers. Is everyone but Willem Pinckars incompetent?

I think the one lesson to learn is to treat crypto just like you do cloud providers. Make sure you have an exit strategy from day one; the ability to push a button to roll keys, switch algorithms, switch cert providers, etc.

nolok3• 12 years ago

Apparently, he is not competent enough to understand what "this is not our actual code but merely a POC" means.

hueving2• 12 years ago

It's not exactly a proof of concept if it doesn't protect the keys, is it?

It strongly hints that even if this is 'only a POC' that their actual implementation is still vulnerable since their POC failed to protect against the very attack is was written to protect against.

abus2• 12 years ago

Here's a patch that prevents any exploit ever occurring in OpenSSL:

    void *custom_malloc() {  };

It's only a POC though so you'll have to adapt it. You can go ahead and begin praising me and flaming my critics.

tptacek7• 12 years ago

This would have been a cutting bit of wit indeed, had Akamai not stepped on your moment by confirming Willem's central thesis. Ouch.

yukichan4• 12 years ago

I don't think I've worked anywhere where there was a simple way to switch cloud providers with the push of a button. Even migrating to different locations within a cloud provider isn't trivial. You'd have to invest a lot of engineering time from the beginning to get this kind of capability, and you may never realize any value for all the pain.

delsarto3• 12 years ago

Yep; not easy at all ... if it were easy everyone would do it :) I think with tools like puppet and orchestration tools like heat, etc it is getting closer to being easier.

I did got to a Netflix talk once where they said that while moving from AWS would huge, they had a plan and I think from memory they said it would take about a week.

nraynaud2• 12 years ago

in my team we had the scripts ready for a second provider (but not the hot backups, I couldn't get that far).

tptacek7• 12 years ago

Which of the assertions in the post do you disagree with?

delsarto3• 12 years ago

None, it's good work.

But the whole episode, and this post in particular, highlights the issue that enough review is never enough. When is the the code secure? When it is written by someone with a good track record? When it gets reviewed and committed by knowledgeable parties? When it's been running for several years without incident? When it passes coverity? When Willem has time to review it?

When I just think about the embedded ssh keys I've got in my few toy systems; if I discovered ssh was broken (and it's happened; look at the Debian bug) I'd be sinking a lot of time figuring out just how to change the keys everywhere. I should have designed for this from the start; lesson learnt. I bet there's lots of admins out there who wish they had better ways to update certs on a moments notice over the past week...

jamesaguilar• 12 years ago

I think it's reasonable to disagree with the part he quoted. "A review by a security engineer ..." Well, it was reviewed by security engineers, and the false sense of security remained. The rest seems fine though.

makomk2• 12 years ago

There's been a lot of overly-confident statements about Heartbleed not being a big deal from apparently competent people these last few days. I'm not sure what it is about it that causes security professionals to spout off without testing their assumptions, but...

daeken2• 12 years ago

When in doubt, always asssume a compromise of anything is a compromise of everything. Golden rule.

yeukhon• 12 years ago

Even if the PoC patch was perfect and is approved by many of the top security folks out there, would Akamai ever share their perfect custom memory allocator with the upstream community.

edit:

okay, whats' wrong with my statement? If they do have a perfect allocator, wouldn't it be nice to know exactly what they did?

rgbrenner• 12 years ago

If this code was running on the Akamai networks for years, why did they have a copy laying around that didn't call mprotect?

Given the above problems I wonder how Akamai manages to run this in production.

How could you misread a two paragraph email so badly. Here, let me shorten it up for you:

  This patch is a _variant_ of what we've been using to help 
  protect customer keys for a decade.

  This should really be considered more of a proof of 
  concept than something that you want to put directly into 
  production. Let me restate that: do not just take this 
  patch and put it into production without careful review.

So to answer your question: this version was never running in production.. and it the initial patch didn't call mprotect because they stripped it out from their version when creating the POC.

mst• 12 years ago

I would be more sympathetic if they'd said "do not take this patch and put it into production, it's almost certainly incomplete and buggy". "without careful review" suggests that they actually expected some possibility of it being complete and correct, at which point having somebody verify that currently it is neither is still really rather useful.

It's more an 'exemplar of concept' or something, really - which is still valuable, but I do wish they'd made that more clear.

tptacek7• 12 years ago

This would be a more compelling rebuttal to Willem's post if Akamai hadn't confirmed his central thesis:

https://blogs.akamai.com/2014/04/heartbleed-update-v3.html

daeken2• 12 years ago

As a result, we have begin the process of rotating all customer SSL keys/certificates.

AGHHHHHHH! Even if their code was likely to have worked perfectly, this is a huge mistake. And I mean huge. They should've operated under the assumption that their defense didn't work and immediately rotated all keys. Period.

iiiears• 12 years ago

They used their patch for some length of time, release it to the community and a few hours later a bug is spotted and fixed. Peer review is a good thing.

cscott2• 12 years ago

It comes down to intent. There are two distinct ways to evaluate Akamai's patch:

1. Did Akamai release the PoC patch to start a discussion about how to protect private keys and share their work as a starting point for changing the code? If so, their efforts here should be considered in that vein and any criticism should be used simply to guide the development of a usable and functional patch.

2. On the other hand, if they are supplying this code as assurance that customer keys were properly protected against exposure by the Heartbleed bug and that certificate replacement is not needed, the patch and the subsequent criticism should give Akamai customers pause.

edit: Akamai acknowledges the bug, and has started rotating all customer SSL keys/certificates, per https://blogs.akamai.com/2014/04/heartbleed-update-v3.html

jackalope• 12 years ago

It's a smokescreen. If you were running a version of OpenSSL that supported the HEARTBEAT extension (patched or not) it's easy enough to determine whether or not it was vulnerable, simply by running the exploit code. Any theoretical discussion of how your patch made it invulnerable is pure speculation. They can release packet captures that prove they were invulnerable without releasing any code at all. At this stage, it's all PR.

paulhodge• 12 years ago

Even if their intent was just to start a discussion, they are not being helpful by "submitting" such bogus code. It would be easier for an expert to rewrite this patch from scratch than try to figure out all of their flaws and unfinished sections.

In one of the email responses, someone pointed out a problem with their code and they responded, "Oops we posted the wrong version" ( http://article.gmane.org/gmane.comp.encryption.openssl.user/... ). They aren't exactly being respectful of other people's time with stuff like that.

In secure coding the phrase "the devil is in the details" applies very strongly. So if you don't have the details figured out, you don't have much of anything.

awalton• 12 years ago

Even if their intent was just to start a discussion, they are not being helpful by "submitting" such bogus code.

All code is bogus code until reviewed. That is absolutely central to understand. Linus's Law only works _if_ people are looking at the code. Implicitly thumbs-upping it doesn't solve problems.

Akamai submitted the code, people reviewed it and found flaws. They're taking action to fix their own code, and the community is coming up with various fixes of their own. That's how Open Source Software Development should work.

While I agree with you that it'd probably be better to rewrite the code with a similar approach, it's also important to note that nobody in the OpenSSL community even considered this approach publicly until Akamai published their code.

Any claims they're being disrespectful of people's time is specious - they said from the beginning that this code needed review and shouldn't be merged. This is just one of those issues that comes out in the wash of code review.

TL;DR: Akamai should be lauded on their intentions but like noted by everyone, the code wasn't good enough. Now, with proper review and rewrites, they will be able to protect their customers into the future, and maybe OpenSSL will become a slightly better product for it too.

brians7• 12 years ago

It was released as both: we thought that our old protection against swap protected us against Heartbleed. What a stroke of luck. We did check for key values visible in the heap, and on our implementation, in our lab, didn't find them on any version of our software used since we took OpenSSL 1.0.1.

We weren't looking for the CRT values.

As one part of our response, we decided to publish the code we thought was keeping us safe. If we were right, sure, there's good PR from that. If we were wrong, it's a chance to find out and get right. Less wrong, anyway.

cscott2• 12 years ago

Decisions were made about certificate revocation based on assumptions about this code and Akamai customers ended up being exposed.

Perhaps third-party security validation of such a critical piece of code should be a prerequisite before asserting that no further countermeasures such as key rotation were necessary. Such an action would demonstrate significant diligence as compared to a public release days after you've told customers there was nothing to worry about.

Additionally, that public release didn't directly encourage security review, the deprecating comments on the post were primarily around portability and design.

csoandy2• 12 years ago

Willem, Thanks for the well-reasoned set of claims for us to evaluate. We are still looking at them, but a critical one is accurate: our own implementation of the secure memory area did not include the CRT values.

More here: https://blogs.akamai.com/2014/04/heartbleed-update-v3.html

comice• 12 years ago

Kind of amused at how Akamai smugly keep their openssl security improvements private for years, only for them to be shown to be broken days after open sourcing.

With enough eyeballs, all bugs are indeed shallow!

nraynaud2• 12 years ago

I like how he pounds on one company who made a mistake on one network, while having no word for the developer who made a mistake for almost the entire internet.

Willem Pinckaers on Akamai's flawed OpenSSL allocator patch

Related Stories