Akamai release source to their custom secure_malloc for OpenSSL

http://thread.gmane.org/gmane.comp.encryption.openssl.user/51243?resub=1

by nikcub • 12 years ago

267 80 12 years ago

thread.gmane.org

nikcub• 12 years ago

For background, this came up as Akamai mentioned in their Heartbleed announcement that they didn't think they had any sensitive information[0] disclosed by the bug as they had wrapped OpenSSL's malloc to create two heaps: a secure heap for sensitive information like keys and an ordinary heap:

https://blogs.akamai.com/2014/04/heartbleed-update.html

It prompted a lot of interest in their solution, which prompted them to release the patch.

edit: [0] by "sensitive information" we mean keys, if I understand the implementation correctly, you could still overrun into web server allocation and get back HTTP headers.

gphilip• 12 years ago

From reading that blog post it is not clear to me if their wrapping code existed prior to the discovery of the heartbleed bug: The post summary says:

Akamai patched the announced Heartbleed vulnerability prior to its public announcement. We, like all users of OpenSSL, could have exposed passwords or session cookies transiting our network from August 2012 through 4 April 2014. Our custom memory allocator protected against nearly every circumstance by which Heartbleed could have leaked SSL keys.

So: did the custom memory allocator exist already in August 2012? From reading the post this looks to be the case. Could it be that someone at Akamai took a look at the heartbeat (or other OpenSSL) code, decided that it could lead to memory leaks, and wrote their own memory allocator wrapper code to guard against this?

Edit: SSL -> OpenSSL

computer2• 12 years ago

How I read it is that this code already protected their private key, but the Heartbleed bug still disclosed other private details (such as submitted user data).

alexkus• 12 years ago

From the original post:

This patch is a variant of what we've been using to help protect customer keys for a decade.

So it protects keys against heartbleed, but not other HTTP related data (urls, cookies, headers, etc).

Dylan168074• 12 years ago

Keys are not passwords and session cookies.

yukichan• 12 years ago

Akamai's patch is older than the heartbleed issue in OpenSSL so it wasn't a response to heartbleed, and they were still vulnerable for non-key data.

brians6• 12 years ago

We definitely leaked session cookies and passwords, until we patched. This only protected SSL private keys.

ccurtsinger2• 12 years ago

You could do better with a probabilistically secure allocator instead: http://people.cs.umass.edu/~emery/pubs/ccs03-novark.pdf

Randomized allocation makes it nearly impossible to forge pointers, locate sensitive data in the heap, and makes reuse unpredictable.

This is strictly more powerful than ASLR, which does nothing prevent Heartbleed. Moving the base of the heap doesn't change the relative addresses of heap objects with a deterministic allocator. A randomized allocator does change these offsets, which makes it nearly impossible to exploit a heap buffer overrun (and quite a few other heap errors).

bello2• 12 years ago

Disclaimer: I'm not too familiar with the code.

"This arena is mmap'd, with guard pages before and after so pointer over- and under-runs won't wander into it."

Doesn't that mean that this will only protect against overreads of a certain max length (such that the int16 length in heardbleed)? Seems like that wouldn't help with a length defined as a bigger int. I wonder if there's any better ways of doing this.

ramidarigaz• 12 years ago

The beginning and end of the allocated section of memory (the guard pages they refer to) are marked as PROT_NONE with mprotect, meaning that any access to them will cause a segfault. It's possible that a misbehaving process could jump straight into the unprotected memory, but it would have to not read from the guard pages at all. Buffer overruns don't have that problem (since they access memory sequentially), and would cause the program to crash before any sensitive data could be read (assuming the overrun starts outside the protected area).

bello2• 12 years ago

That makes sense, thank you!

xyzzy123• 12 years ago

The idea is that a read overrun where the base pointer starts in a different arena would have to cross the guard pages to "get to" the secure arena.

okasaki2• 12 years ago

If OpenSSL had been GPL licenced, this would have been public a long time ago.

zobzu• 12 years ago

Actually as long as they don't distribute the code, even under GPL they don't have to share it.

dontmakemelaugh2• 12 years ago

You mean the application? They need to distribute the code if they provide applications based on the code.

midas0073• 12 years ago

Yup. It's pretty common for open source to be used, abused, modified in house and not shared. Just look at Sergey Aleynikov and Goldman Sachs. [0] Further I've seen a shop deploy hundreds of thousands of CentOS nodes to avoid RHEL license fees. Doesn't even begin to scratch the surface of shops not funding FOSS they exploit like OpenSSH, etc. Most of them are stingy, demanding, greedy bastards.

[0] http://blog.garrytan.com/goldman-sachs-sent-a-brilliant-comp...

jpalomaki• 12 years ago

GNU Affero license changes that: https://www.gnu.org/licenses/why-affero-gpl.html

"It [Affero GPL License] has one added requirement: if you run the program on a server and let other users communicate with it there, your server must also allow them to download the source code corresponding to the program that it's running."

okasaki2• 12 years ago

But it's still GPL and anyone who works at the company is free to share it. Whereas with Apache they can just make the entire thing proprietary and forbid sharing. Granged, we don't know that that's what happened here. Maybe their version was still Apache licenced and nobody bothered to share it at any point.

dbaupp• 12 years ago

And Akamai would probably not be using OpenSSL if it were GPL (maybe because of a business decision within Akamai, or just because OpenSSL would never have become so popular).

awalton• 12 years ago

Why not? They're using Linux and it's GPL. Undoubtedly they have made kernel modifications as well.

They're not distributing the code outside of their company, so the source license in this case is entirely irrelevant.

We could make a different argument if they were using AGPL'd code...

dontmakemelaugh2• 12 years ago

I think they (businesses) often pick OpenSSL exactly because it's not GPL licensed so it's kind of chicken and egg

xorgar831• 12 years ago

It would have to be the Affero GPL actually, since they just run it on their servers and don't distribute the software.

jgrahamc• 12 years ago

I would be interested to know if this actually protects against heartbleed in the real world. I can see how it would keep the private key protected, but not sure if it protects intermediate values used during calculation.

Given that it's likely that the private key can be derived from those parts, it would be good to know if this custom allocator is 'enough' (i.e. are the intermediate values stored in memory allocated using the same allocator)?

This week has been full of surprises. Has anyone tested this?

mrud2• 12 years ago

I didn't but akamai itself did [1]:

    To test this belief, our engineers have rebuilt 
    every Akamai configuration since August 2012, 
    when we upgraded to openssl-1.0.1. This included 
    every kernel, hardware, and edge server software  
    combination; and then a careful inspection of the ways 
    in which memory was allocated, to see if any 
    non-long-term memory allocations might border 
    on our secure heap. Most of the configurations 
    were proven safe; but we found one configuration 
    that was not - there was an available memory block in
    range of the secure key store.

    This less safe configuration was active on our network
    for nine days in March 2013

[1] https://blogs.akamai.com/2014/04/heartbleed-update.html

josephagoss• 12 years ago

That is very impressive. I'm amazed they essentially keep a revision history of their system configuration.

makomk2• 12 years ago

It apparently did for Akamai, though it certainly wouldn't be the first time someone's wrongly thought they were protected from Heartbleed key extraction. It doesn't seem to work in general though; I finally got the version without guard pages working on Debian (the version with them is totally broken and crashes during init) and it doesn't seem to make much difference. Also, it doesn't support threaded servers.

brians6• 12 years ago

As Rich said: a prototype that works in our world, for our use cases. It needs lots of work to generalize. For example, we have about ten thousand SSL private keys per machine. This, without guard pages, protects all but the first few hundred perfectly. Something else protects those first few hundred (and that something else is a lucky freak accident).

If you load 1000 keys, can you extract anything past the 256th?

nly• 12 years ago

Why can't long-term private keys be protected, and used solely by, the kernel?

Linux CryptoAPI seems to have an asymmetric key interface[0], RSA signatures are implemented[1], and there's even a x509/ASN1 parser[1]. Shouldn't this be the default on Linux rather than furthering the NIH syndrome with crude library-local allocators? In the very least, shouldn't there be a secure malloc in glibc? If not, why doesn't OpenSSL use an existing secure malloc library[2]?

[0] https://www.kernel.org/doc/Documentation/crypto/asymmetric-k...

[1] https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux....

[2] http://www.jabberwocky.com/software/secmalloc/

thefreeman• 12 years ago

There have been a lot of posts about the custom malloc. From what i understand, parts of OpenSSL actually depend on the buggy implementation (specifically, the fact that you can the last thing free'd can be malloc'd again).

awda• 12 years ago

If not, why doesn't OpenSSL use an existing secure malloc library?

Any extra library dependency makes OpenSSL less portable. It doubles the work needed to get it running on your new embedded system. shrug It may also increase the code size and attack surface (although probably a good trade-off).

I don't disagree that it sounds like a good idea :).

eliteraspberrie• 12 years ago

OpenSSL aims to be cross-platform, so their options are to use kernel-specific cryptography where possible, and a generic implementation where one isn't available; or always use the generic implementation. They probably reasoned at one point that the latter is less work.

DevX101• 12 years ago

If Akamai realized there was a major security vulnerability more than a year ago why didn't they share this information with the OpenSSL team?

matthavener• 12 years ago

I don't think they knew about heartbleed. This is stopgap to protect against leaking private keys via memory bugs.

ams6110• 12 years ago

I think it's a fair question. If they had what they believed to be an improved (more secure) OpenSSL why not contribute the patch back to the community? After all they are standing on the shoulders of giants here, it seems a bit selfish to take an open-source project, improve it, and then not share that back.

Yes I know that many open-source licenses do not obligate one to do this, but it still seems like the right thing to do to me.

ithkuil• 12 years ago

in other words, this is a general protection against a whole class of vulnerabilities

tptacek• 12 years ago

Because they didn't realize there was a major security vulnerability. Instead, they decided they weren't comfortable with the allocation policies of mainline OpenSSL and rewrote them.

As for why they didn't "share" earlier, and assuming they didn't: the OpenSSL project would probably not have accepted this changeset anyways. It's extremely intrusive and the problem it addresses was, at the time this was written, speculative.

brianpgordon2• 12 years ago

A CDN writing a custom malloc sounds kind of crazy, but I guess it paid off.

judk3• 12 years ago

An open source SSL library writing a custom malloc sounds kind of crazy, but I guess, oh, it was crazy.

lobster_johnson• 12 years ago

Here's a good blog article which looks at their patch: http://blog.nullspace.io/akamai-ssl-patch.html.

herf• 12 years ago

Guard pages address lots of issues with really small overhead, nice. But if you know where to look they're not foolproof.

Wondering what the cost of putting each primitive (e.g., RSA private signing + key) in a separate (child) process would be? Mailservers (qmail and postfix) seem to do designs like this.

You could also imagine a page type that would only be readable to certain code segments (would take CPU support to do it.)

richardwhiuk2• 12 years ago

Expensive, and it's not clear that it would help.

Context switching between applications is costly in terms of cache hit and CPU time.

The other problem is you then have to IPC the sensitive data to the other process, which will mean putting in temporary memory, which would expose it to stack crash, or heap attacks. You've also got to worry about synchronizing the different processes.

whoopdedo• 12 years ago

You could also imagine a page type that would only be readable to certain code segments (would take CPU support to do it.)

Isn't this what TPM is supposed to provide? It received an unfavorable reputation because of its association with DRM, but it addresses the same problem.

teacup50• 12 years ago

Intel SGX would allow implementing this efficiently in-process: https://software.intel.com/en-us/blogs/2013/09/26/protecting...

Akamai release source to their custom secure_malloc for OpenSSL

Related Stories