Why did the OpenSSL punycode vulnerability happen?

https://words.filippo.io/dispatches/openssl-punycode/

by stargrave • 4 years ago

197 95 4 years ago

words.filippo.io

dang• 4 years ago

Ongoing related thread:

The latest OpenSSL vulns were added fairly recently - https://news.ycombinator.com/item?id=33437158 - Nov 2022 (56 comments)

tester7566• 4 years ago

It feels like issues like those are more common in parsers, this specific kind of software.

But why?

Why is parsing so hard? or is it just in low lvl languages? or maybe languages with poor string primitives?

I've written parsers in high level languages and it didnt felt dangerous or insanely hard

nightfly• 4 years ago

Probably low level + safe + performant == hard

tester7566• 4 years ago

But why? where does the complexity come from

insanitybit3• 4 years ago

I think it's easy to hand write a parser that can get into a weird state. Some formats are also easier to write a parser for safely vs others.

For example, if you have a Tag Length Value data format that says "5ABCDE" maybe that means "the next 5 bytes are a string, then there's something else afterwards or it's the end", you don't want to allocate a buffer of 5 values and just keep writing into it without checking that you're reading valid values.

Doing this sort of thing efficiently may also mean that you're tempted to remove something like a bounds check. After all, if you already know that the value is 5 bytes, why check on every access? People really want parsers to be fast. Ideally in the above example I wouldn't even need to copy those bytes out, I could just reference those values, which now leads to potential lifetime issues.

Further, in C, buffers don't have lengths attached to them. In every other language you typically don't work with null terminated strings. So now you have to manage sizes of things throughout your parser.

One way to view this is that the programmer's mental model of the parsing machine can easily drift from the implementation of the parsing machine, leading to vulnerabilities.

Basically it ends up being very easy to accidentally end up with out of bounds reads/writes + there's pressure to be fast + formats can be very complex.

That's my view on it at least.

tester7566• 4 years ago

Doing this sort of thing efficiently may also mean that you're tempted to remove something like a bounds check. After all, if you already know that the value is 5 bytes, why check on every access? People really want parsers to be fast.

In what kind of software bounds check have this significant perf. penalty?

The only people I've heard talking about such a stuff were firmware devs.

Further, in C, buffers don't have lengths attached to them. In every other language you typically don't work with null terminated strings. So now you have to manage sizes of things throughout your parser.

Cannot C have some wrapper over those poor strings

that leads to better safety? dev's experience, etc, etc?

ninkendo• 4 years ago

Poor string primitives is the answer, I think. Or more generally, C’s tendency to require a buffer of sufficient size to be passed for output, and the complexity involved with:

    - What size to pick for the buffer
    - Making sure the buffer is of the size you think it is
    - What to do when the buffer isn’t big enough
    - How to know for sure that the buffer is big enough

A parser runs right into these problems quickly: An input string may have some formatting specifiers (%d, etc), may involve control characters that cause the output buffer to expand, etc etc… Any function that deals with this kind of thing has to do be able to correctly tell the caller that their buffer wasn’t big enough, and by how much, and has to make sure it didn’t accidentally write past the end of the buffer before doing this.

Callers also have to make sure the buffer size they’re passing to the parser is actually accurate… you don’t want to malloc 100 bytes and tell the parser the buffer is 200. They also have to make sure that if they say the buffer is 100 bytes, that the string actually terminates with a \0 by the end of it, or it gets even more confusing.

It’s overall a complicated problem, and arises specifically because of the tendency of C library functions to avoid allocations… because the ownership model of idiomatic C is that callers should be the ones that allocate (because they probably know best how to allocate and when to free, etc.)

alserio3• 4 years ago

I'll give you a tautological and useless answer. I hope you don't mind. A parser is a kind of interpreter, where code is executed based on external input. When user input controls how code is executed you have opened the doors of hell: it's hard to guarantee that all of the possible executions are safe.

tester7566• 4 years ago

But why?

Parsers executes safe code fragments in order based on the input

How safe operations result in unsafe results?

I'm not talking about side channels here.

toast0• 4 years ago

Here's an even more tautological answer. You can't misparse input unless you're parsing input. That's why parsers have trouble with parsing input.

bhk• 4 years ago

It's not hard to use a "vector" library or even just a small set of macros that ensure memory safety when you are manipulating slices of strings or buffers.

Unfortunately it's very tempting to play fast and loose with raw pointers, with ad hoc validation logic all over the place.

OTOH, maybe SSL maintainers consider punycode parsing performance more important than security.

green_on_black• 4 years ago

tl;dr: Using C/C++ and being human => memory safety problems.

cratermoon2• 4 years ago

Also, as we learned back when Heartbleed was discovered, the OpenSSL code is not in good shape. It "suffers from maintenance", as one clever wag said about legacy code. There's a reason LibreSSL forked the code. More distributions need to switch away from OpenSSL.

And before anyone pipes up, I'm not claiming LibreSSL does not and will not ever haver vulnerabilities. I'm saying that ripping stuff like punycode out of the library reduces the attack surface. https://isc.sans.edu/diary/rss/29208

pshc• 4 years ago

Parsing isn’t all that hard, it’s about working with the right tools and level of abstractions. Messing around in C with null-terminated strings is just hilariously error prone.

oconnor663• 4 years ago

I think the biggest difference is bounds checking. If you write off the end of your array in Java, you get an exception. In C, you get a CVE. Other things like manual memory management and fiddly string types don't help, but simple bounds checks would catch so many things.

fulafel2• 4 years ago

Your questions are less interrelated than you might think.

Why is general parsing hard? It's sense-making from free form input. We have so many different languages and syntaxes and there are different algorithmic approaches needed to parse them depending on the format and language, frequently made as dense as possible for various more or less (usually more) misguided "optimization" reasons.

Why is parsing in low level languages hard? This is more about incidental features of low level languages. C just happens to be exceptionally unsuited and unsafe for the tasks involved in parsing. From a safety and security POV C is chainsaw juggling, but C for parsing untrusted data is chainsaw juggling while running through a mine field.

mikepavone2• 4 years ago

There is a function, ossl_punycode_decode that decodes Punycode. Punycode is a way to encode Unicode as ASCII, used to represent Unicode strings in the ASCII-only world of DNS. ossl_punycode_decode takes an output buffer, and if the buffer runs out it keeps parsing and verifying the Punycode but discards the rest of the output.
I did not have the heart to figure out why it works like this. Maybe there's a good reason to do progressive parsing. Maybe it's an artifact of how C makes you do memory management or of OpenSSL's C style. Anyway.

Not involved in OpenSSL, but this is a fairly common pattern in a lot of C APIs. You want to decode some data, but you're not sure how big the output is going to be ahead of time. You could write a separate function to calculate the length, but that function has to do most of the work of actual decoding to figure out that length. A lot of times the output is small enough it can fit in some conservatively sized buffer so you can save a fair bit of work by having a (potentially stack-allocated) buffer of some fixed size and then allocating a precisely sized buffer on the heap if it turns out to not be big enough. Further, having a separate length function means you typically end up with two similar but separate decode implementations which has its own problems.

In most other languages, you have some sort of growable container in the standard library so you just avoid the problem entirely at the expense of having less control over memory allocations.

j-bos• 4 years ago

I am by no means a rust developer and asking in ignorance, would this have been a problem had it been written in rust?

vore2• 4 years ago

Rust would take the growable container approach here, so it wouldn't be vulnerable in the same way. There's nothing in C that prevents you from doing this either, but as the parent post mentions it's common style to do this (see also things like snprintf).

mike_d3• 4 years ago

It is an exercise in navel gazing until an organization steps up to fund: 1) a linkable clib harness that exports Rust functions to replace OpenSSL functions. and 2) pays to get the resulting harness and underlying code FIPS certified so people can use it.

Ygg2• 4 years ago

Not in this case as far as I can tell.

newpavlov• 4 years ago

Rust code would probably return an error on output buffer overflow instead of siltently discarding data. Something like this: https://docs.rs/base16ct/latest/base16ct/mixed/fn.decode.htm...

simcop2387• 4 years ago

Probably not. It's definitely possible to write this function exactly as it is in openssl with this issue, nothing about Rust's safety claims prevent it. However, Rust doesn't have the same issues that lead to this kind of pattern being common like they are in C. Result<> types being so common, better handling of dynamic allocations, and at least for now a generally (it can't last forever) more security conscious community mean that other patterns will be chosen unstead. I'm not sure any language can actually prevent it from being possible outright, but you possibly pair it with some kind of formal proof system like using Coq and TLA+ might let you prove thag an issue like this doesn't exist in a specific implementation. However it's incredibly hard to do this.

tptacek3• 4 years ago

In no other mainstream language other than C (and maybe non-idiomatic C++) would this have ever been a problem.

cesarb4• 4 years ago

would this have been a problem had it been written in rust?

The answer would be "it depends on whether you consider denial-of-service a problem". The key detail which makes all the difference is that, unless you're playing with raw pointers (which can only be dereferenced in "unsafe" blocks), the pointer to a buffer slice is always kept together with its length (in a "fat pointer"). Attempting to write through it past the buffer's bounds will result in a Rust panic, which usually aborts the whole process (there are ways to abort just one thread, or even to treat it similarly to a C++ exception, but a library cannot depend on them since the program might be compiled in the panic=abort mode). While that's obviously better than allowing for remote code execution, it still could be considered an issue.

Of course, that's assuming you want a similar API and are writing the code in a similar style, just replacing the direct pointer manipulation with slice manipulation. I don't know whether, in this particular case, more idiomatic Rust code would have avoided the issue. And, of course, the "growable container" approach would completely avoid it, but Rust is also used in places where memory allocation is not allowed, so having a non-allocating API still makes sense.

tptacek3• 4 years ago

Even in C, you can write an abstraction for a growable buffer. The problem is, you have to make all the rest of the code work with it, rather than a char* or whatnot.

wahern• 4 years ago

BoringSSL and libressl added CBS (bytestring) and CBB (bytebuilder) interfaces in 2015. Converting everything over is a chore, but that's exactly what those projects have been methodically doing over the years.

Unfortunately, I don't see those interfaces in OpenSSL 3.0, though maybe (hopefully) they're working toward something similar.

kevin_thibedeau2• 4 years ago

Then you have the new problem of restricting its growth and that adds a new failure mode to deal with. The C idiom works fine if you don't have implementation bugs.

fulafel2• 4 years ago

Or you can use char * and realloc (or a callback) for it to grow, if you're not forced to use caller allocated buffers.

magicalhippo2• 4 years ago

Not involved in OpenSSL, but this is a fairly common pattern in a lot of C APIs.

A better approach IMHO, typically found in the Win32 API for example, is to take an extra argument which receives the buffer size needed to hold the whole output, in addition to the size of the buffer you pass.

This allows the caller to detect the condition and decide if it's an error or not if the required buffer size returned is greater than the size of the buffer, and potentially call it again with a sufficiently large buffer.

fbdab103• 4 years ago

Isn't that roughly how heartbleed works? Trusting the provided length?

mikepavone2• 4 years ago

A better approach IMHO, typically found in the Win32 API for example, is to take an extra argument which receives the buffer size needed to hold the whole output, in addition to the size of the buffer you pass.

This is exactly what the function in question does and is what lead to the buffer overflow. Basically, this sort of API requires you to continue decoding (so you can calculate the length), but stop writing to the output buffer once the buffer is full. The function had a bug that lead it to keep writing to the buffer in some cases even after it was full.

AceJohnny22• 4 years ago

As curl author Daniel Stenberg said, "I've never even considered to decode punycode. Why does something like OpenSSL need to decode this?"
[...]
Internationalization is not the issue, internationalization is the job.

I want to cheer at this.

Internationalization is hard, really hard, especially in a computing world so defined by its english-language dominance. The little amount of effort demonstrated in defining and testing this feature demonstrates that.

For all I admire Stenberg's work and focus on quality, that was a rather poor take from him.

FiloSottile• 4 years ago

Oh I don't think Daniel was asking why we're doing i18n. My remark was not at him, at all. I was just pre-empting a possible made up objection. He's in fact correct to wonder why punycode decoding ended up in OpenSSL, as the rest of that section explores. The point being that OpenSSL could do its job and still support i18n domains and emails without having to ever decode punycode if only the spec had made different tradeoffs.

AceJohnny22• 4 years ago

ah. Thanks for the clarification!

tialaramex• 4 years ago

Exactly this. I had the very image of this discussion with Python folks for DNSname SANs back when Python was learning not to just rely on CN.

Python folks felt like the correct thing was: 1) Implement a Punycode decoder. 2) Decode hostnames and certificate information to get Unicode 3) Compare the Unicode Strings. They were fretting about how complicated it would be to arrange all this, the months of work needed and the need to bring in teams who understood about i18n issues...

And I was like No, don't do any of that, take the bytes and compare the bytes. If the bytes aren't identical that's not a match, you are done. It seemed to take a while for it to sink in that this is correct and simpler and thus better.

MBCook• 4 years ago

So the blog mentions that in certain cases you have to decode the punycode from one field to compare it to the value in another field.

Would it have been safer to encode the data in the other field to punycode and compare the encoded values?

That way a hacker can’t mess with your decoder (where bugs like to lie). But at the other end you risk that your encoder has an issue. I do t know how to judge if those are equal risks or not.

Thoughts?

mike_d3• 4 years ago

Encoding and decoding are both transforms on attacker supplied input.

I don't think its helpful to armchair quarterbacking other peoples mistakes that ended up in a vulnerability when the overall code quality is good. There are a dozen ways they could have done it differently, which may or may not have resulted in different exotic bugs.

Punycode was the footgun here. Not the language, or the implementation, or the code. They were forced to do something stupid and complex and dangerous deep within the bowels of a critical library.

moring• 4 years ago

If you want to avoid risks both in the encoder and decoder, you might have success with a function that takes an encoded strict and a decoded string and compares them directly, encoding/decoding (whatever works better) on the fly. This avoids the allocation altogether, at the cost of having another less-composable piece of code that must be maintained.

I'm not sure if that is the way to go. An allocation per se isn't hard to get right, especially in this case where its scope is well-defined and very limited. What happened here is more the fact that while a single piece of code is easy to get right, a whole codebase isn't. So the only value added by my proposed solution above is that it works in an environment where you can't allocate (e.g. memory-constrained systems where you need to know the required amount of memory statically).

Bottom line: I don't think you can really solve this problem on a technical level.

edit: typo

nraynaud• 4 years ago

About the classification: I would add that the people choosing the severity are a bit alone because of secrecy, so they also can’t really ask too much for advice.

That’s probably where the NSA could be useful because of their big number of competent and sworn to secrecy employees, but nobody can trust that the zero day sent for an opinion will not be used to fuck with a foreign country on day one.

ahoog422• 4 years ago

I decided to review SBOMs from about 3,800 popular mobile apps to see if any included vulnerable versions of OpenSSL v3.0.x. No mobile apps did (not surprised) but what did surprise me was 98% of the OpenSSL versions included in these apps were vulnerable to older CVEs. About 16% of the apps included OpenSSL, mostly as a transitive dependency.

I posted additional details in this blog+video: https://www.andrewhoog.com/post/how-to-detect-openssl-v3-and...

cesarb4• 4 years ago

No mobile apps did (not surprised) but what did surprise me was 98% of the OpenSSL versions included in these apps were vulnerable to older CVEs.

Just the openssl version is not enough, since it could be patched to fix vulnerabilities without increasing the version (this is very common on Linux distributions, which often apply security patches instead of migrating to a new version; for instance, Fedora released a patched 3.0.5 instead of going to 3.0.7).

And using an older openssl version does not necessarily mean it's using vulnerable code; according to your blog post, the most common use is SQLCipher, which from a quick look at its README.md seems to use openssl only for the encryption algorithms. Unless the vulnerability was on the basic algorithms used (AES, HMAC, etc), it won't affect this usage.

ahoog422• 4 years ago

Great points.

Static binary analysis looks for the version string but doesn’t currently do deeper analysis of reversed code to see if it’s patched. Could go either way.

And determining if the code is triggered and exploitable is quite challenging. Dynamic analysis can help here, provided you have the coverage.

More generally tho, istm that there will be instances when the version is unpatched and there is some exploitable vector (even if it’s just crashing the app). My hope is to raise awareness for developers (and security) about 1) transitive dependencies and 2) some really old OpenSSL versions in very popular mobile apps. I don’t believe most folks think about this and awareness can lead to shipping safer apps.

jesprenj• 4 years ago

Why would a TLS library even parse punycode? The sole reason we still use this hack is so that core infrastructure does not have to change when we use internationalized domains.

If TLS libraries, DNS servers, HTTP servers, ... needed to be patched anyways for the use of IDNs, then why did we not do it properly and just use UTF-8?

No matter how many vulnerabilities we've introduced into software, "Š" in my name still cannot be represented in domain names and is instead written with american letters xn--pga.

I'm sorry for poorly articulating my thoughts here.

boxy_brown• 4 years ago

Why was this code even necessary
The answer is: an explicit IETF design choice, that made punycode decoding part of X.509 verification, without even a line of acknowledgement in the Security Considerations.

cryptonector4• 4 years ago

The idea is that back when IDNA was created sending just-UTF-8 in places that had historically been ASCII-only would break software. The scale of the breakage was probably unknown. The real question is: would a flag day have been needed to just use UTF-8 where previously only ASCII was allowed. I think the answer to that is "no". The punycode thing certainly avoided the question altogether, which is why it was done, but there's no question that had we bitten the bullet then and just gone with just-use-UTF-8 life today would be easier.

(There's also Unicode normalization issues here. One of the problems with just-use-UTF-8 is that dumb servers would only match on octet-wise label equality, but with Unicode you want normalization-insensitive matching. Now, at the time the notion of normalization-insensitive matching hadn't been invented... Requiring something like IDNA made it possible to slip in a normalization requirement.)

cryptonector4• 4 years ago

So, punycode I do think was silly. We should just have used UTF-8 in DNS and have left it at that.

Using UTF-8 would not have required a flag day. It would have required upgrading some servers in order to be able to have non-ASCII domainnames, but it wouldn't have broken anyone not using non-ASCII domainnames.

tinus_hn2• 4 years ago

So it would have broken the whole internet for anyone using a UTF-8 domain name and this system would be adopted by no one. It’s like IPv6, using IPv6 doesn’t break anything for anyone not using it. It’s just that if you use ipv6, you can’t talk to half of the internet including this site.

cryptonector4• 4 years ago

It wouldn't have broken the whole internet though. It's hyperbolic to suggest that it would have.

Why did the OpenSSL punycode vulnerability happen?

Related Stories