'Heartbleed' contributor denies he inserted it deliberately

http://www.smh.com.au/it-pro/security-it/man-who-introduced-serious-heartbleed-security-flaw-denies-he-inserted-it-deliberately-20140410-zqta1.html

by jaboutboul • 12 years ago

57 83 12 years ago

smh.com.au

systematical2• 12 years ago

Yeah most of my bugs are deliberate. I like the feeling of making mistakes and seeing my boss run out of his office frantic.

venomsnake2• 12 years ago

Just wait till you have been told - it is running too fast so the client won't perceive it as a "serious" software ... slow it down.

iwwr2• 12 years ago

The problem is not that bugs happen, but that certain categories of bugs still happen when we have means to avoid them altogether.

sillysaurus35• 12 years ago

I believe it. Crafting a weapon that can be used by both you and your enemies would be pyrrhic.

The lesson out of this ordeal is probably to be as skeptical as possible of everything you take for granted. How do you know that you're secure? What if your assumptions are wrong? Try to invent ways to break your own assumptions. The best way to protect yourself is to try to defeat yourself.

Unfortunately, "ain't nobody got time for that," as they say. But if you find time, it's quite rewarding. And disconcerting. You'll wonder why we're still wrestling with these fundamental problems in 2014, and then you'll start questioning the foundations we've been relying on until now.

flatline• 12 years ago

Crafting a weapon that can be used by both you and your enemies would be pyrrhic.

In the long run, yes. In the short run, only you know about it and can exploit it from day 0, others are presumably going to take some time to find it, and it may go unnoticed for years or never be found (I understand the error was only present in some versions of OpenSSL and both older and more recent versions were free from it). I'm inclined to think the author made a simple mistake but this does not seem like sufficient incentive against someone introducing such a vulnerability.

iwwr2• 12 years ago

It may be impossible to distinguish genuine bugs from bug-backdoors, which is why it's important to start developing crypto in safer frameworks and languages. C considered harmful.

sillysaurus35• 12 years ago

The problem is that GCs are considered harmful to crypto code, though that may be changing with Go. We won't know for about 5 years whether it's trustworthy, and even then, the side channel threat posed by GC may be worth worrying about.

Some big-name cryptographers have started implementing some useful crypto services in Go, so we'll see whether it catches on.

EDIT: Is the JVM generally trusted by cryptographers? I just realized that TextSecure is java code. Is the side channel threat posed by GC not too serious, then?

gsnedders• 12 years ago

Rust provides memory safety without a GC, which seems more promising than anything else for this sort of application.

pornel2• 12 years ago

C vs GC is a false dichotomy.

There can be safer-than-C languages that don't use GC, e.g. you don't need GC for array bounds checking. You can use reference counting for more deterministic garbage collection.

jerf• 12 years ago

"The problem is that GCs are considered harmful to crypto code, though that may be changing with Go."

I'm curious what about Go's GC leads you to say that? Is it that when you get a new chunk of bytes, they've been pre-zeroed? That's not specifically a characteristic of the GC but it's my best guess. Otherwise I don't know what it would be, which is why I ask.

Shish2k• 12 years ago

The problem is that GCs are considered harmful to crypto code

Could a GC'ed language not just pause the GC thread while it executes a given block of code?

dexen• 12 years ago

Compare with the attempted 2003 Linux kernel backdoor [1], which gave root when sys_wait4() is passed uncommon combination of flags. The bug looked inconspicuous and honest enough.

[1] http://lkml.iu.edu//hypermail/linux/kernel/0311.0/0635.html --

weavie• 12 years ago

I can't imagine what this guy must be feeling right now. I find it embarrassing enough when I am outed in my small team for producing a bug that makes it into production. To be known around the entire internet to have caused the largest security bug in recent times must be quite a slammer.

I really hope it doesn't affect his career..

Karunamon• 12 years ago

The armchair crypto brigade aren't helping much, either, by passing around these completely unfounded accusations of malfeasance.

I'm all for being aware of the possibility that shenanigans are involved, but until proof comes out, it is nothing but a possibility, and a remote one at that.

kmkemp• 12 years ago

"Never attribute to malice that which is adequately explained by stupidity." Obviously "stupidity" is way too strong here, but the principle still applies.

rext• 12 years ago

I'd be amazed if he doesn't break from the pressure and/or abuse.

thefreeman• 12 years ago

Let the programmer who has never written a bug cast the first stone.

protomyth2• 12 years ago

I'm actually more worried that this will result in a civil lawsuit from one of the affected companies. In an ideal world[1], it would get thrown out immediately, but I'm a bit worried. That would have a heck of a chilling affect.

1) and a simple reading of the license

danielweber2• 12 years ago

Is it 'rayiner who tells us to always have the disclaimer for liability in our licenses? Whoever it is, this is why.

I think any company that is damaged by this should get all their money back, all $0 of it.

_ak• 12 years ago

He works for T-Systems, which belongs to Deutsche Telekom. His job is pretty safe, I would say. People working for Deutsche Telekom work under conditions similar to German civil servants in terms of terminability.

mikeash3• 12 years ago

I wouldn't be surprised if it was good for his career in the long run. Smart people will know that bugs like this can happen to anyone, and he'll have huge name recognition. As long as he's humble about it, "Yeah, I'm the guy who broke OpenSSL. Man, that sucked." I could see it being a big plus in interviews after a year or two.

krstck• 12 years ago

It we start naming and shaming the coder for each flaw instead of working on fixing the process that allowed it to sneak through, we'll see a chilling effect on open source software. There's a reason we have tests and code reviews and security audits...

talmand2• 12 years ago

Unfortunately the tech industry seems to be in mob mode recently so I would think the chilling effect is already well underway. The fact that it has been suggested this was done intentionally before they know the whole story suggests this to me.

Never mind the fact that the code was apparently reviewed, so I guess the reviewer would have been in on it as well.

There are always people who prefer to play the blame game instead of actually working to solve the problem.

danielweber2• 12 years ago

Part of me wants to see the team give the industry a big middle finger and quit the project, and now companies relying on it have to start spending money to keep it up instead of expecting it to be provided for free.

If something is important to you, you should spend more resources than zero on it.

nikcub3• 12 years ago

I just happen to be speaking to the very reporter who wrote this article prior to him publishing this report. I told him that in 'our world' we don't really emphasize or make much of who introduced the bug, since it is understood that writing secure code is hard.

But what required the denial from the developer (and I feel horrible for him) is that this bug gained so much mainstream attention and speculation. It was only a matter of time before a reporter wanting to further the story got in touch with him.

He was in a bad situation - don't respond and have that misconstrued or respond and deny and give credence to the theory.

We all know that he didn't do it intentionally and in absence of any supporting evidence in all cases these are simple mistakes, but the general public who are currently whipped up in a frenzy around NSA revelations don't know that.

mcherm• 12 years ago

We all know that he didn't do it intentionally

Actually, I hadn't known that. But this article gave me strong reasons to believe that it was, indeed, accidental. I certainly recognize that this is the sort of mistake that is extremely easy to make.

chiph• 12 years ago

Any C programmers who have never failed to check their bounds? Anyone? Anyone? Because everyone has done it.

Now would be a nice time for one of the Lint vendors to donate copies of their product to some of the OpenSSL team members, and for them to dedicate some resources into fixing the more important stuff it finds.

bunderbunder• 12 years ago

Or better yet, start seriously talking about moving to a language that doesn't make it so stupendously easy for trivial bugs like these to get so far out of hand.

Bounds checking incurs such a trivial amount of overhead, and the potential cost of straying out of bounds is so high. I understand we've got decades' worth of C culture built up around coding idioms that are incompatible with type safety and bounds checking. I understand that means that developing and migrating to a dialect of C or C++ that offers something in the way of a safety net is anything but trivial. But we've also got decades' worth of painful examples of just how catastrophic - and difficult to prevent - these kinds of errors can be, and that alone should justify the cost.

At the very least, consider the example set by C#: Be type safe and do bounds checks by default, but provide an `unsafe` keyword for those cases where it really is necessary to disable them.

tptacek5• 12 years ago

Strangely, nobody's tracking down the nginx developer who inserted the exact same bug into nginx just a few years back (a NUL in a header would cause the header copies, done using strncpy, to abort early and expose uninitialized memory). He must have been an NSA plant too, right?

anonbanker4• 12 years ago

Why aren't we doing exactly that? While its best not to attribute to malice what could easily be explained by human stupidity, shouldn't we try to rule out malice?

tptacek5• 12 years ago

Yes! More witch hunts! That's what I was suggesting!

Seriously, this place is like Lord of the Flies sometimes.

Symmetry• 12 years ago

My priors on this being deliberate are pretty low, but I'd expect this denial either way so it really isn't evidence against the idea.

fit2rule2• 12 years ago

I just can't understand why such critical components as OpenSSL just don't use Code Coverage tools like Coverity to find such things as this? Testing, coverage certification, static analysis: this would have been caught if these tools were being used.

peeters• 12 years ago

It would shock me if those tools would actually have caught this bug. That would imply that no benevolent contributor or researcher had run them against OpenSSL in the last two years, which I find incredibly hard to believe, as static analysis is a logical first step in researching a code for vulnerabilities, and OpenSSL is probably in the top 5 most valuable pieces of software to validate.

fit2rule2• 12 years ago

I just ran Coverity on code that is similar to Heartbleed's bug, and it caught it. So, I think this is really just a glaring omission on a lot of peoples' part - but like you I also find it hard to believe nobody ran this code through static analysis. Its mind-boggling.

ajanuary4• 12 years ago

Someone commented (I can't find it now) that post disclosure they ran 5 or 6 coverage tools and one of them flagged it.

They didn't say how much else got flagged, and how many false positives there were. It's very easy to retrospectively look through the list and say "oh yeah, there it is, amongst this big pile of nonsense".

svaha1728• 12 years ago

Does someone know the actual line of source where the error appears? The goto fail vulnerability was easy to catch with static analysis, but I haven't seen the actual lines of code for this one yet.

ThursdayNext• 12 years ago

This thread here:

http://openssl.6102.n7.nabble.com/Coverity-coverage-of-OpenS...

mentions that one OpenSSL developer used to see defect reports from Coverity (probably through Coverity's scan project). He states:

"Coverity used to, and perhaps still do, run scans of OpenSSL, which we had (have?) access to. I used to look at them and fix relevant ones, but got irritated with the false positive level in the end.

If Coverity were interested in fixing their bugs, I might get interested in looking at their reports again."

Of course this doesn't demonstrate that Coverity found this particular problem, and since he doesn't state what the false positive rate was it's difficult to know how reasonable his statement is.

shagie• 12 years ago

According to Andy Chou of Coverity on Twitter https://twitter.com/_achou/status/454287263917043712 - "We looked at the code and it turns out the bug is caused by something we don't currently detect." and then goes on to say "But we're going to look into it further. In cases like this it is not only about finding the defect it's also about not flagging hundreds/thousands of false positives at the same time"

So no, Coverity wouldn't find it.

wglb• 12 years ago

The problem with most static analysis processes is the typically vast number of false positives.

notacoward6• 12 years ago

Maybe the silver lining here is that it puts the final nail in the coffin for "many eyes make all bugs shallow" - which was always total BS from the day it was uttered. There's so much code out there, much of it highly specialized and even project-specific, that there are very few eyes looking at any particular piece of code, and not all eyes are connected to the greatest of brains.

Most static code analyzers could have caught this particular bug, as it requires only a fairly simple kind of reasoning about allocated vs. used length. In fact I believe it probably was found by static analysis before being reported by a human, which is a shame because it misses an opportunity to highlight the value of such tools. Maybe next time people will spend a little less time designing a logo and a little more time doing things that actually help (though that's a wish and what I expect is the exact opposite).

The real lesson here is that we should apply as many different tools and processes as we can at improving code quality for critical infrastructure. Code reviews are nice. Static analysis is nice. Detailed tests are nice. However, none of these alone is sufficient even when pursued with fanatical devotion.

nikcub3• 12 years ago

Maybe the silver lining here is that it puts the final nail in the coffin for "many eyes make all bugs shallow"

The OpenSSL project doesn't seem to have a 'review' step[0] in code commits, unlike FreeBSD, OpenBSD and Chromium (i'm just picking projects i'm familiar with, no bias).

So it might not have been 'many eyes', but the eyes of just one.

edit: [0] a proper review step, what the openssl project does is have feedback on commits (which is what the OP article is referring to). see below.

sp332• 12 years ago

The article says there was a reviewer who looked at this code before it was moved to the release branch.

notacoward6• 12 years ago

The "many eyes" canard isn't just about formal reviews, though. It's also about the idea that when something's open source the reviews aren't limited to maintainers or close colleagues who would be involved regardless of open vs. proprietary. What actually seems to happen in the great majority of cases is that the circle of people who actually understand any particular commit is only very slightly greater than it would be otherwise. The bugs are only a very tiny bit shallower, if at all. The fact that it was just one reviewer in this case, despite the code being widely available, only underscores that point.

dangayle• 12 years ago

> Maybe next time people will spend a little less time designing a logo and a little more time doing things that actually help (though that's a wish and what I expect is the exact opposite).

So you want the graphic designers contributing to crypto code?

As someone who is a mid level programmer but a decent/good designer, the only way I'm able to contribute to some projects is through things like logos.

Don't use this straw man argument to make your point, which is otherwise valid.

tptacek5• 12 years ago

It sure would be nice if some of those designers would take a whack at the Mozilla certificate configuration UI. It hasn't meaningfully changed in a decade, it's one of the Internet's biggest security flaws, and it's something any UX person could safely propose changes to.

mikeash3• 12 years ago

A lot of projects have no designers and the programmers spend a lot of time tweaking atrocious logos. I don't think that was meant to be a slight against designers. Probably the opposite, really: don't waste time screwing about with a logo if you don't have the talent, concentrate on the programming and either go without or find someone who can do it well.

orblivion• 12 years ago

Most static code analyzers could have caught this particular bug

Does that mean nobody out there in the crypto world ran a static analyzer on OpenSSL in the last 2 years?

nuncanada• 12 years ago

OpenSSL's code uses particularly bad MACRO's ifdefs that defeat static analyzer's capabilities to analyze the code...

DanBC2• 12 years ago

... And wasn't the Debian bug caused by someone running a tool against the code and trusting the tool?

tfinniga• 12 years ago

A lot of what Linus says only really applies to linux kernel development.

And some of that only applies when you're Linus.

anonbanker4• 12 years ago

I don't know; this seems like a clear-cut case of "not enough eyes" to me.

raesene3• 12 years ago

Hopefully one positive thing that will come out of this whole Heartbleed thing is that companies making extensive use of Open Source software for security critical purposes will consider contributing to ensure that security reviews are carried out on them.

The cost of regularly reviewing codebases like OpenSSL would likely not be that high when compared to the potential impact of a breach because of a flaw in the software.

JPKab• 12 years ago

I work with both private and public sectors in the DC area.

One of the things that sucks about the Federal sector is that they are dominated by Microsoft and Oracle shills (the kind of IT pros who can't learn new skills unless its spoon fed pre-digested in the form of industry certification training) who do nothing but scream about the danger of open source. Now of course we all know that the only difference between enterprise and open source security holes is that the former go undiscovered for longer, and when discovered by the code owners aren't disclosed despite them having knowledge that black-hats know about it.....

But make no mistake. This fucking idiot [Edit: Ok, this isn't fair, this could happen to anyone, so he's not an idiot, but why not use a static analysis tool?] who did this to OpenSSL and the idiots who let it happen are going to set open source in the Federal government back YEARS. Not because its an actual threat, but because it will be used by the enterprise assholes as a weapon to keep selling their shitware to the risk averse morons who make up the giant pile of middle manager idiots that composes the Federal government.

I've already told my boss I'm not doing any more public sector work after my current project ends, and this is the nail in the coffin for me touching it ever again.

tptacek5• 12 years ago

So your theory is that nobody in the last couple years ever ran Coverity or Fortify on one of the top 5 security codebases on the whole Internet?

'Heartbleed' contributor denies he inserted it deliberately

Related Stories