OpenSSH Backdoors

https://blog.isosceles.com/openssh-backdoors/

by benhawkes • 2 years ago

257 129 2 years ago

blog.isosceles.com

DonHopkins• 2 years ago

The "many eyes" theory of open source security isn't popular right now, but it certainly seems like bigger targets have smaller margins for error.

https://news.ycombinator.com/item?id=20383529

DonHopkins on July 8, 2019 | parent | context | favorite | on: Contributor Agreements Considered Harmful

And then there's Linus's Law, which he made up, then tried to blame on Linus.

"Given enough eyeballs, all bugs are shallow." -Eric S Raymond

"My favorite part of the "many eyes" argument is how few bugs were found by the two eyes of Eric (the originator of the statement). All the many eyes are apparently attached to a lot of hands that type lots of words about many eyes, and never actually audit code." -Theo De Raadt

https://en.wikipedia.org/wiki/Linus%27s_Law

In Facts and Fallacies about Software Engineering, Robert Glass refers to the law as a "mantra" of the open source movement, but calls it a fallacy due to the lack of supporting evidence and because research has indicated that the rate at which additional bugs are uncovered does not scale linearly with the number of reviewers; rather, there is a small maximum number of useful reviewers, between two and four, and additional reviewers above this number uncover bugs at a much lower rate.[4] While closed-source practitioners also promote stringent, independent code analysis during a software project's development, they focus on in-depth review by a few and not primarily the number of "eyeballs".[5][6]
Although detection of even deliberately inserted flaws[7][8] can be attributed to Raymond's claim, the persistence of the Heartbleed security bug in a critical piece of code for two years has been considered as a refutation of Raymond's dictum.[9][10][11][12] Larry Seltzer suspects that the availability of source code may cause some developers and researchers to perform less extensive tests than they would with closed source software, making it easier for bugs to remain.[12] In 2015, the Linux Foundation's executive director Jim Zemlin argued that the complexity of modern software has increased to such levels that specific resource allocation is desirable to improve its security. Regarding some of 2014's largest global open source software vulnerabilities, he says, "In these cases, the eyeballs weren't really looking".[11] Large scale experiments or peer-reviewed surveys to test how well the mantra holds in practice have not been performed.

The little experience Raymond DOES have auditing code has been a total fiasco and embarrassing failure, since his understanding of the code was incompetent and deeply tainted by his preconceived political ideology and conspiracy theories about global warming, which was his only motivation for auditing the code in the first place. His sole quest was to discredit the scientists who warned about global warming. The code he found and highlighted was actually COMMENTED OUT, and he never addressed the fact that the scientists were vindicated.

http://rationalwiki.org/wiki/Eric_S._Raymond

During the Climategate fiasco, Raymond's ability to read other peoples' source code (or at least his honesty about it) was called into question when he was caught quote-mining analysis software written by the CRU researchers, presenting a commented-out section of source code used for analyzing counterfactuals as evidence of deliberate data manipulation. When confronted with the fact that scientists as a general rule are scrupulously honest, Raymond claimed it was a case of an "error cascade," a concept that makes sense in computer science and other places where all data goes through a single potential failure point, but in areas where outside data and multiple lines of evidence are used for verification, doesn't entirely make sense. (He was curiously silent when all the researchers involved were exonerated of scientific misconduct.)

More context:

https://news.ycombinator.com/item?id=20382640

toast05• 2 years ago

Regarding some of 2014's largest global open source software vulnerabilities, he says, "In these cases, the eyeballs weren't really looking".

This makes a lot of sense, because for the most part, you only go looking for bugs when you've run into a problem.

Looking for bugs you haven't run into is a lot harder (especially in complex software like OpenSSL), you might get lucky and someone sees a bug while looking for something else, but mostly things go unlooked at until they cause a problem that attracts attention.

Even when you pay for a professional audit, things can be missed; but you'll likely get better results for security with organized and focused reviews than by hoping your user base finds everything.

cbsmith• 2 years ago

Large open source projects are regularly subjected to security audits.

I think the reality is that closed source software is vulnerable to the same attack, the only difference is fewer eyes to see it and more likely a profit motive will keep those eyes directed in other ways.

poikroequ• 2 years ago

It's not a complete fallacy. In events like this, after the news hits, there are a flurry of eyeballs looking, at least for a little while. The heartbleed bug got people to look at the openssl code and realize what a mess that code is. Inspectre and meltdown has led to the discovery of many more CPU vulnerabilities. After Chatgpt hit the market, there has been lots of new research on AI security, such as into prompt injection attacks.

davidfiala8• 2 years ago

Even with the best intentions, can a volunteer-driven project like OpenSSH truly guarantee the same level of security as a commercial solution with dedicated resources and a financial stake in preventing backdoors?

chgs2• 2 years ago

Better.

Imagine a closed source company with cost pressures employing a random developer who can commit code, perhaps without any peer review, but certainly limited peer review from harried employees.

Now imagine why a nation state would want to get staff working in such a company.

Now if companies like Microsoft or Amazon or Google want to pay people to work on these open source projects that’s a different thing, and a great thing for them to do given how much they rely on the code.

wannacboatmovie2• 2 years ago

Your argument is a model that does no vetting of contributors whatsoever, which resulted in the catastrophe that is the topic of discussion, is better than a hypothetical company which is full of compromised developers that have free reign to commit to the source tree with no oversight? That sounds extremely contrived.

davidfiala8• 2 years ago

There's a ton of great truth here. It's hard to bite the bullet and believe that insiders already exist (everywhere), but I can share that from my experience working in big tech:

- There 100% will be bad actors. Many of them.

- But not always nationstate. Instead, they do it for (dumb) personal reasons, too. Also, don't forget lulzsec as a great example of just doing it for fun. So we cannot presume to know anything about the 'why'. The bad guys I caught did it for the most asinine reasons...

But the good news is that we have options:

- Strategic: Develop processes and systems that account for the perpetual existence of unknown bad actors and allow for successful business operation even when humans are compromised.

- Reactive: Structural logging that makes sense in the context of the action. Alerts and detection systems too.

- Reduction: Reduce access to only what is needed, when it is needed.

- Proactive (not always necessary): Multi party approvals (a la code review and production flag changes or ACL changes, too)

- Social: Build a culture of security practices and awareness of bad actors. Don't make people feel guilty or accusatory, just empower them to make good design and process decisions. It's a team sport.

Bonus: By guarding against evil actors, you've also got some freebie good coverage for when an innocent employee gets compromised too!

---

Companies like Google and Amazon do the techniques above. And they don't generally rely on antiquated technology that cannot and will not change to meet the modern standards.

I know because I was the person that built and Google's first time-based access system and rational-based access systems. And multi party approval systems for access. (Fun fact: The organizational challenge is harder than the technical).

And, those strategies work. And they increase SRE resilience too!

---

But even with the best UX, the best security tooling, the best everything, etc there's no guarantees that it matters if we just reject anything except the old system we're used to.

It's like a motorcycle helmet: Only works if you use it.

jasonjayr• 2 years ago

How can a commercial solution prevent backdoors?

A sensitive product like this would have to defend against well funded, patient, well resourced threats, including but not limited to infiltrating an organzation in order to plant code that only a few people may even be able to notice.

2OEH8eoCRo0• 2 years ago

Well for one they need to show up in person. They can't be some anon anime character who hides their identity for totally legitimate reasons.

yjftsjthsd-h2• 2 years ago

Thankfully, we aren't limited to asking leading questions and then hand waving at it; we have a rather lot of empirical evidence. OpenSSH is 24 years old; has it ever been successfully backdoored?

davidfiala8• 2 years ago

We don't know. We won't know the negative case, but we may someday in some circumstance find out the positive (bugged) case.

But we do know some sane things:

- The stakes couldn't be higher.

- Good: Don't allow inbound SSH connections, even through a fancy $100k firewall.

- Best: Don't let people login with SSH (treat SSH like we treat the serial port: a debugging option of last resort)

dijit3• 2 years ago

Reality has shown that the least secure systems tend to be:

A) The ones with financial stakes in the game.

combined with:

B) Completely closed systems.

Contrarily, the most secure seem to be the ones volunteer led, with no financial stakes.

It doesn't matter what you think is true, this is clearly what is consistently happening.

davidfiala8• 2 years ago

tldr; your statement overlooks the reality of businesses with high ethical and financial obligations, like Google, Amazon, and Azure.

- These companies underpin much of the internet's infrastructure.

- Their security practices are far more advanced than typical businesses, with SSH being a heavily restricted last resort. That's not to imply that everyone else shouldn't strive to do meet that (modern) bar too.

- Dedicated teams focus on minimizing access through time-based, role-based, and purpose-based controls.

- They actively develop new security methodologies, often closed-source, but with public evidence of their impact (e.g., https://cloud.google.com/docs/security/production-services-p... ).

- They rarely experience conventional hacks due to reduced blast radius from attacks and insider threats.

- Leading security experts in both major tech companies and niche organizations are driving new strategies and ways to think about security... their focus includes access reduction, resilience, and reliability, regardless of whether the solutions are closed or commercial for them. The ideas spread. (looking at you, Snapchat, for some odd reason)

- This is key: This evolution may not be obvious unless you actively engage with those at the forefront. I think it's what makes people think like the comment above. We cannot see everything.

- It's crucial to recognize that security is a dynamic field... with both open-source and closed-source solutions contributing.

So, the notion that volunteer-led projects are inherently more secure overlooks the significant investments in security made by major corporations that host the internet, and their relative success in doing so. Their advacements are coming to the rest of the world (eventually).

toast05• 2 years ago

Of course not. OpenSSH comes with no warranty, read the license.

Historically, it's been pretty good though.

If you would consider a commercial alternative, consider how much you would need to pay to get an actionable warranty, and consider if you could find someone to do a warrantied audit of OpenSSH (or especially of OpenSSH in your environment) for that amount. It might be a better use of your money.

dns_snek3• 2 years ago

Did you forget to disclose that you're a founder of a YC-backed commercial solution that wants to compete with SSH?

cirrus3• 2 years ago

David Fiala CEO & Founder at Teclada Inc | Former Google Security Leader | Supercomputing PhD

https://www.linkedin.com/company/teclada?trk=public_profile_...

tptacek7• 2 years ago

Please don't post insinuations about astroturfing, shilling, brigading, foreign agents, and the like. It degrades discussion and is usually mistaken. If you're worried about abuse, email hn@ycombinator.com and we'll look at the data.

https://news.ycombinator.com/newsguidelines.html

whydoyoucare• 2 years ago

We must first precisely define "level of security" that is expected from OpenSSH and a commerical version. Only then the discussion about who can guarantee what would make sense.

dessimus• 2 years ago

Has any open source project taken down the majority of single OS install base as quickly as CrowdStrike? Seems like they would have the "dedicated resources and a financial stake" to prevent such as situation.

tptacek7• 2 years ago

OpenSSH, as load-bearing infrastructure for much of the Internet, is heavily sponsored by tech companies. It empirically has one of the best security records in all of general-purpose off-the-shelf software. If for some benighted reason I found myself competing directly with OpenSSH, the very last thing I would pick as a USP would be security.

SoftTalker• 2 years ago

USP is Unique Selling Proposition?

GJim• 2 years ago

Why the hell is a genuine question being downvoted?

Downvoters, what are you trying to achieve?

udev40962• 2 years ago

HN audience is full of hypocrisy and ignorance

gmuslera• 2 years ago

Sometimes commercial companies have "incentives" to put backdoors, like i.e. secret orders by intelligence agencies. Snowden papers and all related information from that time set a baseline on what you may consider safe.

mmsc2• 2 years ago

Juniper had a backdoor in their firmware in 2015 which gave ssh access, inserted by hackers: https://blog.cryptographyengineering.com/2015/12/22/on-junip...

There were some updates: https://www.zdnet.com/article/congress-asks-juniper-for-the-... and according to https://www.reuters.com/article/world/spy-agency-ducks-quest... China was behind it (but without evidence).

Vecr4• 2 years ago

Was there ever a writeup of exactly how the XZ exploit worked? I mean exactly, I get the general overview and even quite a few of the specifics, but last time I checked no one had credibly figured out exactly how all the obfuscated components went together.

4llan• 2 years ago

Gynvael Coldwind made a great analysis about it: https://gynvael.coldwind.pl/?lang=en&id=782

https://news.ycombinator.com/item?id=39878681

xz/liblzma: Bash-stage Obfuscation Explained

mananaysiempre• 2 years ago

That is, as it says in the title, about the Bash-stage obfuscation. That’s fun but it’d also be interesting to know what capabilities the exploit payload actually provided to the attacker. Last I looked into that a month or so ago there were at least two separate endpoints already discovered, and the investigation was still in progress.

kva-gad-fly• 2 years ago

https://www.openwall.com/lists/oss-security/2024/03/29/4

Vecr4• 2 years ago

Yeah, what's posted by you and other users so far is stuff I know, build scripts, injection, obfuscation. I'm more looking for a careful reverse engineering of the actual payload.

toast05• 2 years ago

I think this is sufficiently detailed?

https://lwn.net/Articles/967192/

But if there's a part that's still unclear, maybe there's another writeup somewhere that addresses it?

Lammy• 2 years ago

The social-engineering aspect of pressuring the old maintainer is way more interesting than the actual software IMHO https://securelist.com/xz-backdoor-story-part-2-social-engin...

Vecr4• 2 years ago

I already got all that. Yes, I think it's interesting, but I wanted to see a final (non-interim) analysis of the payload going byte-by-byte.

pdonis2• 2 years ago

I agree 1000% with this. One thing I don't see addressed in the article you reference, though, is whether any OpenSSH maintainers spotted the addition of a co-maintainer to xz utils and did any due diligence about it.

thmstcls4• 2 years ago

Random aside to the other commenter's linked articles, I find it a bit coincidental that the supposed "kill switch" environment variable, yolAbejyiejuvnup=Evjtgvsh5okmkAv, decodes from UTF-16LE to UTF-8 as 潹䅬敢祪敩番湶灵䔽橶杴獶㕨歯歭癁 which google translates to "You can't do it without a soul."

simonmysun2• 2 years ago

Amazing. How did you find it out?

yorwba2• 2 years ago

Any even-length alphabetic ASCII string decodes to random Chinese characters in UTF-16LE. Digits and = unlock some Japanese hiragana, Korean hangeul and assorted punctuation, but those only make up a small fraction of the total.

For example, 'backdoor'.encode('ascii').decode('utf_16_le') == '慢正潤牯', which Google Translate turns into "Slow and positive", but it's just nonsense.

rwmj2• 2 years ago

However, it's interesting to note that in both 2002 and 2024 we got a backdoor rather than a bugdoor.

As far as we know.

Related, there was a pretty interesting backdoor-by-bug attempt on the Linux kernel (at least, one that we know of) back in 2003: https://lwn.net/Articles/57135/

The Linux "bug" was unsophisticated by modern standards, but you could imagine a modern equivalent that's harder to spot:

Make the "bug" happen across several lines of code, especially if some of those lines are part of existing code (so don't appear in the patch being reviewed). Ensure the compiler doesn't warn about it. Make the set of triggering events very unlikely unless you know the attack. It would be very surprising to me if three letter agencies hadn't done or attempted this.

mmsc2• 2 years ago

And in 2010, a similar backdoor appeared in UnrealICRD: https://lwn.net/Articles/392201/. Also in proftpd: https://www.aldeid.com/wiki/Exploits/proftpd-1.3.3c-backdoor. Both were done by ac1db1tch3z who the author of OP's post, Ben Hawkes, got a shoutout from for another local privilege escalation vulnerability from over a decade ago :-).

Anyways, in response to the backdoor in unrealircd, Core Security came up with a "hiding backdoors in plain sight" challenge: https://seclists.org/fulldisclosure/2010/Jul/66

"Bugdoors" are not new, and I'm sure some have been patched without anybody realizing they were introduced maliciously.

baby• 2 years ago

And there was the socat backdoor

cqqxo4zV46cp2• 2 years ago

Oh man! I’d forgotten all about the UnrealIRCd backdoor! I was running an IRC network at the time. What a blast from the past.

fouronnes3• 2 years ago

To think that there's a safe somewhere in a TLA basement with a "how to get root anywhere" tutorial.

bigiain2• 2 years ago

I imagine there's a cross-TLA committee that analyses all the available zerodays, and meets quarterly to prioritise what order and which agencies can burn those zerodays when needed. And probably KPIs for each agency to add new zerodays which move their agency higher up the list to potentially burn the really good ones.

ehhthing• 2 years ago

The problem with these is that bugdoors require you to target way more valuable stuff compared to backdoors. With a backdoor you can target practically any library or binary that is being run with root privileges, while with a bugdoor you can only really target code that is directly interacting with a network connection.

Direct network facing code is much more likely to have stringent testing and code review for all changes, so as of now it seems a bit easier to target codebases with very little support and maintenance compared to an attack that would target higher value code like OpenSSH or zlib.

egberts12• 2 years ago

Ya think that the tests subditectory would contain sufficient "integrity" test cases to ensure that it doesn't fail, but alas ... nooooooo.

tedunangst2• 2 years ago

What does this mean?

saagarjha2• 2 years ago

What exactly would you test here?

egberts12• 2 years ago

Permutation, value range, logic testing ...

CODE COVERAGE!

jmakov• 2 years ago

At this point what makes us think all major contributors are not on a payroll of one or the other state agency? The attack surface of the whole SW supply chain is huge.

Sesse__• 2 years ago

All major, and not a single one of them has leaked it?

Jerrrrrrry2• 2 years ago

Besides the literal constant torrent of evidence?

tedunangst2• 2 years ago

Sounds like a good social experiment. Work your way up to major contributor for a project, see how long until you're approached by the MIB.

mmh0000• 2 years ago

Whenever these stories come up, I like to remind people about the Underhanded C Code Contest[1] and the IOCCC[2]

TL;DR: Clever C programmers can hide very evil shit in plain sight.

[1] https://www.underhanded-c.org/ [2] https://www.ioccc.org/

dkga• 2 years ago

Wow, I was definitely not aware of this.

cortesoft• 2 years ago

What happened to the underhanded c contest? Seems the last one was in 2016?

thelastparadise• 2 years ago

Maybe because absolutely no one trusts any C code anymore.

ruk_booze• 2 years ago

How sad to see that this is no more.

I helped out reviewing the winning contribution[1] by Linus Åkesson, for what I now know was the final contest.

Linus, aka ”LFT”, is a remarkable and truly talented person in so many ways. If you haven’t heard of him before, I suggest you check out all of his projects.

[1] https://www.linusakesson.net/programming/underhanded/2015.ph...

daghamm• 2 years ago

Even further back, someone claimed that a three letter agency had paid some developer to introduce backdoor to openbsd (or possibly openssh).

Theo did not belive this and publicly disputed these claim and even revealed the name of the whistle-blower. But I have always felt the story rang true and Theo sound not have been so dismissing.

Can't find the story, but it should be on the mailing lists somewhere

jmclnx• 2 years ago

I remember this, it was the FBI. OpenBSD people did a huge audit and nothing was found. That was also like 20 years ago.

Also, other articles stated that never happened.

Plus, the "backdoor" in OpenSSH was a Linux only thing possibly related to systemd. It never affected OpenBSD. That is because of Linux people patching OpenSSH with "dependency hell". I believe systemd people is doing something about these dependency chains.

Arch-TK• 2 years ago

The "thing to do" about the dependencies is not to have them in the first place. Distributions where patching OpenSSH to add a libsystemd dependency instead of adding 15 lines of code.

fijiaarone• 2 years ago

So you’re saying more than 1 person was paid to put the back door in.

daneel_w2• 2 years ago

It was talk about a backdoor in OpenBSD's IPsec stack. The software was audited and nothing was found. The person who stepped forward (after being named) claimed on Twitter to have been formerly involved with the FBI and a supposed project of theirs looking into the feasibility of infiltrating the OpenBSD developer sphere in order to plant a backdoor, but that the project never reached planning.

Edit: the discussions and auditing of the IPsec stack happened around 2011 if memory serves me right. The supposed backdoor "happened" a decade earlier. The "agent" was named Greg Perry.

TacticalCoder• 2 years ago

There's another venue for backdoors in most Linux distros. It's insanely complex by design (so that an endless supply of backdoors can be implanted), touches absolutely everything, was required for the XZ backdoor to work, etc.

But it you mention its name, you get downvoted.

bawolff• 2 years ago

Must be the only time checksums were actually useful.

udev40962• 2 years ago

I wonder if every major open source project should adopt the way SQLite operates. No outside contributions at all

imhoguy• 2 years ago

What is the meaning of "outside"? What if SQLite founder decides to step down. In case of `xz` a fresh insider was a malicious actor who was supposed to help keep up the maintenance. I think open-community projects like Linux Kernel have benefits as they won't die with founder interest waning, still there is more eyes to find issues very early.

gorgoiler• 2 years ago

In terms of its level of severity (and all round insanity) attacking OpenSSH with a backdoor is as if someone had hoax-mailed a packet labelled “Anthrax” to every tech business in the world.

Brazenly stupid and pointlessly broad: the motivation could just as easily have been to cause mass societal disruption (“terrorism”) instead of a targeted attack that just happened to sweep up everyone else in its arms.

hi-v-rocknroll2• 2 years ago

My wish is that FOSS software test engineering approaches and tools evolve to appropriately model and formally-verify code behavior in the spirit of what the seL4 project did and maybe further. Similarly, system behavior should also be formally-specified and provable. Until we get there, it's one removed from whipping up untested code ad-hoc and then YOLO'ing to hope it all works out. It's going to take a lot more work and a new way of defining constraints, relationships, and nonvariants but it's unavoidable to prove that code behaves as intended.

PS: Neither "Just rewrite everything in X where X = Rust", "just use fuzzing", or "just use MISRA coding standards" doesn't get us there. Holistic improvements help, but not with the fundamental deficiency above.

weinzierl• 2 years ago

The important word is "just", when X facilitates formal verification by a considerable margin.

MaxBarraclough2• 2 years ago

That would be a great solution, but full formal verification is a very high bar.

The more realistic answer is to use safer languages than C for this sort of critical work. Rust and Ada spring to mind. They could even expose a C ABI/API. From a quick search, it looks like this has already been achieved for TLS, but it sees little real usage. [0] There are also Rust implementations of SSH (Russh and Thrussh), but I don't think they expose a C ABI/API.

I'm surprised this rather obvious solution doesn't seem to even receive serious consideration. I'd be surprised if performance was seriously impaired. This blog post [1] found Rustls to be faster than OpenSSL. I couldn't find a similar performance evaluation for Russh or Thrussh.

[0] https://github.com/rustls/rustls-ffi

[1] https://bencher.dev/learn/case-study/rustls/

hi-v-rocknroll2• 2 years ago

Well, the important things that run the world demand rigor. The "it's a hobby" defense rings hollow when decades of slip-shod processes have repeated led to a class of failures that repeated dozens and dozens of times.

codethief• 2 years ago

Regardless, I'm not convinced we can defend against this with the way we're currently thinking about operating system design.

Amen!

On that note: Why is it so so difficult to set up a (rootless) container/sandbox correctly? (I mean, look at what runc does – shit's incredibly complex!) And why is it next to impossible to nest containers without privileges and arcane knowledge of how some of the underlying kernel syscalls work? Even those 250 lines of code to landlock `make` that the author mentions sound awful to me.

I don't want to have to set up a sandbox for every single application by hand, let alone set up rules for all things that a malicious application could possibly exploit. Instead, I want security-by-default! Have every application run in a tight sandbox by default and let the application specify what permissions it needs, so that I only need to review those and can grant them as I like. Meanwhile, deny access to everything else!

Clearly, we are not (Linux is not) ready for this yet – we lack both a good UI for all of this permission handling and an agreed-upon contract that all application developers can follow.

In fact, for the vast majority of applications we don't even really know (have documented) what permissions / access to kernel syscalls they would need, so it'd be incredibly hard to switch to a principle of least privilege-based approach over night.

But man, one can always dream…

bogota• 2 years ago

Sounds like IAM for the OS and we all know that just leads to wild card permissions everywhere because the developer doesn’t know what they need

Gibbon1• 2 years ago

The solution to that is linkers need to generate those permissions and create a manifest section in the elf file so the OS can handle it transparently.

sylware• 2 years ago

Backdoor, bugdoor or or "convenient" bug... you can spot in the source code...

Now, have a look at machine code which is spit out by compilers... and there, right there... no bug or backdoor in the source code though, but yet... how would you call that? compiler-injected-door?

To fight this you just need to trust and audit those small pieces of s..oftware which are gcc/clang(llvm).

OpenSSH Backdoors

Related Stories