How the XZ Backdoor Works

https://lwn.net/SubscriberLink/967192/6c39d47b5f299a23/

by chmaynard • 2 years ago

200 77 2 years ago

If the backdoor relies on IFUNC then it would not work on distributions using musl. Like the one I use. Always amusing when HN commenters try to belittle use of musl and promote use of glibc. I like musl. I like choice.

lifthrasiir• 2 years ago

IFUNC was probably chosen because most systems run glibc anyway so that is enough for the widespread vulnerability. The exploit would have used other mechanism tailored for musl if it was a majority instead.

realusername• 2 years ago

xz is bundled in millions of combination of OS/architecture/tooling/config, even the attacker had to make a choice.

SV_BubbleTime2• 2 years ago

Lends some credit to the idea that they had a target or series of targets in mind when deploying this seemingly general attack vector.

probably_wrong• 2 years ago

If you really want to bait HN you should point out instead that this backdoor exclusively affected distros using systemd.

dist-epoch• 2 years ago

Hey, you should put a trigger-warning on that

adql• 2 years ago

Backdoor relied more on compromising supply chain than on particular glibc feature. And the fact the notify support in sshd wasn't implemented directly but via libsystemd that is the one that dragged the lzma dependency into SSH

fl0ki3• 2 years ago

A lot of people who don't like musl just prefer not to have the slowest memory allocator on the planet, especially when multiple threads are involved which they usually are in modern code.

I've watched release-optimized builds with musl run orders of magnitude slower than unoptimized builds with mimalloc, and this was on only 20 cores, it wasn't exactly pushing the envelope of big iron scaling.

Even after waiting years for its mallocng, which is better, it is still the slowest. It's no longer just that it wasn't a priority, it's inherent fallout from musl's practice of reinventing everything without learning nearly enough about why other implementations were built that way in the first place.

With memory allocators there were several legendary implementations to learn from. My pick would be mimalloc because it's not just fast, it's also hardened, which seems relevant in a thread about security.

At least once you know this, you can substitute the allocator for your program even if you otherwise use musl. That's common practice when producing static binaries from Rust code.

The problem remains that far too many people just use musl without actually benchmarking their program under a real workload to see just how much they've sacrificed in return for, well, what exactly, because it also wasn't hardening.

_joel• 2 years ago

Agreed, it's not an insignificant addition to the time of a pipeline run (in our use case) and adds up if you're running a good number. Spent a bit of time porting some images to alpine and it wasn't really worth the effort. If you're restricted by device resource, which what it was created for I guess, then fine.

deng4• 2 years ago

Yes, using a niche system is a good way to not getting hacked, but then, why would you use Linux at all? Looks like you'd be happier with OpenBSD, or maybe Plan9?

1vuio0pswjnm75• 2 years ago

Linux is not the only system I use. It is just one option. I compile sshd from source. It's not always OpenSSH, but when it is, it does not include any patches for systemd.

1vuio0pswjnm75• 2 years ago

What HN commenters fail to grasp about what might drive one's choice to use musl: It's not that musl is so good, it's that glibc is so bad. At least this is what drives the choice for me.

fl0ki3• 2 years ago

I'll take this bait in good faith: What about glibc is so bad that you'd rather use the slower, buggier, less compatible, less throughly reviewed musl in production?

If I had to pick something, it'd be that glibc is behind musl on supporting 64-bit timestamps on 32-bit targets; and at least for my targets, that's a stretch.

In return, musl's (past?) problems with domain resolution alone seem like more than enough of a downside to avoid it on all targets.

https://twitter.com/RichFelker/status/994629795551031296 vs https://datatracker.ietf.org/doc/html/rfc7766#section-5

Disclaimer: I'm targeting these from Rust, not C or C++ directly.

somemisopaste2• 2 years ago

(...) the backdoor adds an audit hook. The dynamic linker calls all the registered audit hooks when it is resolving a symbol.

How was this possible without also modifying the LD_AUDIT var? Haven't seen that mentioned yet, or perhaps I'm missing something.

rwmj• 2 years ago

When you're running inside the binary you can do mostly whatever you want. Especially in this case where the back door could run before mprotect(2) has been used to write-protect critical structures like the GOT and PLT (not that that is watertight either).

tgv• 2 years ago

I guess this is the answer (from https://sourceware.org/glibc/wiki/GNU_IFUNC):

Symbols of type STT_GNU_IFUNC (GNU-specific extension) are treated differently from normal symbols. Such IFUNC symbols point to the resolver function, and all calls to such functions are delayed until runtime.

nwellnhof• 2 years ago

It's probably as easy as modifying "extern struct rtld_global_ro _rtld_global_ro", exported from ld-linux, the dynamic linker/loader. During IFUNC resolution this struct seems to be writable.

somemisopaste2• 2 years ago

So in other words LD_AUDIT is useless? If it's that easy to overwrite the GOT I fail to see the purpose in audit functionality.

Topfi• 2 years ago

Great summary, especially since I missed this yesterday. To be honest, it's far above my knowledge, so I'd just like to thank Andres Freund and everyone else for investing their time, effort, and knowledge into keeping all of us secure.

bsmartt• 2 years ago

To be honest, it's far above my knowledge

Just means it's an even better opportunity to learn!

H8crilA3• 2 years ago

Do we have any idea at all who has done it? From what I've read there's no credible attribution in public sources.

kencausey• 2 years ago

No. I'm sure it is being looked into by multiple parties.

bjoli• 2 years ago

Pretty much this. Some of the actors that would be affected by this are as large as can be.

Sshd is used everywhere.

npteljes3• 2 years ago

I read an analysis that creates a profile, but that's as far as public information goes at the moment.

https://lunduke.locals.com/post/5467061/xz-backdoor-i-did-a-...

INTPenis2• 2 years ago

OpenWRT also uploaded backdoored packages.

1oooqooq• 2 years ago

https://forum.openwrt.org/t/project-statement-about-xz-5-6-1...

you might want to tell them that.

INTPenis2• 2 years ago

I think they know seeing as they renamed the files with the .backdoored suffix.

But I think you're referring to the fact that it was dormant code.

I still thought it worth mentioning in the context of a website that seemed to list all distros affected.

Debian sid, Fedora Rawhide, the Fedora 40 beta, openSUSE Tumbleweed, and Kali Linux all briefly shipped the compromised package. NixOS unstable also shipped the compromised version, but was not vulnerable because it does not patch OpenSSH to link libsystemd.

zekica• 2 years ago

Not really, they updated to 5.6.x but used code from the git tag, not from the tarball, and only in their snapshot (not stable) version. The git repository never contained <i>build-to-host.m4</i>, so the injected build step was never executed and not backdoored library was produced.

dmitrygr• 2 years ago

I do now wonder if systemd itself is part of a similar long game by someone. It is part of everything. One compromise there and it is game over.

It has all the signs:

- Replaces old perfectly working systems? Check

- Large and inscrutable? Check

- Touches a lot of things an init system has no business touching? Check

- Pushed in rather suddenly, in a coordinated fashion, and against much reasonable opposition? Check

andrewshadura• 2 years ago

Same old arguments debunked many times already.

1. Old systems it replaced were difficult to use and maintain.

2. It is in fact modular and the code is actually easy to understand. When I ran into a bug, I managed to find and fix it.

3. Systemd itself replaces the init and service management (initscripts etc), that's quite in the scope. Other bits and pieces like timers and resource management, are also in the scope. Other things like network management, are separate things that are only developed under the umbrella of the systemd project.

4. Systemd was not pushed suddenly, it was many years in brewing, and it became gradually adopted when it was actually ready for production use.

swader999• 2 years ago

Real time sophisticated astro turfing system in place to stifle any criticism - Check.

Just kidding, I think the defense given holds up.

npteljes3• 2 years ago

One compromise there and it is game over.

Consider that every agency like the NSA, and private groups like the NSO Group (authors of the Pegasus spyware) already sit on a pile of unpatched vulnerabilities, for various systems. If one vuln in systemd is game over, then the game is already over. And if what we currently experience is the game over state, then it's manageable.

fl0ki3• 2 years ago

It looks manageable to most of us because we don't know who was targeted and what would have happened if they weren't.

We don't even know what whistleblowers were caught before they could publish their information, and we don't know what information that would have been.

Imagine if nobody ever leaked what Edward Snowden did. Imagine that something similarly important, anywhere in the world, has not been leaked because of successfully exploited vulnerabilities.

throwaway4good2• 2 years ago

Is the original git repository still available somewhere?

rrdharan• 2 years ago

Yes: https://git.rootprojects.org/root/xz

deng4• 2 years ago

Note the malicious m4 build scripts were not checked into git, but only put in the released tar balls. You can see the original content here:

https://salsa.debian.org/debian/xz-utils/-/tree/debian/5.6.0...

throwaway4good2• 2 years ago

The test file containing the backdoor seems to be here:

https://git.rootprojects.org/root/xz/commit/6e636819e8f07033...

coleca• 2 years ago

This may be a dumb question, but is law enforcement investigating this? Is it even technically a crime?

H8crilA3• 2 years ago

Of course it is a crime. This is in fact more than a crime, it's a counter-intelligence problem, even if done by a non-state actor.

magic_hamster• 2 years ago

This is 100% a state actor. We can also kind of narrow down who.

alickz• 2 years ago

What would the crime be? Misuse of computers? Espionage?

Curious as to the legal angle of it

brookst• 2 years ago

I’m not sure that all counter-espionage problems are necessarily crimes, in the sense that a specific law was violated.

lambersley• 2 years ago

In Canada, it would fall into a number of federal laws (Criminal Code)

Unauthorized use - 342(1) Mischief in Relation to Data - 430(1.1) Interception private communications - 184(1) Deceit/fraud - 380(1)

1. https://laws-lois.justice.gc.ca/eng/acts/C-46/section-342.1.... 2. https://laws-lois.justice.gc.ca/eng/acts/c-46/section-430.ht... 3. https://laws-lois.justice.gc.ca/eng/acts/c-46/section-184.ht... 4. https://laws-lois.justice.gc.ca/eng/acts/c-46/section-380.ht...

EasyMark• 2 years ago

Doubtful if law enforcement is, you can bet that the CIA and NSA and SS are looking into it though hoping to find a thread to pull on the sweater.

smitelli• 2 years ago

The file, supposedly a corrupted XZ file, is actually a valid XZ stream with some bytes swapped — for example, 0x20 is swapped with occurrences of 0x09 and vice versa.

My under-caffeinated brain can't unsee that as a silly reference to the old tabs vs. spaces squabble. Maybe it's simply a rational obfuscation choice, or maybe it's indeed a masterstroke of satire.

bonyt• 2 years ago

Wait, you're right, 0x20 is space and 0x09 is ht/tab... they've swapped spaces for tabs...

pdpi• 2 years ago

The payload is supposedly a corrupted xz file. An overzealous "replace spaces with tabs" formatter could easily cause this sort of corruption, so it seems like a perfectly reasonable test case.

acdha• 2 years ago

I think it’s a very shrewd use of that history: their goal was to get a potential reviewer to ignore it and pretty everyone qualified to look at that code has seen those specific values enough to likely tune it out as more tabs vs. spaces noise or perhaps think the original test case started with a file corrupted by some whitespace conversion process, which would be a very easy explanation since very similar things have legitimately happened to other projects.

Whoever did this shows signs of being very familiar with programming culture and I’d bet that this was another deliberate attempt to hide in the background noise.

y_xc2• 2 years ago

"The exploit was caught promptly, so almost no users were affected. Debian sid, Fedora Rawhide, the Fedora 40 beta, openSUSE Tumbleweed, and Kali Linux all briefly shipped the compromised package."

One thing that is noteworthy is that Kali Linux integrated the backdoor briefly.

So we should have a discussion about security of distros packaging speed. Is it more secure to integrate all patches as fast as possible or is it more secure to wait a bit until it is being tested elsewehere. The "elsewhere" would catch the bad effects then. Integrating security patches as fast as possible could be very costy.

hypeatei• 2 years ago

It probably depends on what's being updated in the new version. If it's a vulnerability fix, then fast is good. If not, then waiting a little bit is ideal.

Not sure how that would work in an automatic way without some sort of tagging system marking it as "critical" but that could be abused by bad actors too.

greggsy2• 2 years ago

Most major distros include mechanisms to install packages not security updates, either via the package manager, or through specific repositories.

ajross2• 2 years ago

Is it more secure to integrate all patches as fast as possible or is it more secure to wait a bit until it is being tested elsewehere.

That's a chicken/egg argument. Someone needs to be "elsewhere". Integration testing is absolutely part of security defense in depth architecture. In point of fact this was caught by running Debian testing. Delaying inclusion would have delayed discovery.

y_xc2• 2 years ago

Exactly that is my fear, too. It's like zero trust in everything. - Don't trust that any other party do the security testing for you - Don't trust any developer, it could have malicious intents - Don't trust the patches, they could introduce bugs/backdoors/...

So we're straight going in the direction of protectionism and isolation. Wouldn't that be a big step back for the whole OSS/FOSS community?

One thing I read the xc-writeups last days (I think it was here https://gist.github.com/thesamesam/223949d5a074ebc3dce9ee78b...) was that we at least shouldnt trust any devs/maintainers that have no online identity. Though the argument that you could create a fake ID is possible. So I'm still obsessed from the question, what are the lessons learned for the community from this and how restrictive the participation in OSS will become.

jeltz3• 2 years ago

Wasn't Debian testing also affected? That should have affected some people's PCs.

deng4• 2 years ago

Yes, it was. And by that, for instance also all docker images that derive from debian:testing, which is not that uncommon.

plg94• 2 years ago

that was the whole reason this was caught this early, because Andres Freund was runnig Debian Testing (or Unstable)

miraculixx• 2 years ago

Next attack will create a CVE and provide a "solution". Nothing spreads faster.

1vuio0pswjnm75• 2 years ago

The backdoor relies on IFUNC which means it does not work on distributions using musl. Like the one I use. Always amusing when HN commenters try to belittle use of musl and promote use of glibc. I like musl. I like choice.

How the XZ Backdoor Works

Related Stories