Deep Dive into XZ Utils Backdoor [video]

https://www.youtube.com/watch?v=Q6ovtLdSbEA

by peacefullemon • 2 years ago

216 84 2 years ago

xyst7• 2 years ago

I would have paid money to see the look on the team’s face behind this back door as they realize the years of effort and planning go down the drain because of 1 person doing due diligence.

throwaway4good7• 2 years ago

Speaking of that. Any updates on finding the person(s) behind this attack?

INTPenis2• 2 years ago

Nothing but speculation so far based on the times they were active. No real signatures left behind.

All we know is they were active during business hours of the UTC+1/3 timezones. They used a chinese sounding name as one of their front figures, the one who made most of the commits of malicious code. They wrote english messages that some people claim appear to be a Russian trying to write in English. But that last argument is pretty weak as they could have just masked their origin using Google translate, or even ChatGPT.

One of the biggest clues is actually one little commit that they let slip with a middle name to the chinese persona. And some people claim the names don't match up with different chinese cultures.

After that you can use all your psych 101 experience to try and analyze them but you'll just be guessing.

Personally I think using a chinese front name is a very interesting detail. They could have used any name in the world, there are open source contributors from all over the world.

Using a name in a country where most people have a very little insight is a good way of not being uncovered as fake of course. But why not use a Ugandan name? Also using a Chinese name can be perceived as pointing the blame at China, which might be sensitive geopolitically.

mcmoor• 2 years ago

Is there anyone who have made complication on the most famous (searched by the whole world) anonymous people. No 1 must be Satoshi Nakamoto, and No 2 maybe this one? Is there any others?

rwmj• 2 years ago

CISA (https://cisa.gov) are investigating. They got the emails between me and "Jia Tan" and other information we have. They can subpoena Google to get IP addresses behind the emails, but it depends on whether and where a VPN was used if they'll get any further than that. If we will ever see a fully public report is anyone's guess.

vsnf• 2 years ago

"Aw fuck. Oh well, the Chrono, Boost, libcurl and libsodium backdoors are still active. Can't believe they found them lol"

chx2• 2 years ago

While libcurl is an alluring target it is really ill suited for this, Daniel Stenberg is paid to work on it full time and his track record speaks. The situation is really different from xz.

lrvick2• 2 years ago

I am sure they are happy this one was noticed instead of one of the dozen worse ones they got in years ago.

nan60• 2 years ago

Amazing lecture. Really awesome to hear more about the nitty gritty of this whole debacle.

vasco2• 2 years ago

A reasonable person would expect many more open source projects have been infiltrated and have similar exploits waiting to be found. Hard to prove until the next one is found, but if a second one like this is discovered we probably have a huge problem on our hands.

Since the part that takes longest is getting commit rights and there's a lot of waiting to do, what are the chances the attackers put all their eggs on infiltrating one specific project? I'd be trying to infiltrate at least 10 or 20 even if I were alone, to increase my chances. If you dedicate one or two years to build reputation with the current maintainers by doing a PR here and there, you can do that for a few projects in parallel.

_nalply• 2 years ago

I would start with projects where the maintainer(s) suddenly got a very helpful contributor. I am sure there's a typical pattern. Then if a maintainer seems suspect investigate a little bit more. Here some balance is needed because probably many maintainers are not Mallory.

hyperman1• 2 years ago

Assuming this would be smart to do. But xz is a very good target. It seems a lot of care went into selecting it. Consider:

* It is extremely critical. Used at multiple places in the boot process, with root or kernel mode

* Nevertheless an unnoticeable dependency. Nobody thougt about its security implications

* Stable and trustworthy for years (decades?)

* Managed mostly by 1 person

* That person was more than average vulnerable for pressure, but not in a very visible way.

vasco2• 2 years ago

I'm not saying it was a bad target. It was a perfect target. How many more would fullfil these criteria? Those would be the type of repositories I'd go and investigate.

For there to be another repository affected by the same attack you don't even need to think this specific attacker did it twice. You can just consider that multiple attackers had the same idea. I think it's way more likely that there's more repositories affected by the same type of attack than not.

I'd go as far as saying this is an issue of national / global security and there's space for an agency like the IMF but for software security that would be funded by all the countries of the world to bring more resources and assuredness to this type of dependencies.

Because even if this attacker didn't do more than N=1 and other attackers didn't have the idea before, they surely have it now.

geertj• 2 years ago

I would add to that list:

* Has random binary test files in its repo, ideal for hiding exploit payloads.

honzaik• 2 years ago

I watched the TC interview with Durov (Telegram) and apart from it being a big Telegram ad what struck me is that he told a story about his employee being approached by "an intelligence agency" and asked to reveal information about what open-source libraries they use.

It is kind of strange since their apps are supposed to be open source, maybe he meant the backend? Nonetheless, it seems this has been their modus operandi for a long time.

bayindirh2• 2 years ago

Remember that NSA is openly interested in systemd and how it works. It's a double edged sword. They wanted to be sure that it's hardened as they like it, and note any "useful features" that might come handy later.

The thing is, as computers proliferate and we start to use them in more places, the effects of possible holes moves closer to our homes. From distant infra to near infra; from borderlines to our homes and transportation we use everyday. Even to our pockets via smartphones and other smart devices we host in our homes.

screwt• 2 years ago

Yes, this has happened. See this example from OpenJSF:

"The OpenJS Foundation Cross Project Council received a suspicious series of emails with similar messages, bearing different names and overlapping GitHub-associated emails. These emails implored OpenJS to take action to update one of its popular JavaScript projects to “address any critical vulnerabilities,” yet cited no specifics" [0,1].

[0] https://openjsf.org/blog/openssf-openjs-alert-social-enginee... [1] https://www.schneier.com/blog/archives/2024/04/other-attempt...

michelsedgh• 2 years ago

Its a year from now, 2025, you open Hacker News and see on the front page another article or lecture about XZ. Maybe 2 years from now who knows :_)

xyst7• 2 years ago

Better to get this information out there to as many people. Heightened awareness of the problem is always a good thing.

stabbles2• 2 years ago

`sudo ldd /usr/sbin/sshd` why would you put `sudo ldd` on a slide.

xyst7• 2 years ago

The lecturer did state this was very dangerous and should never be done in a real machine.

ldd has an unfortunate side effect of actually executing code within some libraries (the exact details, I don’t know. Maybe it’s library specific)

stabbles2• 2 years ago

On very old glibc ldd used to set an env variable and run the executable. The assumption being that PT_INTERP is set to glibcs dynamic linker. The dynamic linker would dump resolved libraries and stop execution if the variable was set. My guess is they wanted one and only one code path, whereas a separate resolver in `ldd` could get out of sync with the dynamic linker and therefore report incorrect results. But obviously binaries do not have to have an interpreter set at all, or they may point it to any other executable that's not glibcs dynamic linker, and the result is it would just execute like normal.

The fact that the lecturer is aware of it makes `sudo ldd` look even more odd.

not_a_dane• 2 years ago

looks like this is now the next-stuxnet

xyst7• 2 years ago

More like a wannabe stuxnet.

Whoever was trying to get this into the wild was really sloppy in execution of an otherwise “beautiful” attack.

Stuxnet effectively put a stopgap in Iranian’s nuclear program. This backdoor could barely make it out of the alpha phase of development.

infogulch2• 2 years ago

They may have been rushed. Elsewhere itt:

A patch was ready to be introduced into Linux to prevent this type of attack (dynamically loading unneeded libraries).

spacemanspiff01• 2 years ago

So do we know what the cause of the 500 ms slowdown was?

It seems crazy to me that they spend so much time keeping this hidden and then never check for performance degradation.

glandium• 2 years ago

AIUI, it was the dynamic loader audit hook checking symbol names against strings stored in a trie as a way to hide them. Tons of symbols, and slow process for each one.

thelastparadise• 2 years ago

AIUI?

aiiotnoodle• 2 years ago

The postgres developer had disabled a bunch of optimization in order to create a low noise floor on his test machine that increased the delay too.

Normally it would be much quicker but potentially could have been picked up downstream by other developers when the code became more mainstream.

He did an interview for Risky Business #743.

xyst7• 2 years ago

My working theory is they got sloppy. A patch was ready to be introduced into Linux to prevent this type of attack (dynamically loading unneeded libraries).

They were rushing to get it out into the wild and thus made many mistakes, including the performance degradation.

From what I recall, some distros were reporting build errors in bleeding edge versions. Actors were essentially debugging issues in production, and recommending to distro maintainers ways to “fix” (really hide) the build issue with xz

EasyMark3• 2 years ago

I've worked on long projects for a while, you can be relatively invisible for a year and let them know you're getting close (meaning a couple months) and the management will freak out and want it yesterday if it's near the end of a quarter and they think it will look good on quarterly reports. I assume the same can happen with government actors and "security companies".

jrpelkonen• 2 years ago

I hope that one positive development from this saga will be that open source users do not want to been as the next Jugar Kumar and choose to act less entitled and demanding towards maintainers.

xyst7• 2 years ago

Is it really hard to just ignore those types of people? I would say it’s a 1 in 200 interaction (unless the project was recently hyped up somewhere)

I have just closed issues where the reporter or “user”, as you put it, is just unhelpful, rude, or non responsive.

No response needed most of the time.

EasyMark3• 2 years ago

I've never minded ignoring -anyone- who wasn't paying me if they're being disrespectful. I try my best generally, so I don't feel guilty, and if that's not good enough, they can go pound sand and cry about it; a good cry can definitely help you when you're at peak emotions is my thinking. A lot of people have other things going on and just snap at the smallest thing when they get frustrated.

lenerdenator2• 2 years ago

Worth a watch. There's also this GitHub where there's reverse engineering going on: https://github.com/smx-smx/xzre

We now know the answer to the question of "who would win?"

1) A years-long nation-state-backed hacking effort to infiltrate a software project and compromise most servers in the Western world

2) A German's obsession with efficiency and precision in engineering

optimalsolver• 2 years ago

nation-state-backed

Is that actually known?

xyst7• 2 years ago

From what I have read so far: speculative, at best.

It’s all based on the sophistication, the level of psychological manipulation, years of effort, multiple actors, and a working theory based on commit timings.

lenerdenator2• 2 years ago

Generally speaking, with attacks of this sophistication, if it's not a state-backed APT, it's someone who operates with the unofficial backing of a government to give plausible deniability.

I suppose it could be a few amateurs who took on a passion project, but... it's unlikely.

infogulch2• 2 years ago

I wonder if the certificate has shown up anywhere.

Deep Dive into XZ Utils Backdoor [video]

Related Stories