Backdoor in upstream xz/liblzma leading to SSH server compromise

https://www.openwall.com/lists/oss-security/2024/03/29/4

by rkta • 2 years ago

4549 1773 2 years ago

Rucadi• 2 years ago

Saw this on nix, which was using a compromised version in the unstable channel, I hope not too many systems are affected.

jeffbee4• 2 years ago

Safety through obscurity and weirdness! If you disable ifunc, like any sensible person, this backdoor disables itself.

prydt• 2 years ago

I'm curious now. What is ifunc? (Had difficulty finding it through a search)

jeffbee4• 2 years ago

ifunc is a GNU method of interposing function calls with platform-optimized versions of the function. It is used to detect CPU features at runtime and insert, for example, AVX2-optimized versions of memcmp. It is seen in crypto a lot, because CPUs have many crypto-specific instructions.

However, I don't like it much and I think software should be compiled for the target machine in the first place. My 1 hardened system that is reachable from the public network is based on musl, built mostly with llvm, and with ifunc disabled.

aaronmdjones• 2 years ago

https://sourceware.org/glibc/wiki/GNU_IFUNC

canada_dry• 2 years ago

Is there a way to easily/reliably disable ifunc globally on a system (e.g. ubuntu/debian) without breaking a bunch of things?

FYI this looks for pkgs with liblzma:

dpkg -l |grep liblzma

Versions >= 5.6 are compromised

selckin• 2 years ago

https://github.com/google/oss-fuzz/pull/10667

travem• 2 years ago

Interesting, I used https://ossinsight.io/analyze/JiaT75 to identify contributions from the account used by author of the backdoor. It looks like the account made other potentially problematic contributions to other projects.

The disabling of ifunc in this PR against Google's oss-fuzz project maybe one way they tried to prevent this particular backdoor being flagged by that tool? https://github.com/google/oss-fuzz/pull/10667

mxmlnkn2• 2 years ago

There is a related issue for LLVM/clang by this person:

https://github.com/llvm/llvm-project/issues/63957

afh14• 2 years ago

Interestingly: https://github.com/tukaani-project/xz/issues/62#issuecomment...

_mlbt• 2 years ago

openssh does not directly use liblzma. However debian and several other distributions patch openssh to support systemd notification, and libsystemd does depend on lzma.

It looks to be limited to Linux systems that are running certain patches. macOS and BSD seem unaffected?

delphij• 2 years ago

FreeBSD is not affected as the payloads in question were stripped out, however we are looking into improvements to our workflow to further improve the import process.

rasengan• 2 years ago

One portion of the backdoor is solely in the distributed tarballs. For easier reference, here's a link to debian's import of the tarball, but it is also present in the tarballs for 5.6.0 and 5.6.1:

Ubuntu 22.04 version:

dpkg -l |grep liblzma ii liblzma5:amd64 5.2.5-2ubuntu1 amd64 XZ-format compression library

Whew!

yogorenapan2• 2 years ago

Very strange behavior from the upstream developers. Possible government involvement? I have a feeling LANG is checked to target servers from particular countries

_lvbh• 2 years ago

One thing to note is that the person that added the commits only started contributing around late 2022 and appears to have a Chinese name. Might be required by law to plant the backdoor.

That would be quite scary considering they have contributed to a wide variety of projects including C++ https://learn.microsoft.com/en-us/cpp/overview/whats-new-cpp...

yorwba2• 2 years ago

I don't think you need to worry about the C++ contribution: https://github.com/MicrosoftDocs/cpp-docs/commit/9a96311122a...

ComputerGuru2• 2 years ago

No one is being “required by law” to add vulnerabilities, it’s more likely they are foreign agents to begin with.

8organicbits2• 2 years ago

appears to have a Chinese name

Given the complexity of the attack, I'd assume the name is fake.

bdd8f1df777b2• 2 years ago

I would think a Chinese state hacker would not assume a Chinese name, just in case he was discovered like now.

dotty-• 2 years ago

The contribution to C++ is just a simple markdown change: https://github.com/MicrosoftDocs/cpp-docs/pull/4716 C++ is fine.

anarazel5• 2 years ago

LANG only needs to have some value, the concrete value does not seem to matter.

cf100clunk3• 2 years ago

  I am *not* a security researcher, nor a reverse engineer.  There's lots of
  stuff I have not analyzed and most of what I observed is purely from
  observation rather than exhaustively analyzing the backdoor code.

I love this sort of technical writing from contributors outside the mainstream debugging world who might be averse to sharing. What an excellently summarized report of his findings that should be seen as a template.

anarazel5• 2 years ago

FWIW, it felt intimidating as hell. And I'm fairly established professionally. Not sure what I'd have done earlier in my career (although I'd probably not have found it in the first place).

internetter• 2 years ago

Not sure what I'd have done earlier in my career

To anybody in this sorta situation, you should absolutely share whatever you have. It doesn’t need to be perfect, good, or 100% accurate, but if there’s a risk you could help a lot of people

RockRobotRock2• 2 years ago

I hope you've hired a PR person for all the interviews :)

aerhardt• 2 years ago

This story is an incredible testament to how open-source software can self-regulate against threats, and more broadly, it reminds us that we all stand on the shoulders of contributors like you. Thank you!

bonzini13• 2 years ago

For what it's worth the author is a PostgreSQL committer, he's not a security researcher but he's a pretty damn good engineer!

vhiremath42• 2 years ago

Honestly, you only get this kind of humility when you're working with absolute wizards on a consistent basis. That's how I read that whole analysis. Absolutely fascinating.

agwa5• 2 years ago

openssh does not directly use liblzma. However debian and several other distributions patch openssh to support systemd notification, and libsystemd does depend on lzma.

The systemd notification protocol could have been as simple as just writing a newline to a pipe, but instead you have to link to the libsystemd C library, so now security-critical daemons like openssh have additional dependencies like liblzma loaded into their address space (even if you don't use systemd as PID 1), increasing the risks of supply chain attacks. Thanks, systemd.

capitainenemo3• 2 years ago

FWIW, I did a quick check on a Devuan system. The sshd in Devuan does link to a libsystemd stub - this is to cut down on their maintenance of upstream packages. However that stub does not link to lzma.

cf100clunk3• 2 years ago

On an MX Linux (non-systemd Debian-derived distro) box I ran ldd on /sbin/ssh and also ran:

[EDIT: this string gives cleaner results:]

  lsof -w -P -T -p $(pgrep sshd)|grep mem

and saw liblzma in the results of both, so there is some sort of similar trickery going on.

delroth• 2 years ago

The systemd notification protocol could have been as simple as just writing a newline to a pipe

It basically is. libsystemd links to liblzma for other features not related to notifications.

(The protocol is that systemd passes the path to a unix socket in the `NOTIFY_SOCKET` env variable, and the daemon writes "READY=1" into it.)

agwa5• 2 years ago

Is that protocol documented/stable? For whatever reason, daemons are choosing to link to libsystemd instead of implementing it themselves.

It doesn't matter that libsystemd links to liblzma for other reasons. It's still in the address space of any daemon that is using libsystemd for the notification protocol.

tonyg• 2 years ago

Strange protocol. Why not pass a path to a file that should be `touch`d and/or written to, I wonder? Would avoid the complexity of sockets.

wiml3• 2 years ago

libsystemd links to liblzma for other features not related to notifications

Which is pretty emblematic of systemd's primary architectural fault!

yrro2• 2 years ago

The funny thing is that libsystemd _used_ to be split into several different libraries. I certainly remember libsystemd-journal (which is presumably the part of libsystemd that pulls in liblzma) being separate to libsystemd-daemon (which is the part that implements sd_notify, as used by OpenSSH [after patching by distros]).

If that split had never happened, then liblzma wouldn't have ended up being linked into sshd...

fullstop10• 2 years ago

Also thanks to Debian for modifying openssh.

cassianoleal5• 2 years ago

You're not wrong. Had Debian not patched it in this way, OP might have never found it, leaving all other distros who do the same vulnerable.

Note that OP found this in Debian sid as well, which means it's highly unlikely this issue will find its way into any Debian stable systems.

Jasper_• 2 years ago

That is all the protocol is. From https://www.freedesktop.org/software/systemd/man/latest/sd_n...:

These functions send a single datagram with the state string as payload to the socket referenced in the $NOTIFY_SOCKET environment variable.

The simplest implementation (pseudocode, no error handling, not guaranteed to compile), is something like:

    const char *addrstr = getenv("NOTIFY_SOCKET");
    if (addrstr) {
        int fd = socket(AF_UNIX, SOCK_DGRAM, 0);
        struct sockaddr_un addr = { .sun_family = AF_UNIX };
        strncpy(addr.sun_path, sizeof(addr.sun_path), addrstr);
        connect(fd, (struct sockaddr*) &addr);
        write(fd, "READY=1");
        close(fd);
    }

throwaway71271• 2 years ago

goddamnit leftpad got us too :)

iforgotpassword5• 2 years ago

This is what I did for a daemon I'm maintaining. Type=notify support was requested but I'm really allergic to adding new libs to a project until they really do some heavy lifting and add enough value. I was pleasantly surprised the protocol was that simple and implemented it myself. I think systemd should just provide a simple standalone reference implementation and encourage people to copy it into their project directly. (But maybe they already do, I did that almost a decade ago IIRC when the feature was relatively new.)

Repulsion951314• 2 years ago

Whoops, you forgot `vsock:`, `@`, `SO_PASSCRED` (I think)... oh and where is that example provided? But yep that's all the protocol is for sure (and forever)!

b1125• 2 years ago

One of the objections that many people do not understand, is that systemd adds complexity. Unnecessary complexity. Boats full, loads full, mountains full of complexity.

Yes, there are things delivered with that complexity. However, as an example, sysvinit is maybe, oh, 20k lines of code including binaries, heck including all core init scripts.

What's systemd? 2M lines? It was >1M lines 4+ years ago.

For an init system, a thing that is to be the core of stability, security, and most importantly glacial, stable change -- that is absurdly complex. It's exceedingly over engineered.

And so you get cases like this. And cases like that, and that over there, and that case over there too. All which could not exist, if systemd didn't try to overengineer, over complicate everything.

Ah well. I'm still waiting for someone to basically fork systemd, remove all the fluff (udev, ntp, dns, timers, restart code, specialized logging, on and on and on), and just end up with systemd compatible service files.

But not yet. So... well, oh well.

gavinhoward4• 2 years ago

I have a design in the works to do just this.

The problem? It's on the backburner because I don't think I could find a business model to make money from it.

I don't think offering support for a price would work, for example.

jorvi• 2 years ago

Ah well. I'm still waiting for someone to basically fork systemd, remove all the fluff (udev, ntp, dns, timers, restart code, specialized logging, on and on and on)

Most of the things you named there are modular and can be easily disabled.

Furthermore, udev precedes systemd and systemd has in fact its own replacement for it (though the name escapes me).

Kind of a classic, people loving harping on systemd without properly understanding it.

matheusmoreira7• 2 years ago

Systemd is actually pretty damn good and it's GPL licensed free software.

I understand that people don't like the way it seems to work itself into the rest of Linux user space as a dependency but that's actually our own fault for not investing the man power that Red Hat invests. We have better things to do than make our own Linux user space and so they have occupied that niche. It's free software though, we always have the freedom to do whatever we want.

By the way, all the stuff you mentioned is not really part of the actual init system, namely PID 1. There's an actual service manager for example and it's entirely separate from init. It manages services really well too, it's measurably better than all that "portable" nonsense just by virtue of using cgroups to manage processes which means it can actually supervise poorly written double forking daemons.

dralley6• 2 years ago

This is a bit like complaining that the Linux kernel has 30 million lines of code, while ignoring that 3/4 of that is in hardware support (drivers) or filesystems that nobody is actually required to use at any given time.

systemd is a collection of tools, one of which is an init system. Nobody accused GNU yes of being bloated just because it's in a repository alongside 50 other tools.

bananapub9• 2 years ago

One of the objections that many people do not understand, is that systemd adds complexity. Unnecessary complexity. Boats full, loads full, mountains full of complexity.

this is and always has been such a dumb take.

if you'd like to implement an init (and friends) system that doesn't have "unnecessary complexity" and still provides all the functionality that people currently want, then go and do so and show us? otherwise it's just whinging about things not being like the terrible old days of init being a mass of buggy and racey shell scripts.

marcosdumay• 2 years ago

As long as Gnome requires bug-compatibility with systemd, nobody will rewrite it.

quotemstr• 2 years ago

One of the objections that many people do not understand, is that systemd adds complexity. Unnecessary complexity. Boats full, loads full, mountains full of complexity.

Complexity that would otherwise be distributed to a sea of ad-hoc shell scripts? systemd is a win

bennyhill• 2 years ago

so now security-critical daemons like openssh have additional dependencies like liblzma

Systemd itself seems security-critical to me. Would removing other dependencies on libsystemd really make a secure system where systemd was compromised through its library?

agwa5• 2 years ago

1. systemd (at least the PID 1 part) does not talk to the network, so a remotely-accessible backdoor would need to be more complex (and thus more likely to be detected) than a backdoor that can be loaded into a listening daemon like openssh.

2. You can run Debian systems without systemd as PID 1, but you're still stuck with libsystemd because so many daemons now link with it.

poettering2• 2 years ago

Uh. systemd documents the protocol at various places and the protocol is trivial: a single text datagram sent to am AF_UNIX socket whose path you get via the NOTIFY_SOCKET. That's trivial to implement for any one with some basic unix programming knowledge. And i tell pretty much anyone who wants to listen that they should just implement the proto on their own if thats rhe only reason for a libsystemd dep otherwise. In particular non-C environments really should do their own native impl and not botjer wrapping libsystemd just for this.

But let me stress two other things:

Libselinux pulls in liblzma too and gets linked into tons more programs than libsystemd. And will end up in sshd too (at the very least via libpam/pam_selinux). And most of the really big distros tend do support selinux at least to some level. Hence systemd or not, sshd remains vulnerable by this specific attack.

With that in mind libsystemd git dropped the dep on liblzma actually, all compressors are now dlopen deps and thus only pulled in when needed.

o11c5• 2 years ago

Deferring the load of the library often just makes things harder to analyze, not necessarily more secure. I imagine many of the comments quoting `ldd` are wrongly forgetting about `dlopen`.

(I really wish there were a way to link such that the library isn't actually loaded but it still shows in the metadata, so you can get the performance benefits of doing less work but can still analyze the dependency DAG easily)

iforgotpassword5• 2 years ago

And i tell pretty much anyone who wants to listen that they should just implement the proto on their own if thats rhe only reason for a libsystemd dep otherwise.

That's what I think too. Do the relevant docs point this out too? Ages ago they didn't. I think we should try to avoid that people just google "implement systemd notify daemon" and end up on a page that says "link to libsystemd and call sd_notify()".

agwa5• 2 years ago

And i tell pretty much anyone who wants to listen that they should just implement the proto on their own if thats rhe only reason for a libsystemd dep otherwise

Could you point out where the man page (https://www.freedesktop.org/software/systemd/man/latest/sd_n...) says this?

b1125• 2 years ago

And will end up in sshd too (at the very least via libpam/pam_selinux).

Inaccurate.

It's not pulled in on any sysvinit Debian system I run. It is on stable, oldstable, and oldoldstable systems via systemd.

Not systemd:

# ldd $(which sshd) linux-vdso.so.1 (0x00007ffcb57f5000)

        libcrypt.so.1 => /lib/x86_64-linux-gnu/libcrypt.so.1 (0x00007fbad13c9000)

        libwrap.so.0 => /lib/x86_64-linux-gnu/libwrap.so.0 (0x00007fbad13bd000)

        libaudit.so.1 => /lib/x86_64-linux-gnu/libaudit.so.1 (0x00007fbad138c000)

        libpam.so.0 => /lib/x86_64-linux-gnu/libpam.so.0 (0x00007fbad137a000)

        libsystemd.so.0 => /lib/x86_64-linux-gnu/libsystemd.so.0 (0x00007fbad12d5000)

        libselinux.so.1 => /lib/x86_64-linux-gnu/libselinux.so.1 (0x00007fbad12a5000)

        libgssapi_krb5.so.2 => /lib/x86_64-linux-gnu/libgssapi_krb5.so.2 (0x00007fbad1253000)

        libkrb5.so.3 => /lib/x86_64-linux-gnu/libkrb5.so.3 (0x00007fbad1179000)

        libcom_err.so.2 => /lib/x86_64-linux-gnu/libcom_err.so.2 (0x00007fbad1173000)

        libcrypto.so.3 => /lib/x86_64-linux-gnu/libcrypto.so.3 (0x00007fbad0c00000)

        libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fbad1154000)

        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fbad0a1f000)

        libnsl.so.2 => /lib/x86_64-linux-gnu/libnsl.so.2 (0x00007fbad1137000)

        libcap-ng.so.0 => /lib/x86_64-linux-gnu/libcap-ng.so.0 (0x00007fbad112f000)

        libcap.so.2 => /lib/x86_64-linux-gnu/libcap.so.2 (0x00007fbad1123000)

        /lib64/ld-linux-x86-64.so.2 (0x00007fbad156a000)

        libpcre2-8.so.0 => /lib/x86_64-linux-gnu/libpcre2-8.so.0 (0x00007fbad1089000)

        libk5crypto.so.3 => /lib/x86_64-linux-gnu/libk5crypto.so.3 (0x00007fbad09f2000)

        libkrb5support.so.0 => /lib/x86_64-linux-gnu/libkrb5support.so.0 (0x00007fbad09e4000)

        libkeyutils.so.1 => /lib/x86_64-linux-gnu/libkeyutils.so.1 (0x00007fbad09dd000)

        libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007fbad09cc000)

        libtirpc.so.3 => /lib/x86_64-linux-gnu/libtirpc.so.3 (0x00007fbad099e000)

systemd:

# ldd $(which sshd) linux-vdso.so.1 (0x00007ffc4d3eb000)

        libcrypt.so.1 => /lib/x86_64-linux-gnu/libcrypt.so.1 (0x00007feb8aa35000)

        libwrap.so.0 => /lib/x86_64-linux-gnu/libwrap.so.0 (0x00007feb8aa29000)

        libaudit.so.1 => /lib/x86_64-linux-gnu/libaudit.so.1 (0x00007feb8a9f8000)

        libpam.so.0 => /lib/x86_64-linux-gnu/libpam.so.0 (0x00007feb8a9e6000)

        libsystemd.so.0 => /lib/x86_64-linux-gnu/libsystemd.so.0 (0x00007feb8a916000)

        libselinux.so.1 => /lib/x86_64-linux-gnu/libselinux.so.1 (0x00007feb8a8e6000)

        libgssapi_krb5.so.2 => /lib/x86_64-linux-gnu/libgssapi_krb5.so.2 (0x00007feb8a894000)

        libkrb5.so.3 => /lib/x86_64-linux-gnu/libkrb5.so.3 (0x00007feb8a7ba000)

        libcom_err.so.2 => /lib/x86_64-linux-gnu/libcom_err.so.2 (0x00007feb8a7b4000)

        libcrypto.so.3 => /lib/x86_64-linux-gnu/libcrypto.so.3 (0x00007feb8a200000)

        libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007feb8a795000)

        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007feb8a01f000)

        libnsl.so.2 => /lib/x86_64-linux-gnu/libnsl.so.2 (0x00007feb8a778000)

        libcap-ng.so.0 => /lib/x86_64-linux-gnu/libcap-ng.so.0 (0x00007feb8a770000)

        libcap.so.2 => /lib/x86_64-linux-gnu/libcap.so.2 (0x00007feb8a764000)

        libgcrypt.so.20 => /lib/x86_64-linux-gnu/libgcrypt.so.20 (0x00007feb89ed8000)

        liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007feb8a735000)

        libzstd.so.1 => /lib/x86_64-linux-gnu/libzstd.so.1 (0x00007feb89e1c000)

        liblz4.so.1 => /lib/x86_64-linux-gnu/liblz4.so.1 (0x00007feb8a70d000)

        /lib64/ld-linux-x86-64.so.2 (0x00007feb8abb5000)

        libpcre2-8.so.0 => /lib/x86_64-linux-gnu/libpcre2-8.so.0 (0x00007feb89d82000)

        libk5crypto.so.3 => /lib/x86_64-linux-gnu/libk5crypto.so.3 (0x00007feb8a6e0000)

        libkrb5support.so.0 => /lib/x86_64-linux-gnu/libkrb5support.so.0 (0x00007feb8a6d2000)

        libkeyutils.so.1 => /lib/x86_64-linux-gnu/libkeyutils.so.1 (0x00007feb8a6c9000)

        libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007feb8a6b8000)

        libtirpc.so.3 => /lib/x86_64-linux-gnu/libtirpc.so.3 (0x00007feb8a68a000)

        libgpg-error.so.0 => /lib/x86_64-linux-gnu/libgpg-error.so.0 (0x00007feb89d5a000)

# ldd $(which sshd) | grep liblz

        liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007fd1e647a000)

        liblz4.so.1 => /lib/x86_64-linux-gnu/liblz4.so.1 (0x00007fd1e6398000)

DPA2• 2 years ago

The correct thing to do would be to put different unrelated APIs into their own library, instead of everything into libsystemd0. This has always been one of my biggest issues with it. It makes it hard to replace just one API from that library, because on a binary distribution, only one package can provide it. And as a nice side effect, surprises like this one could then be avoided.

3v1n03• 2 years ago

IIRC sshd loads libpam only if specifically configured for it. So while it's not wrong, it's also a more edge case for the backdoor to work.

mardifoufs4• 2 years ago

What? I don't get it? Isn't it on Debian if they modified the package to do something like this? Why would you blame systemd for maintainers doing something that upstream has never required or recommended?

hnald• 2 years ago

It's unfortunate that the anti-systemd party lost the war... years ago. But I don't blame systemd, Lennart Pottering or the fanboys (though it would have been so much better if the guy never worked in open source or wasn't such a prolific programmer). I blame Debian and its community for succumbing to this assault on Unix philosophy (again, years ago).

1attice• 2 years ago

Sometimes things evolve in ways that make us feel a little obsolete.

I've been learning NixOS for a few years now, and it would have been impossible without systemd. It's one heck of a learning curve, but when you get to the other side, you know something of great power and value. Certain kinds of complexity adds 'land' (eg. systemd) that can become 'real estate' (eg. NixOS), which in turn hopes to become 'land' for the next innovation, and so forth.

Whether this happens or not (whether it's the right kind of complexity) is really hard to assess up-front, and probably impossible without knowing the complex new technology in question very well. (And by then you have the bias of depending, in part, yourself on the success of the new tech, as you've committed significant resources to mastering it, so good luck on convincing skeptical newcomers!)

It's almost like a sort of event horizon -- once you know a complex new technology well enough to see whether or not it's useful, the conflict-of-interest makes your opinion unreliable to outsiders!

Nevertheless, the assessment process itself, while difficult to get right, is worth getting better at.

It's easy for impatience and the sensation of what I've taken to calling 'daunt' -- that intrinsic recoil that the mind has from absorbing a large amounts of information whose use case is not immediately relevant -- to dissuade one from exploring. But then, one never discovers new 'land', and one never builds new real estate!

[ Aside: This is why I'm a little skeptical of the current rebellion against frontend frameworks. Certainly some of them, like tailwind, are clearly adding fetters to an otherwise powerful browser stack. But others, like Svelte, and to some extent, even React, bring significant benefits.

The rebellion has this vibe like, well, users _should_ prefer more simply-built interfaces, and if they don't, well, they just have bad taste. What would be more humble would be to let the marketplace (e.g. consumers) decide what is preferable, and then build that. ]

shirro• 2 years ago

The notify protocol isn't much more complicated than that. From memory you send a string to a unix socket. I have written both systemd notify and listenfd in a few languages for little experiments and it is hard to imagine how the protocols could be simpler.

Looking at most popular projects these days they are a mass of dependencies and I think very few of them can be properly audited and verified by the projects that use them. Rust and Go might be more memory safe than C but look at the number of cargo or go modules in most projects. I have mostly stopped using node/npm on my systems.

esbranson• 2 years ago

Not a programmer, but couldn't the distribution's sshd patches for systemd (and all other distro patches for privileged daemons) use static includes? Wouldn't that have only pulled in the simple client-side communication API? Would that have defeated this vector? Would it be doable?

0x06• 2 years ago

Homebrew is currently shipping 5.6.1 (and was shipping 5.6.0 as well). Hopefully not affected on mac?

stephenr4• 2 years ago

The issue is caused by patches to add integration with systemd, so no, this won't affect SSH on a Mac.

0x06• 2 years ago

Just because macs don't use systemd, doesn't mean the backdoor won't work. The oss-sec post talks about liblzma having backdoors in crc32_resolve() and crc64_resolve() and that it has not been fully reversed. This could perhaps affect more than just sshd on x86-64 linux?

woodruffw3• 2 years ago

Homebrew reverted to 5.4.6 once the maintainers became aware. The current understanding is that macOS is not affected, but that's not certain.

[1] https://github.com/Homebrew/homebrew-core/pull/167512

asveikau2• 2 years ago

That's completely crazy, the backdoor is introduced through a very cryptic addition to the configure script. Just looking at the diff, it doesn't look malicious at all, it looks like build script gibberish.

zb34• 2 years ago

Yeah, now imagine they succeeded and it didn't cause any performance issues...

Can we even be sure no such successful attempt has already been made?

gpvos• 2 years ago

No, we can't.

coldpie• 2 years ago

You can be certain it has happened, many times. Now think of all the software we mindlessly consume via docker, language package managers, and the like.

Remember, there is no such thing as computer security. Make your decisions accordingly :)

agwa5• 2 years ago

Thanks to autoconf, we're now used to build scripts looking like gibberish. A perfect place to hide a backdoor.

rwmj10• 2 years ago

This is my main take-away from this. We must stop using upstream configure and other "binary" scripts. Delete them all and run "autoreconf -fi" to recreate them. (Debian already does something like this I think.)

demizer• 2 years ago

Maybe the US Government needs to put its line in the sand and mandate the end of autotools. :D

pornel• 2 years ago

Maybe it's time to dramatically simplify autoconf?

How long do we need to (pretend to) keep compatibility with pre-ANSI C compilers, broken shells on exotic retro-unixes, and running scripts that check how many bits are in a byte?

hgs32• 2 years ago

Autoconf is m4 macros and Bourne shell. Most mainstream programming languages have a packaging system that lets you invoke a shell script. This attack is a reminder to keep your shell scripts clean. Don't treat them as an afterthought.

hypnagogic8• 2 years ago

I'm wondering is there i.e. no way to add an automated flagging system that A/B / `diff` checks the tarball contents against the repo's files and warns if there's a mismatch? This would be on i.e. GitHub's end so that there'd be this sort of automated integrity test and subsequent warning? Just a thought, since tainted tarballs like these might be altogether be (and become) a threat vector, regardless of the repo.

phendrenad2• 2 years ago

Not just autoconf. Build systems in general are a bad abstraction, which leads to lots and lots of code to try to make them do what you want. It's a sad reality of the mismatch between a prodecural task (compile files X, Y, and Z into binary A) and what we want (compile some random subset of files X, Y, and Z, doing an arbitrary number of other tasks first, into binary B).

For fun, you can read the responses to my musing that maybe build systems aren't needed: https://news.ycombinator.com/item?id=35474996 (People can't imagine programming without a build system - it's sad)

omoikane• 2 years ago

It looks like an earlier commit with a binary blob "test data" contained the bulk of the backdoor, then the configure script enabled it, and then later commits patched up valgrind errors caused by the backdoor. See the commit links in the "Compromised Repository" section.

Also, seems like the same user who made these changes are still submitting changes to various repositories as of a few days ago. Maybe these projects need to temporarily stop accepting commits until further review is done?

20after42• 2 years ago

"Given the activity over several weeks, the committer is either directly involved or there was some quite severe compromise of their system. Unfortunately the latter looks like the less likely explanation, given they communicated on various lists about the "fixes" mentioned above."

Crazy indeed.

tetromino_2• 2 years ago

A big part of the problem is all the tooling around git (like the default github UI) which hides diffs for binary files like these pseudo-"test" files. Makes them an ideal place to hide exploit data since comparatively few people would bother opening a hex editor manually.

acdha8• 2 years ago

How many people read autoconf scripts, though? I think those filters are symptom of the larger problem that many popular C/C++ codebases have these gigantic build files which even experts try to avoid dealing with. I know why we have them but it does seem like something which might be worth reconsidering now that the tool chain is considerably more stable than it was in the 80s and 90s.

johnny224• 2 years ago

I don't see how showing the binary diffs would help. 99.99999% of people would just scroll right past them anyways.

londons_explore10• 2 years ago

testdata should not be on the same machine as the build is done. testdata (and tests generally) aren't as well audited, and therefore shouldn't be allowed to leak into the finished product.

Sure - you want to test stuff, but that can be done with a special "test build" in it's own VM.

bangoodirro2• 2 years ago

00011900: 0000 4883 f804 7416 b85f 5f5f 5f33 010f ..H...t..____3.. │ 00011910: b651 0483 f25a 09c2 0f84 5903 0000 488d .Q...Z....Y...H. │ 00011920: 7c24 40e8 5875 0000 488b 4c24 4848 3b4c |$@.Xu..H.L$HH;L │ 00011930: 2440 7516 4885 c074 114d 85ff 0f84 3202 $@u.H..t.M....2. │ 00011940: 0000 498b 0ee9 2c02 0000 b9fe ffff ff45 ..I...,........E │ 00011950: 31f6 4885 db74 0289 0b48 8bbc 2470 1300 1.H..t...H..$p.. │ 00011960: 0048 85ff 0f85 c200 0000 0f57 c00f 2945 .H.........W..)E │ 00011970: 0048 89ac 2470 1300 0048 8bbc 2410 0300 .H..$p...H..$... │ 00011980: 0048 8d84 2428 0300 0048 39c7 7405 e8ad .H..$(...H9.t... │ 00011990: e6ff ff48 8bbc 24d8 0200 0048 8d84 24f0 ...H..$....H..$. │ 000119a0: 0200 0048 39c7 7405 e893 e6ff ff48 8bbc ...H9.t......H.. │ 000119b0: 2480 0200 0048 8d84 2498 0200 0048 39c7 $....H..$....H9. │ 000119c0: 7405 e879 e6ff ff48 8bbc 2468 0100 004c t..y...H..$h...L │ Please tell me what this code does, Sheldon

lyu072822• 2 years ago

in this case the backdoor was hidden in a nesting doll of compressed data manipulated with head/tail and tr, even replacing byte ranges inbetween. Would've been impossible to find if you were just looking at the test fixtures.

ptx7• 2 years ago

The use of "eval" stands out, or at least it should stand out – but there are two more instances of it in the same script, which presumably are not used maliciously.

A while back there was a discussion[0] of an arbitrary code execution vulnerability in exiftool which was also the result of "eval".

Avoiding casual use of this overpowered footgun might make it easier to spot malicious backdoors. Usually there is a better way to do it in almost all cases where people feel the need to reach for "eval", unless the feature you're implementing really is "take a piece of arbitrary code from the user and execute it".

[0] https://news.ycombinator.com/item?id=39154825

bonzini13• 2 years ago

Unfortunately eval in a shell script has an effect on the semantics but is not necessary to do some kind of parsing of the contents of a variable, unlike Python or Perl or JavaScript. A

    $goo

line (without quotes) will already do word splitting, though it won't do another layer of variable expansion and unquoting, for which you'll need

    eval "$goo"

(This time with quotes).

jwilk6• 2 years ago

eval in autoconf macros is nothing unusual.

In (pre-backdoor) xz 5.4.5:

  $ grep -wl eval m4/*
  m4/gettext.m4
  m4/lib-link.m4
  m4/lib-prefix.m4
  m4/libtool.m4

lyu072822• 2 years ago

Usually there is a better way to do it in almost all cases where people feel the need to reach for "eval"

unfortunately thats just standard in configure scripts, for example from python:

``` grep eval Python-3.12.2/configure | wc -l 165 ```

and its 32,958 lines of code, plenty of binary fixtures as well in the tarball to hide stuff.

who knows, but I have feeling us finding the backdoor in this case was more of a happy accident.

youainti• 2 years ago

Summary: "The upstream xz repository and the xz tarballs have been backdoored."

It is known to be in version 5.6.0 and 5.6.1, and the obfuscated code is found in the test directory.

buildbot8• 2 years ago

This potentially could be a full automated rootkit type breach right? Great - is any system with 5.6.1 possibly vulnerable?

Also super weird a contributor thought they could slip this in and not have it be noticed at some point. It may point to burning that person (aka, they go to jail) for whatever they achieved with this. (And whoever they are…)

q3k2• 2 years ago

NixOS/Pkgs 23.11 unaffected, unstable contains backdoored implementations (5.6.0, 5.6.1) but their OpenSSH sshd does not seem to link against systemd/liblzma, and the backdoor doesn't get configured in (only happens on .deb/.rpm systems).

jchw• 2 years ago

It may not have really mattered much for NixOS:

b) argv[0] needs to be /usr/sbin/sshd

For once, the lack of FHS interoperability is a benefit, if only on accident.

q3k2• 2 years ago

Right, but in this case it's not even compiled it, which is arguably better than compiled in but assumed dormant :) (at least until someone actually does a full analysis of the payload).

o11c5• 2 years ago

Note that NixOS has a unique advantage in that `dlopen` is easier to analyze, but you do have to check for it. A lot of people are looking only at `ldd` and missing that they can be vulnerable at runtime.

dandanua• 2 years ago

That's one of the advantages of NixOS - viruses and mass hacks have lesser chance to function due to how different this OS is. Until it gets more popular, of course.

rany_8• 2 years ago

It's actually not an advantage. The reason why the exploit wasn't included is because the attacker specifically decided to only inject x86_64 Debian and RHEL to reduce the chances of this getting detected.

lambdanil• 2 years ago

That's just security by obscurity, not something I'd consider a good security measure.

lostmsu4• 2 years ago

Not affected by the latest CVE, but the author had unrestricted access to xz for 2 years, so I would say it is affected until the other contributions are proven safe (never gonna happen) or it reverts to pre-adversarial actor version.

AdmiralAsshat3• 2 years ago

Red Hat assigned this issue CVE-2024-3094.

Does that mean this affects RHEL and Fedora?

formerly_proven5• 2 years ago

RHEL no, Fedora 41 and Rawhide yes.

https://www.redhat.com/en/blog/urgent-security-alert-fedora-...

https://lists.debian.org/debian-security-announce/2024/msg00...

dralley6• 2 years ago

Note that Fedora 40 isn't even released yet, it's in beta, Fedora 41 / rawhide is basically a development branch used only by a small number of people.

jethro_tell9• 2 years ago

RHEL won't get this bug for 2 years =)

fargle2• 2 years ago

i knew there was an advantage to being 8-10 years out of date at all times...

and when they do port finally backport this bug in 2026, they will probably implement the systemd integration with openssl (pbthththt...) via 600 patch files in some nonstandard divergent manner that thwarts the payload anyhow. see? i knew they were super duper secure.

richardwhiuk3• 2 years ago

Red Hat helps to do the job of making sure OSS has CVEs so there's common vernacular for the problem.

lpapez• 2 years ago

So many security companies publishing daily generic blog posts about "serious supply chain compromises" in various distros on packages with 0 downloads, and yet it takes a developer debugging performance issues to find an actual compromise.

I worked in the software supply chain field and cannot resist feeling the entire point of that industry is to make companies pay for a security certificate so you can shift the blame onto someone else when things go wrong.

r0ckarong• 2 years ago

cannot resist feeling the entire point of that industry is to make companies pay for a security certificate so you can shift the blame onto someone else when things go wrong.

That's the entire point. You did everything you could by getting someone else look at it and saying it's fine.

numpad0• 2 years ago

This needs a Rust joke. You know, the problem with the whole certification charade is it slows down jobs and prevents __actual_problems getting evaluated. But is it safe?

bawolff8• 2 years ago

the entire point of that industry is to make companies pay for a security certificate so you can shift the blame onto someone else when things go wrong.

That is actually a major point of a lot of corporate security measures (shifting risk)

landownersubgrp3• 2 years ago

And that's a good thing.

keepamovin3• 2 years ago

If you installed xz on macOS using brew, then you have

  xz (XZ Utils) 5.6.1
  liblzma 5.6.1

which are within the release target for the vuln. As elsewhere in these comments, people say macOS effect is uncertain. If concerned you can revert to 5.4.6 with

  brew upgrade xz

neodypsis• 2 years ago

It's been reverted now: https://github.com/Homebrew/homebrew-core/blob/9a0603b474804...

pjl• 2 years ago

Similarly if you're using MacPorts, make sure to sync and upgrade xz if you have it installed.

5.6.1 was available for a few days and just rolled back ~20 minutes ago: https://github.com/macports/macports-ports/commit/a1388aee09...

quinncom• 2 years ago

Thank you for this tip. `brew upgrade xz` worked.

I was going to uninstall but it's used by so many things…

     brew uninstall xz
    Error: Refusing to uninstall /opt/homebrew/Cellar/xz/5.6.1
    because it is required by aom, composer, curl, ffmpeg, gcc, gd, ghostscript, glib, google-cloud-sdk, grc, harfbuzz, httpie, img2pdf, jbig2enc, jpeg-xl, leptonica, libarchive, libavif, libheif, libraw, libtiff, libzip, little-cms2, numpy, ocrmypdf, openblas, openjpeg, openvino, php, pillow, pipx, pngquant, poppler, python@3.11, python@3.12, rsync, tesseract, tesseract-lang, unpaper, webp, wp-cli, yt-dlp and zstd, which are currently installed.

CableNinja• 2 years ago

Thats basically the whole point actually... A company pays for insurance for the business. The insurance company says sure we will insure you, but you need to go through audits A B and C, and you need certifications X and Y to be insured by us. Those audits are often industry dependent, mostly for topics like HIPAA, PCI, SOC, etc.

Insurance company hears about supply chain attacks. Declares that insured must have supply chain validation. Company goes and gets a shiny cert.

Now when things go wrong, the company can point to the cert and go "it wasnt us, see we have the cert you told us to get and its up to date". And the company gets to wash their hands of liability (most of the time).

77pt773• 2 years ago

And the company gets to wash their hands of liability (most of the time).

Certification theater.

It's completely performative.

landownersubgrp3• 2 years ago

What you describe is a normal process in order to minimise damage from attacks. The damage of hacking is ultimately property damage. The procedures you've described allow you to minimise it.

And that's a good thing.

markus_zhang2• 2 years ago

That's the entire point of certification, and any certification at all. Certification does not guarantee performance. Actually, I would always cast a suspect glance to anyone who is FOCUSED on getting certification after certification without any side project.

fourfour3• 2 years ago

Looks like Arch Linux shipped both compromised versions - and 5.6.1-2 is out to hopefully resolve it.

tutfbhuf3• 2 years ago

I upgraded Arch Linux on my server a few hours ago. Arch Linux does not fetch one of the compromised tarballs but builds from source and sshd does not link against liblzma on Arch.

  [root@archlinux ~]# pacman -Qi xz | head -n2  
  Name            : xz  
  Version         : 5.6.1-2  
  [root@archlinux ~]# pacman -Qi openssh | head -n2
  Name            : openssh
  Version         : 9.7p1-1
  [root@archlinux ~]# ldd $(which sshd) | grep liblzma
  [root@archlinux ~]#

It seems that Arch Linux is not affected.

gpm11• 2 years ago

5.6.1-1 was built from what I understand to be one of the affected tarballs. This was patched in 5.6.1-2: https://gitlab.archlinux.org/archlinux/packaging/packages/xz...

I agree on the sshd linking part.

gpm11• 2 years ago

On arch, `ldd $(which sshd)` doesn't list lzma or xz, so I think it's unaffected? Obviously still not great to be shipping malicious code that just happens to not trigger.

fullstop10• 2 years ago

My Arch setup is the same, they must not patch openssh.

altairprime2• 2 years ago

Deleted per below

Macha• 2 years ago

5.6.1-2 is not an attempted fix, it's just some tweaks to Arch's own build script to improve reproducibility. Arch's build script ultimately delegates to the compromised build script unfortunately, but it also appears the payload itself is specifically targeting deb/RPM based distros, so a narrow miss for Arch here.

(EDIT: as others have pointed out, part of the exploit is in the artifact from libxz, which Arch is now avoiding by switching to building from a git checkout)

gpm11• 2 years ago

Are you sure about that? The diff moves away from using the compromised tarballs to the not-compromised (by this) git source. The comment message says it's about reproducibility, but especially combined with the timing it looks to me like that was just to avoid breaking an embargo.

mook2• 2 years ago

The writeup indicates that the backdoor only gets applied when building for rpm or deb, so Arch probably would have been okay either way? Same with Nix, Homebrew, etc.

aquova• 2 years ago

The project has made an official post on the subject

https://archlinux.org/news/the-xz-package-has-been-backdoore...

bawolff8• 2 years ago

The terrifying part is that this was primarily found because the backdoor was poorly made and causing performance problems.

Makes you wonder what more competent actors can do.

rwmj10• 2 years ago

I've analysed the backdoor myself and it's very sophisticated, not poorly made at all. The performance problem is surprising in this context, but I think next time they won't make that mistake.

Nextgrid• 2 years ago

Do you have a writeup or any details as to what it does? The logical thing based on this post is that it hooks the SSH key verification mechanism to silently allow some attacker-controlled keys but I wonder if there's more to it?

bawolff8• 2 years ago

I guess it seems like the operational parts are a bit poorly done. Valgrind issues, adding a new version with symbols removed, the aforementioned performance issues. Like i would assume the type of person who would do this sort of thing, over a 2 year period no less, would test extensively and be sure all their i's are dotted. Its all kind of surprising given how audacious the attack is.

pinko3• 2 years ago

s/can do/have done/

aidenn0• 2 years ago

So many malicious actors have been caught because they accidentally created a mild annoyance for someone that went on to bird-dog the problem.

cpach2• 2 years ago

Case in point: https://news.ycombinator.com/item?id=39843930

londons_explore10• 2 years ago

Which is why a really good backdoor is a one line logic bug somewhere which is fiendishly difficult to trigger.

entropie2• 2 years ago

Unrelated: as a dog/pointer lover i really like the term "to bird-dog the problem". Never heard of it (iam from germany though)

yard2010• 2 years ago

You must mean, "Makes you wonder what more competent actors are doing"

rwmj10• 2 years ago

Very annoying - the apparent author of the backdoor was in communication with me over several weeks trying to get xz 5.6.x added to Fedora 40 & 41 because of it's "great new features". We even worked with him to fix the valgrind issue (which it turns out now was caused by the backdoor he had added). We had to race last night to fix the problem after an inadvertent break of the embargo.

He has been part of the xz project for 2 years, adding all sorts of binary test files, and to be honest with this level of sophistication I would be suspicious of even older versions of xz until proven otherwise.

gigatexal4• 2 years ago

Name and shame this author. They should never be allowed anywhere near any open projects ever again.

Lichtso3• 2 years ago

They might have burnt the reputation built for this particular pseudonym but what is stopping them from doing it again? They were clearly in it for the long run.

0xbadcafebee• 2 years ago

Please don't?

1. You don't actually know what has been done by whom or why. You don't know if the author intended all of this, or if their account was compromised. You don't know if someone is pretending to be someone else. You don't know if this person was being blackmailed, forced against their will, etc. You don't really know much of anything, except a backdoor was introduced by somebody.

2. Assuming the author did do something maliciously, relying on personal reputation is bad security practice. The majority of successful security attacks come from insiders. You have to trust insiders, because someone has to get work done, and you don't know who's an insider attacker until they are found out. It's therefore a best security practice to limit access, provide audit logs, sign artifacts, etc, so you can trace back where an incursion happened, identify poisoned artifacts, remove them, etc. Just saying "let's ostracize Phil and hope this never happens again" doesn't work.

3. A lot of today's famous and important security researchers were, at one time or another, absolute dirtbags who did bad things. Human beings are fallible. But human beings can also grow and change. Nobody wants to listen to reason or compassion when their blood is up, so nobody wants to hear this right now. But that's why it needs to be said now. If someone is found guilty beyond a reasonable doubt (that's really the important part...), then name and shame, sure, shame can work wonders. But at some point people need to be given another chance.

yieldcrv• 2 years ago

I wonder who the target was!

juliusdavies• 2 years ago

Every Linux box inside AWS, Azure, and GCP and other cloud providers that retains the default admin sudo-able user (e.g., “ec2”) and is running ssh on port 22.

I bet they intended for their back door to eventually be merged into the base Amazon Linux image.

swagmoney1606• 2 years ago

Probably less of an individual and more of an exploit to sell.

njsg• 2 years ago

Distro build hosts and distro package maintainers might not be a bad guess. Depends on whether getting this shipped was the final goal. It might have been just the beginning, part of some bootstrapping.

formerly_proven5• 2 years ago

I think this has been in the making for almost a year. The whole ifunc infrastructure was added in June 2023 by Hans Jansen and Jia Tan. The initial patch is "authored by" Lasse Collin in the git metadata, but the code actually came from Hans Jansen: https://github.com/tukaani-project/xz/commit/ee44863ae88e377...

Thanks to Hans Jansen for the original patch.

https://github.com/tukaani-project/xz/pull/53

There were a ton of patches by these two subsequently because the ifunc code was breaking with all sorts of build options and obviously caused many problems with various sanitizers. Subsequently the configure script was modified multiple times to detect the use of sanitizers and abort the build unless either the sanitizer was disabled or the use of ifuncs was disabled. That would've masked the payload in many testing and debugging environments.

The hansjans162 Github account was created in 2023 and the only thing it did was add this code to liblzma. The same name later applied to do a NMU at Debian for the vulnerable version. Another "<name><number>" account (which only appears here, once) then pops up and asks for the vulnerable version to be imported: https://www.mail-archive.com/search?l=debian-bugs-dist@lists...

zb34• 2 years ago

Also I saw this hans jansen user pushing for merging the 5.6.1 update in debian: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1067708

zb34• 2 years ago

Also I see this PR: https://github.com/tukaani-project/xz/pull/64

bluecheese33• 2 years ago

because the ifunc code was breaking with all sorts of build options and obviously caused many problems with various sanitizers

for example, https://github.com/google/oss-fuzz/pull/10667

bed994• 2 years ago

1 week ago "Hans Jansen" user "hjansen" was created in debian and opened 8 PRs including the upgrade to 5.6.1 to xz-utils

From https://salsa.debian.org/users/hjansen/activity

Author: Hans Jansen <hansjansen162@outlook.com>

- [Debian Games / empire](https://salsa.debian.org/games-team/empire): opened merge request "!2 New upstream version 1.17" - March 17, 2024

- [Debian Games / empire](https://salsa.debian.org/games-team/empire): opened merge request "!1 Update to upstream 1.17" - March 17, 2024

- [Debian Games / libretro / libretro-core-info](https://salsa.debian.org/games-team/libretro/libretro-core-i...): opened merge request "!2 New upstream version 1.17.0" - March 17, 2024

- [Debian Games / libretro / libretro-core-info](https://salsa.debian.org/games-team/libretro/libretro-core-i...): opened merge request "!1 Update to upstream 1.17.0" - March 17, 2024

- [Debian Games / endless-sky](https://salsa.debian.org/games-team/endless-sky): opened merge request "!6 Update upstream branch to 0.10.6" - March 17, 2024

- [Debian Games / endless-sky](https://salsa.debian.org/games-team/endless-sky): opened merge request "!5 Update to upstream 0.10.6" - March 17, 2024

- [Debian / Xz Utils](https://salsa.debian.org/debian/xz-utils): opened merge request "!1 Update to upstream 5.6.1" - March 17, 2024

amluto• 2 years ago

Wow, what a big pile of infrastructure for a non-optimization.

An internal call via ifunc is not magic — it’s just a call via the GOT or PLT, which boils down to function pointers. An internal call through a hidden visibility function pointer (the right way to do this) is also a function pointer.

The even better solution is a plain old if statement, which implements the very very fancy “devirtualization” optimization, and the result will be effectively predicted on most CPUs and is not subject to the whole pile of issue that retpolines are needed to work around.

snvzz• 2 years ago

Hans Jansen and Jia Tan

Are they really two people conspiring?

Unless proven otherwise, it is safe to assume one is just a pseudonym alias of the other.

formerly_proven5• 2 years ago

Make it two years.

Jia Tan getting maintainer access looks like it is almost certainly to be part of the operation. Lasse Colling mentioned multiple times how Jia has helped off-list and to me it seems like Jia befriended Lasse as well (see how Lasse talks about them in 2023).

Also the pattern of astroturfing dates back to 2022. See for example this thread where Jia, who has helped at this point for a few weeks, posts a patch, and a <name><number>@protonmail (jigarkumar17) user pops up and then bumps the thread three times(!) lamenting the slowness of the project and pushing for Jia to get commit access: https://www.mail-archive.com/xz-devel@tukaani.org/msg00553.h...

Naturally, like in the other instances of this happening, this user only appears once on the internet.

tootie• 2 years ago

Does anybody know anything about Jia Tan? Is it likely just a made up persona? Or is this a well-known person.

Ill_Yam_6894• 2 years ago

It seems like Hans Jansen has also an account on proton.me (hansjansen162@proton.me) with the Outlook address configured as recovery-email.

jonathanspw• 2 years ago

Yesterday sure was fun wasn't it :p Thanks for all your help/working with me on getting this cleaned up in Fedora.

w4ffl352• 2 years ago

Is it normal that when I try to uninstall xz it is trying to install lzma?

speleding• 2 years ago

PSA: I just noticed homebrew installed the compromised version on my Mac as a dependency of some other package. You may want to check this to see what version you get:

   xz --version

Homebrew has already taken action, a `brew upgrade` will downgrade back to the last known good version.

KingOfCoders• 2 years ago

Sleeper.

smeehee• 2 years ago

Debian have reverted xz-utils (in unstable) to 5.4.5 – actual version string is “5.6.1+really5.4.5-1”. So presumably that version's safe; we shall see…

xorcist• 2 years ago

Is that version truly vetted? "Jia Tan" has been the official maintainer since 5.4.3, could have pushed code under any other pseudonym, and controls the signing keys. I would have felt better about reverting farther back, xz hasn't had any breaking changes for a long time.

kzrdude6• 2 years ago

There are suggestions to roll back further

sgarland5• 2 years ago

TIL that +really is a canonical string. [0]

[0] https://www.debian.org/doc/debian-policy/ch-controlfields.ht...

Jommi• 2 years ago

the account was either sold or stolen

account4252• 2 years ago

That's pure speculation and there are plenty of hints to the contrary.

userbinator5• 2 years ago

because of it's "great new features"

"great" for whom? I've seen enough of the industry to immediately feel suspicious when someone uses that sort of phrasing in an attempt to persuade me. It's no different from claiming a "better experience" or similar.

SilasX2• 2 years ago

You can find more examples of that kind of puffer if you go to a website's cookie consent pop-up and find the clause after "we use cookies to...".

LtWorf15• 2 years ago

I made a library where version 2 is really really much faster than version 1. I'd want everyone to just move to version 2.

kapouer2• 2 years ago

Github accounts of both xz maintainers have been suspended.

miduil4• 2 years ago

Not true, the original author wasn't suspended: https://github.com/Larhzu

https://github.com/JiaT75 was suspended for a moment, but isn't anymore?

logicziller2• 2 years ago

These shouldn't be suspended, and neither should their repositories. People might want to dig through the source code. It's okay if they add a warning on the repository, but suspending _everything_ is a stupid thing to do.

mongol• 2 years ago

Interesting that one of the commits commented on update of the test file that it was for better reproducibility for having been generated by a fixed random seed (although how goes unmentioned). For the future, random test data better be generated as part of the build, rather than being committed as opaque blobs...

WhyNotHugo2• 2 years ago

I agree on principle, but sometimes programmatic generating test data is not so easy.

E.g.: I have a specific JPEG committed into a repository because it triggers a specific issue when reading its metadata. It's not just _random_ data, but specific bogus data.

But yeah, if the test blob is purely random, then you can just commit a seed and generate in during tests.

api• 2 years ago

I’m surprised there isn’t way more of this stuff. The supply chain is so huge and therefore represents so much surface area.

SoftTalker• 2 years ago

There probably is. Way more than anyone knows. I bet every major project on github is riddled with state actors.

cozzyd2• 2 years ago

Imagine if sshd was distributed by PyPI or cargo or npm instead of by a distro.

junon9• 2 years ago

GitHub has suspended @JiaT75's account.

EDIT: Lasse Collin's account @Larhzu has also been suspended.

EDIT: Github has disabled all Tukaani repositories, including downloads from the releases page.

EDIT: Just did a bit of poking. xz-embedded was touched by Jia as well and it appears to be used in the linux kernel. I did quick look and it doesn't appear Jia touched anything of interest in there. I also checked the previous mirror at the tukaani project website, and nothing was out of place other than lagging a few commits behind:

https://gist.github.com/Qix-/f1a1b9a933e8847f56103bc14783ab7...

Here's a mailing list message from them ca. 2022.

https://listor.tp-sv.se/pipermail/tp-sv_listor.tp-sv.se/2022...

MinGW w64 on AUR was last published by Jia on Feb 29: https://aur.archlinux.org/cgit/aur.git/log/?h=mingw-w64-xz (found by searching their public key: 22D465F2B4C173803B20C6DE59FCF207FEA7F445)

pacman-static on AUR still lists their public key as a contributor, xz was last updated to 5.4.5 on 17-11-2023: https://aur.archlinux.org/cgit/aur.git/?h=pacman-static

EDIT: I've emailed the maintainer to have the key removed.

Alpine was patched as of 6 hours ago.

https://git.alpinelinux.org/aports/commit/?id=982d2c6bcbbb57...

OpenSUSE is still listing Jia's public key: https://sources.suse.com/SUSE:SLE-15-SP6:GA/xz/576e550c49a36... (cross-ref with https://web.archive.org/web/20240329235153/https://tukaani.o...)

EDIT: Spoke with some folks in the package channel on libera, seems to be a non-issue. It is not used as attestation nor an ACL.

Arch appears to still list Jia as an approved publisher, if I'm understanding this page correctly.

https://gitlab.archlinux.org/archlinux/packaging/packages/xz...

EDIT: Just sent an email to the last committer to bring it to their attention.

EDIT: It's been removed.

jiatan's Libera info indicates they registered on Dec 12 13:43:12 2022 with no timezone information.

    -NickServ- Information on jiatan (account jiatan):
    -NickServ- Registered : Dec 12 13:43:12 2022 +0000 (1y 15w 3d ago)
    -NickServ- Last seen : (less than two weeks ago)
    -NickServ- User seen : (less than two weeks ago)
    -NickServ- Flags : HideMail, Private
    -NickServ- jiatan has enabled nick protection
    -NickServ- *** End of Info ***

/whowas expired not too long ago, unfortunately. If anyone has it I'd love to know.

They are not registered on freenode.

EDIT: Libera has stated they have not received any requests for information from any agencies as of yet (30th Saturday March 2024 00:39:31 UTC).

EDIT: Jia Tan was using a VPN to connect; that's all I'll be sharing here.

junon9• 2 years ago

Just for posterity since I can no longer edit: Libera staff has been firm and unrelenting in their position not to disclose anything whatsoever about the account. I obtained the last point on my own. Libera has made it clear they will not budge on this topic, which I applaud and respect. They were not involved whatsoever in ascertaining a VPN was used, and since that fact makes anything else about the connection information moot, there's nothing else to say about it.

Phenylacetyl3• 2 years ago

The alpine patch includes gettext-dev which is likely also exploited as the same authors have been pushing gettext to projects where their changes have been questioned

mook2• 2 years ago

FWIW, that's mingw-w64-xz (cross-compiled xz utils) in AUR, not ming-w64 (which would normally refer to the compiler toolchain itself).

reisse• 2 years ago

EDIT: Github has disabled all Tukaani repositories, including downloads from the releases page.

Why? Isn't it better to freeze them and let as many people as possible analyze the code?

hypnagogic8• 2 years ago

Asking this here too: why isn't there an automated A/B or diff match for the tarball contents to match the repo, auto-flag with a warning if that happens? Am I missing something here?

mikolajw2• 2 years ago

I've posted an earlier WHOWAS of jiatan here: https://news.ycombinator.com/item?id=39868773

junon9• 2 years ago

It appears to be an RCE, not a public key bypass: https://news.ycombinator.com/item?id=39877312

menomatter2• 2 years ago

account is back online https://github.com/JiaT75

drazk• 2 years ago

After reading the original post by Andres Freund, https://www.openwall.com/lists/oss-security/2024/03/29/4, his analysis indicates that the RSA_public_decrypt function is being redirected to the malware code. Since RSA_public_decrypt is only used in the context of RSA public key - private key authentication, can we reasonably conclude that the backdoor does not affect username-password authentication?

cbolton2• 2 years ago

Isn't it rather that the attacker can log in to the compromised server by exploiting the RSA code path?

thayne2• 2 years ago

Do you know if it was actually the commit author, of if their commit access was compromised?

bpye2• 2 years ago

If it was a compromise it also included the signing keys as the release tarball was modified vs the source available on GitHub.

LispSporks22• 2 years ago

Nice. I worked on a Linux disto when I was a wee lad and all we did was compute a new md5 and ship it.

nodesocket• 2 years ago

Can legal action be taken against the author if it's found he maliciously added the backdoor?

saltedtangerine• 2 years ago

Good luck with that. We don't even know what country is he from. Probably from China but even if so. Good luck finding him among 1.5 Billions.

menomatter2• 2 years ago

his account is active again on github https://github.com/JiaT75

landownersubgrp3• 2 years ago

It is not good to take into consideration something with any unreadable text instead of the open text of the programme. It should be excluded.

thesnide• 2 years ago

The discussion to upload it to Debian is interesting on its own https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1067708

0x06• 2 years ago

Wow, that's a lot of anonymous accounts adding comments there urging for a fast merge!

And this "Hans Jansen" guy is apparently running around salsa.debian.org pushing for more updates in other projects as well: https://salsa.debian.org/users/hjansen/activity

matthews2• 2 years ago

running around salsa.debian.org pushing for more updates in other projects as well

This is quite common in most (all?) distributions. People are going through lists of outdated packages, updating them, testing them, and pushing them.

2OEH8eoCRo05• 2 years ago

And now we see why I don't trust anons, aliases, or anime characters to make contributions.

My GitHub says exactly who I am!

ValdikSS3• 2 years ago

that's a lot of anonymous accounts

Just FYI, krygorin4545@proton.me (the latest message before the upload) was created Tue Mar 26 18:30:02 UTC 2024, about an hour earlier than the message was posted.

Proton generates PGP key upon creating the account, with the real datetime of the key (but the key does not include the timezone).

nammi• 2 years ago

That name jumped out at me, Hans Jansen is the name Dominic Monaghan used when posing as a German interviewer with Elijah Woods. Not that it can't be a real person

https://youtu.be/IfhMILe8C84

joeyh4• 2 years ago

See comments about "Hans Janson" upthread, he appeared to collaborate on the exploit in other ways as well.

rsync• 2 years ago

Hans Gruber would have Been a much more stylish choice…

fullstop10• 2 years ago

I get the feeling that a number of the comments are all the same person / group.

oasisaimlessly• 2 years ago

For anyone else feeling some deja vu about ifunc / Valgrind errors, this Red Hat issue [1] was previously linked from HN 12 days ago [2].

[1] https://bugzilla.redhat.com/show_bug.cgi?id=2267598

[2] https://news.ycombinator.com/item?id=39733185

formerly_proven5• 2 years ago

Quite ironic: The most recent commit in the git repo is "Simplify SECURITY.md", committed by the same Github account which added the backdoor.

https://github.com/tukaani-project/xz/commit/af071ef7702debe...

rany_8• 2 years ago

It's not ironic, this change is really sinister IMO. They want you to waste more time after you've submitted the security report and maximize the amount of back and forth. Basically the hope is that they'd be able to pester you with requests for more info/details in order to "resolve the issue" which would give them more time to exploit their targets.

arp2425• 2 years ago

I've long since said that if you want to hide something nefarious you'd do that in the GNU autoconf soup (and not in "curl | sh" scripts).

Would be interesting to see what's going on here; the person who did the releases has done previous releases too (are they affected?) And has commits going back to 2022 – relatively recent, but not that recent. Many are real commits with real changes, and they have commits on some related projects like libarchive. Seems like a lot of effort just to insert a backdoor.

Edit: anyone with access can add files to existing releases and it won't show that someone else added it (I just tested). However, the timestamp of the file will be to when you uploaded it, not that of the release. On xz all the timestamps of the files match with the timestamp of the release (usually the .tar.gz is a few minutes earlier, which makes sense). So looks like they were done by the same person who did the release. I suspected someone else might have added/altered the files briefly after the release before anyone noticed, but that doesn't seem to be the case.

sslayer• 2 years ago

I would be curious if their commits could be analyzed for patterns that could then be used to detect commits from their other account

carom• 2 years ago

There was a DARPA program on this topic called Social Cyber. [1]

1. https://www.darpa.mil/program/hybrid-ai-to-protect-integrity...

bombcar18• 2 years ago

One thing that is annoying is that many open source projects have been getting "garbage commits" apparently from people looking to "build cred" for resumes or such.

Easier and easier to hide this junk in amongst them.

bawolff8• 2 years ago

I mean, a backdoor at this scale (particularly if it wasn't noticed for a while and got into stable distros) could be worth millions. Maybe hundreds of millions (think of the insider trading possibilities alone, not to mention espionage). 2 years doesn't seem like that much work relative to the potential pay off.

This is the sort of case where america's over the top hacking laws make sense.

jethro_tell9• 2 years ago

And what law would you use to target someone who wrote some code and posted it for free on the internet that was willingly consumed?

eigenvalue• 2 years ago

Every single commit this person ever did should immediately be rolled back in all projects.

gopher_space2• 2 years ago

It's weird and disturbing that this isn't the default perspective.

andruby• 2 years ago

Hoe will you do that practically though? That’s probably thousands of commits upon which tens or hundred thousand commits from others were built. You can’t just rollback everything two years and expect it not to break or bring back older vulnerabilities that were patched in those commits.

neurostimulant• 2 years ago

Rolling back two years worth of commits made by a major contributor is going to be hell. I'm looking forward to see how they'll do this.

planb• 2 years ago

I don’t thinks that’s necessary: there are enough eyes on this person’s work now.

ptx7• 2 years ago

Couldn't the autoconf soup be generated from simpler inputs by the CI/CD system to avoid this kind of problem? Incomprehensible soup as a build artifact (e.g. executables) is perfectly normal, but it seems to me that such things don't belong in the source code.

(This means you too, gradle-wrapper! And your generated wrapper for your generated wrapper. That junk is not source code and doesn't belong in the repo.)

bonzini13• 2 years ago

Yes, it's usually regenerated already. However even the source is often pretty gnarly.

And in general, the build system of a large project is doing a lot of work and is considered pretty uninteresting and obscure. Random CMake macros or shell scripts would be just as likely to host bad code.

This is also why I like meson, because it's much more constrained than the others and the build system tends to be more modular and the complex parts split across multiple smaller, mostly independent scripts (written in Python or bash, 20-30 lines max). It's still complex, but I find it easier to organize.

mysidia3• 2 years ago

The other thing besides the autoconf soup is the XZ project contains incomprehensible binaries as "test data"; the "bad-3-corrupt_lzma2.xz" part of the backdoor that they even put in the repo.

It's entirely possible they could have got that injection through review, even if they had that framwork and instead put it in source files used to generate autoconf soup.

WatchDog• 2 years ago

gradle-wrapper is just a convenience, you can always just build the project with an installed version of gradle. Although I get your point, it’s a great place to hide nefarious code.

AeroNotix2• 2 years ago

Pure speculation but my guess is a specific state actor ahem is looking for developers innocently working with open source to then strongarm them into doing stuff like this.

dec0dedab0de2• 2 years ago

Or hiring them to do it for years without telling them why until they need a favor.

Bulat_Ziganshin7• 2 years ago

many people are patriots of their countries. if state agency would approach them proposing to have paid OSS work and help their country to fight terrorism/dictatorships/capitalists/whatever-they-believe, they will feel like killing two birds with one job

bodyfour• 2 years ago

I've long since said that if you want to hide something nefarious you'd do that in the GNU autoconf soup (and not in "curl | sh" scripts).

Yeah, I've been banging on that same drum for ages too... for example on this very site a decade ago: https://news.ycombinator.com/item?id=7213563

I'm honestly surprised that this autoconf vector hasn't happened more often... or more often that we know of.

pretzel5297• 2 years ago

Given that this was discovered by sheer luck, I'd expect way more such exploits in the wild.

dist-epoch5• 2 years ago

they have commits on some related projects like libarchive

Windows started using libarchive to support .rar, .7z, ...

https://arstechnica.com/gadgets/2023/05/cancel-your-winrar-t...

IshKebab2• 2 years ago

Yeah this was my first thought too. Though I think the case against autoconf is already so overwhelming I think anyone still using it is just irredeemable; this isn't going to persuade them.

stabbles• 2 years ago

How about wheels in the python ecosystem

WesolyKubeczek• 2 years ago

I've long since said that if you want to hide something nefarious you'd do that in the GNU autoconf soup

If I recall correctly, xz can be built with both autoconf and cmake, are cmake configs similarly affected?

amiga3864• 2 years ago

Yes, there is evidence of sabotage on the CMake configs too.

https://git.tukaani.org/?p=xz.git;a=commit;h=f9cf4c05edd14de...

bonyt2• 2 years ago

For those panicking, here are some key things to look for, based on the writeup:

- A very recent version of liblzma5 - 5.6.0 or 5.6.1. This was added in the last month or so. If you're not on a rolling release distro, your version is probably older.

- A debian or RPM based distro of Linux on x86_64. In an apparent attempt to make reverse engineering harder, it does not seem to apply when built outside of deb or rpm packaging. It is also specific to Linux.

- Running OpenSSH sshd from systemd. OpenSSH as patched by some distros only pulls in libsystemd for logging functionality, which pulls in the compromised liblzma5.

Debian testing already has a version called '5.6.1+really5.4.5-1' that is really an older version 5.4, repackaged with a newer version to convince apt that it is in fact an upgrade.

It is possible there are other flaws or backdoors in liblzma5, though.

treffer6• 2 years ago

Ubuntu still ships 5.4.5 on 24.03 (atm).

I did a quick diff of the source (.orig file from packages.ubuntu.com) and the content mostly matched the 5.4.5 github tag except for Changelog and some translation files. It does match the tarball content, though.

So for 5.4.5 the tagged release and download on github differ.

It does change format strings, e.g.

   +#: src/xz/args.c:735
   +#, fuzzy
   +#| msgid "%s: With --format=raw, --suffix=.SUF is required unless writing to stdout"
   +msgid "With --format=raw, --suffix=.SUF is required unless writing to stdout"
   +msgstr "%s: amb --format=raw, --suffix=.SUF és necessari si no s'escriu a la sortida estàndard"

There is no second argument to that printf for example. I think there is at least a format string injection in the older tarballs.

[Edit] formatting

fransje264• 2 years ago

Thanks for the heads up.

chasil2• 2 years ago

RHEL9 is shipping 5.2.5; RHEL8 is on 5.2.4.

mort962• 2 years ago

FYI, your formatting is broken. Hacker News doesn't support backtick code blocks, you have to indent code.

Anyway, so... the xz project has been compromised for a long time, at least since 5.4.5. I see that this JiaT75 guy has been the primary guy in charge of at least the GitHub releases for years. Should we view all releases after he got involved as probably compromised?

jwilk6• 2 years ago

"#, fuzzy" means the translation is out-of-date and it will be discarded at compile time.

hostyle• 2 years ago

$ dpkg-query -W liblzma5

liblzma5:amd64 5.4.1-0.2

fransje264• 2 years ago

I did notice that my debian-based system got noticeably slower and unresponsive at times the last two weeks, without obvious reasons. Could it be related?

I read through the report, but what wasn't directly clear to me was: what does the exploit actually do?

My normal internet connection has such an appalling upload that I don't think anything relevant could be uploaded. But I will change my ssh keys asap.

anarazel5• 2 years ago

I did notice that my debian-based system got noticeably slower and unresponsive at times the last two weeks, without obvious reasons. Could it be related?

Possible but unlikely.

I read through the report, but what wasn't directly clear to me was: what does the exploit actually do?

It injects code that runs early during sshd connection establishment. Likely allowing remote code execution if you know the right magic to send to the server.

cpach2• 2 years ago

Are you on stable/testing/unstable?

With our current knowledge, stable shouldn’t be affected by this.

idoubtit• 2 years ago

The article gives a link to a simple shell script that detects the signature of the compromised function.

Running OpenSSH sshd from systemd

I think this is irrelevant.

From the article: "Initially starting sshd outside of systemd did not show the slowdown, despite the backdoor briefly getting invoked." If I understand correctly the whole section, the behavior of OpenSSH may have differed when launched from systemd, but the backdoor was there in both cases.

Maybe some distributions that don't use systemd strip the libxz code from the upstream OpenSSH release, but I wouldn't bet on it if a fix is available.

bonyt2• 2 years ago

I think the distributions that do use systemd are the ones that add the libsystemd code, which in turn brings in the liblzma5 code. So, it may not be entirely relevant how it is run, but it needs to be a version of OpenSSH patched.

anarazel5• 2 years ago

From the article: "Initially starting sshd outside of systemd did not show the slowdown, despite the backdoor briefly getting invoked." If I understand correctly the whole section, the behavior of OpenSSH may have differed when launched from systemd, but the backdoor was there in both cases.

It looks like the backdoor "deactivates" itself when it detects being started interactively, as a security researcher might. I was eventually able to circumvent that, but unless you do so, it'll not be active when started interactively.

However, the backdoor would also be active if you started it with an shell script (as the traditional sys-v rc scripts did) outside the context of an interactive shell, as TERM wouldn't be set either in that context.

Maybe some distributions that don't use systemd strip the libxz code from the upstream OpenSSH release, but I wouldn't bet on it if a fix is available.

There's no xz code in openssh.

nwallin• 2 years ago

Maybe some distributions that don't use systemd strip the libxz code from the upstream OpenSSH release, but I wouldn't bet on it if a fix is available.

OpenSSH is developed by the OpenBSD project, and systemd is not compatible with OpenBSD. The upstream project has no systemd or liblzma code to strip. If your sshd binary links to liblzma, it's because the package maintainers for your distro have gone out of their way to add systemd's patch to your sshd binary.

From the article: "Initially starting sshd outside of systemd did not show the slowdown, despite the backdoor briefly getting invoked." If I understand correctly the whole section, the behavior of OpenSSH may have differed when launched from systemd, but the backdoor was there in both cases.

From what I understand, the backdoor detects if it's in any of a handful of different debug environments. If it's in a debug environment or not launched by systemd, it won't hook itself up. ("nothing to see here folks...") But if sshd isn't linked to liblzma to begin with, none of the backdoor's code even exists in the processes' page maps.

I'm still downgrading to an unaffected version, of course, but it's nice to know I was never vulnerable just by typing 'ldd `which sshd`' and not seeing liblzma.so.

blcknight2• 2 years ago

Debian testing already has a version called '5.6.1+really5.4.5-1' that is really an older version 5.4, repackaged with a newer version to convince apt that it is in fact an upgrade.

I'm surprised .deb doesn't have a better approach. RPM has epoch for this purpose http://novosial.org/rpm/epoch/index.html

pja2• 2 years ago

Debian packages can have epochs too. I’m not sure why the maintainers haven’t just bumped the epoch here.

Maybe they’re expecting a 5.6.x release shortly that fixes all these issues & don’t want to add an epoch for a very short term packaging issue?

nicolas_174• 2 years ago

.deb has epochs too, but I think Debian developers avoid it where possible because 1:5.4.5 is interpreted as newer than anything without a colon, so it would break eg. packages that depend on liblzma >= 5.0, < 6. There may be more common cases that aren't coming to mind now.

stefanor• 2 years ago

Debian has epochs, but it's a bad idea to use them for this purpose.

Two reasons:

1. Once you bump the epoch, you have to use it forever. 2. The deb filename often doesn't contain the epoch (we use a colon which isn't valid on many filesystems), so an epoch-revert will give the same file name as pre-epoch, which breaks your repository.

So, the current best practice is the +really+ thing.

5p4n9112• 2 years ago

I really like the XBPS way of the reverts keyword in the package template that forces a downgrade from said software version. It's simple but works without any of the troubles RPM epochs have with resolving dependencies as it's just literally a way to tell xbps-install that "yeah, this is a lower version number in the repository but you should update anyway".

NotPractical• 2 years ago

If you're not on a rolling release distro, your version is probably older.

Ironic considering security is often advertised as a feature of rolling release distros. I suppose in most instances it does provide better security, but there are some advantages to Debian's approach (stable Debian, that is).

javajosh• 2 years ago

Ironic considering security is often advertised as a feature of rolling release distros.

Security is a feature of rolling release. But supply-chain attacks like this are the exception to the rule.

leeoniya• 2 years ago

i mean, rolling implies rolling 0-days, too.

pdw2• 2 years ago

Focusing on sshd is the wrong approach. The backdoor was in liblzma5. It was discovered to attack sshd, but it very likely had other targets as well. The payload hasn't been analyzed yet, but _almost everything_ links to libzma5. Firefox and Chromium do. Keepassxc does. And it might have made arbitrary changes to your system, so installing the security update might not remove the backdoor.

junon9• 2 years ago

From what I'm understanding it's trying to patch itself into the symbol resolution step of ld.so specifically for libcrypto under systemd on x86_64. Am I misreading the report?

That's a strong indication it's targeting sshd specifically.

saagarjha6• 2 years ago

It checks for argv[0] == "sshd"

rathermucker• 2 years ago

Tumbleweed has a package: liblzma5-5.6.1.revertto5.4-3.2.x86_64 FYI

account4252• 2 years ago

revertto probably just means "revert to" but it does sound quite italian lol.

colanderman2• 2 years ago

The latest commit from the user who committed those patches is weirdly a simplification of the security reporting process, to not request as much detail:

https://github.com/tukaani-project/xz/commit/af071ef7702debe...

Not sure what to make of this.

caelum19• 2 years ago

Potentially the purpose is that if someone goes to the effort to get those details together, they are more likely to send the same report to other trusted individuals. Maybe it was originally there to add legitimacy, then they got a report sent in, and removed it to slow the spread of awareness

londons_explore10• 2 years ago

Affected versions of XZ Utils

Most people, to find the affected versions, would either have to bisect or delve deep enough to find the offending commit. Either of which would reveal the attacker.

By not asking for the version, there is a good chance you just report "It's acting oddly, plz investigate".

rany_8• 2 years ago

I think the reason is pretty obvious. They want you to waste more time after you've submitted the security report and maximize the amount of back and forth. Basically the hope is that they'd be able to pester you with requests for more info/details in order to "resolve the issue" which would give them more time to exploit their targets.

colanderman2• 2 years ago

That repository is now disabled. But here's a similar change to the .github repository of tukaani-project from @JiaT75 to the bug report template:

    + or create a private Security Advisory instead.

Under a commit titled "Wrap text on Issue template .yaml files."

[1] https://github.com/tukaani-project/.github/commit/44b766adc4...

AdmiralAsshat3• 2 years ago

Yikes! Do you have any info on the individual's background or possible motivations?

b1125• 2 years ago

Yikes indeed. This fix is being rolled out very fast, but what about the entire rest of the codebase? And scripts? I mean, years of access? I'd trust no aspect of this code until a full audit is done, at least of every patch this author contributed.

(note: not referring to fedora here, a current fix is required. But just generally. As in, everyone is rolling out this fix, but... I mean, this codebase is poison in my eyes without a solid audit)

b1125• 2 years ago

This seems to be the account, correct me if wrong (linked from the security email commit link):

https://github.com/JiaT75

I hope authors of all these projects have been alerted.

STest - Unit testing framework for C/C++. Easy to use by simply dropping stest.c and stest.h into your project!

libarchive/libarchive - Multi-format archive and compression library

Seatest - Simple C based Unit Testing

Everything this account has done should be investigated.

Woha, is this legit or some sort of scam on Google in some way?:

https://github.com/google/oss-fuzz/pull/11587

edit: I have to be missing something, or I'm confused. The above author seems to be primary contact for xz? Have they just taken over?? Or did the bad commit come from another source, and a legit person applied it?

A bit confused here.

formerly_proven5• 2 years ago

Well that account also did most of the releases since 5.4.0.

rwmj10• 2 years ago

I handed over all the emails I received to the security team, who I guess will send them "higher". I'll let them analyse it.

5kg3• 2 years ago

There is zero web presence for this person and associated email address.

Looks more likely a fake identity than compromised account.

Retr0id3• 2 years ago

Did you check Chinese social media?

mrb6• 2 years ago

Actually the "jiat0218" user part in his email address jiat0218@gmail.com has a bunch of matches on Taiwanese sites:

https://char.tw/blog/post/24397301

https://forum.babyhome.com.tw/topic/167439

https://bmwcct.com.tw/forums/thread1828.html

johnny224• 2 years ago

I've never had a web presencse for my associated emails due to wanting to avoid spammers. I don't have a false identity.

junon9• 2 years ago

This is all I can find on them.

    carrd.co jiat0218@gmail.com business https://jiat0218@gmail.com.carrd.co
    eBay JiaT75 shopping https://www.ebay.com/usr/JiaT75
    giters jiat0218 coding https://giters.com/jiat0218
    giters JiaT75 coding https://giters.com/JiaT75
    GitHub jiat0218 coding https://github.com/jiat0218
    GitHub JiaT75 coding https://github.com/JiaT75
    Mastodon-meow.so.. jiat0218@gmail.com social https://meow.social/@jiat0218@gmail.com

Beyond that, nothing surefire. (This is all publicly queryable information, if anyone is curious).

heee399• 2 years ago

I am more interest about his git commits https://github.com/JiaT75?tab=overview&from=2021-12-01&to=20... If JiaT75 is a Chinese, then his working log should follow Chinese Holiday, especially Spring Festival and National Holiday. Chinese usually not work on first 3 days of Spring Festival and National Holiday - 2021 2/11 - 2/13 (few commits), 2021 10/1 - 10/3 (nothing) 2022 1/31 - 2/2 (huge commits on 1/31, suspect), 2022 10/1 - 10/3 (nothing) 2023 and 2024, not very much commits. So 2022 1/31 huge commits is a proof that he is not follow Chinese holiday.

But wait, 2021 is his active year, but he missed almost all Aug. Is he on holiday? Who can have such a long holiday? What i can think is a solider who has a long vacation (探亲假). So let's guess he is a solider then it's sense that he worked on Spring Holiday because they need on duty. Let's double check again, if he is a solider, then they will have a holiday on every Aug. 1 because it's liberation army day. I check and no commits on all 4 years Aug. 1.

codear• 2 years ago

i wonder if that avatar, familiarity with C/C++ and Git, and "offering help with open source projects" is just coincidence

https://github.com/JiaT75

https://twitter.com/JiaTan1337/status/1774931375994319244

kind of interesting also to see this account was set up ~2 months ago. if it's a troll, it's a somewhat poor joke.

wpietri• 2 years ago

I get why people are focusing on this bad actor. But the question that interests me more: how many other apparent individuals fit the profile that this person presented before caught?

Phenylacetyl3• 2 years ago

Apparently, many.

It looks like gettext may be containing a part of their attack infrastructure.

https://github.com/microsoft/vcpkg/pull/37199#pullrequestrev...

https://github.com/microsoft/vcpkg/pull/37356/files#diff-e16...

https://github.com/MonicaLiu0311

mrb6• 2 years ago

I would presume it's a state actor. Generally in the blackhat world, attackers have very precise targets. They want to attack this company or this group of individuals. But someone who backdoors such a core piece of open source infrastructure wants to cast a wide net to attack as many as possible. So that fits the profile of a government intelligence agency who is interested in surveilling, well, everything.

Or it could in theory be malware authors (ransomware, etc). However these guys tend to aim at the low hanging fruits. They want to make a buck quickly. I don't think they have the patience and persistence to infiltrate an open source project for 2 long years to finally gain enough trust and access to backdoor it. On the other hand, a state actor is in for the long term, so they would spend that much time (and more) to accomplish that.

So that's my guess: Jia Tan is an employee of some intelligence agency. He chose to present an asian persona, but that's not necessarily who he truly represents. Could be anyone, really: Russia, China, Israel, or even the US, etc.

Edit: given that Lasse Collin was the only maintainer of xz utils in 2022 before Jia Tan, I wouldn't be surprised if the state actor interfered with Lasse somehow. They could have done anything to distract him from the project: introduce a mistress in his life, give him a high-paying job, make his spouse sick so he has to care for her, etc. With Lasse not having as many hours to spend on the project, he would have been more likely to give access to a developer who shows up around the same time and who is highly motivated to contribute code. I would be interested to talk to Lasse to understand his circumstances around 2022.

janc_3• 2 years ago

Or they have just one or a small number of targets, but don’t want the target(s) to know that they were the only target(s), so they backdoor a large number of victims to “hide in the crowd”.

I agree that this is likely a state actor, or at least a very large & wealthy private actor who can play the long game…

dist-epoch5• 2 years ago

According to top comment he committed multiple binary files to xz for the last two years.

Most likely this is not the first backdoor, just the first one to be discovered, so it wasn't two years of work until there were results.

But I still agree that he's probably a state actor.

fullstop10• 2 years ago

If anyone here happens to know Lasse, it might be good to check up on him and see how he's doing.

Delk• 2 years ago

I wouldn't be surprised if the state actor interfered with Lasse somehow

People could also just get tired after years of active maintainership or become busier with life. Being the sole maintainer of an active open source project on top of work and perhaps family takes either a lot of enthusiasm or a lot of commitment. It's not really a given that people want to (or can) keep doing that forever at the same pace.

Someone then spots the opportunity.

I have no idea what the story is here but it might be something rather mundane.

hk__2• 2 years ago

I haven't lost interest but my ability to care has been fairly limited mostly due to longterm mental health issues but also due to some other things. Recently I've worked off-list a bit with Jia Tan on XZ Utils and perhaps he will have a bigger role in the future, we'll see.

https://www.mail-archive.com/xz-devel@tukaani.org/msg00567.h...

jnxx10• 2 years ago

They want to attack this company or this group of individuals. But someone who backdoors such a core piece of open source infrastructure wants to cast a wide net to attack as many as possible.

The stuxnet malware, which compromised Siemens industrial controls to attack specific centrifuges in uranium enrichment plants in Iran, is a counterexample to that.

2OEH8eoCRo05• 2 years ago

It's ridiculous to think it's the US as it would be an attack on Red Hat a US company and an attack on Americans. It's a good way to be dragged in front of Congress.

occamsrazorwit4• 2 years ago

Given the details from another comment [1], it sounds like both maintainers are suspicious. Lasse's behavior has changed recently, and he's been pushing to get Jia Tan's changes into the Linux kernel. It's possible both accounts aren't even run by the original Lasse Collin and Jia Tan anymore.

Edit: Also, Github has suspended both accounts. Perhaps they know something we don't.

[1] https://news.ycombinator.com/item?id=39865810#39866275

Repulsion951314• 2 years ago

Generally in the blackhat world, attackers have very precise targets

Lol, what

wants to cast a wide net to attack as many as possible. So that fits the profile of a government intelligence agency

That's quite backwards. Governments are far more likely to deploy a complex attack against a single target (see also: Stuxnet); other attackers (motivated primarily by money) are far more likely to cast a wide net.

pyrolistical• 2 years ago

Literally this https://xkcd.com/2347/

Havoc• 2 years ago

Bit much speculating about mistresses and poisoned spouses with well anything to go on...

ibic2• 2 years ago

Yes, I believe it's an state actor, and the intention of choosing a typical Chinese name Jia Tan is intentially and malicious.

Muntzer• 2 years ago

Adding some unreadable binary to the source code is a really dangerous thing to do. We also need tools to quickly detect the addition of indentation symbols that can be easily overlooked.

BYW,I had a classmate who used to play DOTA1(on war3) under this name at the University of Science and Technology of China a long time ago, and this was his first girlfriend name (maybe) . His father was a high-ranking official. Then he joined the parent department of the Internal Security Detachment, a secret service that has gained a lot of power in the last few years. I hope I'm not awake . lol.

jpalomaki• 2 years ago

Seems to be a perfect project to hijack. Not too much happening, widely used, long history, single maintainer who no longer has time to manage the project and wants to pass it over.

dang4• 2 years ago

We detached this subthread from https://news.ycombinator.com/item?id=39866275. (It's fine; I'm just trying to prune the top-heavy subthread.)

AviationAtom• 2 years ago

Not a developer but reading the changelogs and commit history from this person seem interesting, as they appear to be some effort consolidate control and push things in the direction of supporting wider dissemination of their backdoor code:

Discussing commits that the other author has since reverted, IFUNC change with Project Zero tests, a focus on embedded, etc.:

https://www.mail-archive.com/xz-devel@tukaani.org/msg00642.h...

Trimming security reporting details:

https://git.tukaani.org/?p=xz.git;a=commitdiff;h=af071ef7702...

electronwill• 2 years ago

"crazytan" is the LinkedIn profile of a security software engineer named Jia Tan in Sunnyvale working at Snowflake, who attended Shanghai Jiao Tong University from 2011 to 2015 and Georgia Institute of Technology from 2015 to 2017. However, this Jia Tan on LinkedIn might not be the same Jia Tan who worked on XZ Utils. Also, the person who inserted the malicious code might be someone else who hijacked the account of the Jia Tan who worked on XZ Utils.

wtznc• 2 years ago

I found a user who seems suspicious to me.

https://github.com/snappyJack/CVE-request-XZ-5.2.5-has-denia...

He understood the software architecture quite early on while working on the following repository. He connected the dots from his other projects and went rogue. (probably to benefit from crypto?). Take a look at his other repositories and code style and recent likes on github. Is he our Jia Tan?

b7kich• 2 years ago

May or may not be related: https://www.linkedin.com › crazytan Jia Tan - Snowflake | LinkedIn

molaeiali• 2 years ago

Has Jia in any way posted a response to the incident?

AdmiralAsshat3• 2 years ago

My assumption would be that he knows the jig is up, and is probably going to do everything he can to jettison the JiaTan account, lest any IPs he uses be turned over to authorities.

source_inform• 2 years ago

An Indian with the name, Jigar (meaning heart) would never address himself as Jigar, as seen in the citation. This would be culturally a bit weird. Unless he is being sarcastic or writing this on some comic note.

Secondly, the use of English is not consistent in what should be from typical Indian. He should be from a foreign background or a very reputed English medium.

The language though seemingly simple for a native English speaker but it seems in this case; a person whose first language: likely is not English.

It is possible that Grammarly or auto correct could have been used to write these. But can't be certain of anything stated above.

I do think that this is a sabotage account with 60% chances unless Mr. Kumar comes out clean, publicly. He is likely a state sponsored actor.

ParetoOptimal• 2 years ago

If you have a recently updated NixOS unstable it has the affected version:

    $ xz --version
    xz (XZ Utils) 5.6.1
    liblzma 5.6.1

EDIT: I've been informed on the NixOS matrix that they are 99% sure NixOS isn't affected, based on conversations in #security:nixos.org

20after42• 2 years ago

"Docs: Simplify SECURITY.md."

https://github.com/tukaani-project/xz/commit/af071ef7702debe...

Removes instructions about details relevant to security reports. Heh, nice one.

PedroBatista• 2 years ago

Given the recent ( not so recent ) attacks/"bugs" I feel there is a need to do more than the already hard task of investigating and detecting attacks but also to bring IRL consequences to these people.

My understanding is that right now it's pretty much a name and shame of people who most of the time aren't even real "people" but hostile agents either working for governments or criminal groups ( or both )

Getting punched in the face is actually a necessary human condition for a healthy civilization.

buildbot8• 2 years ago

In the article it says CISA was notified - that sounds like it's going to be a federal investigation if nothing else. If I was this person, I wouldn't be in the USA (or any US friendly nation) ASAP.

graemep2• 2 years ago

One of Jia Tan's recent contributions is "Speed up CRC32 calculation on LoongArch" I would guess the odds are that this is not someone in the US.

sneak• 2 years ago

What law do you think is being broken here?

computerfriend4• 2 years ago

From their Git commits, they're in China's time zone.

rubymamis• 2 years ago

We shouldn't rule out the probability that this account is from a U.S. agency as well.

offmycloud• 2 years ago

CISA Advisory: https://www.cisa.gov/news-events/alerts/2024/03/29/reported-...

Note that it say "Fedora 41" in the CISA page link to Red Hat, but Red Hat changed the blog title to "Fedora 40" and left the HTML page title as "Fedora 41".

weinberg• 2 years ago

And I bet if it ended up on a NATO system, things escalate quickly for the person / nation states being scrutinized (https://www.nato.int/cps/en/natohq/topics_78170.htm)

oceanplexian3• 2 years ago

A federal investigation into what, itself? The primary actors doing this type of thing are the US Government.

progbits2• 2 years ago

Getting punched in the face is actually a necessary human condition for a healthy civilization.

Aside from signed commits, we need to bring back GPG key parties and web of trust. When using a project you would know how many punches away from the committers you are.

woodruffw3• 2 years ago

PGP is more famous for "web of trust" topologies, not chains of trust.

For all of their nerd cred, key parties didn't accomplish very much (as evidenced by the fact that nothing on the Internet really broke when the WoT imploded a few years ago[1]). The "real" solution here is mostly cultural: treating third-party software like the risky thing it actually is, rather than a free source of pre-screened labor.

[1] https://inversegravity.net/2019/web-of-trust-dead/

EthanHeilman3• 2 years ago

The web of punches?

throw10920• 2 years ago

Getting punched in the face is actually a necessary human condition for a healthy civilization.

This is factually false - in fact, it's literally the direct opposite of the truth. "Getting punched in the face" is base violence that is incompatible with a healthy civilization. A good government with a robust justice system is what is actually needed for a healthy civilization.

bagels4• 2 years ago

Is this a crime? Has anyone been prosecuted for adding a backdoor like this?

pvg2• 2 years ago

Has anyone been prosecuted for adding a backdoor

Google up Randal Schwartz. Caution: clickhole.

bagels4• 2 years ago

Seems a little different. Based on a quick read, he gained unauthorized access to systems.

In this case, backdoor code was offered to and accepted by xz maintainers.

amiga3864• 2 years ago

As far as I remember, he added no backdoors.

He was a consultant/sysadmin for Intel, and he did 3 things which he thought his employer would support, and was astonished to find that not only did his employer not support, but actively had him prosecuted for doing it. Ouch.

1. He ran a reverse-proxy on two machines so he could check in on them from home.

2. He used the crack program to find weak passwords.

3. He found a weak password, and used it to log into a system, which he copied the /etc/shadow file from to look for additional weak passwords.

https://www.giac.org/paper/gsec/4039/intel-v-randal-l-schwar...

https://web.archive.org/web/20160216204357/http://www.lightl...

He didn't try and hide his activities, and didn't do anything else untoward, it was literally just these things which most people wouldn't bat an eyelid at. These days, it is completely normal for a company to provide VPNs for their employees, and completely normal to continually scan for unexpected user accounts or weak passwords. But... because he didn't explain this to higher-ups and get their buy-in, they prosecuted him instead of thanking him.

move-on-by• 2 years ago

Fascinating. Just yesterday the author added a `SECURITY.md` file to the `xz-java` project.

If you discover a security vulnerability in this project please report it privately. *Do not disclose it as a public issue.* This gives us time to work with you to fix the issue before public exposure, reducing the chance that the exploit will be used before a patch is released.

Reading that in a different light, it says give me time to adjust my exploits and capitalize on any targets. Makes me wonder what other vulns might exist in the author's other projects.

xyst2• 2 years ago

90 day dark window for maintainers is SOP though. Then after 90 days, it’s free game for public disclosure

szundi• 2 years ago

How many of people like this one exist?

ldayley• 2 years ago

If this question had a reliable (and public) answer then the world would be a very different place!

That said, this is an important question. We, particularly those us who work on critical infrastructure or software, should be asking ourselves this regularly to help prevent this type of thing.

Note that it's also easy (and similarly catastrophic) to swing too far the other way and approach all unknowns with automatic paranoia. We live in a world where we have to trust strangers every day, and if we lose that option completely then our civilization grinds to a halt.

But-- vigilance is warranted. I applaud these engineers who followed their instincts and dug into this. They all did us a huge service!

EDIT: wording, spelling

hulitu2• 2 years ago

How many of people like this one exist?

I guess every 3 letter agency has at least one. You can do the math. They havent't learned anything after Solar Winds.

ncr1004• 2 years ago

Security Researchers: Is this request-for-private-disclosure + "90-days before public" reasonable?

It's a SEVERE issue, to my mind, and 90 days seems too long to me.

bawolff8• 2 years ago

Whether its reasonable is debatable, but that type of time frame is pretty normal for things that aren't being actively exploited.

This situation is perhaps a little different as its not an accidental bug waiting to be discovered but an intentionally placed exploit. We know that a malicious person already knows about it.

cjbprime5• 2 years ago

In this particular case, there is a strong reason to expect exploitation in the wild to already be occurring (because it's an intentional backdoor) and this would change the risk calculus around disclosure timelines.

But in the general case, it's normal for 90 days to be given for the coordinated patching of even very severe vulnerabilities -- you are giving time not just to the project maintainers, but to the users of the software to finish updating their systems to a new fixed release, before enough detail to easily weaponize the vulnerability is shared. Google Project Zero is an example of a team with many critical impact findings using a 90-day timeline.

sterlind• 2 years ago

I think you have to take the credibility of the maintainer into account.

If it's a large company, made of people with names and faces, with a lot to lose by hacking its users, they're unlikely to abuse private disclosure. If it's some tiny library, the maintainers might be in on it.

Also, if there's evidence of exploitation in the wild, the embargo is a gift to the attacker. The existence of a vulnerability in that case should be announced, even if the specifics have to be kept under embargo.

decoy78• 2 years ago

imho it depends on the vuln. I've given a vendor over a year, because it was a very low risk vuln. This isn't a vuln though - this is an attack.

sidewndr462• 2 years ago

I've always laughed my ass off at the idea of a disclosure window. It takes less than a day to find RCE that grants root privileges on devices that I've bothered to look at. Why on earth would I bother spending months of my time trying to convince someone to fix something?

BartjeD• 2 years ago

The fraudulent author must have enjoyed the 'in joke' -- He's the one create vulnerabilities..

tw042• 2 years ago

Honestly it seems like a state-based actor hoping to get whatever high value target compromised before it's made public. Reporting privately buys them more time, and allows them to let handlers know when the jig is up.

returningfory23• 2 years ago

A couple of years ago I wrote a Go library that wraps the xz C code and allows you to do xz compression in Go: https://github.com/jamespfennell/xz

About a week ago I received the first PR on that repo, to upgrade to 5.6.1. I thought it was odd to get such a random PR...it's not the same GitHub account as upstream though.

pinko3• 2 years ago

it's not the same GitHub account as upstream

This is valuable information, and a sign that this may be the tip of an iceberg.

Bromeo• 2 years ago

I don't want to read too much into it, but the person (supposedly) submitting the PR seems to work at 1Password since December last year, as per his Linkedin. (And his Linkedin page has a link to the Github profile that made the PR).

returningfory23• 2 years ago

Yeah the GitHub account looks really really legitimate. Maybe it was compromised though?

bombcar18• 2 years ago

If I were trying to compromise supply chains, getting into someplace like 1Password would be high up on the list.

Poor guy, he's probably going to get the third degree now.

switch007• 2 years ago

As a 1Password user, I just got rather nervous.

lelandbatey• 2 years ago

They're definitely a real person. I know cause that "1Password employee since December" is a person I know IRL and worked with for years at their prior employer. They're not a no-name person or a fake identity just FYI. Please don't be witch hunting; this genuinely looks like an unfortunate case where Jared was merely proactively doing their job by trying to get an externally maintained golang bindings of XZ to the latest version of XZ. Jared's pretty fantastic to work with and is definitely the type of person to be filing PRs on external tools to get them to update dependencies. I think the timing is comically bad, but I can vouch for Jared.

https://github.com/jamespfennell/xz/pull/2

arp2425• 2 years ago

As a bit of an aside, I would never accept a PR like this, and would always update $large_vendored_dependency myself. This is unreviewable, and trivial to insert any backdoor (unless you go through the motions of updating it yourself and diffing, at which point the PR becomes superfluous). I'd be wary even from a well-known author unless I knew them personally on some level (real-life or via internet). Not that I wouldn't trust them, but people's machines or accounts can get compromised, people can have psychotic episodes, things like that. At the very least I'd like to have some out-of-band "is this really you?" signal.

This is how I once inserted a joke in one of our (private) repos that would randomly send cryptic messages to our chat channel. This was pretty harmless and just a joke (there's some context that made it funny), but it took them years to find it – and that was only because I told them after I quit.

That said, looking at the GitHub account I'd be surprised if there's anything nefarious going on here. Probably just someone using your repo, seeing it's outdated, and updating it.

LVB• 2 years ago

The (most?) popular SQLite driver for Go often gets PRs to update the SQLite C amalgamation, which the owner politely declines (and I appreciate him for that stance, and for taking on the maintenance burden it brings).

e.g., https://github.com/mattn/go-sqlite3/pull/1042#issuecomment-1...

creatonez2• 2 years ago

In this case, the project is using Git submodules for its vendored dependencies, so you can trivially cryptographically verify that they have vendored the correct dependency just by checking the commit hash. It looks really crazy on Github but in most git clients it will just display the commit hash change.

5kg3• 2 years ago

The backdoor (test binary blob and autoconf) is not part of the pull request.

cbmuser• 2 years ago

There was also a bug report in Debian which requested updating xz-utils to 5.6.1: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1067708

roflmaostc• 2 years ago

That's the same Hans Jansen mentioned here: https://boehs.org/node/everything-i-know-about-the-xz-backdo...

icambron• 2 years ago

IMO your prior on this should be that it's most likely just someone innocently updating a dependency.

jaredallard22• 2 years ago

Hey all, I’m the author of that PR. Just posted to Github with additional context: https://github.com/jamespfennell/xz/pull/2#issuecomment-2027...

blueflow• 2 years ago

That sucks to have people write mails to your employer...

SheinhardtWigCo• 2 years ago

I appreciated your detailed update!

ikekkdcjkfke2• 2 years ago

The dopamine hits from updating stuff should come to an end, it should be thought of as adding potentially new bugs or exploits, unless the update fixes a CVE. Also Github needs to remove the green colors and checkmarks in PR's to prevent these dopamine traps from overriding any critical thinking

squigz• 2 years ago

Internet detectives at work in this thread!

baxtr• 2 years ago

Suddenly anything like that becomes super suspicious.

I wonder how this will affect the OS community in general.

account4252• 2 years ago

More caution might not be a bad thing.

notyoutube• 2 years ago

Is the solution against such attacks in the future only to scrutinize more, or are there other reasonable options in terms of hardening?

JanisErdmanis3• 2 years ago

The lesson here seems to not depend on tools written in languages that have complex, obscure build systems and no one is either able or interested to read. Using tools rewritten in Rust, Go or any other languege which resolves dependencies within project seems the only way to do hardening here.

Lichtso3• 2 years ago

Once somebody actually does this people are gonna complain the same as always: "The sole purpose of your project is to rewrite perfectly fine stuff in Rust for the sake of it" or something along these lines.

blcknight2• 2 years ago

I agree there's safer languages than C, but nobody reads the 50,000 lines changed when you update the vendoring in a random golang project. It would be easy to introduce something there that nobody notices too.

ok12345611• 2 years ago

Wouldn't a supply chain attack like this be much worse with Rust and Cargo because of the fact it's not just a single dynamic library that needs to be reinstalled system-wise, but, instead, every binary would require a new release?

klysm• 2 years ago

People are going to be upset with this perspective but I completely agree. The whole autoconf set of tools is a complete disaster.

arp2425• 2 years ago

You don't need a complex obscure build system for most C code. There's a lot of historical baggage here, but many projects (including xz, I suspect) can get away with a fairly straight-forward Makefile. Double so when using some GNU make extensions.

bonzini13• 2 years ago

Rust is the worst in terms of build system transparency. Ever heard of build.rs? You can hide backdoors in any crate, or in any crate's build.rs, or the same recursively.

msm_6• 2 years ago

Is this really the lesson here? We are talking about a maintainer here, who had access to signing keys and a full access to the repository. Deb packages which were distributed are also different than the source code. Do you honestly believe that the (arguably awful) autotools syntax is the single root cause of this mess, Rust will save us from everything, and this is what we should take away from this situation?

delfinom5• 2 years ago

I call bullshit.

The fundamental problem here was a violation of chain of trust. Open source is only about the source being open. But if users are just downloading blobs with prebuilt binaries or even _pre-generated scripts_ that aren't in the original source, there is nothing a less-obscure build system will save you from as you are putting your entire security on the chain of trust being maintained.

joshhansen5• 2 years ago

Am I crazy thinking libraries shouldn't be able to provide _other libraries'_ symbols without the other libraries' "permission"? What am I missing?

perihelions• 2 years ago

Imagine a more competent backdoor attempt on xz(1)—one that wouldn't have been noticed this quickly. xz is everywhere. They could pull off a "reflections on trusting trust": an xz which selectively modifies a tiny subset of the files it sees, like .tar.xz software tarballs underlying certain build processes. Not source code tarballs (someone might notice)—tarballs distributing pre-compiled binaries.

edit to add: Arch Linux' entire package system used to run on .tar.xz binaries (they switched to Zstd a few years ago [0]).

[0] https://news.ycombinator.com/item?id=19478171 ("Arch Linux propose changing compression method from xz to zstd (archlinux.org)")

nolist_policy5• 2 years ago

deb packages are xz compressed...

1oooqooq3• 2 years ago

my freaking kernels/initrd are xz or zstd compressed!

nolist_policy5• 2 years ago

... and Debian is very serious about it: https://fulda.social/@Ganneff/112184975950858403

joeyh4• 2 years ago

A backdoored xz could also run payloads hidden inside other xz files, allowing targeted attacks.

Phenylacetyl3• 2 years ago

The same authors have also contributed to Zstd

joeyh4• 2 years ago

details please? I do not see any such contributions to https://github.com/facebook/zstd

alright2565• 2 years ago

https://github.com/tukaani-project/tukaani-project.github.io...

Note: GitHub automatically includes two archives Source code (zip) and Source code (tar.gz) in the releases. These archives cannot be disabled and should be ignored.

The author was thinking ahead! Latest commit hash for this repo: 8a3b5f28d00ebc2c1619c87a8c8975718f12e271

o11c5• 2 years ago

For a long time, there was one legitimately annoying disadvantage to the git-generated tarballs though - they lost tagging information. However, since git 2.32 (released June 2021; presumably available on GitHub by August 2021 when they blogged about it) you can use `$Format:%(describe)$` ... limited to once per repository for performance reasons.

rany_8• 2 years ago

Except this change was made in 2023, it is just scary how good this threat actor was.

legobmw99• 2 years ago

I believe they also do not include sub modules, which is a big disadvantage for some projects

rom1v• 2 years ago

Btw, this is not the only project providing a source tarball different from the git repo, for example libusb also does this (and probably others):

- https://github.com/libusb/libusb/issues/1468#issuecomment-19...

- https://github.com/orgs/community/discussions/6003

cryptonector7• 2 years ago

It's very common in autoconf codebases because the idea is that you untar and then run `./configure ...` rather than `autoreconf -fi && ./configure ...`. But to do that either you have to commit `./configure` or you have to make a separate tarball (typically with `make dist`). I know because two projects I co-maintain do this.

londons_explore10• 2 years ago

I think the lesson here for packagers is that binary testdata should not be present while doing the build.

It is too easy to hide things in testdata.

yencabulator4• 2 years ago

Nice idea, but then you just hide the attack in logo.png that gets embedded in the binary. Less useful for libraries, works plenty good for web/desktop/mobile.

consumer4512• 2 years ago

This entire thread is above my pay grade, but isn’t minimizing the attack surface always a good thing?

wood_spirit• 2 years ago

A lot of eyes will be dissecting this specific exploit, and investigating this specific account, but how can we find the same kind of attack in a general way if it’s being used in other projects and using other contributor names?

londons_explore10• 2 years ago

Note that the malicious binary is fairly long and complex.

This attack can be stopped by disallowing any binary testdata or other non-source code to be on the build machines during a build.

You could imagine a simple process which checks out the code, then runs some kind of entropy checker over the code to check it is all unminified and uncompressed source code, before finally kicking off the build process.

autogenerated files would also not be allowed to be in the source repo - they're too long and could easily hide bad stuff. Instead the build process should generate the file during the build.

trulyrandom• 2 years ago

This requires a more comprehensive redesign of the build process. Most Linux distributions also run the tests of the project they're building as part of the build process.

treffer6• 2 years ago

1. Everything must be visible. A diff between the release tarball and tag should be unacceptable. It was hidden from the eyes to begin with.

2. Build systems should be simple and obvious. Potentially not even code. The inclusion was well hidden.

3. This was caught through runtime inspection. It should be possible to halt any Linux system at runtime, load debug symbols and map _everything_ back to the source code. If something can't map back then regard it as a potentially malicious blackbox.

There has been a strong focus and joint effort to make distributions reproducible. What we haven't managed though is prove that the project compromises only of freshly compiled content. Sorta like a build time / runtime "libre" proof.

This should exist for good debugging anyway.

It wouldn't hinder source code based backdoors or malicious vulnerable code. But it would detect a backdoor like this one.

Just an initial thought though, and probably hard to do, but not impossibly hard, especially for a default server environment.

Avamander• 2 years ago

More reproducible builds, maybe even across distributions? Builds based on specific commits (no tarballs like in this case), possibly signed (just for attribution, not for security per se)? Allow fewer unsafe/runtime modifications The way oss-fuzz ASAN was disabled should've been a warning on its own, if these issues weren't so common.

I'm not aware of any efforts towards it, but libraries should also probably be more confined to only provide intended functionality without being able to hook elsewhere?

JonChesterfield• 2 years ago

The Guix full source bootstrap is looking less paranoid as time goes on

account4252• 2 years ago

I haven't looked at Guix but in the discussions around this exploit for NixOS they mentioned that regenerating autoshit for xz-utils would not be something they can/want to do because that would add a lot more dependencies to the bootstrap before other packages can be build. Kind of funny how a requirement for bootstrapped builds can add a requirement for trusting not-quite-binaries-but-also-not-really-source blobs.

mac-chaffee• 2 years ago

Build-related fixes are only treating the symptoms, not the disease. The real fix would be better sandboxing and capability-based security[1] built into major OSes which make backdoors a lot less useful. Why does a compression library have the ability to "install an audit hook into the dynamic linker" or anything else that isn't compressing data? No amount of SBOMs, reproducible builds, code signing, or banning binaries will change the fact that one mistake anywhere in the stack has a huge blast radius.

[1] https://en.wikipedia.org/wiki/Capability-based_security

mkleczek• 2 years ago

That's why I always raise concerns about JEP 411 - removal of SecurityManager from Java without any replacement.

someguydave2• 2 years ago

Just ban autotools

afiodorov2• 2 years ago

We should be able to produce a tar and a proof that tar was produced from a specific source code.

Quote from the article:

    That line is not in the upstream source of build-to-host, nor is build-to-host used by xz in git.

Zero Knowledge virtual machines, like cartesi.io, might help with this. Idea is to take the source, run a bunch of computational steps (compilation & archiving) and at the same time produce some kind of signature that certain steps were executed.

The verifiers can then easily check that the signature and indeed be convinced that the code was executed as it is claimed and source code wasn't tampered with.

The advantage of Zero-Knowledge technology in this case is that one doesn't need to repeat the computational steps themselves nor rely on a trusted party to do it for them (like automated build - that can also be compromised by the state actors). Just having the proof solves this trust problem mathematically: if you have the proof & the tar, you can quickly check source code that produced the tar wasn't modified.

renonce• 2 years ago

I don’t think zero knowledge systems are practical at the moment. It will take over around 8 orders of magnitude more compute and memory to produce a ZKP proof of generic computation like compilation. Even 2 orders of magnitude is barely acceptable.

Tenobrus• 2 years ago

It looks like the person who added the backdoor is in fact the current co-maintainer of the project (and the more active of the two): https://tukaani.org/about.html

kzrdude6• 2 years ago

Makes me wonder if he's an owner of the github organization, and what happens with it now?

kzrdude6• 2 years ago

In various places they say Lasse Collin is not online right now, but he did make commits a week ago https://git.tukaani.org/?p=xz.git;a=summary

Scaevolus• 2 years ago

It's wild that this could have laid dormant for far longer if the exploit was better written-- if it didn't spike slow down logins or disturb valgrind.

frankjr2• 2 years ago

Now consider that your average Linux distribution pulls in tens of thousands of packages, each of which can be similarly compromised. Pretty scary to think about.

RGamma• 2 years ago

The terrible desktop software security model of weak/essentially non-existent security boundaries at run and compile time makes this all the more spicy.

Computer security for billions runs on the simultaneous goodwill of many thousand contributors. Optimistically said it's actually a giant compliment to the programming community.

And this is not even talking about hardware backdoors that are a million times worse and basically undetectable when done well. The myriad ways to betray user trust at any level of computation make me dizzy...

afh14• 2 years ago

I have exactly 719 packages on my Gentoo box, just rebuilt everything as part of the profile 23 upgrade.

Luker88• 2 years ago

@people who write github scanners for updates and security issues (dependabot and the like)

Can we start including a blacklist of emails and names of contributors (with reasons/links to discussions)?

I can't track them and I don't want them in my projects.

Might not be very helpful as it is easy to create new identities, but I see no reason to make it easier for them. Also, I might approach differently someone with lots of contributions to known projects than a new account, so it still helps.

arp2425• 2 years ago

It takes a minute to create a new email address. And you can change or fake an email address on a git commit trivially. You, too, can writing code as anyone you want by just doing "git commit --author='Joe Biden <icecream@whitehouse.gov>'". On the internet nobody knows you're Joe Biden.

nine_k• 2 years ago

You can write a rather simple GitHub action that would do that: look at a PR and reject / close it if you don't like it for some reason. AFAIK open-source projects have a free quota of actions.

OTOH sticking to the same email for more than one exploit might be not as wise for a malicious agent.

the84724• 2 years ago

github already suspended the account

pfortuny• 2 years ago

Unfortunately, this is how good bad actors work: with a very long-term point of view. There is no “harmless” project any more.

ametrau• 2 years ago

Probably a state actor. You can look far into the future when you’re working for the party.

DrewRWx• 2 years ago

And that long term perspective could be used constructively instead!

calvinmorrison• 2 years ago

Which like, also wouldn't be totally weird if I found out that the xz or whatever library maintainer worked for the DoE as a researcher? I kind of expect governments to be funding this stuff.

hangonhn4• 2 years ago

I imagine it might be easier to just compromise a weakly protected account than to actual put in a 2 years long effort with real contributions. If we mandated MFA for all contributors who contribute to these really important projects then we can know with greater certainty if it was really a long con vs. a recently compromised account.

__s• 2 years ago

This seems like a great way to invest in supporting open source projects in meantime if these projects are being used by these actors. Just have to maintain an internal fork without the backdoors

Maybe someone can disrupt the open source funding problem by brokering exploit bounties /s

the84724• 2 years ago

github already mandates MFA for members of important projects

guinea-unicorn3• 2 years ago

I find it funny how MFA is treated as if it would make account takeover suddenly impossible. It's just a bit more work, isn't it? And a big loss in convenience.

I'd much rather see passwords entirely replaced by key-based authentication. That would improve security. Adding 2FA to my password is just patching a fundamentally broken system.

LtWorf15• 2 years ago

As I said recently in a talk I gave, 2FA as implemented by pypy or github is meaningless, when in fact all actions are performed via tokens that never expire, that are saved inside a .txt file on the disk.

tw042• 2 years ago

For some random server, sure. For a state sponsored attack? Having an embedded exploit you can use when convenient, or better yet an unknown exploit affecting every linux-based system connected to the internet that you can use when war breaks out - that's invaluable.

EasyMark2• 2 years ago

they might not have been playing the long con. maybe approached by actors willing to pay them a lot of money to try and slip in a back door. I'm sure a deep dive into code contributions would clear that up for anyone familiar with the code base and some free time.

gamer1914• 2 years ago

This PR from July 8 2023 is suspicious, so it was very likely a long con: https://github.com/google/oss-fuzz/pull/10667

beginner_• 2 years ago

Not MFA but git commit signing. I don't get why such core low-level projects don't mandate it. MFA doesn0t help if a github access token is stolen and I bet most of use such a token for pushing from an IDE.

Even if an access token to github is stolen, the sudden lack of signed commit should raise red flags. github should allow projects to force commit signing (if not already possible).

Then the access token plus the singing key would need to be stolen.

But of course all that doesn't help in the here more likley scenario of a long con by a state-sponsored hacker or in case of duress (which in certain countries seems pretty likley to happen)

bobba273• 2 years ago

This is a state sponsored event. Pretty poorly executed though as they were tweaking and modifying things in their and other tools after the fact though.

As a state sponsored project. What makes you think this is their only project and that this is a big setback? I am paranoid myself to think yesterdays meeting went like : "team #25 has failed/been found out. Reallocate resources to the other 49 teams."

jnxx10• 2 years ago

And, Joey Hess has counted at least 750 commits to xz from that handle.

https://hachyderm.io/@joeyh/112180715824680521

This does not look trust-inspiring. If the code is complex, there could be many more exploits hiding.

ebfe1• 2 years ago

clickhouse has pretty good github_events dataset on playground that folks can use to do some research - some info on the dataset https://ghe.clickhouse.tech/

Example of what this user JiaT75 did so far:

https://play.clickhouse.com/play?user=play#U0VMRUNUICogRlJPT...

pull requests mentioning xz, 5.6 without downgrade, cve being mentioned in the last 60 days:

https://play.clickhouse.com/play?user=play#U0VMRUNUIGNyZWF0Z...

waynesonfire• 2 years ago

750 commits... is xz able to send e-mails yet?

indigodaddy2• 2 years ago

Anyone have any level of confidence that for example EL7/8 would not be at risk even if more potential exploits at play?

jnxx10• 2 years ago

If this is a conspiracy or a state-sponsored attack, they might have gone specifically for embedded devices and the linux kernel. Here archived from tukaani.org:

https://web.archive.org/web/20110831134700/http://tukaani.or...

XZ Embedded is a relatively small decompressor for the XZ format. It was developed with the Linux kernel in mind, but is easily usable in other projects too.
*Features*
* Compiled code 8-20 KiB
[...]
* All the required memory is allocated at initialization time.

This is targeted at embedded and real-time stuff. Could even be part of boot loaders in things like buildroot or RTEMS. And this means potentially millions of devices, from smart toasters or toothbrushes to satellites and missiles which most can't be updated with security fixes.

codedokode3• 2 years ago

If the code is complex, there could be many more exploits hiding.

Then the code should not be complex. Low-level hacks and tricks (like pointer juggling) should be not allowed and simplicity and readability should be preferred.

pmarreck3• 2 years ago

I have some questions.

1) Are there no legit code reviews from contributors like this? How did this get accepted into main repos while flying under the radar? When I do a code review, I try to understand the actual code I'm reviewing. Call me crazy I guess!

2) Is there no legal recourse to this? We're talking about someone who managed to root any linux server that stays up-to-date.

moritonal• 2 years ago

Warning, drunk brain talking. But a LLM driven email based "collaborator" could play a very long gMw adding basic features to a code made whilst earning trust backed by a generated online presence. My money is on a resurgance in the Web of Trust.

w4ffl352• 2 years ago

Clearly a human is even better at it.

llmblockchain3• 2 years ago

State level actor? China?

jnxx10• 2 years ago

The web of trust is a really nice idea, but it works badly against that kind of attacks. Just consider that in the real world, most living people (all eight billions) are linked by only six degrees of separation. It really works, for code and for trusted social relations (like "I lend you 100 bucks and you pay me them back when you get your salary") mostly when you know the code author in person.

This is also not a new insight. In the beginning of the naughties, there was a web site named kuro5hin.org, which experiemented with user ratings and trust networks. It turned out impossible to prevent take-overs.

cyanydeez• 2 years ago

State actors have equaly long horizons to compromise

lucasRW• 2 years ago

More likely that the account of that dev was breawched, dont you think ?

dang4• 2 years ago

(I detached this subthread from https://news.ycombinator.com/item?id=39866275, for the sake of pruning the top heavy thread.)

n3uman• 2 years ago

https://github.com/tukaani-project/tukaani-project.github.io... Does this mean anything that it changed to a parameter??

danielhlockard• 2 years ago

no. unlikely.

Aissen• 2 years ago

Looks like one of the backdoor authors even went and disabled the feature the exploit relied on directly on oss-fuzz to prevent accidental discovery: https://social.treehouse.systems/@Aissen/112180302735030319 https://github.com/google/oss-fuzz/pull/10667

But luckily there was some serendipity: "I accidentally found a security issue while benchmarking postgres changes." https://mastodon.social/@AndresFreundTec/112180083704606941

miduil4• 2 years ago

This is getting addressed here: https://github.com/google/oss-fuzz/issues/11760

nialv7• 2 years ago

This in of itself can be legitimate. ifunc has real uses and it indeed does not work when sanitizer is enabled. Similar change in llvm: https://github.com/llvm/llvm-project/commit/1ef3de6b09f6b21a...

kzrdude6• 2 years ago

Because of the exploit, so, why should we use configurations in production that were not covered by these tests?

lostmsu4• 2 years ago

Could that commit also be made by a bad actor?

throwaway2902• 2 years ago

and that was in mid 2023. Very funny that Wikipedia on this issue says

It is unknown whether this backdoor was intentionally placed by a maintainer or whether a maintainer was compromised

Yeah, if you've been compromised for a year your attacker is now your identity. Can't just wave hands, practice infosec hygiene

mdip• 2 years ago

Anyone keeping current with OpenSUSE Tumbleweed got a update...downgrade. Prior to `zypper dup --no-allow-vendor-change` I had 5.6.0, now I'm at 5.4.6.

intel_brain2• 2 years ago

I see `5.6.1.revertto5.4-3.2`

weinzierl2• 2 years ago

The backdoor is not in the C source directly, but a build script uses data from files in the test dir to only create the backdoor in the release tars. Did I summarize that correctly?

soneil• 2 years ago

That's how I understand it. A build script that's in the releases tarballs but not the git repo, checks to see if it's being run as part of the debian/build or rpm build processes, and then injects content from one of the "test" files.

bombcar18• 2 years ago

I could imagine another similar attack done against an image processing library, include some "test data" of corrupted images that should "clean up" (and have it actually work!) but the corruption data itself is code to be run elsewhere.

sylware2• 2 years ago

This is why the less the better... even if it means less comfortable... to a certain point obviously. And that includes SDKs...

hgs32• 2 years ago

I don't understand why you were downvoted. Having fewer moving parts does make it easier to catch issues.

sylware2• 2 years ago

Everything which is not engaging in licking Big Tech balls (open source or not) on HN is served with severe downvoting, that most of the time (probably real trash human beings or AI trolls with headless blink/geeko|webkit).

crispyambulance3• 2 years ago

I am not embarrassed to say... is there anything in there that someone who runs a server with ssh needs to know?

I literally can't make heads or tails of the risk here. All I see is the very alarming and scary words "backdoor" and "ssh server" in the same sentence.

If I am keeping stuff up to date, is there anything at all to worry about?

pxx• 2 years ago

You should probably not be running your own publicly-accessible ssh servers if this email is not sufficient to at least start figuring out what your next actions are.

The email itself comes with an evaluation script to figure out if anything is currently vulnerable to specifically this discovery. For affected distributions, openssh servers may have been backdoored for at least the past month.

crispyambulance3• 2 years ago

Yet here I am, getting up every morning and getting dressed and tying my shoes all by myself, and then maintaining a small number of servers that have openssh on them!

Thanks, though, for pointing out the little script at the very end of that technical gauntlet of an email intended for specialists. I had gotten through the first 3 or 4 paragraphs and had given up.

What I should have done is just googled CVE-2024-3094, whatever, still glad I asked.

ShamelessC• 2 years ago

You should probably not be running your own publicly-accessible ssh servers if this email is not sufficient to at least start figuring out what your next actions are.

That seems like a fairly unreasonable stance.

dualbus2• 2 years ago

I literally can't make heads or tails of the risk here. All I see is the very alarming and scary words "backdoor" and "ssh server" in the same sentence.

From what I've read, there is still lots of unknowns about the scope of the problem. What has been uncovered so far indicates it involves bypassing authentication in SSH.

In https://gist.github.com/thesamesam/223949d5a074ebc3dce9ee78b..., Sam James points out

If this payload is loaded in openssh sshd, the RSA_public_decrypt function will be redirected into a malicious implementation. We have observed that this malicious implementation can be used to bypass authentication. Further research is being done to explain why.

Thus, an attacker maybe could use this to connect to vulnerable servers without needing to authenticate at all.

crispyambulance3• 2 years ago

Thanks, that gist is a really lucid explanation for normal folks.

sschueller• 2 years ago

So much for a quiet Easter holiday. Fuck

multimoon• 2 years ago

It seems like based on the (very well written) analysis that this is a way to bypass ssh auth, not something that phones out which would've been even scarier.

My server runs arch w/ a LTS kernel (which sounds dumb on the surface, but was by far the easiest way to do ZFS on Linux that wasn't Ubuntu) and it seems that since I don't have SSH exposed to the outside internet for good reason, and my understanding is Arch never patched shhd to begin with that I and most people who would be in similar situations to me are unaffected.

Still insane that this happened to begin with, and I feel bad for the Archlinux maintainers who are now going to feel more pressure to try to catch things like this.

NekkoDroid4• 2 years ago

Being included via libsystemd isn't the only way ssh can load liblzma, it can come as an indirect dependency of Selinux (and its PAM stack) IIUC. Which makes it even a bit more funny (?) since Arch also doesn't officially support any Selinux stuff.

There might be other ways sshd might pull in lzma, but those are the 2 ways I saw commonly mentioned.

On a different note, pacman/makepkg got the ability to checksum source repository checkouts in 6.1.

LeoPanthera• 2 years ago

xz is just a horribly designed format, and always has been. If you use it, please switch to Lzip. Same compression level, but designed by someone competent.

https://www.nongnu.org/lzip/

https://www.nongnu.org/lzip/xz_inadequate.html

someguydave2• 2 years ago

Thanks for that link, lzip sounds useful

account4252• 2 years ago

Someone competent? More like a drama queen butthurt that his pet project did not win the popularity contest. Not the kind of person I want to rely on for important tools.

psi-jack2• 2 years ago

Be an asshole elsewhere. This makes me LESS want to use lzip because of such aggressive non-useful slander and just plain nonsense.

gmnon• 2 years ago

Funny how Lasse Collin started to ccing himself and Jia Tan from 2024-03-20 (that was a day of tons of xz kernel patches), he never did that before. :)

https://lore.kernel.org/lkml/20240320183846.19475-2-lasse.co...

ncr1004• 2 years ago

Also interesting, to me, how the GMail account for the backdoor contributor ONLY appears in the context of "XZ" discussions. Google their email address. Suggests a kind of focus, to me, and a lack of reality / genuineness.

fullstop10• 2 years ago

This also means that Google might know who they are, unless they were careful to hide behind VPN or other such means.

bombcar18• 2 years ago

This is extremely suspicious.

It looks like someone may have noticed a unmaintained or lightly maintained project related to various things, and moved to take control of it.

Otherwhere in the discussion here someone mentions the domain details changed; if you have control of the domain you have control of all emails associated with it.

ui2RjUen875bfFA6• 2 years ago

those pipe usages are quite suspicious

https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-n...

pipeing into this shell script which now uses "eval"

https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-n...

i guess this will be revisited and removed soon

Hackbraten8• 2 years ago

pipeing into this shell script which now uses "eval"

I don’t actually see an issue with that `eval`. Why would one consider running `xz` followed by `eval`-ing its output more insecure than just running `xz`? If `xz` wants to do shenanigans with the privileges it already has, then it wouldn’t need `eval`’s help for that.

bonzini13• 2 years ago

"started to cc himself" seems to be simply "contributing to a new project and not having git-send-email fully set up". By default git-send-email Cc the sender, though in practice it's one of the first options one changes.

xyst2• 2 years ago

Time for another OS wipe. Glad I keep bleeding edge versions VMd

pushedx• 2 years ago

Mirror of the report, since the Openwall servers appear to be down.

https://web.archive.org/web/20240329182300/https://www.openw...

markus_zhang2• 2 years ago

Keeps one wonder how many similar backdoors are there in the wild. What is the best way to execute such a move? This is sophisticated enough, but not good enough to stay unnoticed for a long while. If I were a state actor I'd think about at least 6-12 months.

wannacboatmovie3• 2 years ago

Really disappointed in the number of posters here who are playing down rushing to judgement and suggesting perhaps a legitimate developer was compromised, when it's very clear this is sophisticated and not the work of a single person.

I'm recalling bad memories of the Juniper backdoor years ago.

Whoever did this, was playing the long game. As the top post pointed out, there was an effort to get this into Fedora.... which eventually makes its way into RHEL (read: high value targets). This was not for short term payoffs by some rogue developer trying to mine crypto or other such nonsense. What you are seeing here is the planting of seeds for something months or a year down the road.

jcalvinowens3• 2 years ago

Oof, this is on my Sid laptop:

  {0}[calvinow@mozart ~] dpkg-query -W liblzma5
  liblzma5:amd64  5.6.0-0.2
  {0}[calvinow@mozart ~] hexdump -ve '1/1 "%.2x"' /lib/x86_64-linux-gnu/liblzma.so.5 | grep -c f30f1efa554889f54c89ce5389fb81e7000000804883ec28488954241848894c2410
  1

Glad I stopped running sshd on my laptop a long time ago... still probably going to reinstall :/

msm_6• 2 years ago

No obvious need to reinstall if you didn't use ssh and expose it publicly and are not a politically important person. All signs suggest that it was a nation state attack, and you are likely not a target.

jcalvinowens3• 2 years ago

We'll see... given that sshd is just one of many possible argv[0] it may have choosen to act on, I'm going to be a little paranoid until it's been fully analyzed. It just takes half an hour to reinstall, I have some shows to catch up on anyway :)

liveoneggs• 2 years ago

The best part is everyone disabling security tests that started failing

c_rrodriguez2• 2 years ago

Everybody here In jumping into the pure malice bandwagon, I have a better hypothesis.

Abandonment and inaction, the actual developers of these tools are elsewhere, oblivious to this drama, trying to make living because most of the time you are not compensated nor any corporation cares about making things sustainable at all. This is the default status of everything your fancy cloud depends on underneath.

An attacker took over of the project slowly and stayed dormant until recently.

johnklos• 2 years ago

Except that doesn't match reality.

Someone has worked on xz for several years. Are you saying that this somewhat active contributor was likely actively contributing, then all of a sudden stopped, also stopped paying attention, and also allowed their account to be compromised or otherwise handed it over to a nefarious party?

That fails the sniff test.

c_rrodriguez2• 2 years ago

See, people drop dead from OSS projects pretty frecuently, usually because they take on other life responsabilities and there is no cushion or guard against a bus factor. Then it is very easy to get credentials compromised or have your project took over by someone else.

dkarras• 2 years ago

funding model of OSS work is obviously a problem, but these problems are deeper than that. even a very well compensated OSS developer can get a knock on the door from a government agency (or anyone with a "$5 wrench")[1] and they might feel "compelled" to give up their maintainer creds.

[1] https://xkcd.com/538/

ColonelPhantom• 2 years ago

Well, yeah. The attacker, operating largely under the name Jia Tan, has successfully manipulated the original author (Lasse Collin) to become a maintainer.

The attacker indeed laid dormant for two years, pretending to just be maintaining xz.

I really don't see any way how this wasn't malice on Jia's part. But I do think your hypothesis applies to Lasse, who was just happy someone could help him maintain xz.

elchief• 2 years ago

"Amazon Linux customers are not affected by this issue, and no action is required. AWS infrastructure and services do not utilize the affected software and are not impacted. Users of Bottlerocket are not affected."

https://aws.amazon.com/security/security-bulletins/AWS-2024-...

ikekkdcjkfke2• 2 years ago

Github should probably remove the dopamine hits of green checkmarks etc. like in serious stock broker apps

Nathanba2• 2 years ago

They should also remove the emojis, there is no need to have people feel good about upvotes. I've long felt uncomfortable with emojis on Slack as well. Responding to a coding or infrastructure issue should not be a social activity, I respond because it's my job and if the issue is worth it, not because a human being should feel appreciated (either them or me).

Jonnax• 2 years ago

Many people write code for fun and slack is a social communications platform.

If you can't imagine people using these tools for other reasons than pure unemotional business value then you don't understand their market.

Your suggestions would lose those platforms users and revenue.

dpkirchner5• 2 years ago

The emojis reduce (but not eliminate) the number of "me too!"s PRs will get, which IMO is a good thing.

elintknower• 2 years ago

Candidly how would someone protect against a vulnerability like this?

anononaut3• 2 years ago

Compile all your packages from source would be a start.

Hackbraten8• 2 years ago

You’re not wrong. However, building from source wouldn’t have protected you against this specific backdoor. The upstream source tarball itself was compromised in a cleverly sneaky way.

devttyeu2• 2 years ago

Build from source AND run an Ai agent that reviews every single line of code you compile (while hoping that the any potential exploit doesn’t also fool / exploit your AI agent)

kosolam• 2 years ago

Jesus! Does anyone know if Debian stable is affected?

ValdikSS3• 2 years ago

It's not. Neither Ubuntu.

anononaut3• 2 years ago

Do you have a source my friend? I thought Ubuntu was built off of Debian testing or unstable

djao• 2 years ago

The stable releases don't have this particular backdoor, but they're still using older versions of the library that were released by the same bad actor.

bhaak6• 2 years ago

I looked at the differences between the GitHub repository and released packages. About 60 files are in a release package that are not in the repo (most are generated files for building) but also some of the .po files have changes.

That's devastating.

If you don't build your release packages from feeding "git ls-files" into tar, you are doing it wrong.

icommentedtoday• 2 years ago

Why not `git archive`?

bhaak6• 2 years ago

Because I didn't know about it.

Although if I look at its documentation, it's already a somewhat complicate invocation with unclear effects (lots of commandline options). Git seems to not be able to do KISS.

git ls-files and tar is a simple thing everybody understands and can do without much issues.

cryptonector7• 2 years ago

https://news.ycombinator.com/item?id=39872062

sesuximo• 2 years ago

I think this is unfortunately very common practice

oxymoron290• 2 years ago

Jai Tan's commit history on his github profile suggests he took off for Christmas, new years, and spring break. I smell an American.

bloak• 2 years ago

Interesting. Is there also a pattern in the times of day? (I don't so much mean the times in commits done by the developer because they can be fake. I'd be more interested in authentic times recorded by GitHub, if any such times are publicly accessible.)

Another thing would be to examine everything ever written by the user for linguistic clues. This might point towards particular native languages or a particular variant of English or towards there being several different authors.

bombcar18• 2 years ago

Someone said commits lined up with Beijing time, but I've not verified that.

But that wouldn't count for much, someone employed by anyone could work any hours.

rdtsc3• 2 years ago

Sometimes you smell an American because someone wanted you to smell an American.

Operating on a target region schedule doesn't seem particularly sophisticated, at least compared to the all the efforts put into this exploit.

returningfory23• 2 years ago

Another interesting data point: about 2 years ago there was a clear pressure campaign to name a new maintainer: https://www.mail-archive.com/xz-devel@tukaani.org/msg00566.h...

At the time I thought it was just rude, but maybe this is when it all started.

leosanchez• 2 years ago

How many people are involved in this ?

masklinn2• 2 years ago

Could be just a single person with a bunch of identities.

JaDogg• 2 years ago

Good cop bad cop play maybe.

jamespo• 2 years ago

"Jigar Kumar" seems to have disappeared

Nathanba2• 2 years ago

true, that is suspicious as well. A person that hasn't even created any bugs or issues suddenly has a big problem with the speed of development? Especially the way this was phrased: "You ignore the many patches bit rotting away on this mailing list. Right now you choke your repo. Why wait until 5.4.0 to change maintainer? Why delay what your repo needs?"

"Why delay what your repo needs?" This sounds like scammer lingo

matsemann• 2 years ago

Wow, people suck. I almost hope it's fake profiles urging the maintainer to take on a new member as a long con. Because I sincerely hope Jigar Kumar is not a real person behaving like that towards volunteers working for free.

MaximilianEmel2• 2 years ago

Has this affected OpenBSD at all?

ikmckenz2• 2 years ago

Seems the backdoor relied on Debian and others patching their copies of openssh to support systemd notifications, and this would obviously not be the case on OpenBSD.

To be sure the current ports version of xz is 5.4.5: https://cvsweb.openbsd.org/cgi-bin/cvsweb/~checkout~/ports/a...

Although the maintainer was working on updating to 5.6.1, but this news broke before the diff was landed: https://marc.info/?l=openbsd-ports&m=171174441521894&w=2

port443• 2 years ago

I think its much more likely this was not a bad actor, given their long history of commits.

It's a known fact that China will "recruit" people to operate them. A quote:

They talk to them, say my friend, I see you like our special menu. Are you from China? Are you here on a VISA? Do you have family back there? Would you like your family to stay alive? Is your loyalty to this temporary employer or is your loyalty to your motherland? You know, a whole bunch of stuff like that. That’s how Chinese intelligence operations acts...

This just gives feelings of less "compromised account" and more "Your account is now our account"

Johnny555• 2 years ago

Isn't that still a "bad actor" even if they are coerced into it?

foobiekr• 2 years ago

Yes.

Terr_• 2 years ago

For the purposes of security discussions, I would say yes. You often don't know their real identity let alone their motivations and tribulations.

However if we were critiquing characters in a book-- especially ones where narrative voice tells us exactly their true motivations--then maybe not, and they get framed as a "dupe" or "manipulated" etc.

Almondsetat• 2 years ago

"bad actor" doesn't mean "bad faith", it's not a value judgement

ip26• 2 years ago

I believe your parent is trying to make a distinction that the handle's history may not be suspect, only recent activity, positing a rubber-hose type compromise.

zeroCalories18• 2 years ago

I think we should seriously consider something like a ts clearance as mandatory for work on core technologies. Many other projects, both open and closed, are probably compromised by foreign agents.

Meetvelde• 2 years ago

That's hard to do when the development of these libraries is so international. Not to mention that it's already so hard to find maintainers for some of these projects. Given that getting a TS clearance is such a long and difficult process, it would almost guarantee more difficulty in finding people to do this thankless job.

rwmj10• 2 years ago

That just means the bad actors will all have clearance while putting in a bunch of hurdles for amateur contributors. The only answer is the hard one, constant improvement in methods to detect and mitigate bugs.

cesarb8• 2 years ago

I think we should seriously consider something like a ts clearance as mandatory for work on core technologies.

Was xz/lzma a core technology when it was created? Is my tiny "constant time equality" Rust crate a core technology? Even though it's used by the BLAKE3 crate? By the way, is the BLAKE3 crate a core technology? Will it ever become a core technology?

With free software in general, things do not start a "core technology"; they become a "core technology" over time due to usage. At which point would a maintainer have to get a TS clearance? Would the equivalent of a TS clearance from my Latin America country be acceptable? And how would I obtain it? Is it even available to people outside the military and government (legit question, I never looked)?

csdreamer7• 2 years ago

The Linux kernel is complaining about a lack of funding for CI-one of the highest visibility projects out there. Where will the money come from for this?

Corps? Aside from Intel most of them barely pay to upstream their drivers.

The govt? The US federal government cut so much of it's support since the 70s and 80s.

isbvhodnvemrwvn2• 2 years ago

This only ensures the backdoors are coming from governments that issued the clearances, nothing more. I prefer more competition, at least there is incentive to detect those issues.

vmladenov• 2 years ago

This seems infeasible for projects like LLVM that depend on international collaboration.

AnonymousPlanet3• 2 years ago

That's a very US centric view and would practically split the open source community along the atlantic at best and fracture it globally at worst. Be careful what you wish for.

colinsane• 2 years ago

how many people in PRISM had such clearance? and how many of them would i trust? precisely zero.

failbuffer• 2 years ago

Killing your pipeline for innovation and talent development doesn't make you secure, it makes you fall behind. The Soviet Union found this out the hard way when they made a policy decision to steal chip technology instead of investing in their own people. They were outpaced and the world came to use chips, networks, and software designed by Americans.

okasaki• 2 years ago

A quote from... your arse?

joveian3• 2 years ago

That is what I thought too but it wasn't hard to find:

https://darknetdiaries.com/transcript/21/

threeseed2• 2 years ago

It's also a known fact that China will coerce people by threatening family and friends.

Seen this happen to friends here in Australia who were attending pro-Taiwan protests.

dang4• 2 years ago

We detached this subthread from https://news.ycombinator.com/item?id=39867106.

k8svet3• 2 years ago

Wait, I'm on mobile. Did this partially slip by because of the ABSURD PRACTICE of publishing release.tarballs that do not 1:1 correspond with source?

Let me guess, autotools? I want to rage shit post but I guess I'll wait for confirmation first.

EDIT: YUP, AT LEAST PARTIALLY. Fucking god damn autotools.

hypnagogic8• 2 years ago

Been saying this the whole day now, GitHub really needs an automated diff / A/B check-up on tarballs against the actual repo, flag everything with at least a warning (+[insert additional scrutiny steps here]) when the tarball isn't matching the repo.

5p4n9112• 2 years ago

The author (Jia Tan) also changed the xz.tukaani.org (actually the github.io, where the main contributor is, surprise, also them) release description to state all new releases are signed by their OpenPGP key. I'd guess that was one of the first steps to a complete project takeover.

I hope Lasse Collin still has control of his accounts, though the CC on the kernel mailing list looks kind of suspicious to me.

0xthr0w4• 2 years ago

Out of curiosity I looked at the list of followers of the account who committed the backdoor.

Randomly picked https://github.com/Neustradamus and looked at all their contributions.

Interestingly enough, they got Microsoft to upgrade ([0],[1]) `vcpkg` to liblzma 5.6.0 3 weeks ago.

[0] https://github.com/microsoft/vcpkg/issues/37197

[1] https://github.com/microsoft/vcpkg/pull/37199

sroussey4• 2 years ago

OMG: look at the other contributions. He is trying to take over projects and pushing some change to sha256 in a hundred projects.

Example: https://github.com/duesee/imap-flow/issues/96

masklinn2• 2 years ago

This guy's interactions seem weird but it might just be because of the non-native english or a strange attitude, or he's very good at covering his track e.g. found a cpython issue where he got reprimanded for serially opening issues: https://github.com/python/cpython/issues/115195#issuecomment...

But clicking around he seems to mostly be interacting with interest around these bits e.g. https://github.com/python/cpython/issues/95341#issuecomment-... or pinging the entire python team to link to the PR... of a core python developer: https://github.com/python/cpython/issues/95341#issuecomment-...

If I saw that on a $dayjob project I'd pit him as an innocuous pain in the ass (overly excited, noisy, dickriding).

Here's a PR from 2020 where he recommends / requests the addition of SCRAM to an SMTP client: https://github.com/marlam/msmtp/issues/36 which is basically the same thing as the PR you found. The linked documents seem genuine, and SCRAM is an actual challenge/response authentication method for a variety of protocols (in this case mostly SMTP, IMAP, and XMPP): https://en.wikipedia.org/wiki/Salted_Challenge_Response_Auth...

Although, and that's a bit creepy, he shows up in the edition history for the SCRAM page, the edit mostly seem innocent though he does plug his "state of play" github repository.

gowthamgts122• 2 years ago

reported the account to github, just in case.

arp2425• 2 years ago

What? They're just asking for some features there?

Ya'll need to calm down; this is getting silly. Half the GitHub accounts look "suspicious" if you start scrutinizing everything down the the microscopic detail.

gaucheries• 2 years ago

I appreciate the way that duesee handled that whole issue.

_cenw• 2 years ago

Hey, I remember this guy! Buddy of someone who tried to get a bunch of low quality stuff into ifupdown-ng, including copying code with an incompatible license and removing the notice. He's in every PR, complaining the "project is dead". He even pushes for the account to be made "team member".

https://github.com/ifupdown-ng/ifupdown-ng/pulls/easynetdev

He follows 54k accounts though, so it may indeed just be coincidence.

dikei• 2 years ago

I wouldn't be surprised if that is just a bot.

He even follows me, though I have never published any open-source project on my own.

account4252• 2 years ago

The PR + angry user pushing for the PR author to gain commit access spiel is definitely suspiciously similar to what happened with xz-utils. Possible coincidence but worth investigating further.

neustradamus• 2 years ago

Dear @0xthr0w4, do you attack me because I have requested the XZ update?

Do not mix, I am not linked to the XZ project.

resonious2• 2 years ago

The parent comment doesn't read like an attack to me. Just an observation. Would be curious why you wanted the update though.

gouggoug• 2 years ago

List of pull request requesting the updating to liblzma 5.6.0 [0]

I wonder what amount of scrutiny all the accounts that proposed the upgrade should be put under.

[0] https://github.com/search?q=liblzma+5.6.0&type=pullrequests

mikolajw2• 2 years ago

Tukaani website states "jiatan" as the nickname of the malicious code committer on Libera Chat.

WHOWAS jiatan provided me the following information:

jiatan ~jiatan 185.128.24.163 * :Jia Tan jiatan 185.128.24.163 :actually using host jiatan jiatan :was logged in as jiatan tungsten.libera.chat :Fri Mar 14:47:40 2024

WHOIS yields nothing, the user is not present on the network at the moment.

Given that 185.128.24.163 is covered with a range-block on the English Wikipedia, it appears this is a proxy.

chrononaut• 2 years ago

it appears this is a proxy.

Yes, that IP address appears associated with witopia[.]net, specifically vpn.singapore.witopia[.]net points to that IP address.

circusfly• 2 years ago

Waiting for the new YouTube videos on this. "Woah! Linux has a back door dudes!". My distribution, Ubuntu (now Kubuntu) 2022 isn't affected.

fullstackchris2• 2 years ago

not sure why you're being downvoted. this is exactly what is going to happen.

Lockal3• 2 years ago

Still better than TwoMinuteToiletPapers and other AI-bamboozled channels hyping over proprietary OpenAI crap (text/photo/video), what a time to be alive!

autoexecbat• 2 years ago

I'm really curious about if the act of injecting a backdoor into OSS software is legal/illegal ?

Are they somehow in the clear unless we can show they actively exploited it?

mnau2• 2 years ago

Probably depends on criminal code a country. Mine does (EU country):

Section 231 Obtaining and Possession of Access Device and Computer System Passwords and other such Data
(1) Whoever with the intent to commit a criminal offence of Breach of secrecy of correspondence [...] or a criminal offence of Unauthorised access to computer systems and information media [...] produces, puts into circulation, imports, exports, transits, offers, provides, sells, or otherwise makes available, obtains for him/herself or for another, or handles
a) a device or its component, process, instrument or any other means, including a computer programme designed or adapted for unauthorised access to electronic communications networks, computer system or a part thereof, or
b) a computer password, access code, data, process or any other similar means by which it is possible to gain access to a computer system or a part thereof,

shall be sentenced .. (1 year as an individual, 3 years as a member of a organized group)

Culonavirus• 2 years ago

The way I see it: People are being charged for their speech all the time. Especially outside the US, but even in the US. And code is speech.

And that is even before all the hacking/cracking/espionage laws get involved.

There's a reason all the (sane) people doing grey/black hat work take their security and anonymity extremely seriously.

vhiremath42• 2 years ago

My favorite part was the analysis of "I'm not really a security researcher or reverse engineer but here's a complete breakdown of exactly how the behavior changes."

You only get this kind of humility when you're working with absolute wizards on a consistent basis.

rpigab• 2 years ago

I'd love to be at Microsoft right now and have the power to review this user's connection history to Github, even though VPN exists, many things can be learned from connection habits, links to ISPs, maybe even guess if VPNs were used, roundtrip time on connections can give hints.

I really don't think some random guy wants to weaken ssh just to extract some petty ransomware cash from a couple targets.

qecez• 2 years ago

I really don't think some random guy wants to weaken ssh just to extract some petty ransomware cash from a couple targets.

Which is why there's probably nothing remotely interesting in them logs.

mhh__• 2 years ago

Intelligence agencies get caught red handed all the time so I wouldn't be too sure.

If it was an organised group I'm sure they were careful, of course, but it only takes one fuckup.

alpb2• 2 years ago

That’d be illegal for an employee to do.

sapiogram• 2 years ago

Why?

optimalsolver• 2 years ago

I'm guessing Microsoft just got a call from the Government telling them not to look too deeply into it.

RockRobotRock2• 2 years ago

Nah. I'm sure Microsoft got a call from the alphabet boys and nobody, not even internal employees are allowed to look at the logs right now.

megous• 2 years ago

Oh my, another reason not to use github. :D So many reasons poping up just in this comment section alone.

bananapub9• 2 years ago

people are mis-reading the Debian bug report: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1067708

it wasn't the apparently newly-created identity "Hans Jansen" just asking for a new version to be uploaded, it was "Hans Jansen" providing a new version to be uploaded as a non-maintainer-upload - Debian-speak for "the maintainer is AWOL, someone else is uploading their package". if "Hans Jansen" is another attacker then they did this cleverly, providing the new - compromised - upstream tarballs in an innocent-looking way and avoiding anyone examining the upstream diff.

vasili111• 2 years ago

Could anyone please tell me if current stable version of Debian has that backdoor or not?

yabones• 2 years ago

Debian stable has 5.4.1, the backdoored versions are 5.6.0-5.6.1

https://packages.debian.org/bookworm/xz-utils

https://packages.debian.org/bookworm/liblzma5

anononaut3• 2 years ago

Debian Stable seems to be in the clear.

https://lists.debian.org/debian-security-announce/2024/msg00...

teddyh2• 2 years ago

It does not contain the backdoor: <https://security-tracker.debian.org/tracker/CVE-2024-3094<

kazinator• 2 years ago

Doesn't this call for criminal charges?

mnau2• 2 years ago

Good luck finding him/her.

GitHub probably already gave feds all logs and IPs, but I would bet 100:1 that it's all going to be a VPN or something like that.

ozgune• 2 years ago

I read through the entire report and it gradually got more interesting. Then, I got to the very end, saw Andres Freund's name, and it put a smile on my face. :)

Who else would have run a PostgreSQL performance benchmark and discover a major security issue in the process?

mik1998• 2 years ago

Personally, I use lzip ever since I read https://www.nongnu.org/lzip/xz_inadequate.html Seems like the complexity of XZ has backfired severely, as expected.

bananapub9• 2 years ago

Seems like the complexity of XZ has backfired severely, as expected.

this is a very bad reading of the current situation.

ta8645• 2 years ago

This kind of shallow dismissal is really unhelpful to those of us trying to follow the argument. You take a tone of authoritative expert, without giving any illuminating information to help outsiders judge the merit of your assertion. Why is it a very bad reading of the current situation? What is a better reading?

lacoolj• 2 years ago

What a disappointment.

It's something always in the back of our minds as developers using public libraries, but when something like this happens, non-developers that hear about it start to associate it with the rest of the open-source community.

It's essentially a terrorist attack on developer experience. Thankfully, management doesn't follow the same approach as the TSA.

Zigurd2• 2 years ago

"Lasse Collin," as other posters here have found, does not seem to exist as an experienced coder. Oddly, there is a Swedish jazz musician named Lasse Collin, which would otherwise be one of those names, especially the last name, that would stick out. Instead it is buried under a lot of mentions of a musician.

rany_8• 2 years ago

Searching for my real name on Google doesn't return anything either, I don't think this means anything.

Zigurd2• 2 years ago

Lasse Collin the contributor is findable, especially if you add "tukaani" to the search. But not in any other context, unless that's what old jazz musicians do in their retirement.

akyuu• 2 years ago

Lasse Collin has been working on xz for decades: https://sourceforge.net/p/sevenzip/discussion/45797/thread/0...

Now, whether his GitHub account is currently being controlled by him is another question.

Also, for some more context: In 2022, Lasse said he was struggling to work on xz and was looking for maintainers, and mentioned Jia Tan: https://www.mail-archive.com/xz-devel@tukaani.org/msg00567.h...

nateskulic• 2 years ago

Fairly deep bugs for a Bazaar.

alathers• 2 years ago

Thank the gods I didn't plan on having a life this weekend

userbinator5• 2 years ago

Looking at how many requests to update to the backdoored version have been made, I wonder if the fact that many people (including developers) have been conditioned to essentially accept updates as "always-good" is a huge contributing factor in how easy it is to spread something like this.

The known unknowns can be better than the unknown unknowns.

frenchman995• 2 years ago

Totally agree. With things like Dependabot encouraged by GitHub, people now get automated pull requests for dependency updates, increasing the speed of propagation of such vulnerabilities.

kapouer2• 2 years ago

Both https://github.com/tukaani-project members accounts have been suspended. (to see that, you can list the followers of each account).

dmarto3• 2 years ago

Kinda relevant, as I saw few comments about how safer languages are the solution.

Here[0] is a very simple example, that shows how easy such supply chain attacks are in Rust; and lets not forget that there was a very large python attack just a few days ago[1].

[0] https://github.com/c-skills/rust1

[1] https://checkmarx.com/blog/over-170k-users-affected-by-attac...

mrcus2• 2 years ago

I am very concerned about Rust.

Rust’s “decision” to have a very slim standard library has advantages, but it severely amplifies some other issues. In Go, I have to pull in zero dependencies to make an HTTP request. In Rust, pulling reqwest pulls in at least 30 distinct packages (https://lib.rs/crates/reqwest). Date/time, “basic” base64, common hashing or checksums, etc, they all become supply chain vectors.

The Rust ecosystem’s collective refusal to land stable major versions is one of the amplifying issues. “Upgrade fatigue” hits me, at least. “Sure, upgrade ring to 0.17” (which is effectively the 16th major version). And because v0.X versions are usually incompatible, it’s not really possible to opt not to upgrade, because it only takes a short while before some other transitive dependency breaks because you are slow to upgrade. I recently spent a while writing my code to support running multiple versions of the `http` library, for example (which, to be fair, did just land version 1.0). My NATS library (https://lib.rs/crates/async-nats) is at version 34. My transitive base64 dependency is at version 22 (https://lib.rs/crates/base64).

This makes it nearly impossible for me to review these libraries and pin them, because if I pin foo@0.41.7, and bar needs foo@0.42.1, I just get both. bar can’t do =>0.41, because the point of the 0.X series is that it is not backwards compatible. It makes this process so time consuming that I expect people will either just stop (as if they did) reviewing their dependencies, or accept that they might have to reinvent everything from URL parsing to constructing http headers or doing CRC checks.

Combine this with a build- and compile-time system that allows completely arbitrary code execution, which is routinely just a wrapper for stuff like in the zx attack (look at a lot of the low-level libs you inevitably pull in). Sure, the build scripts and the macro system enables stuff like the amazing sqlx library, but said build and macro code is already so hard to read, it really takes proper wizardry to properly understand.

dmarto3• 2 years ago

You have perfectly put into words, all my thoughts.

I have been thinking about ways to secure myself, as it is exhausting to think about it every time there is an update or some new dependency.

After this attack, I think the only sure way is to unplug the computer and go buy goats.

The next best thing? Probably ephemeral VMs or some Codespaces/"Cloud Dev Env thingy". (except neither would save me in the xz case)

Brian_K_White13• 2 years ago

Or you vendor everything.

You don't automatically download anything at build or install time, you just update your local source copies when you want to. Which to be clear I know means rarely.

It's 1970 all over again!

7373737373• 2 years ago

In Rust, pulling reqwest pulls in at least 30 distinct packages

This would be less of a problem if each dependency (and in turn, their dependencies) were individually sandboxed, and only allowed to access specific inputs/files at runtime in the capability security (https://en.wikipedia.org/wiki/Capability-based_security) fashion.

This way the attack surface would be hollowed out as much as possible, and exploits limited to the (sub)program output or specific accessible (writable) files.

mrcoffee4u• 2 years ago

can someone ELI5 ?

QuantumG2• 2 years ago

House of cards experiences strong wind.

jchoksi• 2 years ago

The two active maintainers seem to be: Lasse Collin <lasse.collin@tukaani.org> and Jia Tan <jiat0218@gmail.com>

Searching DDG for "jiat0218" I came across a blog post which I found weird. Seems to be dated: 2006-05-03

Blog post: "Kuso拍賣．有靈氣的筷子 - 闕小豪" <https://char.tw/blog/post/24397301>

Internet Archive link: <https://web.archive.org/web/20240329182713/https://char.tw/b...>

The contents of the page when translated seems to be about jiat0218 auctioning a pair of spiritual chopsticks as a prank.

The blog entry is basically a QA between jiat0218 and various other people about these chopsticks.

If Jia Tan does turn out to be a compromised maintainer working for a state actor then some of the content on the blog page can be viewed in a more sinister way (i.e. spycraft / hacks for sale etc.).

Example question 38:

    Question 38
    accounta066 (3): Are these chopsticks really that good? I kind of want to buy
    them! But I recently sent money for online shopping but didn’t receive anything.
    It’s very risky; currently jiat0218 you don’t have any reviews, you can
    interview me. Do you want to hand it over?! … A sincere buyer will keep it.
    
    Reply to
    jiat0218 (4): First of all, I would like to express my condolences to you for
    your unfortunate experience! What can I say about this kind of thing...My little
    sister has always been trustworthy. What’s more, this is a pair of spiritual 
    chopsticks, so I hope to have a good one. It’s the beginning! As you can see,
    my little sister is very careful and takes her time when answering your
    questions. Except for the two messages that were accidentally deleted by her,
    she always answers your questions. If this still doesn’t reassure you, then I
    can only say that I still have room to work hard. You are still welcome
    to bid... ^_^

Note however, it could all just be what it purports to be which is a prank auction of spiritual chopsticks.

fragmede7• 2 years ago

Chopsticks could also be a codeword for something. Maybe some sort a backdoor into a system somewhere.

alwayslikethis2• 2 years ago

This is likely just a coincidence. 0218 looks like a birthday and jiat is probably the name + initial. 18 years is also too long of a time horizon for this.

dimgl• 2 years ago

Crazy to think that the time horizon for these kinds of attacks span decades. This absolutely does not read like a coincidence. Chopsticks, little sister, "room to work hard", all sound like codewords.

astrange6• 2 years ago

Do you say that about every word commonly used in Asia?

slowmotiony• 2 years ago

Sounds to me like google translate gibberish

dboreham• 2 years ago

Something about this I found surprising is that Linux distros are pulling and packaging pre-built binaries from upstream projects. I'd have expected them to build from source.

richardwhiuk3• 2 years ago

They were pulling a tarball from upstream and building it - the tarball was compromised.

Lockal3• 2 years ago

The answer is not complete. There were 2 ways to pull sources:

bad - https://github.com/tukaani-project/xz/releases/download/...

or:

good - https://github.com/tukaani-project/xz/archive/refs/tags/...

Specifically in Gentoo, there is a note in https://github.com/gentoo/gentoo/blob/master/app-arch/xz-uti...

  # Remember: we cannot leverage autotools in this ebuild in order
  #           to avoid circular deps with autotools

Namely, to unpack autoconf-2.72e.tar.xz from gnu.org you need xz-tools. And this is just the shortest circle. It is not very common, but xz-utils was one of few rare cases where regeneration of autohell files was considered as unnecessary complication (it backfired).

frankjr2• 2 years ago

Not in this case as the other commenter pointed out but for example Vivaldi on Arch Linux is just a repackaged upstream build.

https://gitlab.archlinux.org/archlinux/packaging/packages/vi...

haolez• 2 years ago

I'm not trying to troll, but I'm wondering if a distro like Gentoo is less susceptible to such attacks, since the source code feels more transparent with their approach. But then again, it seems that upstream was infected in this case, so I'm not sure if a culture of compiling from source locally would help.

StressedDev2• 2 years ago

It is not going to make a difference. If you run malicious code, you will get hacked. Compiling the code yourself does not prevent the code from being malicious.

The one it might help is it might make it easier to find the back door once you know there is one.

stephc_int132• 2 years ago

I guess that rewriting liblzma in Rust would not have prevented this backdoor. But would have likely increased the confidence in its safety.

Using the build system (and potentially the compiler) to insert malicious backdoors is far from a new idea, and I don't see why this example would the only case.

anonymous-panda• 2 years ago

Don’t know all the details and rust isn’t immune to a build attack, but stuff like that tends to stand out a lot more I think in a build.rs than it would in some m4 automake soup.

minetest2048• 2 years ago

There was a drama back then where serde tried to ship its derive macro as a precompiled binary: https://news.ycombinator.com/item?id=37189462

yencabulator4• 2 years ago

The backdoor hinged on hiding things in large shell scripts, obscure C "optimizations", and sanitizer disabling. I'd expect all of those would be a much bigger red flag in the Rust world.

nullifidian• 2 years ago

It would have made it worse, because there would be 300 crates with 250 different maintainers, all pulled in by several trivial/baseline dependencies. More dependencies = higher the probability that a malicious maintainer has gotten maintainer's rights for one of them, especially because many original authors/maintainers of rust style microdepencency crates move on with their lives and eventually seek to exit their maintainer role. At least for classic C/C++ software, by the virtue of it being very inconvenient to casually pull 300 dependencies for something trivial, there are fewer dependencies, i.e. separate projects/repos, and these tend to be more self-contained. There are also "unserious" distributions like Fedora and something like stable/testing/unstable pipeline in Debian, which help with catching the most egregious attempts. Crates.io and npm are unserious by their very design, which is focused on maximizing growth by eliminating as many "hindrances" as possible.

yogorenapan2• 2 years ago

Why is rust beginning to sound like JavaScript?

im3w1l4• 2 years ago

This hack exploited a fairly unique quirk in the linux C ecosystem / culture. That packages are built from "tarballs" that are not exact copies of the git HEAD as they also contain generated scripts with arbitrary code.

It would not have happened in any modern language. It probably wouldn't have even happened in a Vistual Studio C-project for windows either.

everybackdoor4• 2 years ago

Funny you should say that, given they definitely have exploit code in `vcpkg`

Denvercoder94• 2 years ago

It would not have happened in any modern language.

It would. pip for example installs from tarballs uploaded to PyPi, not from a git repository.

joshhansen5• 2 years ago

If it were using Cargo as its build system, it might make such manipulations more obvious / understandable?

dang4• 2 years ago

Related ongoing threads:

Xz: Disable ifunc to fix Issue 60259 - https://news.ycombinator.com/item?id=39869718

FAQ on the xz-utils backdoor - https://news.ycombinator.com/item?id=39869068

Everything I Know About the XZ Backdoor - https://news.ycombinator.com/item?id=39868673

notmysql_• 2 years ago

Interestingly on of the accounts that the GitHub account who introduced the backdoor follows was suspended very recently [1] who is also part of the org who runs XZ

[1] https://github.com/JiaT75?tab=following

rany_8• 2 years ago

That JiaT75 account is also suspended, if you check https://github.com/Larhzu?tab=following you'll see that they're suspended as well. It's pretty weird that it's that hard to find out whether a user is suspended.

fullstackchris2• 2 years ago

pRoBaBlY a StaTe AcToR

zero definition of what that means...

egos of people who just like to say cool words they don't understand

lol

this comment will probably get deleted, but let the action of this comment being deleted stand that in 2024 we're all allowed to use big words with no definition of what they mean -> bad

state actor? who? what motive? what country? all comments involving "state actor" are very broad and strange... i would like people to stop using words that have no meaning, as it really takes away from the overall conversation of what is going on.

i mean you're seriously going to say "state actor playing the long game" to what end? the issue was resolved in 2 hours... this is stupid

Hackbraten8• 2 years ago

For starters, the backdoor was technically really sophisticated.

For example, the malicious code circumvents a hardening technique (RELRO) in a clever way, which would otherwise have blocked it from manipulating the sshd code in the same process space at runtime. This is not something that script kiddies usually cook up in an afternoon to make a quick buck. You need experts and a lot of time to pull off feats like that.

This points to an organization with excellent funding. I’m not surprised at all that people are attributing this to some unknown nation-level group.

dlenski2• 2 years ago

A lot of software (including https://gitlab.com/openconnect/openconnect of which I'm a maintainer) uses libxml2, which in turn transitively links to libzma, using it to load and store compressed XML.

I'm not *too* worried about OpenConnect given that we use `libxml2` only to read and parse uncompressed XML…

But I am wondering if there has been any statement from libxml2 devs (they're under the GNOME umbrella) about potential risks to libxml2 and its users.

bananapub9• 2 years ago

only to read and parse uncompressed XML…

how does libxml2 know to decompress something?

does it require you, as the caller, to explicitly tell it to?

or does it look at the magic bytes or filename or mimetype or something?

dlenski2• 2 years ago

how does libxml2 know to decompress something? > does it require you, as the caller, to explicitly tell it to?

In the entry point/function that we use, `xmlReadMemory` (https://gnome.pages.gitlab.gnome.org/libxml2/devhelp/libxml2...), it doesn't handle compressed XML at all.

But there are indeed others where it attempts to auto-detect compression, although as I understand it from the docs only ZLib compression is autodetected… though I suspect these may be out-of-date and it may autodetect any/all compiled -in compression algorithms.

Regardless, the fact that it links with liblzma is cause for concern, given the mechanism of operation of the liblzma/xz backdoor.

enedil• 2 years ago

This doesn't matter, if libxml2 loads .so and the library is malicious, you are already potentially compromised, as it is possible to run code on library load.

0x06• 2 years ago

Interesting commit in January where the actual OpenPGP key was changed: https://github.com/tukaani-project/tukaani-project.github.io...

illusive40806• 2 years ago

GitHub suspended this project

gertvdijk• 2 years ago

They just signed each other's keys around that time, and one needs to redistribute the public keys for that; nothing suspicious about it I think. The key fingerprint 22D465F2B4C173803B20C6DE59FCF207FEA7F445 remained the same.

before:

    pub   rsa4096/0x59FCF207FEA7F445 2022-12-28 [SC] [expires: 2027-12-27]
        22D465F2B4C173803B20C6DE59FCF207FEA7F445
    uid                             Jia Tan <jiat0218@gmail.com>
    sig        0x59FCF207FEA7F445 2022-12-28   [selfsig]
    sub   rsa4096/0x63CCE556C94DDA4F 2022-12-28 [E] [expires: 2027-12-27]
    sig        0x59FCF207FEA7F445 2022-12-28   [keybind]

after:

    pub   rsa4096/0x59FCF207FEA7F445 2022-12-28 [SC] [expires: 2027-12-27]
        22D465F2B4C173803B20C6DE59FCF207FEA7F445
    uid                             Jia Tan <jiat0218@gmail.com>
    sig        0x59FCF207FEA7F445 2022-12-28   [selfsig]
    sig        0x38EE757D69184620 2024-01-12   Lasse Collin <lasse.collin@tukaani.org>
    sub   rsa4096/0x63CCE556C94DDA4F 2022-12-28 [E] [expires: 2027-12-27]
    sig        0x59FCF207FEA7F445 2022-12-28   [keybind]

Lasse's key for reference:

    pub   rsa4096/0x38EE757D69184620 2010-10-24 [SC] [expires: 2025-02-07]
        3690C240CE51B4670D30AD1C38EE757D69184620
    uid                             Lasse Collin <lasse.collin@tukaani.org>
    sig        0x38EE757D69184620 2024-01-08   [selfsig]
    sig        0x59FCF207FEA7F445 2024-01-12   Jia Tan <jiat0218@gmail.com>
    sub   rsa4096/0x5923A9D358ADF744 2010-10-24 [E] [expires: 2025-02-07]
    sig        0x38EE757D69184620 2024-01-08   [keybind]

MaximilianEmel2• 2 years ago

We need to get these complex & bloated build-systems under control.

77pt773• 2 years ago

What we need is to move away from 1970s build tools.

0x06• 2 years ago

All these older (4.x, 5.0.x etc) releases that were suddenly uploaded a few months ago should probably also be considered suspect: https://github.com/tukaani-project/tukaani-project.github.io...

shortsunblack• 2 years ago

Pretty much proof that OSS != automatically more secure. And proof that OSS projects can get backdoored. See this for more ideas on this issue: https://seirdy.one/posts/2022/02/02/floss-security/

derkades• 2 years ago

The malware was hidden inside an opaque binary. If anything, this shows that we need more open source and more reproducibility.

llmblockchain3• 2 years ago

Was Debian 12/stable unaffected? Only sid?

joshhansen5• 2 years ago

My understanding is that that's correct. I'm on fully upgraded stable (Debian 12) and my xz is 5.4.2 and liblzma as well.

xvilka3• 2 years ago

Maybe it's finally time to start sunsetting LZMA and xz all together in favor of newer algorithms like Zstandard that also offer better performance but compression rates on par with LZMA.

illusive40806• 2 years ago

Yes but don’t start thinking they’re immune to compromise

xvilka3• 2 years ago

Nobody is. But it's a great opportunity window.

mrbluecoat• 2 years ago

I am *not* a security researcher, nor a reverse engineer.

Could have fooled me - impressive write-up!

neoneye2• 2 years ago

Damn. I'm on macOS and use homebrew. To my surprise I had "xz" version 6.5.1 installed on my computer!

I ran "brew upgrade" and that downgraded to version 5.4.6.

afh14• 2 years ago

Potentially malicious commit by same author on libarchive: https://github.com/libarchive/libarchive/pull/1609

andix• 2 years ago

Is there already a list of distributions that included the affected versions in non-prereelase channels?

illusive40806• 2 years ago

None that I could find have included it. Not even NixOS 23.11.

devttyeu2• 2 years ago

Wouldn’t be surprised that the ssh auth being made slower was deliberate - that makes it fairly easy to index all open ssh servers on the internet, then to see which ones get slower to fail preauth as they install the backdoor

A1kmm2• 2 years ago

Looks like GitHub has suspended access to the repository, which while it protects against people accidentally compiling and using the code, but certainly complicates forensic analysis for anyone who doesn't have a clone or access to history (which is what I think a lot of people will be doing now to understand their exposure).

gpm11• 2 years ago

Well that's inconvenient, I was (probably, time permitting) going to propose to some of my friends that we attempt to reverse this for fun tomorrow.

Anyone have a link to the git history? I guess we can use the ubuntu tarball for the evil version.

A1kmm2• 2 years ago

It looks like git clone https://git.tukaani.org/xz.git still works for now (note: you will obviously be cloning malware if you do this) - that is, however, trusting the project infrastructure that compromised maintainers could have had access to, so I'm not sure if it is unmodified.

HEAD (git rev-parse HEAD) on my result of doing that is currently 0b99783d63f27606936bb79a16c52d0d70c0b56f, and it does have commits people have referenced as being part of the backdoor in it.

gowthamgts122• 2 years ago

https://git.tukaani.org/xz.git

it's throwing 403 now.

gpm11• 2 years ago

Apparently there's a wayback machine for git repos and it "just coincidentally" archived this repo the day before the news broke:

https://archive.softwareheritage.org/browse/origin/visits/?o...

sn2• 2 years ago

For bad-3-corrupt_lzma2.xz, the claim was that "the original files were generated with random local to my machine. To better reproduce these files in the future, a constant seed was used to recreate these files." with no indication of what the seed was.

I got curious and decided to run 'ent' https://www.fourmilab.ch/random/ to see how likely the data in the bad stream was to be random. I used some python to split the data into 3 streams, since it's supposed to be the middle one that's "bad":

I used this regex to split in python, and wrote to "tmp":

    re.split(b'\xfd7zXZ', x)

I manually used dd and truncate to strip out the remaining header and footer according to the specification, which left 48 bytes:

    $ ent tmp2 # bad file payload
    Entropy = 4.157806 bits per byte.
    
    Optimum compression would reduce the size
    of this 48 byte file by 48 percent.
    
    Chi square distribution for 48 samples is 1114.67, and randomly
    would exceed this value less than 0.01 percent of the times.
    
    Arithmetic mean value of data bytes is 51.4167 (127.5 = random).
    Monte Carlo value for Pi is 4.000000000 (error 27.32 percent).
    Serial correlation coefficient is 0.258711 (totally uncorrelated = 0.0).
    
    $ ent tmp3 # urandom
    Entropy = 5.376629 bits per byte.
    
    Optimum compression would reduce the size
    of this 48 byte file by 32 percent.
    
    Chi square distribution for 48 samples is 261.33, and randomly
    would exceed this value 37.92 percent of the times.
    
    Arithmetic mean value of data bytes is 127.8125 (127.5 = random).
    Monte Carlo value for Pi is 3.500000000 (error 11.41 percent).
    Serial correlation coefficient is -0.067038 (totally uncorrelated = 0.0).

The data does not look random. From https://www.fourmilab.ch/random/ for the Chi-square Test, "We interpret the percentage as the degree to which the sequence tested is suspected of being non-random. If the percentage is greater than 99% or less than 1%, the sequence is almost certainly not random. If the percentage is between 99% and 95% or between 1% and 5%, the sequence is suspect. Percentages between 90% and 95% and 5% and 10% indicate the sequence is “almost suspect”."

supriyo-biswas2• 2 years ago

Now to be fair, such an archive could have been created with a “store” level of compression that doesn’t actually perform any compression.

sn2• 2 years ago

My reading of the commit message is they're claiming the "data" should look random.

CGamesPlay• 2 years ago

Why has Github disabled the (apparently official) xz repository, but left the implicated account open to the world? It makes getting caught up on the issue pretty difficult, when GitHub has revoked everyone's access to see the affected source code.

https://github.com/tukaani-project/xz vs https://github.com/JiaT75

dzaima• 2 years ago

The account has been suspended for a while, but for whatever reason that's not displayed on the profile itself (can be seen at https://github.com/Larhzu?tab=following). Repo being disabled is newer, and, while annoying and realistically likely pointless, it's not particularly unreasonable to take down a repository including a real backdoor.

account4252• 2 years ago

Taking down the repo prevents more people inadvertendly pulling and building the backdoor so that makes sense. They should have immediately rehosted and archived the state at a different URL which makes it clear to not use it.

betaby• 2 years ago

How that backdoor is triggered and what exactly it does?

west0n• 2 years ago

It seems that to counter this type of supply chain attack, the best practices for managing software dependencies are to pin the version numbers of dependencies instead of using `latest`, and to use static linking instead of dynamic linking.

xyzzy_plugh• 2 years ago

This gist summarizes the current situation very well: https://gist.github.com/thesamesam/223949d5a074ebc3dce9ee78b...

Definitely looking like they were most likely some sort of state actor. This is very well done and all in plain sight. It's reassuring that it was discovered but given a simple audit of the release build artifacts would have raised alarms, how prevalent is this behavior in other projects? Terrifying stuff.

Brian_K_White13• 2 years ago

It doesn't really relate to this issue other than that both issues share a common source, but I wish we'd never fallen for xz.

I agree with the lzip guy

https://www.nongnu.org/lzip/xz_inadequate.html

bitwize• 2 years ago

Looks like Jonathan Blow was right about open source.

fwungy• 2 years ago

Brain fart: would it be possible to attach passwords to a crypto based micro transaction such that every time you attempted a password entry your crypto account was charged a small fee for the login attempt?

This would thwart brute force attacks, but not be a significant cost for users. If you could attach your login to the crypto account it would mean the account would have to be funded to allow the attempt. The token wouldn't store passwords it would just be a gatekeeper to the login attempt.

The fees would be paid to the service providers as mining fees.

E.g. foo@bar.com needs a password and a token provided from a designated crypto address to gain access to the service.

byearthithatius• 2 years ago

I hope mainstream news cover this so the general population can understand the issue with our software ecoysystems reliance on unpaid open-source maintainers

account4252• 2 years ago

I worry the mainstream news take would just be "open source bad, microsoft closed source and google cloud good"

korginator• 2 years ago

xz is so pervasive, I just discovered on my Mac that the (affected?) version 5.6.1 made it into homebrew. The post in the linked article says that only Linux x86-64 systems are affected, but now I'm left scratching my head whether my Mac is also in trouble, just that we don't know it yet.

BobbyTables2• 2 years ago

Why doesn’t GitHub force “releases” to be a simple repo tarball for sources and with binaries from GitHub actions or such…

I find it incredibly ironic that a “version control” site gives no assurance of reproducible builds (nor reproducible source!!)

The real villain is not the perpetrator, it is Microsoft, and it is all of us.

cryptonector7• 2 years ago

Because then for autoconf codebases you have to commit `./configure` or you have to require that users have autoconf installed and run `autoreconf -fi` first.

Maybe autoconf-using projects should really just require that users have autoconf installed.

Not that that would prevent backdoors, mind you.

dpkirchner5• 2 years ago

If committing configure is objectionable, perhaps there could be "service" repositories that are not directly writable and are guaranteed to be nothing more than the base repo + autoconf cruft used to generate the releases.

Brian_K_White13• 2 years ago

Too inflexible ideological. There are infinite things that most properly belong in a release file and not in the source, that can't be generated from that source by github actions, and seperately no one should be compelled to use github actions.

65a2• 2 years ago

Is there a proper reverse engineering of the payload yet?

17e55aab• 2 years ago

a user offered 5.6.0 and 5.4.5 in an issue to microsoft/vcpkg

5.4.5 can be compromised

https://github.com/microsoft/vcpkg/issues/37197

shp0ngle• 2 years ago

we should take this diagram and change "random person in nebraska" to "possibly a state-level attacker"

https://xkcd.com/2347/

nice

secondary_op• 2 years ago

Github making suspect repository private and hiding recent account activity is wrong move and is interfering with citizens investigation efforts.

frenchman995• 2 years ago

Going forward this will require more than a citizens investigation. Law enforcement will surely be granted access. Also, tarballs are still available in package managers if you really want to dig into the code.

zamalek4• 2 years ago

It's a crime scene. It effectively has the "police" yellow tape around it.

BarbaryCoast• 2 years ago

There's a bug in the detection script. The line:

if [ "$path" == "" ]

should be

if [ "$path" = "" ]

dualbus2• 2 years ago

Bash accepts both variants of the equality operator. So it is not a bug.

kn100• 2 years ago

Here's a handy bash script I threw together to audit any docker containers you might be running on your machine. It's hacky, but will quickly let you know what version, if any, of xz, is running in your docker containers.

``` #!/bin/bash

# Get list of all running Docker containers containers=$(docker ps --format "{{.Names}}")

# Loop through each container for container in $containers; do # Get container image image=$(docker inspect --format='{{.Config.Image}}' "$container")

    # Execute xz --version inside the container
    version=$(docker exec "$container" xz --version)

    # Write container name, image, and command output to a text file
    echo "Container: $container" >> docker_container_versions.txt
    echo "Image: $image" >> docker_container_versions.txt
    echo "xz Version:" >> docker_container_versions.txt
    echo "$version" >> docker_container_versions.txt
    echo "" >> docker_container_versions.txt

done

echo "Output written to docker_container_versions.txt" ```

Rhea_Karty4• 2 years ago

Notes on time stamps and time zones.

A few interesting bits that I haven't fully fleshed out. TLDR: Some people have been throwing around that Jia is from “China,” but it seems also quite possible that Jia is from somewhere in Eastern Europe pretending to be from China. In addition, Lasse Collin and Hans Jansen are from the same EET time zone.

The following analysis was conducted on JiaT75’s (https://github.com/JiaT75?tab=overview&from=2021-12-01&to=20...) commits to the XZ repository, and their time stamps.

Observation 1: Time zone basic analysis

Here is the data on Jia’s time zone and the number of times he was recorded in that time zone: 3: + 0200 (in winter: February and November) 6: +0300 (in summer: in Jun, Jul, early October) 440: +0800

1. The +800 is likely CST. China (or Indonesia or Philippines), given that Australia does daylight savings time and almost no one lives in Siberia and the Gobi dessert. 2. The +0200/+0300, if we are assuming that this is one location, is likely on EET (Finland, Estonia, Latvia, Lithuania, Ukraine, Moldavia, Romania, Bulgaria, Greece, Turkey). This is because we see a switch from +300 in the winter (past the last weekend of October) and +200 in the summer (past the last Sunday in March). 1. Incidentally, this seems to be the same time zone as Lasse Collin and Hans Jansen…

Observation 2: Time zone inconsistencies

Let’s analyze the few times where Jia was recorded in a non +800 time zone. Here, we notice that there are some situations where Jia switches between +800 and +300/+200 in a seemingly implausible time. Indicating that perhaps he is not actually in +800 CST time, as his profile would like us to believe.

Jia Tan Tue, 27 Jun 2023 23:38:32 +0800 —> 23:38 + 8 = 7:30 (+ 1) Jia Tan Tue, 27 Jun 2023 17:27:09 +0300 —> 17:27 + 3 = 20:30 —> about a 9 hour difference, but a flight from China to anywhere in Eastern Europe is at a min 10 hours

Jia Tan Thu, 5 May 2022 20:53:42 +0800 Jia Tan Sat, 19 Nov 2022 23:18:04 +0800 Jia Tan Mon, 7 Nov 2022 16:24:14 +0200 Jia Tan Sun, 23 Oct 2022 21:01:08 +0800 Jia Tan Thu, 6 Oct 2022 21:53:09 +0300 —> 21:53 + 3 = 1:00 (+1) Jia Tan Thu, 6 Oct 2022 17:00:38 +0800 —> 17:00 + 8 = 1:00 (+1) Jia Tan Wed, 5 Oct 2022 23:54:12 +0800 Jia Tan Wed, 5 Oct 2022 20:57:16 +0800 —> again, given the flight time, this is even more impossible

Jia Tan Fri, 2 Sep 2022 20:18:55 +0800 Jia Tan Thu, 8 Sep 2022 15:07:00 +0300 Jia Tan Mon, 25 Jul 2022 18:30:05 +0300 Jia Tan Mon, 25 Jul 2022 18:20:01 +0300 Jia Tan Fri, 1 Jul 2022 21:19:26 +0800 Jia Tan Thu, 16 Jun 2022 17:32:19 +0300 Jia Tan Mon, 13 Jun 2022 20:27:03 +0800 —> the ordering of these time stamps and the switching back and forth between time zones looks strange.

Jia Tan Thu, 15 Feb 2024 22:26:43 +0800 Jia Tan Thu, 15 Feb 2024 01:53:40 +0800 Jia Tan Mon, 12 Feb 2024 17:09:10 +0200 Jia Tan Mon, 12 Feb 2024 17:09:10 +0200 Jia Tan Tue, 13 Feb 2024 22:38:58 +0800 —> this travel time is possible, but the duration of stay is unlikely

Observation 3: Strange record of time stamps

It seems that from the commits, often the time stamps are out of order. I am not sure what would cause this other than some tampering.

Observation 4: Bank holiday inconsistencies

We notice that Jia’s work schedule and holidays seems to align much better with an Eastern European than a Chinese person.

Disclaimer: I am not an expert in Chinese holidays, so this very well could be inaccurate. I am referencing this list of bank holidays:(https://www.bankofchina.co.id/en-id/service/information/late...)

Chinese bank holidays (just looking at 2023): - Working on 2023, 29 September: Mid Autumn Festival - Working on 2023, 05 April: Tomb Sweeping Day - Working on 2023, 26, 22, 23, 24, 26, 27 Jan: Lunar New Year

Eastern European holidays: - Never working on Dec 25: Christmas (for many EET countries) - Never working Dec 31 or Jan 1: New Years

Observation 5: Little weekend work —> salary job?

The most common working days for Jia were Tue (86), Wed (85), Thu (89), and Fri (79). If we adjust his time zone to EET, then that means he is usually working 9 am to 6 pm. This makes much more sense than someone working at midnight and 1 am on a Tuesday night.

These times also line up well with Hans Jansen and Lasse Collin.

I think it is more likely that Jia does this as part of his work… somewhere in Eastern Europe. Likely working with, or in fact being one and the same as, Hans Jansen and Lasse Collin.

Rhea_Karty4• 2 years ago

TLDR: Some people have been throwing around “China,” but it seems also quite possible that Jia is from somewhere in Eastern Europe pretending to be from China. In addition, Lasse Collin and Hans Jansen are from the same EET time zone.

These are my notes on time stamps/zones. There are a few interesting bits that I haven't fully fleshed out.

The following analysis was conducted on JiaT75’s (https://github.com/JiaT75?tab=overview&from=2021-12-01&to=20...) commits to the XZ repository, and their time stamps.

Observation 1: Time zone basic analysis

Here is the data on Jia’s time zone and the number of times he was recorded in that time zone:

3: + 0200 (in winter: February and November)

6: +0300 (in summer: in Jun, Jul, early October)

440: +0800

1. The +800 is likely CST. China (or Indonesia or Philippines), given that Australia does daylight savings time and almost no one lives in Siberia and the Gobi dessert.

2. The +0200/+0300, if we are assuming that this is one location, is likely on EET (Finland, Estonia, Latvia, Lithuania, Ukraine, Moldavia, Romania, Bulgaria, Greece, Turkey). This is because we see a switch from +300 in the winter (past the last weekend of October) and +200 in the summer (past the last Sunday in March).

Incidentally, this seems to be the same time zone as Lasse Collin and Hans Jansen…

Observation 2: Time zone inconsistencies

Jia Tan Tue, 27 Jun 2023 23:38:32 +0800 —> 23:38 + 8 = 7:30 (+ 1) Jia Tan Tue, 27 Jun 2023 17:27:09 +0300 —> 17:27 + 3 = 20:30 —> about a 9 hour difference, but flight from China to anywhere in Eastern Europe is at a min 10 hours

Jia Tan Thu, 5 May 2022 20:53:42 +0800

Jia Tan Sat, 19 Nov 2022 23:18:04 +0800

Jia Tan Mon, 7 Nov 2022 16:24:14 +0200

Jia Tan Sun, 23 Oct 2022 21:01:08 +0800

Jia Tan Thu, 6 Oct 2022 21:53:09 +0300 —> 21:53 + 3 = 1:00 (+1)

Jia Tan Thu, 6 Oct 2022 17:00:38 +0800 —> 17:00 + 8 = 1:00 (+1)

Jia Tan Wed, 5 Oct 2022 23:54:12 +0800

Jia Tan Wed, 5 Oct 2022 20:57:16 +0800

—> again, given the flight time, this is even more impossible

Jia Tan Fri, 2 Sep 2022 20:18:55 +0800

Jia Tan Thu, 8 Sep 2022 15:07:00 +0300

Jia Tan Mon, 25 Jul 2022 18:30:05 +0300

Jia Tan Mon, 25 Jul 2022 18:20:01 +0300

Jia Tan Fri, 1 Jul 2022 21:19:26 +0800

Jia Tan Thu, 16 Jun 2022 17:32:19 +0300

Jia Tan Mon, 13 Jun 2022 20:27:03 +0800

—> the ordering of these time stamps, and the switching back and forth looks strange.

Jia Tan Thu, 15 Feb 2024 22:26:43 +0800

Jia Tan Thu, 15 Feb 2024 01:53:40 +0800

Jia Tan Mon, 12 Feb 2024 17:09:10 +0200

Jia Tan Tue, 13 Feb 2024 22:38:58 +0800

—> this travel time is possible, but the duration of stay is unlikely

Observation 3: Strange record of time stamps It seems that from the commits, often the time stamps are out of order. I am not sure what would cause this other than some tampering.

Observation 4: Bank holiday inconsistencies

We notice that Jia’s work schedule and holidays seem to align much better with an Eastern European than a Chinese person.

Disclaimer: I am not an expert in Chinese holidays, so this very well could be inaccurate. I am referencing this list of bak holidays:(https://www.bankofchina.co.id/en-id/service/information/late...)

Chinese bank holidays (just looking at 2023):

- Working on 2023, 29 September: Mid Autumn Festival

- Working on 2023, 05 April: Tomb Sweeping Day

- Working on 2023, 26, 22, 23, 24, 26, 27 Jan: Lunar New Year

Eastern European holidays:

- Never working on Dec 25: Christmas (for many EET countries)

- Never working Dec 31 or Jan 1: New Years

Observation 5: No weekend work —> salary job?

The most common working days for Jia was Tue (86), Wed (85), Thu (89), and Fri (79). If we adjust his time zone to be EET, then that means he is usually working 9 am to 6 pm. This makes much more sense than someone working at midnight and 1 am on a Tuesday night.

These times also line up well with Hans Jansen and Lasse Collin.

I think it is more likely that Jia does this as part of his work… somewhere in Eastern Europe. Likely working with, or in fact being one and the same as, Hans Jansen and Lasse Collin.

mborch• 2 years ago

This 2011 addition to the XZ Utils Wikipedia page is interesting because a) why is this relevant, b) who is Mike Kezner since he's not mentioned on the Tukaani project page (https://tukaani.org/about.html) under "Historical acknowledgments".

https://en.wikipedia.org/w/index.php?title=XZ_Utils&diff=pre...

Arch Linux played an important role in making this compression software trusted and depended upon. Perhaps not a coincidence, but at the very least, such a big project should more carefully consider the software they distribute and rely on, whether it's worth the risk.

ui2RjUen875bfFA6• 2 years ago

Arch Linux played an important role in making this compression software trusted and depended upon.

because of the way arch distributes packages? then what you think about freebsd?

prissy04• 2 years ago

If you check the history of that IP address, it added Mike Kezner to other pages. No clue why.

frenchman995• 2 years ago

You say yourself that the time data could be tampered. It's trivial to change commit dates in git. So this analysis means nothing by itself, unfortunately.

mimop• 2 years ago

I wouldn't say that. This guy seems to have tried hard to appear Chinese (and possibly tampered the time stamps this way) – but based on that analysis, it seems plausible they did a bad job and were actually based out of Eastern Europe.

Rhea_Karty4• 2 years ago

Made a more detailed write-up on this: https://rheaeve.substack.com/p/xz-backdoor-times-damned-time...

bed994• 2 years ago

I asked ChatGPT 4 based on Jia's Github avatar image:

The timezones that ChatGPT thinks the avatar comes from aligns with +2 and +3, see what how it ranked and at the end the description of Jia's avatar:

---

Rank, Score, Country, City, Timezone, Criteria

1, 10, Saudi Arabia, Mecca, AST (UTC+3), Heartland of Islam, deeply rooted calligraphic traditions.

2, 9.5, Iran, Tehran, IRST (UTC+3:30), Integral Persian calligraphy with a distinct style and history.

3, 9, Turkey, Istanbul, TRT (UTC+3), Historical significance of Ottoman calligraphy, actively preserved.

4, 8.5, Egypt, Cairo, EET (UTC+2), Home to Al-Azhar University, with calligraphy in the curriculum.

5, 8, Morocco, Marrakech, WET (UTC+0), Calligraphy integrated into architecture and crafts.

6, 7.5, United Arab Emirates, Abu Dhabi, GST (UTC+4), Promotes Islamic arts through festivals and museums.

7, 7, Syria, Damascus, EET (UTC+2), Historical center of Arabic calligraphy, despite recent conflicts.

8, 6.5, Pakistan, Islamabad, PKT (UTC+5), Rich tradition, hosts several institutions and events dedicated to calligraphy.

9, 6, Indonesia, Jakarta, WIB (UTC+7), Largest Muslim-majority country with calligraphy in art and monuments.

10, 5.5, Spain, Cordoba, CET (UTC+1), Legacy of Islamic culture and appreciation for calligraphy, particularly in Andalusia.

GPT4: This image appears to be a stylized representation of the letter 'J' within an intricate border, possibly inspired by the art style of Islamic calligraphy. The ornate background is typical of arabesque patterns, which are characteristic of Islamic art and consist of repeating geometric forms that often echo the shapes of plants, flowers, and sometimes calligraphic writing. The letter 'J' stands out in a vibrant yellow, contrasting with the dark green of the surrounding design.

Rhea_Karty4• 2 years ago

Interesting :). However, I think that EET is the only time zone that works. (This is mostly because is seems that the area follows DST, which most non western countries in the worlf do not).

CanaryLayout3• 2 years ago

Well isn't this an interesting commit. He finished his inject macro to compose the payload at build, so now he can start clearing up the repo so none of that shit gets seen when cruising through it.

https://git.tukaani.org/?p=xz.git;a=commitdiff;h=4323bc3e0c1...

astrange6• 2 years ago

That's not what gitignore does. I can't think of a way it would let you hide this exploit.

zamalek4• 2 years ago

Accidentally committing it.

zeehio• 2 years ago

On Ubuntu there is a bug report asking to sync the 5.6 version from Debian experimental https://bugs.launchpad.net/ubuntu/+source/xz-utils/+bug/2055...

Retr0id3• 2 years ago

The `pack`[0] compression utility that reached the HN front page the other day[1] is setting off my alarm bells right now. (It was at the time too, but now doubly so)

It's written in Pascal, and the only (semi-)documented way to build it yourself is to use a graphical IDE, and pull in pre-compiled library binaries (stored in the git repo of a dependency which afaict Pack is the only dependent of - appears to be maintained by the same pseudonymous author but from a different account).

I've opened an issue[2] outlining my concerns. I'm certainly not accusing them of having backdoored binaries, but if I was setting up a project to be deliberately backdoorable, it'd look a lot like this.

[0] https://pack.ac/

[1] https://news.ycombinator.com/item?id=39793805

[2] https://github.com/PackOrganization/Pack/issues/10

jum4• 2 years ago

Maybe @JiaT75 got forced to do it. Maybe someone has more personal contact with him and can check how he is doing.

rossant• 2 years ago

Incredible. It's like discovering your colleague for 2 years at the secret nuclear weapon facility is a spy for another country, covering his tracks until the very last minute. Feels like a Hollywood movie is coming up.

Should we start doing background checks on all committers to such critical IT infrastructure?

throwaway2902• 2 years ago

Not even background check but a foreground check would already help. Like literally, who dis? any identity at all?

Too often maintainers who have no time just blanket approve PRs and see if stuff breaks.

arter45• 2 years ago

But how? Let's say you're one of 10 maintainers of an open source project. A new user wants to contribute. What do you do? Do you ask them to send you some form of ID? Assuming this is legal and assuming you could ensure the new user is the actual owner of an actual, non counterfeit ID, what do you do? Do you vet people based on their nationality? If so, what nationality should be blackballed? Maybe 3 maintainers are American, 5 are European and 2 are Chinese. Who gets to decide? Or do you decide based on the company they work for?

Open source is, by definition, open. The PR/merge request process is generally meant to accept or refuse commits based on the content (which is why you have a diff), not on the owner.

Building consensus on which commits are actually valid, even in the face of malicious actors, is a notoriously difficult problem. Byzantine fault tolerance can be achieved with a 2/3 + 1 majority, but if anyone can create new identities and have them join the system (Sybil attack) you're going to have to do things differently.

pdimitar• 2 years ago

This was only a matter of time. Open source projects are under-staffed, maintainers are overworked and burned out, and everyone relies on the goodwill of all actors.

Obviously a bad actor will make use of these conditions and the assumption of good will.

We need automated tooling to vet for stuff like this. And maybe migrate away from C/C++ while we are at it because they don't make such scanning easy at all.

Epa095• 2 years ago

I hope Lasse Collin is doing OK! Here is a older message from him [1]

"I haven't lost interest but my ability to care has been fairly limited mostly due to longterm mental health issues but also due to some other things. Recently I've worked off-list a bit with Jia Tan on XZ Utils and perhaps he will have a bigger role in the future, we'll see.

It's also good to keep in mind that this is an unpaid hobby project. "

Github (Microsoft) are in a unique position to figure out if his account is hacked or not, and find a way to reach him. I hope they reach out and offer him some proper support! Economic support (if that's needed), or just help clearing his name.

This is another tale of how we are building multi trillion dollar industries on the back of unpaid volunteers. It's not github 'job', and many other organisations have benefited even more from Lasses work, but they are in a unique position, and would be literally pocket change for them.

1:https://www.mail-archive.com/xz-devel@tukaani.org/msg00567.h...

syslog• 2 years ago

Relevant xkcd:

https://xkcd.com/2347/

cbolton2• 2 years ago

In a movie his mental health issues would likely have been caused intentionally by the attacker, setting the stage for the mole to offer to step in just at the right time. Seems a bit far fetched in this case though for what looks like a tangential attack.

maerF0x04• 2 years ago

or > Recently I've worked off-list a bit with Jia Tan on XZ Utils and perhaps he will have a bigger role in the future, we'll see.

Is actually Jia Tan has him tied up in a basement and is posing as him. State actors can do that kind of thing.

deanresin• 2 years ago

In a movie, he was killed by foreign state actors, and his identity assumed by the foreign state hacker. Actually, someone should check on him.

account4252• 2 years ago

what looks like a tangential attack

Does it? I expect that finding someone vulnerable was the more likely approach rather than messing with the life of a stable maintainer, but it does seem very much like the attacker was acting with malicious intent from the start of his interaction with the xz project.

delfinom5• 2 years ago

He came on IRC, he seemed ok. He did some cleanup of access and signed off for easter.

farmdve• 2 years ago

I mean, he was right at least. Jia Tan did have a bigger role.

400thecat• 2 years ago

which IRC channel ?

slavik81• 2 years ago

Lasse appears to be active and working on undoing the sabotage. https://git.tukaani.org/?p=xz.git;a=blobdiff;f=CMakeLists.tx...

grumpyprole• 2 years ago

I would start restoring trust by reverting all this guys commits. It's the best way to be sure.

k8svet3• 2 years ago

I would like to see more attention given to this. I'm capable of compartmentalization and not over-guilting myself, but holy hell, I really hope he's doing alright. This would kind of destroy me.

I was actually telling my dad about this. I have a project, 500+ users, not quite root access, but enough to cause serious damage. I can think of at least one covert way to backdoor the binary artifacts from it.

About two years ago, someone showed up, started making good commits. In this case, they have some other community rep that goes back a bit further but... man it's an unsettling feeling.

01apr2029• 2 years ago

I'm capable of compartmentalization

teach me how. help me learn how, please. any resources with practical utility you can share? or any class of therapists that are good at teaching this with right frameworks offered? thank you

zh3• 2 years ago

Comment from Andres Freund on how and why he found it [0] and more information on the LWN story about the backdoor. Recommend people read this to see how close we came (and think about what this is going to mean for the future).

[0] https://lwn.net/Articles/967194/

eBombzor• 2 years ago

That man deserves a Nobel Prize

itsTyrion• 2 years ago

that's... creative. and patient. 11/10 concerning - now I'm wondering how many other projects could have shit like this in them or added right as I'm writing this shudder

kzrdude6• 2 years ago

Jia Tan "cleaned up" in all their ZSTD branches some hours ago, probably hiding something https://github.com/JiaT75/zstd/branches/all

zamalek4• 2 years ago

Bad move. Destroying evidence is a felony.

cypress66• 2 years ago

If you are this deep into it, it doesn't matter.

delfinom5• 2 years ago

If only you could prosecute people in adversarial countries for a felony, lol.

JackSlateur• 2 years ago

Not everywhere, and only if you can prove that were evidences :)

sroussey4• 2 years ago

GitHub/Microsoft likely has a backup. I’d be getting those out about now.

dfgdfg345454562• 2 years ago

chmod u+x running detect_sh script just runs with no output on my arch linux box?

https://www.openwall.com/lists/oss-security/2024/03/29/4

Hackbraten8• 2 years ago

Yes, Arch Linux’s OpenSSH binary doesn’t even link to liblzma, which means your installation is not affected by this particular backdoor.

The authors of the `detect_sh` script didn’t have that scenario in mind, so the `ldd` invocation never finds a link and the script bails early without a message.

dfgdfg345454562• 2 years ago

Thanks!

77pt773• 2 years ago

remove the -e option on the script and run it.

Anyway, arch is not affected because they don't modify openssh to link against any of this nonesense.

imanhodjaev2• 2 years ago

now I wonder which browsers link liblzma?

imanhodjaev2• 2 years ago

I wonder which browsers link liblzma and can this lead to https eavesdropping?

hcks3• 2 years ago

It was caught out of luck due to performance degradation. So nobody reads the code - not even once- prior to merging into upstream supply chain?

hcks3• 2 years ago

https://x.com/bl4sty/status/1773780531143925959?s=20

So nobody reads releases notes either.

But I’m sure this was a one off and were safe now

dhx2• 2 years ago

A mirror of the offending repository created by someone else is available at [1]. GitHub should be keeping the evidence in the open (even if just renamed or archived in a safer format) instead of deleting it/hiding it away.

The offending tarball for v5.6.1 is easier to find, an example being.[2]

m4/.gitignore was updated 2 weeks ago to hide build-to-host.m4 that is only present in the release tarball and is used to inject the backdoor at build time.[3]

[1] https://git.phial.org/d6/xz-analysis-mirror

[2] https://mirrors.xtom.ee/gentoo/distfiles/9f/xz-5.6.1.tar.gz

[3] https://git.phial.org/d6/xz-analysis-mirror/commit/4323bc3e0...

8organicbits2• 2 years ago

There's good discussion of the timeline here: https://boehs.org/node/everything-i-know-about-the-xz-backdo...

user20180120• 2 years ago

Why is the Long Range Zip lrzip compression format not used? It gives better compression than xz when using the correct switches.

squarefoot• 2 years ago

State actor or not, let's not ignore that the backdoor has been discovered thanks to the open nature of the projects involved that allowed digging into the code. Just another example like the infamous Borland InterBase backdoor in the early 2K that remained dormant for years and was discovered months after the source code has been released. If the xz malware authors worked for any corp that produced closed source drivers or blobs that can't be properly audited, we would be fucked; I just hope this is not already happening, because the attack surface in all those devices and appliances out there running closed code is huge.

Roark66• 2 years ago

Sadly this is exactly one of the cases where open source is much more vulnerable to a state actor sponsored attack than proprietary software. (it is also easier to find such backdoors in OS software but that's BTW)

Why? Well, consider this, to "contribute" to a proprietary project you need to get hired by a company, go through their he. Also they have to be hiring in the right team etc. Your operative has to be in a different country, needs a CV that checks out, passports/ids are checked etc.

But to contribute to an OS project? You just need an email address. Your operative sends good contributions until they build trust, then they start introducing backdoors in the part of the code "no one, but them understands".

The cost of such attack is a lot lower for a state actor so we have to assume every single OS project that has a potential to get back doored had many attempts of doing so. (proprietary software too, but as mentioned, this is much more expensive)

So what is the solution? IDK, but enforcing certain "understandability" requirements can be a part of it.

alufers• 2 years ago

Is that true? Large companies producing software usually have bespoke infra, which barely anyone monitors. See: the Solarwinds hack. Similarly to the xz compromise they added the a Trojan to the binary artifacts by hijacking the build infrastructure. According to Wikipedia "around 18,000 government and private users downloaded compromised versions", it took almost a year for somebody to detect the trojan.

Thanks to the tiered updates of Linux distros, the backdoor was caught in testing releases, and not in stable versions. So only a very low percentage of people were impacted. Also the whole situation happened because distros used the tarball with a "closed source" generated script, instead of generating it themselves from the git repo. Again proving that it's easier to hide stuff in closed source software that nobody inspects.

Same with getting hired. Don't companies hire cheap contractors from Asia? There it would be easy to sneak in some crooked or even fake person to do some dirty work. Personally I was even emailed by a guy from China who asked me if I was willing to "borrow" him my identity so he could work in western companies, and he would share the money with me. Of course I didn't agree, but I'm not sure if everybody whose email he found on Github did.

https://en.wikipedia.org/wiki/2020_United_States_federal_gov...

throwaway73567• 2 years ago

Well, consider this, to "contribute" to a proprietary project you need to get hired by a company, go through their he.

Or work for a third-party company that gets access to critical systems without any checks. See for example the incident from 2022 here: https://en.wikipedia.org/wiki/Okta,_Inc.

Or a third-party that rents critical infrastructure to the company (Cloud, SaaS solutions).

joveian3• 2 years ago

Or exactly this kind of backdoor in open source but target proprietary software. I don't know of any survey but I'd be surprised if less than half of proprietary software used open source software one way or another and not surprised if it was quite a bit more than that.

sirsinsalot• 2 years ago

I think we have to assume that all community software is a target. The payoff for bad actors is too great.

For every one of these we spot, assume there are two we have not.

Dribble4633• 2 years ago

Hello,

Github just disabled the repo : https://github.com/tukaani-project/xz

Do someone have an up to date fork to see the project history ?

_zephyrus_• 2 years ago

Is there any news concerning the payload analysis? Just curious to see if it can be correlated with something I have in my sshd logs (e.g. login attempt with specific RSA keys).

Randalthorro• 2 years ago

Since GitHub disabled the repos.. I uploaded all GitHub Events from the two suspected users and from their shared project repo as easy to consume CSV files:

https://github.com/emirkmo/xz-backdoor-github

For those who want to see the GitHub events (commits, comments, pull_requets, diffs, etc.)

lostmsu4• 2 years ago

Better make a torrent out of them.

inevitable112• 2 years ago

Surely the real target of this was Tor (which links liblzma) not random SSH servers.

KOLANICH• 2 years ago

Please note: the changes have been made after GitHub has enforced 2FA (certainly not for "better security", but for promotion of FIDO2 and Windows Hello biometric impl of FIDO2, see https://codeberg.org/KOLANICH/Fuck-GuanTEEnomo for more info. Until recent times (for now access via git protocol is blocked for my acc, I guess based on lack of 2FA set up) it was even possible to push into all repos one has access by just using single-factor SSH key even without enabling 2FA in the account). As I have warned, nothing will protect when a backdoor is introduced by a malicious maintainer, or a "smart entrepreneur" who sold his project to a ad-company, or a loyal "patriot" living and earning money within reach of some state, or just a powerless man who got an offer he can't refuse. In general supply chain attacks by "legitimate" maintainers cannot be prevented. "Jia Tan" is just a sockpuppet to mitigate consequences to maintainers to make it look like they are not involved into it. They surely are. At least according to the current info it were they who have given the malicious account the permission to publish releases on behalf of the project and access to the repo.

IMHO all maintainers of the backdooored projects anyhow related to accepting the malicious changes should be considered as accomplices and boycotted. We don't need evidence of their liability, it is they who need to maintain their reputation. We are just free to take our decisions based on their reputation. Even if they were hacked themselves, it is not our problem, it is their problem. Our problem is to keep ourselves safe. It may feel "unjust" to ruin reputation of a person based on the fact he may be cheated or hacked… But if a person can be cheated or hacked, why should he/she have such a good reputation as everyone else?! So, it makes a lot of sense to just exclude and replace everyone, for whome there exists evidence of comprometation, no matter due to unconcern or malice. But FOSS is a doocracy serving products at dumpling prices ($0, free of charge), and for majority backdoored software is completely acceptable given that they get them free of charge. And powerful actors who can afford to pay for software will just hire devs to develop their private versions, while allowing the public to pay $0 for their free versions and use the backdoors placed into them themselves. In other words a complete market failure.

I think that 1. xz project must be shut down completely. I mean projects should stop using it as a dependency, exclude from distros, boycott it. LZMA algo was developed by Igor Pavlov in 7z project, but somehow it has happenned that liblzma was developed and maintained by unrelated folks. liblzma should be developed as a part of 7z project taking no code other than the trivial one for API compatibility adapter from xz. 2. Projects created by compromised authkrs should be boycotted. 3. Other projects touched by the compromised devs/maintainers should be audited. 4. All the projects using autotools should be audited and must replace autotools with cmake/meson. Autotools is a piece of shit, completely uncomprehensible. There is no surprise it was used to hude a backdoor - according to my experience in FOSS noone likes to touch its scripts anyhow. 5. No project should be built from releases. Project should be built from git directly. Implementing full support of SHA256 in git and git forges (GitHub, GitLab, Codeberg, sr.ht) should be accelerated to mitigate attacks using collisions to replace approved commits (I guess the randomness can be concealed from reviewer's eye in binary resource files, like pictures).

throwaway67743• 2 years ago

It's always Debian, like last time when they removed RNG randomness from ssh because of a warning.

zingelshuher• 2 years ago

Why isn't he identified personally? Very likely he is 'contributing' to other projects under different accounts.

Decabytes3• 2 years ago

So when are we going to stop pretending that OSS maintainers/projects are reaping what they sow when they "work for free" and give away their source code away using OSS licensed software, while large companies profit off of them? If they were paid more (or in some cases even actually paid), then they could afford to quit their day jobs, reducing burn out, they could actually hire a team of trusted vetted devs instead of relying on the goodwill of strangers who step up "just to help them out" and they could pay security researchers to vet their code.

Turns out burned out maintainers are a great attack vector and if you are willing to play the long game you can ingratiate yourself with the community with your seemingly innocuous contributions.

kortilla• 2 years ago

Paid people get burnt out as well and they are just as likely to accept free help as an unpaid person.

Decabytes3• 2 years ago

That's true, but many of these maintainers work a day job on top of doing the open source work precisely because the open source work doesn't pay the bills. If they could get back 40 hours of their time I think many would appreciate it

qwery2• 2 years ago

So when are we going to stop pretending ...

I'm not sure that we are. Doesn't everybody know that developing/maintaining free software is largely thankless work, with little to no direct recompense?

I don't think moving towards unfree software is a good way to make free software more secure. It shouldn't be a surprise that proprietary software is less likely to be exploited in this way simply because they don't accept any patches from outside of the team. What you want is more people that understand and care about free software and low barriers to getting involved.

Decabytes3• 2 years ago

Doesn't everybody know that developing/maintaining free software is largely thankless work, with little to no direct recompense?

No I don't think that is a universally acknowledged feeling. Numerous maintainers have detailed recieving entitled demands from users, as if they were paying customers to the open source software projects. Georges Stavracas' interview on the Tech over Tea podcast^1 describes many such experiences. Similarly, when Aseprite transitioned its license^2 to secure financial stability, it faced backlash from users accusing the developer of betraying and oppressing the community.

On the flipside, if everyone truly does know this is the case, then it's a shame that so many people know, and yet are unwilling to financially support developers to change that. See all of the developers for large open source projects who have day jobs, or take huge pay cuts to work on open source projects. I get that not everyone can support a project financially, but I've personally tried to break that habit of expecting everything I use to be free, and go out of my way to look for donation buttons for project maintainers, and raise awareness during fundraisers. Now if only I could donate directly to Emacs development... I'd encourage other people to do the same.

What you want is more people that understand and care about free software and low barriers to getting involved.

This is tough. For example, the intention behind initiatives like DigitalOcean's Hacktoberfest, are designed to do just this. It is a good idea in theory, submit 4 pull requests and win a tshirt, but not in practice. The event has been criticized for inadvertently encouraging superficial contributions, such as minor text edits or trivial commits, which burden maintainers^3, causing many maintainers to just archive their repos for the month of October.

So, while there's a recognition of the need for more people who understand and value free software, along with lower barriers to entry, the current state of affairs often falls short. The path forward should involve not just increasing awareness and participation but also providing meaningful support and compensation to maintainers. By doing so, we can foster a more sustainable, secure, and vibrant open source community. Or at least that is how I feel...

1. https://www.youtube.com/watch?v=kO0V7BE1bEo 2. https://github.com/aseprite/aseprite/issues/1242 3. https://twitter.com/shitoberfest?lang=en

delfinom5• 2 years ago

OSS maintainers aren't reaping anything. Most OSS licenses say the software is provided without warranty.

pinley• 2 years ago

https://imgur.com/WGaK3Tn

7ero2• 2 years ago

is this sev0?

7ero2• 2 years ago

Is this sev0?

jaromilrojo5• 2 years ago

This is another proof that systemd is an anti-pattern for security: with its crawling and ever growing web of dependencies, it extends the surface of vulnerability to orders of magnitude, and once embraced not even large distro communities can defend you from that.

A malware code injection in upstream xz-tools is a vector for remote exploitation of the ssh daemon due to a dependency on systemd for notifications and due to systemd's call to dlopen() liblzma library (CVE-2024-3094). The resulting build interferes with authentication in sshd via systemd.

saagarjha6• 2 years ago

This isn't Twitter you don't have to use hashtags

heptazoid• 2 years ago

This isn't Xitter, you don't have to tell people how to write.

acdha8• 2 years ago

Please take the systemd trolling to Reddit. They likely targeted xz specifically because it’s so widely used but there are dozens of other libraries which are potential candidates for an attack on sshd, much less everything else which has a direct dependency unrelated to systemd (e.g. dpkg).

Rather than distracting, think about how the open source projects you use would handle an attack like this where someone volunteers to help a beleaguered maintainer and spends time helpfully taking on more responsibilities before trying to weaken something.

jaromilrojo5• 2 years ago

You are distracting from facts with speculations and trolling FUD. I refer to what is known and has happened, you are speculating on what is not known.

account4252• 2 years ago

Those other libraries dependend on by sshd are hopefully more closely monitored. The upstream sshd developers probably did not even consider that liblzma could end up being loaded in the process.

Make excuses for systemd all you want but loading multiple additional libraries into crytical system deamons just to write a few bytes into a socket is inexcusable and directly enabled this attack vector.

throwaway73567• 2 years ago

systemd's call to dlopen() liblzma library (CVE-2024-3094)

That's technically wrong, but no surprise. Anti-systemd trolls usually don't understand technical details after all.

jaromilrojo5• 2 years ago

It is 10 and more years that I experience such ad-hominem attacks.

You are so quickly labeling an identifiable professional as troll, while hiding behind your throwaway identity, that I am confident readers will be able to discern.

Meanwhile let us be precise and add more facts https://github.com/systemd/systemd/pull/31550

Our community is swamped by people like you, so I will refrain from answering further provocations, believing I have provided enough details to back my assertion.

geggo98• 2 years ago

Actually you have a point. A collection of shell scripts (like the classical init systems) have obviously a smaller attack surface. In this case the attacker used some integration code with systemd to attack the ssh daemon. So sshd without systemd integration is safe against this specific attack.

In general, I’m not convinced that systemd makes things less secure. I have the suspicion that the attacker would just have used a different vector, if there was no systemd integration. After all it looks like the attacker was also trying to integrate exploits in owner libraries, like zstd.

Still I would appreciate it, if systemd developers would find a better protection against supply chain attacks.

jaromilrojo5• 2 years ago

I really appreciate your tone and dialectic reasoning, thanks for your reply. And yes, as simple as it sounds, I believe that shell scripts help a lot to maintain mission critical tools. One hands-on example is https://dyne.org/software/tomb where I took this approach to replace whole disk encryption which is nowadays also dependent on systemd-cryptsetup.

acdha8• 2 years ago

It’s also tricky to reason about risk: for example, ShellShock caused a bunch of vulnerabilities in things which used shell scripts and the classic SysV Init system was a factor in a ton of vulnerabilities over the years because not having a standard way to do things like drop privileges or namespace things, manage processes, difficulties around managing chroot, etc. meant that you had a bunch of people implementing code which ran with elevated privileges because it needed to do things like bind to a low network port and they either had vulnerabilities in the privileged part or messed up some detail. I think in general it’s been much better in the systemd era where so much of that is builtin but I have been happy to see them starting to trim some of the things like the compression format bindings and I expect this will spur more.

the_errorist• 2 years ago

Looks like Lasse Collin has commented on LKML: https://lkml.org/lkml/2024/3/30/188

Also, some info here: https://tukaani.org/xz-backdoor/

jwilk6• 2 years ago

Or if you can't stand the lkml.org UI:

https://lore.kernel.org/lkml/20240330144848.102a1e8c@kaneli/

hypnagogic8• 2 years ago

In the future: automated `diff` or any other A/B check to see whether or not the tarball matches the source repo (if not, auto-flag with a mismatch warning attribute), is that feasible to implement?

qxfys• 2 years ago

So, it's been almost 24 hours since I read this yesterday. Is it confirmed that Jia Tan is the perpetrator? do we know who he/she really is? Or are we going to live for the rest of our lives only knowing the pseudo name? just like Satoshi Nakamoto did to us. ;)

mise_en_place• 2 years ago

This is why we never upgrade software versions. I’ve been asked by our customers why we use such an old AMI version. This is why.

gkoberger• 2 years ago

This feels like the exact opposite of the takeaway you should have. Old software isn't inherently more secure; you're missing thousands of security and bug fixes. Yes, this was bad, but look how quickly the community came together to catch it and fix it.

It only took 6 days for it to be found and fixed.

hypnagogic8• 2 years ago

- * _ring ring_ * - "Hello?" - "It's Lasse Collin." - "Why are you collin me? Why not just use the backdoor?"

nolist_policy5• 2 years ago

Debian is considering that their infrastructure may be compromised[1].

[1] https://fulda.social/@Ganneff/112184975950858403

evilmonkey19• 2 years ago

Which OS are affected by this compromise?? Is Ubuntu affected?

costco• 2 years ago

Anyone have any idea what the code in the malicious liblzma_la-crc64-fast.o is actually doing? It's difficult to follow statically.

snickerer• 2 years ago

When I search for "digital masquerade" on Google, the first result is a book with this title from the author Jia Tan. I assume that is how the attackers got their fake name. Or they think using this author's name is a joke.

wowserszzzzz• 2 years ago

Wow

https://ubuntu.com/security/notices/USN-5378-2

ptx7• 2 years ago

Python for Windows bundles liblzma from this project, but it appears to be version 5.2.5 [0] vendored into the Python project's repo on 2022-04-18 [1], so that should be fine, right?

[0] https://github.com/python/cpython/blob/main/PCbuild/get_exte...

[1] https://github.com/python/cpython-source-deps/tree/xz

croemer2• 2 years ago

Which nation state (if any) is most likely behind this? China based on name, or is this a red herring?

The perpetrator did most GitHub actions between 10 and 18 UTC, which sort of rules out US based, unless the messages were scheduled. Consistent with Europe to Asia.

See clickhouse for data: https://play.clickhouse.com/play?user=play#U0VMRUNUICogRlJPT...

rieter3• 2 years ago

Could easily be someone in the EST time zone. There aren't that many events that would be earlier than 8am EST.

perryizgr8• 2 years ago

Why are projects like xz and sshd still active? Just freeze it, it works fine. Only changes should be fixes for vulnerabilities. None of this complicated new functionality. If you want something like that make a new project. If it is truly better people will use it.

joshhansen5• 2 years ago

Is it time to deprecate the ability for code to implement linker symbols in other libraries? Shouldn't there be a strict namespace separation between binaries/libraries? liblzma being to implement openssh symbols seems like a symptom of a much larger problem.

krascovict• 2 years ago

Hello everybody.

I am taking the initiative to gather more information regarding the possible precursors and perpetrators of the backdoor.

The purpose of this commentary is focused on open source information (OSINT).

I am not a judge of anyone or any action that may occur, the objective of this comment is to help through accurate and quick information to help the core developers of the affected packages and consequently the Linux kernel (which may have been indirectly or directly affected) take action necessary in relation to the fact that occurred.

NOTE: This comment will always have "edit" so always review it for information.

Information I have so far.

Summary: 1. GitHub Account Suspension: - The accounts of @JiaT75 and @Larhzu were suspended by GitHub. - All Tukaani repositories, including downloads, were disabled. - Investigate the cause of the account suspensions and whether there is any correlation with suspicious activities.

2. Possible Backdoor in xz/liblzma: - There are concerns about the presence of a backdoor in xz/liblzma. - Investigate whether there is evidence of compromise in the source code and recent updates. - Examine potential impacts, especially if the software is used in critical systems.

3. Updates and Patches in Packages: - Note recent updates in packages such as MinGW w64, pacman-static, Alpine, and OpenSUSE. - Review changelogs to understand if these updates are related to security fixes.

4. Jia's Activities on Platforms and Projects: - Investigate Jia's contributions to different projects and platforms, such as Arch Linux, Alpine Linux, and OpenSUSE. - Check for correlations between Jia's activities and reported security issues.

5. Libera Registration Information: - Analyze Jia's registration details on Libera to determine the timeline of their online activities. - Consider correlating this information with other online activities of Jia.

6. VPN Usage: - Confirm Jia's use of VPN and assess its impact on security investigations. - Explore possible reasons for using a VPN and how it may affect the identification and tracking of online activities.

Links related to user JiaT75 [xz] Remove JiaT75 as a contact, determine correct contacts #11760 - Google/oss-fuzz https://github.com/google/oss-fuzz/issues/11760

Tuktest index hash #7 - tukaani-project/xz/pull/7 https://web.archive.org/web/20240329230522/https://github.co...

bheadmaster• 2 years ago

This is exactly why I fight the windmills so hard when it comes automatic updates in Linux software.

So much damage is caused just by adding a single maintainer to a project - imagine how much power you would have to wield the remote execution systems put in place by naive developers for "automatic updates".

All it takes is a single malicious maintainer given access to the new version update of some popular user software, and they have a new botnet of thousands of devices at their disposal. Better yet, after the backdoor installation, they can just release the real update and cover their tracks forever.

Automatic updates are like running web applications, but without any sandboxing or protection usually implemented by the browser.

bicepjai3• 2 years ago

For someone who does not understand the packages used, could you please summarize in layman non technical terms. Thanks I did read the main post.

bicepjai3• 2 years ago

This link helped a little. https://gist.github.com/thesamesam/223949d5a074ebc3dce9ee78b...

ui2RjUen875bfFA6• 2 years ago

i added there something about a possible planned kernel attack which not mentioned so much yet.

https://gist.github.com/thesamesam/223949d5a074ebc3dce9ee78b...

bicepjai3• 2 years ago

This is perfect summary https://research.swtch.com/xz-timeline

3v1n03• 2 years ago

Also the attacker included in the 5.6.0 release the support for the long-awaited multi-threading decompression (and - broken - sandbox) making it very attractive to upgrade to...

It was probably a tactic to give a reason to upgrade. It's not always a fault for those who did or tried to do.

Backdoor in upstream xz/liblzma leading to SSH server compromise

Related Stories