Post Mortem: axios NPM supply chain compromise

https://github.com/axios/axios/issues/10636

by JeanMeche • 4 months ago

291 135 4 months ago

uticus• 4 months ago

March 31, around 01:00 UTC: community members file issues reporting the compromise. The attacker deletes them using the compromised account.

Interesting it got caught when it did.

fraywing• 4 months ago

Incredible uptick in supply chain attacks over the last few weeks.

I feel like npm specifically needs to up their game on SA of malicious code embedded in public projects.

simulator5g2• 4 months ago

That's the reality of modern war. Many countries are likely planting malware on a wide scale. You can't even really prove where an attack originated from, so uninvolved countries would also be smart to take advantage of the current conflict. Like if you primarily wrote German, you would translate your malware to Chinese, Farsi, English, or Hebrew, and take other steps to make it appear to come from one of those warring countries. Any country who was making a long term plan involving malware would likely do it around this time.

altmanaltman• 4 months ago

You can write code in Chinese and Farsi?

ipnon• 4 months ago

NPM is designed to let you run untrusted code on your machine. It will never work. There is no game to step up. It's like asking an ostrich to start flying.

dcrazy5• 4 months ago

It’s far from a complete solution, but to mitigate this specific avenue of supply chain compromise, couldn’t Github/npm issue single-purpose physical hardware tokens and allow projects (or even mandate, for the most popular ones) maintainers use these hardware tokens as a form of 2FA?

lrvick20• 4 months ago

All maintainers need to do is code signing. This is a solved problem but the NPM team has been actively rejecting optional signing support for over a decade now. Even so maintainers could sign their commits anyway, but most are too lazy to spend a few minutes to prevent themselves from being impersonated.

pas5• 4 months ago

code becomes trusted by review, but these crowd sourcing efforts to do so fizzled out, so in practice we have weak proxies like number of downloads

the implicit trust we have in maintainers is easily faked as we see

dgellow3• 4 months ago

npm process to setup OIDC is way too frustrating. There is just so much friction. You need the package to first exists in the registry, meaning you have to first create an API token and push something. And only then can you enable OIDC for that specific package. After adding the repo + workflow names, you have to save. Then finally toggle the “only allow OIDC publishing”.

Before each action you need to enter your 2fa code.

I got so frustrated with npm end of last year that I wrote a whole guide covering that issue: https://npmdigest.com/guides/npm-trusted-publishing

mcintyre19944• 4 months ago

You’re right, but a colleague recently showed me this CLI for it: https://docs.npmjs.com/cli/v11/commands/npm-trust

Still needs to be published first, but looks like it automates all the annoying UI things you mentioned.

charcircuit2• 4 months ago

Does OIDC flow block this same issue of being able to use a RAT to publish a malicious package?

hsbauauvhabzb• 4 months ago

No, once the computer is compromised nothing really helps assuming the attacker is patient enough.

fortuitous-frog2• 4 months ago

No. axios (v1 at least; not v0) were setup to publish via OIDC, but there's no option on npmjs for package maintainers to restrict their package to *only* using OIDC. The maintainer says his machine was infected via RAT, so if he was using software-based 2FA, nothing could have prevented this.

dgellow3• 4 months ago

Actually there is an option to restrict to only OIDC publishing. It is a bit hidden and relies on a different form for reasons I really cannot understand. Npm UX is just so bad.

Point 4 from https://npmdigest.com/guides/npm-trusted-publishing#ux-probl...

(I wrote that guide page for myself because I always get annoyed when dealing with npm OIDC)

mcintyre19944• 4 months ago

Nope, the most restrictive option available is to disallow tokens and require 2FA. I think that using exclusively hardware 2FA and not having the backup codes on the compromised machine probably would have prevented this attack though.

pepve• 4 months ago

Someone in the linked Github thread describes an attack where the attackers waited for the victim to use their Yubikey for an AWS login, giving the attackers access to AWS as well. I don't think hardware 2FA is safe against a RAT.

akersten• 4 months ago

Any good payload analysis been published yet? Really curious if this was just a one and done info stealer or if it potentially could have clawed its way deeper into affected systems.

sheept• 4 months ago

This article[0] investigated the payload. It's a RAT, so it's capable of executing whatever shell commands it receives, instead of just stealing credentials.

[0] https://safedep.io/axios-npm-supply-chain-compromise/

Zopieux2• 4 months ago

Not much we didn't know (you're basically SOL since an owner was compromised), however we now have a small peek into the actual meat of the social engineering, which is the only interesting news imho: https://github.com/axios/axios/issues/10636#issuecomment-418...

hatmanstack• 4 months ago

jasonsaayman and voxpelli had useful write ups from the "head on a swivel" perspective of what to watch out for. Jason mentioned "the meeting said something on my system was out of date." they were using Microsoft meeting and that's how they got RCE. Would love more color on that.

pas5• 4 months ago

they are cloning Zoom and MS Teams, and try to get people to either copy a script (which is in a textarea that's conveniently too small to show the whole script, and scrollbars are hidden by CSS, and there's a copy button, and when you paste it into the terminal you'll see last few lines, also look innocent, but there's a curl | zsh or `mshta` somewhere in there), download and run a binary/.dmg (and it might be even signed by GoogIe LLC. - the name chosen to look good in the usual typeface used on macOS).

...

it seems the correct muscle memory response to train into people is that "if some meeting link someone sent you doesn't work, then you should create one and send them the link"

(and of course never download and execute anything, don't copy scripts into terminals, but it seems even veteran maintainers do this, etc...)

see Infection Chain here https://cloud.google.com/blog/topics/threat-intelligence/unc...

textarea at the bottom of this comment: https://github.com/axios/axios/issues/10636#issuecomment-418...

denalii• 4 months ago

Other comment already said, but it seems it was likely a clone of the web interface rather than the actual teams client. You can see a lot more details in this comment on the github thread (not by the axios maintainer, but goes over the same threat group and campaign) https://github.com/axios/axios/issues/10636#issuecomment-418...

Hamuko• 4 months ago

Makes me glad that I've only ever used my iPad whenever I've had to interview through Microsoft Teams.

NewEntryHN• 4 months ago

He says it mimicks what is described here: https://cloud.google.com/blog/topics/threat-intelligence/unc...

Which is basically phishing:

The meeting link itself directed to a spoofed Zoom meeting that was hosted on the threat actor's infrastructure, zoom[.]uswe05[.]us.
Once in the "meeting," the fake video call facilitated a ruse that gave the impression to the end user that they were experiencing audio issues.
The recovered web page provided two sets of commands to be run for "troubleshooting": one for macOS systems, and one for Windows systems. Embedded within the string of commands was a single command that initiated the infection chain.

lrvick20• 4 months ago

An owner being compromised is absolutely survivable on a responsibly run FOSS project with proper commit/review/push signing.

This and every other recent supply chain attack was completely preventable.

So much so I am very comfortable victim blaming at this point.

This is absolutely on the Axios team.

Go setup some smartcards for signing git push/commit and publish those keys widely, and mandate signed merge commits so nothing lands on main without two maintainer sigs, and no more single points of failure.

TheTaytay• 4 months ago

It wasn’t done through git. It was a direct npm publish from the compromised machine. If you read further down in the comments (https://github.com/axios/axios/issues/10636#issuecomment-418...), it seems difficult to pick the right npm settings to prevent this attack.

If I understand it correctly, your suggestions wouldn’t have prevented it, which is evidence that this is not as trivially fixable as you believe it is.

fortuitous-frog2• 4 months ago

Did you investigate the maintainer compromise and publication path? The malicious version was never committed or pushed via git. The maintainer signs his commits, and v1 releases were using OIDC and provenance attestations. The malicious package versions were published locally using the npm cli after the maintainer's machine was compromised via a RAT; there's no way for package maintainers to disable/forbid local publication on npmjs.

It seems the Axios team was largely practicing what you're preaching. To the extent they aren't: it still wouldn't have prevented this compromise.

Zopieux2• 4 months ago

This only works up to a point. Some human needs some way of changing the publication setup in case something goes wrong or changes. What you're asking is blowing a proverbial e-fuse once the setup is known to be working. This is software, shit will go wrong at some point and you need a way to make changes.

patrakov3• 4 months ago

The "nothing gets on main without two signatures" rule would not have prevented the xz story, where a comaintainer was able to smuggle malicious code past the review as "binary data for new tests" and, effectively, get it signed.

robshippr2• 4 months ago

The interesting detail from this thread is that every legitimate v1 release had OIDC provenance attestations and the malicious one didn't, but nobody checks. Even simpler, if you're diffing your lockfile between deploys, a brand new dependency appearing in a patch release is a pretty obvious red flag.

GCUMstlyHarmls• 4 months ago

To be honest, I would have assumed the tooling would do attestation verification for me. The diffing the lockfile would be on me though.

lrvick20• 4 months ago

I ask this on every supply chain security fail: Can we please mandate signing packages? Or at least commits?

NPM rejected PRs to support optional signing multiple times more than a decade ago now, and this choice has not aged well.

Anyone that cannot take 5 minutes to set up commit signing with a $40 usb smartcard to prevent impersonation has absolutely no business writing widely depended upon FOSS software.

Normalized negligence is still negligence.

4ndrewl4• 4 months ago

Is the onus really on people who write code here? It really should be on those who choose to use this unsigned code, surely?

lorenzohess• 4 months ago

Perhaps, but if it's gotten to the point where millions of people download the unsigned code, signing should probably become required. Even reproducible builds.

lrvick20• 4 months ago

Anyone that maintains code for others to consume has a basic obligation to do the bare minimum to make sure their reputations are not hijacked by bad actors.

Just sign commits and reviews. It is so easy to stop these attacks that not doing so is like a doctor that refuses to wash their hands between patients.

If you are not going to wash your hands do not be a doctor.

If you are not going to sign your code do not be a FOSS maintainer.

eviks2• 4 months ago

"Anyone that cannot spend $40+ to give every FOSS maintainer a smartcard and maybe even separate machines for releases and make the more secure workflow truly 5 minutes has absolutely no business widely depending upon FOSS"

lrvick20• 4 months ago

A $50 used laptop from goodwill and a $40 yubikey will do the job.

If maintainers really cannot afford that, they should flag it as a major big bold print supply chain risk on the readme: "We cannot afford 4 yubikeys for our maintainers and thus all code is signed with software keys in virtual machines as a best effort defense. Donate to our fund [here] to raise $500 for dedicated release hardware"

Friends and I have gotten 100s of yubikeys and nitrokeys donated to FOSS maintainers, but FOSS maintainers have to be willing to say they would use them and signal that they need them.

Honestly though, anyone that cannot afford $40 I expect is at high risk of being bribed or having to give up contributing to take on more work, so we should significantly fund any project signaling that much desperation.

patrakov3• 4 months ago

Anyone that cannot take 5 minutes to set up commit signing with a $40 usb smartcard to prevent impersonation has absolutely no business writing widely depended upon FOSS software.

No. As a user of your package, I want assurance that the package you publish does what it says it does and does not contain malware. This is different from the package having been published by you. I want protection against you going rogue, not only from you being impersonated. 2FA on your side does not protect me against you going rogue. A comaintainer does.

So the correct quote would be: Anyone that cannot find a comaintainer to review all the code and to prevent deliberate sabotage has absolutely no business writing widely depended upon FOSS software.

anematode• 4 months ago

Looks like a very sophisticated operation, and I feel for the maintainer who had his machine compromised.

The next incarnation of this, I worry, is that the malware hibernates somehow (e.g., if (Date.now() < 1776188434046) { exit(); }) to maximize the damage.

ffsm8• 4 months ago

Isn't that already how it is?

I mean the compromised machine registers itself on the command server and occasionally checks for workloads.

The hacker then decides his next actions - depending on the machine they compromised they'll either try to spread (like this time) and make a broad attack or they may go more in-depth and try to exfiltrate data/spread internally if eg a build node has been compromised

nurettin3• 4 months ago

I never understood why all the CAS tutorials pushed axios. This was before vite and build-scripts was how you did react. After the compromise I reviewed some projects and converted them to pure JS fetch and vite.

pas5• 4 months ago

what's CAS?

nurettin3• 4 months ago

oops I meant create react app, no idea how I typed cas.

pianopatrick• 4 months ago

Seems to me the root of the problem was that the guy was using the same device for all sorts of stuff.

Seems to me that one drastic tactic NPM could employ to prevent attacks like this is to use hardware security. NPM could procure and configure laptops with identity rooted in the laptop TPM instead of 2FA. Configure the NPM servers so that for certain repos only updates signed with the private key in the laptop TPM can be pushed to NPM. Each high profile repo would have certain laptops that can upload for that repo. Set up the laptop with a minimal version of Linux with just the command line tools to upload to NPM, not even a browser or desktop environment. Give those laptops to maintainers of high profile repos for free to use for updates.

Then at update time, the maintainer just transfers the code from their dev machine to the secure laptop via USB drive or CD and pushes to NPM from the special laptop.

pas5• 4 months ago

they can simply make an app that requires tapping a button, so people don't end up with TOTP seeds stored in their password manager on the same notebook where they run 'publish' from

momo_dev2• 4 months ago

this is why i pin every dependency hash in my python projects. pip install --require-hashes with a locked requirements file catches exactly this, if the package hash changes unexpectedly the install fails. surprised this isn't the default in the npm ecosystem

minitech• 4 months ago

Npm and the other JavaScript package managers do generate and check lockfiles with hashes by default. This was a new release, not a republishing of an old version (which isn’t possible on the npm registry anyway).

momo_dev2• 4 months ago

i wasn't aware npm lockfiles check hashes by default now. my concern is more about the initial install before a lockfile exists, like in CI from a fresh clone without a committed lockfile. but you're right, once the lockfile is there the hash mismatch would be caught.

aeneas_ory2• 4 months ago

Check if your machine was affected with this tool: https://github.com/aeneasr/was-i-axios-pwned

zwarag• 4 months ago

How do we know this is not the next tool in line to compromise a machine?

aeneas_ory2• 4 months ago

Read the source code

eviks2• 4 months ago

something on my system was out of date. i installed the missing item

Given the "extreme vigilance" of the primitive "don't install unknown something on your machine" level is unattainable, can there really be an effective project-level solutions?

Mandatory involvement of more people to hope not everyone installs random stuff, at least not at same time? (though you might not even have more people...)

robshippr2• 4 months ago

The interesting detail from the GitHub thread is shaanmajid's observation that every legitimate v1 release had OIDC provenance attestations and the malicious one didn't, but nobody checks. Even simpler, if you're diffing your lockfile between deploys, a brand new dependency appearing in a patch release is a pretty obvious red flag without needing any attestation infrastructure.

ncr100• 4 months ago

Dupe comment - double submitted? https://news.ycombinator.com/item?id=47622805

falkensmaize2• 4 months ago

The fetch api has been widely available in browsers for a decade now. And in node since 18. A competent developer could whip up a more axios-like library with fetch in a day easily. You can do all the cool things like interceptors with fetch too.

Yet most developers I work with just use it reflexively. This seems like one of the biggest issues with the npm ecosystem - the complete lack of motivation to write even trivial things yourself.

timcobb2• 4 months ago

Fetch can't do a lot of table stakes stuff...

paustint• 4 months ago

Ok, well have AI write some table stakes for you in 10 minutes with 100% test coverage and only provide exactly what "table stakes" you are missing without any bells and whistles.

dkdbejwi383• 4 months ago

Such as?

neya2• 4 months ago

A competent developer could whip up a more axios-like library with fetch in a day easily.

Then you would have created just an axios clone. AKA re-inventing the wheel. The issue isn't the library itself, but rather the fact that it's popular and provided a large enough attack surface.

You can actually just clone the axios package and use it as is from your private repo and you would not have been affected.

darepublic• 4 months ago

You would have created a smaller axios that only does what you needed it to. Even better

littlestymaar• 4 months ago

AKA re-inventing the wheel.

The wheel is the native fetch API, nobody needs to reinvent it.

All you'd do in that scenario is make your own hubcap to put on top.

bensyverson• 4 months ago

I think we're entering an era where "re-inventing the wheel" is actually a completely valid defensive posture. The cost is so low relative to the reduction in risk.

kro• 4 months ago

I really don't get this either, I've always removed axios when it was preinstalled in a framework.

I use "xhr" via fetch extensively, it can do everything in day to day business for years with minimal boilerplate.

(The only exception known to me being upload progress/status indication)

port112• 4 months ago

Axios really does a lot of other great things. I would argue that Fetch could’ve easily been Axios-lite. Axios handles errors better, has interceptors, parses JSON for you, etc.

The multiple supply chain attacks against NPM packages would, of course, be solved if we simply stop using third-party libraries.

falkensmaize2• 4 months ago

I guess the point I’m making is that a lot of popular JavaScript libraries were created to address deficiencies in the core api that don’t exist anymore, but we keep using these libraries mostly because of entropy and familiarity.

schindlabua• 4 months ago

parse json?

const x = await fetch(...); await x.json();

"intercept" code that runs before every request?

const withAuth = (res, options) => fetch(res, { ... do stuff here });

agumonkey• 4 months ago

Maybe people are too comfy with axios base path url and interceptor api ? or maybe fetch handles that as well ? (through a shim ?)

Xentyon• 4 months ago

This is why I've moved to native fetch for most projects. The fewer dependencies in the chain, the smaller the attack surface. For API clients especially, fetch + a thin wrapper is usually enough.

Chyzwar2• 4 months ago

NPM should fix this mess.

Adding postinstall should require approval from NPM. NPM clients should not install freshly published packages. NPM packages should be scanned after publishing. High profile packages should verify upstream git hash signature. NPM install should run in sandbox and detect any attempt to install outside project directory.

But npm being part of multi trillion company cannot be bothered to fix any of these. Instead they push for tighter integration with GitHub with UX that suck.

cromka2• 4 months ago

NPM clients should not install freshly published packages.

That would be a beautiful example of Cobra effect: what about updates that fix vulnerabilities? You're gonna force users to wait couple days or a week before they can get malware removed?

Chyzwar2• 4 months ago

This could be controlled by npm. Client ask for available versions anyway. If package is security fix then it can be made available instantly. But this delay gives time for security scanners and time to notify maintainers that package was published.

mcintyre19944• 4 months ago

In cases like this that isn’t an issue, NPM takes the malicious package down and you roll back to the previous version.

The problem would be new versions that fix security issues though, and because this is all open source as soon as you publish the fix everyone knows the vulnerability. You wouldn’t want everyone to stay on the insecure version with a basically public vulnerability for a week.

mememememememo• 4 months ago

Just ban postinstall.

Post Mortem: axios NPM supply chain compromise

Related Stories