NPM 6.9.1 is broken due to .git folder in published tarball

https://npm.community/t/npm-6-9-1-is-broken-due-to-git-folder-in-published-tarball/8454/2

by FrankSansC • 7 years ago

195 147 7 years ago

Software this popular doesn't even run a test that installs, upgrades, then uninstalls before deployment? Isn't that the most important thing the software needs to be able to do? Does NPM test anything at all before they release? Another reason to switch to yarn.

floatingatoll• 7 years ago

All autoupdate tests will eventually fail to prevent a self-breaking autoupdate from affecting one or more users. The trick is to prepare effectively for that scenario ahead of time. For example, homebrew’s “brew doctor” command has existed for a long time, but in recent years it now comes with a litany or helpful text and other commands (including autoupdate code logic) reference it when they encounter errors.

fps_doug2• 7 years ago

Welp, not that our product is that popular, we're selling in one country only, but there are exactly zero tests. Everything is manual. And I think this is still how the majority of projects out there do it. Not that this is great, but management, and a lot of devs don't see too much value in it, more like "nice to have if there's extra time".

not_kurt_godel• 7 years ago

That is not how the majority of professional software projects do things, and even if it were it doesn't matter because npm is not your average software project anyway. It is absolute clown status that they don't have integ tests to catch this sort of thing, and adds yet more doubt in my mind about the wisdom of using it (on top of an already considerable doubt pile).

bassman90002• 7 years ago

How old is the product? Have you done any decent refactoring?

de_watcher2• 7 years ago

Another reason to switch to yarn.

Another reason to stop making dozens of different package managers. Or dozens copies of literally anything.

Can you people just talk and work together? These package managers aint rocket science.

Look at GNU/Linux: there are only 2 or 3 variations for everything.

diggan• 7 years ago

Can you people just talk and work together

This is a general problem in technology, not just about package managers

Look at GNU/Linux: there is only 2 or 3 variations for everything.

Not true, there is probably hundreds of versions of everything (probably not bad in itself) but 2-3 popular ones that you know about.

The reason why we keep seeing new package managers is because people think they can do it better than the existing stuff. And I think it's a necessary step to improvement, but some of the suggested solutions will be crap. But then sometimes something really good comes along and the ecosystem self-corrects.

sime2009• 7 years ago

Look at GNU/Linux: there are only 2 or 3 variations for everything.

Mostly false.

For package management on Linux I can think of 4 just off the top of my head:

* apt

* rpm

* apk

* pacman

There are bound to be more and this is not counting other approaches like AppImage and Snap.

eeZah7Ux3• 7 years ago

Another reason to stop making dozens of different package managers.

Even better: decouple what is generic / OS specific / language specific into libraries and formats and reuse them.

I've worked on many package managers and it's astonishing how every new language reinvents the wheel and makes the same mistakes again.

People ignore the history of software packaging every day.

Already__Taken• 7 years ago

lol what? 2 or 300 maybe. rpm, Deb, apt, apt-get, APK, nix, aptitude, dpkg, aur. some of these are different, or the same, or not. dammed if I can tell. hopefully containers eat them all.

pluma3• 7 years ago

There are literally only two package managers for the npm registry right now. This is akin to apt vs dpkg.

There used to be bower, which installed via GitHub directly and has been dead for ages. There are now also several experimental alternatives but they have basically no traction.

The newest contestant is Entropic, which was built by former npm developers as a reaction to the problems of a for-profit venture capital funded startup running a centralised registry and controlling the client.

So for the Linux comparison (ignoring the actual implementations): yarn and npm are apt and dpkg, Entropic is yum and Bower is pacman. Not exactly proliferation if you consider how many packages there are on npm right now.

eeZah7Ux3• 7 years ago

doesn't even run a test that installs, upgrades, then uninstalls before deployment

That would be what Debian does - automatically.

Not hipster enough.

rolltiide• 7 years ago

The interesting thing about semi-decentralized technologies is that every issue seems like a clusterfuck but it actually gets fixed fairly quickly and nobody in the future has to think about it at all

choeger• 7 years ago

More interesting question: How many hip startups could not build/test/ship their latest iteration yesterday because their whole CI pipeline depended on npm working?

gridlockd3• 7 years ago

What CI pipeline depends on the latest release of any piece of software working?

Updates can always break stuff.

eeZah7Ux3• 7 years ago

hip startups could not build/test/ship their latest iteration

This sounds like a feature.

z3t4• 7 years ago

These sort of bugs are hard to find with unit tests. But it's still easy to test, to make sure it never happen again. I think it's called integration tests. It can also be done with live testing, by checking all packages that lands on the public repository - if they include .git folder, passwords/keys, or other credentials.

mrolla2• 7 years ago

Bug aside, it baffles me that it's a real person publishing rather than CI after tagging.

majewsky• 7 years ago

I don't know about their process, but I always make my release tags myself. I wouldn't want to give the signing key to any CI system.

WoahNoun• 7 years ago

That seems fundamentally backwards. The CI system should do the tagging. Allowing manual tagging introduces intentional or unintentional malfeasance in shared projects.

mrolla2• 7 years ago

I am very reluctant myself, but I think at some point you have to compromise. You can self host and go the hard way, trust some 3rd party CI (reputation is key) or go, if available, with the one from your cloud provider. This is exactly why we went with Codebuild at some point. It's not great, but hey we are not giving it anything more they don't already have (we host everything on AWS).

gigatexal6• 7 years ago

I think this probably happens more often in other package repositories than we hear about it’s just that NPM is gigantic and people like to hate on JavaScript.

I’ve done stupid things too. I once committed the s3 keys to our public repo and had to explain why our s3 bill was so high. Never made that mistake again.

thrwaway483732• 7 years ago

Ouch. How long did it take you to figure out what happened, and how high was your bill? How many people abused it, do you think (just one?) and any idea for what?

gigatexal6• 7 years ago

Just one day. Someone uploaded thousands of junk and since we had versioning it took a long time to delete. And I think we were being billed extra on the high load being given to our bucket. So could have been multiple someones. I revoked the keys, deleted the bucket, even my own account, and reset everything. Lucky for me the CTO was understanding. I don’t remember the exact bill amount but it went up in relative terms by something like 2x

fs111• 7 years ago

Which other package repos are those supposed to be?

gigatexal6• 7 years ago

Maybe github’s repo push is going to fix things.

smurv1337• 7 years ago

Been there done that. I Think we were charged some $6000, not my proudest moment ...

gigatexal6• 7 years ago

Oh man I think had our bill been that high I’d have been fired for sure. We were a very cash strapped small op at the time.

lmm5• 7 years ago

Having submitted packages to Maven Central, NPM seems distinctly lax by comparison. No federation and no package signing speaks to an organisation that makes poor choices rather than one that's just been unlucky.

gigatexal6• 7 years ago

That low hanging fruit should be addressed. Having hashes of the data attached to the package that could then be checked on download would be nice. But it wouldn’t have fixed this adding of the .git folder.

awinder• 7 years ago

This doesn’t count? https://blog.npmjs.org/post/172999548390/new-pgp-machinery

pluma3• 7 years ago

Unlike Maven Central or almost every other package registry in the world, NPM is operated by a venture capital founded startup (npm Inc).

So, yeah, their priorities are very different and that leads to questionable technical decisions.

A number of prominent ex-employees have collaborated on a new, federated package repository called Entropic. Frankly, I hope they succeed and npm Inc meets its inevitable fate.

tracker14• 7 years ago

What I really wish that NPM would do, as an initial security addon would be to load each of the JS files and check for all the IP addresses and URLs in the module, and include that in the package information. If it's not an API client, then one can/could be very leery of any addresses and could flag any packages that add/change them between releases, or maybe require a major version bump.

krapp3• 7 years ago

I think this probably happens more often in other package repositories than we hear about it’s just that NPM is gigantic and people like to hate on JavaScript.

This is Hacker News... if other package managers had issues more often than the circus of errors that is NPM, we would definitely hear about it.

We don't, because NPM is qualitatively worse than all of them.

wwright• 7 years ago

I believe NPM is at least an order of magnitude bigger than every other package repository, for what it’s worth.

pcr910303• 7 years ago

This is Hacker News... if other package managers that HN hates had issues more often than the circus of errors that is NPM, we would definitely hear about it.

We don’t, because HNers hate NPM the most than all of them.

jtdev• 7 years ago

Or... maybe those other package repos are actually better managed? And maybe, just maybe, JavaScript might have actual, material language and ecosystem issues.

firethief• 7 years ago

cargo won't publish with a dirty git working directory unless you force it, so that wouldn't have happened with Rust (and .git/ is excluded implicitly).

Sahhaese2• 7 years ago

I see it's been fixed by releasing 6.9.2.

Does this mean you can now "brick" people's projects by sneaking npm@6.9.1 into their package.json?

KhoomeiK• 7 years ago

Certainly seems possible... Looks like I probably won't be installing anything with npm until they unpublish it.

Ajedi32• 7 years ago

I don't think so. NPM packages aren't installed globally by default, they're scoped to the local project via `node_modules`. So by default, the version of npm on the user's PATH wouldn't get updated.

anc84• 7 years ago

The majority of npm based installation instructions I encounter "out there" are using "npm install -g".

MehdiHK• 7 years ago

Probably relevant-

This week's Node.js weekly [1] mentions: "Last week we mentioned the long awaited status of npm 6.9.1 and the possible ‘strike’ [2] in ongoing community work on the project, but npm’s Isaac Z. Schlueter has stepped up, got a release out"

[1] https://nodeweekly.com/issues/294

[2] https://gist.github.com/aeschright/8ed09cbc2a4aee00fcb4ad350...

ch4s3• 7 years ago

Well, they did just fire a bunch of people for complaining about working conditions, so it's no surprise that they're stumbling into other mistakes.

kotxig5• 7 years ago

While this is clearly a bug, the default publish strategy is to publish all the contents of the package including dev/test files. If you look at the node_modules tree on a typical server-side node project it's full of garbage files.

We still think it's a great idea to pull down hundreds if not thousands of copies of modules from a remote every time we generate a build. The community has always been lacking maturity around how it manages modules and releases code for production, preferring to preserve developer productivity and trinket features over best security practices and optimizing for the size and quality of production packages.

I don't want to install hundreds of modules every time I create a new build. I don't want your tests or your README in my production tarball. I don't want your browser compatibility code in my server code. I don't want my node_modules tree to be 9 layers deep. I don't want to have to dedupe multiple copies of modules by hand. I don't want to have to debug where shrinkwrap isn't respected. I don't want to play roulette with the package manager version to figure out which one does the right thing for my package. I don't want to run your dubious pre/post install scripts. I don't want to use npm.

The sad thing is, npm is a commercial entity that has its package manager published by default with an opensource community project. And this is why we can't have nice things.

LeoNatan25• 7 years ago

Maybe "we" should not be writing our server code using a terrible ecosystem? Just an idea.

kotxig5• 7 years ago

Indeed. It's hard to avoid that conclusion, but you can't deprecate things overnight.

halfmatthalfcat5• 7 years ago

Node has a great server-side ecosystem. I'd go as far as saying Node lends itself to certain layers of the backend stack because of it's history being more CPU-efficient then other server-side runtimes.

Can you provide any examples where Node does not provide a good ecosystem for backend programming? Genuinely curious.

drngdds• 7 years ago

Yeah, I don't know why npm defaults to including everything in the folder for the production build instead of having it be a whitelist. It's not that hard to specify "include package.json, package-lock.json, and ./build" in your package.json.

kotxig5• 7 years ago

It promotes certain developer-friendly ideals such as going into the module directly within node_modules or using npm explore. I think this is the wrong priority.

halfmatthalfcat5• 7 years ago

That's why it's a good idea to leverage bundlers and simply roll all of your desired dependencies into a single file and then just execute Node against that.

You can easily do this with Webpack.

edit: To clarify I'm talking about server-side components.

andrew_• 7 years ago

For anyone reading this comment, investigate Rollup. My own anecdotal experience is that it kicks webpack's behind when it comes to bundling/packaging server-side anything, or anything that'll be run via `node`

shados2• 7 years ago

Thats why we never replaced our homegrown, predates-npm package manager. It feels crazy to be maintaining our own when there's such a popular 3rd party one out there, but honestly...it really sucks. Migrating nearly a decade worth of code to something that is often objectively worse doesn't feel worth it. And so the homegrown package manager lives on...

growtofill• 7 years ago

‘We’ being?

tracker14• 7 years ago

`npm ci` will use a local cache, and only pull specifically identified versions.

_Marak_• 7 years ago

NPM still has bugs from v5 which haven't been addressed and are ignored by it's maintainers. Problems with `npm link` were never fixed. There is a discussion on the NPM tracker where the maintainers tell Substack that his ( very correct ) assessment of an issue was wrong and lock the thread.

It's always the same usual suspects at NPM never accepting responsibility or blame for multiple incidents and ongoing quality issues.

NPM is currently on it's last legs before acquisition or buyout.

shados2• 7 years ago

npm link almost feels like a troll attempt. Running 2 packages locally together is one of the most basic features I can think of, and it works so poorly.

mercurysmessage• 7 years ago

Found this out not long ago on my own. Wish I would have known link was so bad before I tried it.

samirm• 7 years ago

If they are on the verge of a buyout, I hope Microsoft gets them.

msoad• 7 years ago

Yup! Even this "amazing" bug was reported and was ignored by the team before:

https://npm.community/t/npm-6-9-1-is-broken-due-to-git-folde...

mustardo• 7 years ago

Fuckit.JS to the rescue?

https://github.com/mattdiamond/fuckitjs/blob/master/README.m...

NPM 6.9.1 is broken due to .git folder in published tarball

Related Stories