When Not to Use Lock Files with Node.js

https://www.twilio.com/blog/lockfiles-nodejs

by fagnerbrack • 6 years ago

43 13 6 years ago

This sounds more like a fix for an issue that should not exist in the first place. Why is npm ignoring the lock file to begin with? Why not publish and respect it on install?

ailideex• 6 years ago

If your package only works with specific versions you should put those versions in package.json - if the versions in package.json is correct then what specific versions within those bounds someone uses is ... well ... unrelated to your package.

syspec• 6 years ago

It can blow up your bundle size if everyone has patch version mismatch . That's probably the only reason I can think of.

In theory semver should help with this, but humans decide the semver version and it's not always possible to know the effects of a change so they may mark it as patch, but it breaks something and you get this sceanrion

paulddraper• 6 years ago

It's a consequence of modularization, and it's not at all unique to Node.js. (Although Node.js has hyper-modularization, so the situation is more pronounced.)

Ruby gem has lock files, Python pip has lock files. I wish everyday that apt/dpkg had lock files.

The balance is between permissive versions (small install size) and pinned versions (reproducibility).

Some situations call for one, some call for the other.

j88439h84• 6 years ago

But doesn't Node give each module its own deps? Python can't do this because it doesn't support multiple versions of the same dep.

furstenheim• 6 years ago

"however, if you want to make sure you have the same experience as your users, I'd recommend to add it to .gitignore."

That's just plainly wrong. Lock files are ignored in libraries so that shared common dependencies can be hoisted and number of dependencies reduced. That means that two users that depend on your library will get different resolutions for you dependencies according to other libraries that they use.

leipert• 6 years ago

The reason why not to use lock files in libraries doesn’t make sense to me.

Yes, you should ignore lock files in the .npmignore, so that they won’t be published to npm, because library users don’t consume them. As a library author, where a lot of people are collaborating on a package, you will gain the same benefits: reproducibility and easier triage of bugs. If you are concerned about updates, you can utilize tools to automatically update your lock files (greenkeeper, dependabot, renovate). Renovate has a mode to completely update your lock file in a defined interval.

Edit: your ci could also have a scheduled job which ignores the lock file (scheduled?), if you want to emulate a "integration" test and a "fresh" install

AgentME2• 6 years ago

I agree. I have a bunch of little modules published on npm, and for all of them, I have a yarn.lock (or package-lock.json) lockfile committed in the git repo, but also listed in .npmignore so it's not part of the package published on npm. I do this so that all of the test/dev dependencies stay pinned to guaranteed working/tested versions, and so that there's a record of which versions of (non-dev) dependencies the library is known to work with. Here's a bunch of painful problems that happen when you don't check in your lockfile:

* One of the most frustrating things in the world is checking out a git repo that's missing any lockfiles and then finding out that its build script or unit tests are broken with the current versions of the dependencies, so you're blocked on doing any work with the repo until you fix that. The extra frustrating thing is that people who previously checked out the repo (including the maintainer) won't run into this issue, so this issue will fall onto new contributors and be an obstacle to them specifically!

* The next worst thing is getting a report from a user that your library no longer works with the current versions of its dependencies, and you don't even know which dependency is the problem or know of a good version of the dependency it worked with.

Putting the lockfile in .gitignore opens you up to the two above issues, and then barely helps. Your CI system isn't going to re-run your tests when your dependencies or sub-dependencies get updates. You're going to have to set that up separately and then you could set that process up to ignore or update your lockfile so it's not a reason to git-ignore your lockfile.

why-oh-why2• 6 years ago

Your CI system isn't going to re-run your tests when your dependencies or sub-dependencies get updates.

Yes and no. Your CI system runs when you want it to. If you want to check the latest dependencies,

- with a lockfile you have to delete it, create a new one and commit it;

- without a lock you just press the “rerun tests” on your CI

why-oh-why2• 6 years ago

What’s the point of reproducibility in your team but not with your users (which is by definition the most common use case)?

Libraries should skip lockfiles because they are ignored by the end user and therefore any testing you do based on them helps no one.

I’ve had lockfiles being broken for no reason* so if you can avoid them, do.

* For example, multiple versions of the same package were installed after several install/updates, but nuking the lock would avoid this issue and solve the problems it caused.

viraptor• 6 years ago

any testing you do based on them helps no one.

Counter example: they help PR authors.

With a lockfile - CI fails which means the issue is most likely in the change.

Without a lockfile - CI fails which means the issue is with the change, or the new dependencies, or the distribution of the new dependencies, or...

Ideally testing new changes and updating deps are two completely separate processes.

jrochkind1• 6 years ago

However, just because we are getting rid of the package-lock.json doesn't mean we won't have the ability to pin the dependencies and child dependencies we have. There is another file we can use called npm-shrinkwrap.json.
It's basically the same file as package-lock.json and is produced by npm shrinkwrap and actually packaged and published to the npm registry.
So by adding npm shrinkwrap to your npm scripts as a prepack script or even a git commit hook, you can make sure that the same versions of dependencies are used in your dev environment, with your users and in your CI.

So you are still pinning for dev and CI, even though your actual users will not be pinned.

Why is this preferable to just using the yarn.lock or package-lock.json, that would appear to do the same thing? Whats the difference that makes using npm shrinkwrap preferable?

It can be confusing that you are pinning in dev/CI, but your downstream users will be using different versions. It seems to be the lesser evil though, neither trying to pin all users to exact versions of every dependency in the tree, nor refraining from pinning at all for dev/CI -- are really feasible.

I don't understand why the OP is suggesting `npm shrinkwrap` is a better solution than a checked-in yarn.lock or package-lock.json that are excluded from the npm module publish.

When Not to Use Lock Files with Node.js

Related Stories