My failed attempt to shrink all NPM packages by 5%

https://evanhahn.com/my-failed-attempt-to-shrink-all-npm-packages-by-5-percent/

by todsacerdoti • 1 year ago

365 243 1 year ago

huqedato• 1 year ago

Try to use this https://github.com/xthezealot/npmprune

My reading of OP is that it’s less about whether zopfli is technically the best way to achieve a 5% reduction in package size, and more about how that relatively simple proposal interacted with the NPM committee. Do you think something like this would fare better or differently for some reason?

sd94• 1 year ago

The final pro/cons list: https://github.com/npm/rfcs/pull/595#issuecomment-1200480148

I don't find the cons all that compelling to be honest, or at least I think they warrant further discussion to see if there are workarounds (e.g. a choice of compression scheme for a library like typescript, if they would prefer faster publishes).

It would have been interesting to see what eventually played out if the author hadn't closed the RFC themselves. It could have been the sort of thing that eventually happens after 2 years, but then quietly makes everybody's lives better.

n4r9• 1 year ago

I felt the same. The proposal wasn't rejected! Also, performance gains go beyond user stories - e.g. they reduce infra costs and environmental impact - so I think the main concerns of the maintainers could have been addressed.

IshKebab2• 1 year ago

The proposal wasn't rejected!

They soft-rejected by requiring more validation than was reasonable. I see this all the time. "But did you consider <extremely unlikely issue>? Please go and run more tests."

It's pretty clear that the people making the decision didn't actually care about the bandwidth savings, otherwise they would have put the work in themselves to do this, e.g. by requiring Zopfli for popular packages. I doubt Microsoft cares if it takes an extra 2 minutes to publish Typescript.

Kind of a wild decision considering NPM uses 4.5 PB of traffic per week. 5% of that is 225 TB/week, which according to my brief checks costs around $10k/week!

I guess this is a "not my money" problem fundamentally.

alt227• 1 year ago

I feel massively increasing publish time is a valid reason not to push this though considering such small gains and who the gains apply to.

scott_w• 1 year ago

I agree, going from 1 second to 2.5 minutes is a huge negative change, in my opinion. I know publishing a package isn't something you do 10x a day but it's probably a big enough change that, were I doing it, I'd think the publish process is hanging and keep retrying it.

rererereferred• 1 year ago

Since it's backwards compatible, individual maintainers could enable it in their own pipeline if they don't have issues with the slowdown. It sounds like it could be a single flag in the publish command.

michaelmior3• 1 year ago

Probably not worth the added complexity, but in theory, the package could be published immediately with the existing compression and then in the background, replaced with the Zopfli-compressed version.

macspoofing• 1 year ago

I don't find the cons all that compelling to be honest

I found it reasonable.

The 5% improvement was balanced against the cons of increased cli complexity, lack of native JS zopfli implementation, and slower compression .. and 5% just wasn't worth it at the moment - and I agree.

or at least I think they warrant further discussion

I think that was the final statement.

sd94• 1 year ago

Yes, but there’s a difference between “this warrants further discussion” and “this warrants further discussion and I’m closing the RFC”. The latter all but guarantees that no further discussion will take place.

jerf2• 1 year ago

"I don't find the cons all that compelling to be honest"

This is a solid example of how things change at scale. Concerns I wouldn't even think about for my personal website become things I need to think about for the download site being hit by 50,000 of my customers become big deals when operating at the scale of npm.

You'll find those arguments the pointless nitpicking of entrenched interests who just don't want to make any changes, until you experience your very own "oh man, I really thought this change was perfectly safe and now my entire customer base is trashed" moment, and then suddenly things like "hey, we need to consider how this affects old signatures and the speed of decompression and just generally whether this is worth the non-zero risks for what are in the end not really that substantial benefits".

I do not say this as the wise Zen guru sitting cross-legged and meditating from a position of being above it all; I say it looking at my own battle scars from the Perfectly Safe things I've pushed out to my customer base, only to discover some tiny little nit caused me trouble. Fortunately I haven't caused any true catastrophes, but that's as much luck as skill.

Attaining the proper balance between moving forward even though it incurs risk and just not changing things that are working is the hardest part of being a software maintainer, because both extremes are definitely bad. Everyone tends to start out in the former situation, but then when they are inevitably bitten it is important not to overcorrect into terrified fear of ever changing anything.

sd94• 1 year ago

I agree with everything you said, but it doesn’t contradict my point

pif• 1 year ago

This is a solid example of how things change at scale.

5% is 5% at any scale.

advisedwang• 1 year ago

The pros aren't all that compelling either. The npm repo is the only group that this would really be remotely significant for, and there seemed to be no interest. So it doesn't take much of a con to nix a solution to a non-problem.

ForOldHack• 1 year ago

Every single download, until the end of time is affected: It speeds up the servers, speeds up the updates, saves disk space on the update servers, and saves on bandwidth costs and usage.

Everyone benefits, the only cost is a ultra microscopic time on the front end, and a tiny cost on the client end, and for a very significant number of users, time and money saved. The examples of compression here...

mootothemax• 1 year ago

I don't find the cons all that compelling to be honest, or at least I think they warrant further discussion

It needs a novel JS port of a C compresison library, which will be wired into a heavily-used and public-facing toolchain, and is something that will ruin a significant number of peoples' days if it breaks.

For me, that kind of ask needs a compelling use case from the start.

orta• 1 year ago

Congrats on a great write-up. Sometimes trying to ship something at that sorta scale turns out to just not really make sense in a way that is hard to see at the beginning.

Another personal win is that you got a very thorough understanding of the people involved and how the outreach parts of the RFC process works. I've also had a few fail, but I've also had a few pass! Always easier to do the next time

stabbles2• 1 year ago

One thing that's excellent about zopfli (apart from being gzip compatible) is how easy it is to bootstrap:

    git clone https://github.com/google/zopfli.git
    cc -O2 zopfli/src/zopfli/*.c -lm

It just requires a C compiler and linker.

stabbles2• 1 year ago

The main downside though, it's impressively slow.

Comparing to gzip isn't really worth it. Combine pigz (threaded) with zlib-ng (simd) and you get decent performance. pigz is used in `docker push`.

For example, gzipping llvm.tar (624MB) takes less than a second for me:

    $ time /home/harmen/spack/opt/spack/linux-ubuntu24.04-zen2/gcc-13.2.0/pigz-2.8-5ptdjrmudifhjvhb757ym2bzvgtcsoqc/bin/pigz -k hello.tar 
    
    real    0m0.779s
    user    0m11.126s
    sys     0m0.460s

At the same time, zopfli compiled with -O3 -march=native takes 35 minutes. No wonder it's not popular.

It is almost 2700x slower than the state of the art for just 6.8% bytes saved.

Levitating2• 1 year ago

2700x slower

That is impressively slow.

In my opinion even the 28x decrease in performance mentioned would be a no-go. Sure the package saves a few bytes but I don't need my entire pc to grind to a halt every time I publish a package.

Besides, storage is cheap but CPU power draw is not. Imagine the additional CO2 that would have to be produced if this RFC was merged.

2 gigabytes of bandwidth per year across all installations

This must be a really rough estimate and I am curious how it was calculated. In any case 2 gigabytes over a year is absolutely nothing. Just my home network can produce a terabyte a day.

Demlmlm• 1 year ago

He should not have been closing this tbh.

5% sounds like a good deal

ape4• 1 year ago

Usually people require more than 5% to make a big change

hinkley7• 1 year ago

That’s why our code is so slow. Dozens of poor decisions that each account for 2-4% of overall time lost, but 30-60% in aggregate.

fergie3• 1 year ago

Props to anyone who tries to make the world a better place.

Its not always obvious who has the most important use cases. In the case of NPM they are prioritizing the user experience of module authors. I totally see how this change would be great for module consumers, yet create potentially massive inconvenience for module authors.

Interesting write-up

atiedebee2• 1 year ago

I think "massive" is overstating it. I don't think deploying a new version of a package is something that happens many times a day, so it wouldn't be a constant pain point.

Also, since this is a case of having something compressed once and decompressed potentially thousands of times, it seems like the perfect tool for the job.

philipwhiuk4• 1 year ago

Every build in a CI system would probably create the package.

This is changing every build in every CI system to make it slower.

fergie3• 1 year ago

Module authors generally have fairly large test suites which are run often- sometimes on each file save. If you have a 1 or 2 second build script its not a huge deal. If that script starts taking 30-60 seconds- you have just hosed productivity. Also you have massively increased the load on your CI server- possibly bumping you out of a free tier.

The fix would then have to be some variation of:

a) Stop testing (so often)

b) Stop bundling before testing

c) Publish to a different package manager

- all of which would affect the overall quality and quantity npm modules.

avodonosov8• 1 year ago

My experiment on how to reduce javascript size of every web app by 30-50% : https://github.com/avodonosov/pocl

Working approach, but in the end I abandoned the project - I doubt people care about such js size savings.

dagelf• 1 year ago

Wdym?? 50% is a big deal

bluGill2• 1 year ago

50% size savings isn't important to the people who pay for it. They pay at most pennies for 100% savings (that is somehow all the functionality in zero bytes - not worth anything to those paying the bills)

avodonosov8• 1 year ago

Why big deal?

50% is just a big O of the original size :)

If we consider overall Internet traffic, which is dominated by video and images, the size of javascript transmitted is negligible.

Savings of memory and cpu for end user's browser? Maybe, but our software at all layers is so bloated, without need, just carelessly, that I'm not sure working of such javascript savings is useful - any resources saved will be eaten by something else immediately.

For an application developer or operator, the 50% savings of javascript size are probably not worth dealing with some tool that dynamically prunes the app code, raising questions about privacy and security. I though through the security and privacy questions, but who would want to even spend attention on these considerations?

As I mention in the README at github, a Microsoft researcher investigated the same approach earlier, but Microsoft haven't taken it anywhere.

There was a commercial company offering this approach of javascript minification as a product, complete product. As well as other optimizations, like image size, etc. Their proxy embedded special "agent" code into the app which inspected the device and reported to server what image sizes are optimal for user, what js functions are invoked. And the server prepared optimized versions of app for various devices. Javascript was "streamed" in batches of only functions needed by the app.

Now the company is dissolved - that's why the website is unavailable. Wikipedia says they were bought out by Akamai - https://en.wikipedia.org/wiki/Instart. But I don't see any traces of this approach in the today's Akamai offerings.

I contacted several companies, in CDN business and others, trying to interest them in the idea and get very modest funding for a couple more months of my time to work on this (I was working on this in the end of a long break from payed work and was running out of savings). Didn't find anyone ready to take part.

This all may be signs that possibility of such an optimization is not valuable enough for users.

philipwhiuk4• 1 year ago

How do you evaluate call usage?

avodonosov8• 1 year ago

By instrumenting the code so that the function records the fact that it is being invoked.

Then the info about the called functions is sent back to the server.

(Only the functions never seen to be called are instrumented, the known active functions are not instrumented).

KTibow3• 1 year ago

I think this is called tree shaking and Vite/Rollup do this by default these days. Of course, it's easy when you explicitly say what you're importing.

avodonosov8• 1 year ago

That's not tree-shaking.

hinkley7• 1 year ago

I got measurable decreases in deployment time by shrinking the node_modules directory in our docker images.

I think people forget that, when you’re copying the same images to dozens and dozens of boxes, any improvement starts to add up to real numbers.

syncsynchalt3• 1 year ago

I've not done it, but have you considered using `pnpm` and volume-mounting a shared persistent `pnpm-store` into the containers? It seems like you'd get near-instant npm installs that way.

avodonosov8• 1 year ago

In this approach the size of deployment bundles / images is not necessarily reduced.

Reduced is the size of javascript loaded into the end user's browser.

jsheard• 1 year ago

I wonder if it would make more sense to pursue Brotli at this point, Node has had it built-in since 10.x so it should be pretty ubiquitous by now. It would require an update to NPM itself though.

silverwind2• 1 year ago

+1 to brotli. Newly published packages could use brotli by default, so old ones stay compatible.

canucker20164• 1 year ago

Here's the Brotli supporter's blog post about adding Brotli support to NPM packages.

https://jamiemagee.co.uk/blog/honey-i-shrunk-the-npm-package...

and the related HN discussion from that time:

https://news.ycombinator.com/item?id=37754489

pornel• 1 year ago

It only doesn't apply to existing versions of existing packages. Newer releases would apply Zopfli, so over time likely the majority of actively used/maintained packages would be recompressed.

choobacker• 1 year ago

Nice write up!

When it was finally my turn, I stammered.
Watching it back, I cringe a bit. I was wordy, unclear, and unconvincing.
You can watch my mumbling in the recording

I watched this, and the author was articulate and presented well. The author is too harsh!

Good job for trying to push the boundaries.

hartator• 1 year ago

I mean 4-5% the size for 10-100x the time is not worth it.

swiftcoder2• 1 year ago

That's not actually so straightforward. You pay the 10-100x slowdown once on the compressing side, to save 4-5% on every download - which for a popular package one would expect downloads to be in the millions.

philipwhiuk4• 1 year ago

The downloads are cached. The build happens on every publish for every CI build.

inglor_cz• 1 year ago

As the author himself said, just React was downloaded half a billion times; that is a lot of saved bandwidth on both sides, but especially so for the server.

Maybe it would make sense to only apply this improvement in images that are a) either very big or b) get downloaded at least million times each year or so. That would cover most of the savings while leaving most packages and developers out of it.

bonzini2• 1 year ago

Assuming download and decompression cost to be proportional to the size of the incoming compressed stream, it would break even at 2000 downloads. A big assumption I know, but 2000 is a really small number.

liontwist8• 1 year ago

It absolutely is. Packages are zipped once and downloaded thousands of times.

cedws2• 1 year ago

Last I checked npm packages were full of garbage including non-source code. There's no reason for node_modules to be as big as it usually is, text compresses extremely well. It's just general sloppiness endemic to the JavaScript ecosystem.

MortyWaves• 1 year ago

Totally agree with you. I wish npm did a better job of filtering the crap files out of packages.

Alifatisk2• 1 year ago

At least, switch to pnpm minimize the bloat

jefozabuss2• 1 year ago

I just installed a project with pnpm about 120 packages mostly react/webpack/eslint/redux related

with prod env: 700MB

without prod env: 900MB

sadly the bloat cannot be avoided that well :/

MBCook5• 1 year ago

As someone who mostly works in Java it continues to floor me that this isn’t the default. Why does every project I work on need an identical copy of possibly hundreds of packages if they’re the same version?

I also like Yarn pnp’s model of leaving node_modules as zip files. CPUs are way faster than storage, they can decompress on the fly. Less disk space at rest, less disk slack, less filesystem bookkeeping.

Every single filesystem is way faster at dealing with one file than dozens/hundreds. Now multiply that by the the hundreds if does, it add up.

vinnymac• 1 year ago

You might be interested in e18e if you would like to see that change: https://e18e.dev/

They’ve done a lot of great work already.

KTibow3• 1 year ago

Does this replace ljharb stuff?

hinkley7• 1 year ago

I believe I knocked 10% off of our node_modules directory by filing .npmignore PRs or bug reports to tools we used.

Now if rxjs weren’t a dumpster fire…

TheRealPomax2• 1 year ago

That's on the package publishers, not NPM. They give you an `.npmignore` that's trivially filled out to ensure your package isn't full of garbage, so if someone doesn't bother using that: that's on them, not NPM.

(And it's also a little on the folks who install dependencies: if the cruft in a specific library bothers you, hit up the repo and file an issue (or even MR/PR) to get that .npmignore file filled out. I've helped folks reduce their packages by 50+MB in some cases, it's worth your own time as much as it is theirs)

silverwind2• 1 year ago

It's much better to allowlist the files meant to be published using `files` in package.json because you never know what garbage the user has in their folder at the time of publish.

On a typical project with a build step, only a `dist` folder would published.

eitau_12• 1 year ago

It's not even funny:

  $ ll /nix/store/*-insect-5.9.0/lib/node_modules/insect/node_modules/clipboardy/fallbacks/*
  /nix/store/…-insect-5.9.0/lib/node_modules/insect/node_modules/clipboardy/fallbacks/linux:
  .r-xr-xr-x 129k root  1 Jan  1970 xsel

  /nix/store/…-insect-5.9.0/lib/node_modules/insect/node_modules/clipboardy/fallbacks/windows:
  .r-xr-xr-x 444k root  1 Jan  1970 clipboard_i686.exe
  .r-xr-xr-x 331k root  1 Jan  1970 clipboard_x86_64.exe

(clipboardy ships executables and none of them can be run on NixOS btw)

cedws2• 1 year ago

Are they reproducible? Shipping binaries in JS packages is dodgy AF - a Jia Tan attack waiting to happen.

dicytea• 1 year ago

I don't know why, but clipboard libraries tend to be really poorly implemented, especially in scripting languages.

I just checked out clipboardy and all they do is dispatch binaries from the path and hope it's the right one (or if it's even there at all). I think I had a similar experience with Python and Lua scripts. There's an unfunny amount of poorly-written one-off clipboard scripts out there just waiting to be exploited.

I'm only glad that the go-to clipboard library in Rust (arboard) seems solid.

hombre_fatal• 1 year ago

One of the things I like about node_modules is that it's not purely source code and it's not purely build artifacts.

You can read the code and you can usually read the actual README/docs/tests of the package instead of having to find it online. And you can usually edit library code for debugging purposes.

If node_modules is taking up a lot of space across a bunch of old projects, just write the `find` script that recursively deletes them all; You can always run `npm install` in the future when you need to work on that project again.

tanepiper• 1 year ago

Yep, I wrote a script that starts at a root `node_modules` folder and iterates through to remove anything not required (dotfiles, Dockerfile, .md files, etc) - in one of our smaller apps this removes about 25Mb of fluff, some packages are up to 60-70mb of crap removed.

glenjamin• 1 year ago

This strikes me as something that could be done for the highest traffic packages at the backend, rather than be driven by the client at pubish-time.

fastest963• 1 year ago

The article talks about this. There are hashes that are generated for the tarball so the backend can't recompress anything.

BonoboIO• 1 year ago

In summary: It’s a nice feature, which gives nice benefits for often downloaded packages, but nobody at npm cares for the bandwidth?

nikeee• 1 year ago

I don't see why it wouldn't be possible to hide behind a flag once Node.js supports zopfli natively. In case of CI/CD, it's totally feasible to just add a --strong-compression flag. In that case, the user expects it to take its time.

TS releases a non-preview version every few months, so using 2.5 minutes for compression would work.

jopsen• 1 year ago

Think of the complexity involved, think of having to fix bugs because something went wrong.

Such effort would be better spent preparing npm/node for a future where packages with a lower-bound npm version constraint can be compressed with zstd or similar.

Every feature you add to a program is complexity, you need to really decide if you want it.

abound• 1 year ago

A few people have mentioned the environmental angle, but I'd care more about if/how much this slows down decompression on the client. Compressing React 20x slower once is one thing, but 50 million decompressions being even 1% slower is likely net more energy intensive, even accounting for the saved energy transmitting 4-5% fewer bits on the wire.

web007• 1 year ago

It's very likely zero or positive impact on the decompression side of things.

Starting with smaller data means everything ends up smaller. It's the same decompression algorithm in all cases, so it's not some special / unoptimized branch of code. It's yielding the same data in the end, so writes equal out plus or minus disk queue fullness and power cycles. It's _maybe_ better for RAM and CPU because more data fits in cache, so less memory is used and the compute is idle less often.

It's relatively easy to test decompression efficiency if you think CPU time is a good proxy for energy usage: go find something like React and test the decomp time of gzip -9 vs zopfli. Or even better, find something similar but much bigger so you can see the delta and it's not lost in rounding errors.

sltkr• 1 year ago

For formats like deflate, decompression time doesn't generally depend on compressed size. (zstd is similar, though memory use can depend on the compression level used).

This means an optimization like this is virtually guaranteed to be a net positive on the receiving end, since you always save a bit of time/energy when downloading a smaller compressed file.

DannyBee5• 1 year ago

I can speak to this - there is no meaningful decompression effect across an insane set of tested data at Google and elsewhere. Zopfli was invented prior to brotli

Zopfli is easiest to think of as something that just tries harder than gzip to find matches and better encodings. Much harder.

decompression speed is linear either way.

It's easiest to think of decompression as a linear time vm executor[1], where the bytecoded instructions are basically

go back <distance> bytes, output the next <length> bytes you see, then output character <c>

(outputting literal data is the instruction <0,0,{character to output}>)

Assuming you did not output a file larger than the original uncompressed file (why would you bother?), you will, worst case, process N bytes during decompression, where N is the size of the original input file.

The practical decompression speed is driven by cache behavior, but it thrashes the cache no matter what.

In practice, reduction of size vs gzip occurs by either finding larger runs, or encodings that are smaller than the existing ones.

After all, if you want the compressed file to shrink, you need output less instructions somehow, or make more of the instructions identical (so they can be represented in less bits by later huffman coding).

In practice, this has almost exclusively positive effects on decompression speed - either the vm has less things to process (which is faster), or more of the things it does look the same (which has better cache behavior).

[1] this is one way archive formats will sometimes choose to deal with multiple compression method support - encode them all to the same kind of bytecode (usually some form of copy + literal instruction set), and then decoding is the same for all of them. ~all compression algorithms output some bytecode like the above on their own already, so it's not a lot of work. This doesn't help you support other archive formats, but if you want to have a bunch of per-file compression options that you pick from based on what works best, this enables you to still only have to have one decoder.

adgjlsfhk1• 1 year ago

This seems like a place where the more ambitious version that switches to ZSTD might have better tradeoffs. You would get similar or better compression, with faster decompression and recompression than zstd.It would lose backward compatibility though...

bufferoverflow• 1 year ago

Brotli and lzo1b have good compression ratios and pretty fast decompression speeds. Compression speed should not matter that much, since you only do it once.

https://quixdb.github.io/squash-benchmark/

There even more obscure options:

https://www.mattmahoney.net/dc/text.html

vlovich1234• 1 year ago

Not necessarily - could retain backward compat by publishing both gzip and zstd variants and having downloaders with newer npm’s prefer to download zstd. Over time, you could require packages only upload zstd going forward and either generate zstd versions of the backlog of unmaintained packages or at least those that see some amount of traffic over some time period if you’re willing to drop very old packages. The ability to install arbitrary versions of packages probably means you’re probably better off reprocessing the backlog although that may cost more than doing nothing.

The package lock checksum is probably a more solvable issue with some coordination.

The benefit of doing this though is less immediate - it will take a few years to show payoff and these kinds of payoffs are not typically made by the kind of committee decisions process described (for better or worse).

MBCook5• 1 year ago

Thats a much higher hurdle to jump. I don’t blame the author for trying this first.

If accepted, it might have been a good stepping stone too. A chance to get to know everyone and their concerns and how they think.

So if you wanted to see how this works (proposal + in prod) and then come back later proposing something bigger by switching off zip that would make sense to me as a possible follow up.

chuckadams• 1 year ago

Switching to a shared cache in the fashion of pnpm would eliminate far more redundant downloads than a compression algorithm needing 20x more CPU.

liontwist8• 1 year ago

The fact that you are pursuing this is admirable.

But this whole thing sounds too much like work. Finding opportunities, convincing entrenched stakeholders, accommodating irrelevant feedback, pitching in meetings — this is the kind of thing that top engineers get paid a lot of money to do.

For me personally open source is the time to be creative and free. So my tolerance for anything more than review is very low. And I would have quit at the first roadblock.

What’s a little sad, is NPM should not be operating like a company with 1000+ employees. The “persuade us users want this” approach is only going to stop volunteers. They should be proactively identifying efforts like this and helping you bring it across the finish line.

coliveira• 1 year ago

The problem is that this guy is treating open source as if it was a company where you need to direct a project to completion. Nobody in open source wants to be told what to do. Just release your work, if it is useful, the community will pick it up and everybody will benefit. You cannot force your improvement into the whole group, even if it is beneficial in your opinion.

liontwist8• 1 year ago

where you need to direct a project to completion

Do you want to get a change in, or not?

Is this a project working with the community or not?

Just release your work

What motivation exists to optimize package formats if nobody uses that package format? There are no benefits unless it’s in mainline.

Nobody in open source wants to be told what to do

He’s not telling anybody to do work. He is sharing an optimization with clear tradeoffs - not a new architecture.

You cannot force your improvement into the whole group

Nope, but communication is key. “put up a PR and we will let you know whether it’s something we want to pursue”.

Instead they put him through several levels of gatekeepers where each one gave him incorrect feedback.

“Why do we want to optimize bandwidth” is a question they should have the answer to.

If this PR showed up on my project I would say “I’m worried about X,Y,Z” we will set up a test for X and Y and get back to you. Could you look into Z?

gjsman-10004• 1 year ago

What’s a little sad, is NPM should not be operating like a company with 1000+ employees. The “persuade us users want this” approach is only going to stop volunteers. They should be proactively identifying efforts like this and helping you bring it across the finish line.

Says who?

Says an engineer? Says a product person?

NPM is a company with 14 employees; with a system integrated into countless extremely niche and weird integrations they cannot control. Many of those integrations might make a professional engineer's hair catch fire - "it should never be done this way!" - but the real world is that the wrong way is the majority of the time. There's no guarantee that many of the downloads come from the official client, just as one example.

The last thing they need, or I want, or any of their customers want, or their 14 employees need, is something that might break backwards compatibility in an extremely niche case, anger a major customer, cause countless support tickets, all for a tiny optimization nobody cares about.

This is something I've learned here about HN that, for own mental health, I now dismiss: Engineers are obsessed with 2% optimizations here, 5% optimizations there; unchecked, it will literally turn into an OCD outlet, all for things nobody in the non-tech world even notices, let alone asks about. Just let it go.

liontwist8• 1 year ago

Open source needs to operate differently than a company because people don’t have time/money/energy to deal with bullshit.

Hell. Even 15 employees larping as a corporation is going to be inefficient.

what you and NPM are telling us, is that they are happy to take free labor, but this is not an open source project.

Engineers are obsessed with 2% optimizations here

Actually in large products these are incredible finds. But ok. They should have the leadership to know which bandwidth tradeoffs they are committed to and tell him immediately it’s not what they want, rather than sending him to various gatekeepers.

stonemetal12• 1 year ago

NPM is a webservice. They could package the top 10-15 enhancements call it V2. When 98% of traffic is V2 turn V1 off. Repeat every 10 years or so until they work their way into having a good protocol.

maccard4• 1 year ago

Engineers are obsessed with 2% optimizations here, 5% optimizations there; unchecked, it will literally turn into an OCD outlet, all for things nobody in the non-tech world even notices, let alone asks about. Just let it go.

I absolutely disagree with you. If the world took more of those 5% optimisations here and there everything would be faster. I think more people should look at those 5% optimisations. In many cases they unlock knowledge that results in a 20% speed up later down the line. An example from my past - I was tasked with reducing the running speed of a one shot tool we were using at $JOB. It was taking about 15 minutes to run. I shaved off seconds here and there with some fine grained optimisations, and tens of seconds with some modernisation of some core libraries. Nothing earth shattering but improvements none the less. One day, I noticed a pattern was repeating and I was fixing an issue for the third time in a different place (searching a gigantic array of stuff for a specific entry). I took a step back and realised that if I replaced the mega list with a hash table it might fix every instance of this issue in our app. It was a massive change, touching pretty much every file. And all of a sudden our 15 minute runtime was under 30 seconds.

People used this tool every day, it was developed by a team of engineers wildly smarter than me. But it had grown and nobody really understood the impact of the growth. When it started that array was 30, 50 entries. On our project it was 300,000 and growing every day.

Not paying attention to these things causes decay and rot. Not every change should be taken, but more people should care.

lyu072822• 1 year ago

Says who?
Says an engineer?

I prevent cross-site scripting, I monitor for DDoS attacks, emergency database rollbacks, and faulty transaction handlings. The Internet heard of it? Transfers half a petabyte of data every minute. Do you have any idea how that happens? All those YouPorn ones and zeroes streaming directly to your shitty, little smart phone day after day? Every dipshit who shits his pants if he can't get the new dubstep Skrillex remix in under 12 seconds? It's not magic, it's talent and sweat. People like me, ensuring your packets get delivered, un-sniffed. So what do I do? I make sure that one bad config on one key component doesn't bankrupt the entire fucking company. That's what the fuck I do.

efitz• 1 year ago

I think that the reason NPM responded this way is because it was a premature optimization.

If/when NPM has a problem - storage costs are too high, or transfer costs are too high, or user feedback indicates that users are unhappy with transfer sizes - then they will be ready to listen to this kind of proposal.

I think their response was completely rational, especially given a potentially huge impact on compute costs and/or publication latency.

maccard4• 1 year ago

I disagree with it being a premature optimisation. Treating everything that you haven’t already identified personally as a problem as a premature optimisation is cargo cutting in its own way. The attitude of not caring is why npm and various tools are so slow.

That said, I think NPM’s response was totally correct - explain the problem and the tradeoffs. And OP decided the tradeoffs weren’t worth it, which is totally fair.

Cthulhu_• 1 year ago

While NPM is open source, it's in the awkward spot of also having... probably hundreds of thousands if not millions of professional applications depend on it; it should be run like a business, because millions depend on it.

...which makes it all the weirder that security isn't any better, as in, publishing a package can be done without a review step on the npm side, for example. I find it strange that they haven't doubled down on enterprise offerings, e.g. creating hosted versions (corporate proxies), reviewed / validated / LTS / certified versions of packages, etc.

liontwist8• 1 year ago

Why would a more complex zip slow down decompress? This comment seems to misunderstand how these formats work. OP is right.

dvh• 1 year ago

Imagine being in the middle of nowhere, in winter, on Saturday night, on some farm, knee deep in a cow piss, servicing some 3rd party feed dispenser, only to discover that you have possible solution but it's in some obscure format instead of .tar.gz. Nearest internet 60 miles away. This is what I always imagine happening when some new obscure format come into play, imagine the poor fella, alone, cold, screaming. So incredibly close to his goal, but ultimately stopped by some artificial unnecessary made-up bullshit.

tehjoker2• 1 year ago

I believe zopfli compression is backwards compatible with DEFLATE, it just uses more CPU during the compression phase.

DannyBee5• 1 year ago

Correct - it is just looking for matches harder, and encoding harder.

PaulHoule5• 1 year ago

Years back I came to the conclusion that conda using bzip2 for compression was a big mistake.

Back then if you wanted to use a particular neural network it was meant for a certain version of Tensorflow which expected you to have a certain version of the CUDA libs.

If you had to work with multiple models the "normal" way to do things was use the developer unfriendly [1][2] installers from NVIDIA to install a single version of the libs at a time.

Turned out you could have many versions of CUDA installed as long as you kept them in different directories and set the library path accordingly, it made sense to pack them up for conda and install them together with everything else.

But oh boy was it slow to unpack those bzip2 packages! Since conda had good caching, if you build environments often at all you could be paying more in decompress time than you pay in compression time.

If you were building a new system today you'd probably use zstd since it beats gzip on both speed and compression.

[1] click... click... click...

[2] like they're really going to do something useful with my email address

zahlman5• 1 year ago

But oh boy was it slow to unpack those bzip2 packages! Since conda had good caching, if you build environments often at all you could be paying more in decompress time than you pay in compression time.

For Paper, I'm planning to cache both the wheel archives (so that they're available without recompressing on demand) and unpacked versions (installing into new environments will generally use hard links to the unpacked cache, where possible).

If you were building a new system today you'd probably use zstd since it beats gzip on both speed and compression.

FWIW, in my testing LZMA is a big win (and I'm sure zstd would be as well, but LZMA has standard library support already). But there are serious roadblocks to adopting a change like that in the Python ecosystem. This sort of idea puts them several layers deep in meta-discussion - see for example https://discuss.python.org/t/pep-777-how-to-re-invent-the-wh... . In general, progress on Python packaging gets stuck in a double-bind: try to change too little and you won't get any buy-in that it's worthwhile, but try to change too much and everyone will freak out about backwards compatibility.

PaulHoule5• 1 year ago

I designed a system which was a lot like uv but written in Python and when I looked at the politics I decided not to go forward with it. (My system also had the problem that it had to be isolated from other Pythons so it would not get its environment trashed, with the ability for software developers to trash their environment I wasn't sure it was a problem that could be 100% solved. uv solved it by not being written in Python. Genius!)

kittikitti• 1 year ago

Thank you so much for posting this. The original logic was clear and it had me excited! I believe this is useful because compression is very common and although it might not fit perfectly in this scenario, it could very well be a breakthrough in another. If I come across a framework that could also benefit from this compression algorithm, I'll be sure to give you credit.

bhouston9• 1 year ago

What about a different approach - an optional npm proxy that recompresses popular packages with 7z/etc in the background?

Could verify package integrity by hashing contents rather than archives, plus digital signatures for recompressed versions. Only kicks in for frequently downloaded packages once compression is ready.

Benefits: No npm changes needed, opt-in only, potential for big bandwidth savings on popular packages. Main tradeoff is additional verification steps, but they could be optional given a digital signature approach.

Curious if others see major security holes in this approach?

ndriscoll• 1 year ago

This felt like the obvious way to do things to me: hash a .tar file, not a .tar.gz file. Use Accept-Encoding to negotiate the compression scheme for transfers. CDN can compress on the fly or optionally cache precompressed files. i.e. just use standard off-the-shelf HTTP features. These days I prefer uncompressed .tar files anyway because ZFS has transparent zstd, so decompressed archive files are generally smaller than a .gz.

cesarb• 1 year ago

hash a .tar file, not a .tar.gz file

For security reasons, it's usually better to hash the compressed file, since it reduces the attack surface: the decompressor is not exposed to unverified data. There have already been vulnerabilities in decompressor implementations which can be exploited through malformed compressed data (and this includes IIRC at least one vulnerability in zlib, which is the standard decompressor for .gz).

bhouston9• 1 year ago

You are correct. They should be uploading and downloading dumb tar files and let the HTTP connection negotiate the compression method. All hashes should be based on the uncompressed raw tar dump. This would be proper separation of concerns.

cozzyd2• 1 year ago

Enjoy zipbomb.js

hinkley7• 1 year ago

Does npm even default to gzip -9? Wikipedia claims zopfli is 80 times slower under default settings.

zahlman5• 1 year ago

My experience has been that past -6 or so, gzip files get only a tiny bit smaller in typical cases. (I think I've even seen them get bigger with -9.)

EvanHahn• 1 year ago

Yes: https://github.com/npm/pacote/blob/bf1f60f58bb61f053262f5472...

jefozabuss2• 1 year ago

I wonder what is the tarball size difference on average if you'd for example download everything in one tarball (full package list) instead of 1-by-1 as the gzip compression would work way better in that case.

Also for bigger companies this is not really a "big" problem as they usually have in-house proxies (as you cannot rely on a 3rd party repository in CI/CD for multiple reasons (security, audit, speed, etc)).

Dylan16807• 1 year ago

You might save a little bit by putting similar very small files next to each other in the same tarball, but in general I would not expect significant improvements. Gzip can only compress repetitions that are within 32KB of each other.

snizovtsev• 1 year ago

Yes, but it was expected. It's like prioritising code readability over performance everywhere but the hot path.

Earlier in my career, I managed to use Zopfli once to compress gigabytes of PNG assets into a fast in-memory database supporting a 50K+ RPS web page. We wanted to keep it simple and avoid the complexity of horizontal scaling, and it was OK to drop some rarely used images. So the more images we could pack into a single server, the more coverage we had. In that sense Zopfli was beneficial.

1337shadow• 1 year ago

Ok but why doesn't npm registry actually recompress the archives? It can even apply that retroactively, wouldn't require zopli in npm CLI

aseipp• 1 year ago

Hashes of the tarballs are recorded in the package-lock.json of downstream dependants, so recompressing the files in place will cause the hashes to change and break everyone. It has to be done at upload time.

notpushkin• 1 year ago

But it still can be done on the npm side, right?

bhouston9• 1 year ago

The hashes of the uncompressed tarballs would be great. Then the HTTP connection can negotiate a compression format for transfer (which can change over time at HTTP itself changes) rather than baking it into the NPM package standard (which is incredibly inflexible.)

tonymet2• 1 year ago

I’m more concerned about the 40gb of node_modules. Why hasn’t node supported tgz node_modules. That would save 75% of the space or more.

bhouston9• 1 year ago

isn't this a file system thing? Why bake it into npm?

tonymet2• 1 year ago

efficiency

nopurpose• 1 year ago

It reminds me of an effort to improve docker image format and make it move away from being just a tar file. I can't find links anymore, but it was a pretty clever design, which still couldn't beat dumb tar in efficiency.

bhouston9• 1 year ago

Transferring around dumb tar is actually smart because the HTTPS connection can negotiate a compressed version of it to transfer - e.g. gzip, brotli, etc. No need to bake in an unchangable compression format into the standard.

woadwarrior012• 1 year ago

From the RFC on github[1].

Zopfli is written in C, which presents challenges. Unless it was added to Node core, the CLI would need to (1) rewrite Zopfli in JS, possibly impacting performance (2) rely on a native module, impacting reliability (3) rely on a WebAssembly module. All of these options add complexity.

Wow! Who's going to tell them that V8 is written in C++? :)

[1] https://github.com/npm/rfcs/pull/595

kmacdough• 1 year ago

It's not about C per-se, as much as each native compiled dependency creates additional maintenance concerns. Changes to hardware/OS can require a recompile or even fixes. NPM build system already requires a JavaScript runtime, so is already handled as part of existing maintenance. The point is that Zopfli either needs to be rewritten for a platform-agnostic abstraction they already support, or else Zopfli will be added to a list of native modules to maintain.

woadwarrior012• 1 year ago

It's not about C per-se, as much as each native compiled dependency creates additional maintenance concerns. Changes to hardware/OS can require a recompile or even fixes.

This is a canard. zopfli is written in portable C and is far more portable than the nodejs runtime. On any hardware/OS combo that one can build the nodejs runtime, they certainly can also build and run zopfli.

gweinberg• 1 year ago

I was under the impression that bzip compresses more than gzip, but gzip is much faster, so gzip is better for things that need to be compressed on the fly, and bzip is better fro things that get archived. Is this not true? Wouldn't it have been better to use bzip all along for this purpose?

bangaladore• 1 year ago

I think the main TLDR here [1]:

For example, I tried recompressing the latest version of the typescript package. GNU tar was able to completely compress the archive in about 1.2 seconds on my machine. Zopfli, with just 1 iteration, took 2.5 minutes.

[1] https://github.com/npm/rfcs/pull/595#issuecomment-1200480148

My question of course would be, what about LZ4, or Zstd or Brotli? Or is backwards compatibility strictly necessary? I understand that GZIP is still a good compressor, so those others may not produce meaningful gains. But, as the author suggests, even small gains can produce huge results in bandwidth reduction.

loeg• 1 year ago

It probably makes more sense to save more bytes and compressor time and just switch to zstd (a bigger scoped effort, sure).

imoverclocked2• 1 year ago

But the cons were substantial: ... This wouldn’t retroactively apply to existing packages.

Why is this substantial? My understanding is that packages shouldn't be touched once published. It seems likely for any change to not apply retroactively.

szundi• 1 year ago

This is a nice guy

omoikane• 1 year ago

Integrating Zopfli into the npm CLI would be difficult.

Is it possible to modify "gzip -9" or zlib to invoke zopfli? This way everyone who wants to compress better will get the extra compression automatically, in addition to npm.

There will be an increase in compression time, but since "gzip -9" is not the default, people preferring compression speed might not be affected.

bombcar2• 1 year ago

You'd have more problems here, but you could do it - if you let it take ages and ages to percolate though all environments.

It's been almost 30 years since bzip2 was released and even now not everything can handle tar.bz2

arccy3• 1 year ago

probably because bzip2 isn't a very good format

tehjoker2• 1 year ago

5% improvement is basically the minimum I usually consider worthwhile to pursue, but it's still small. Once you get to 10% or 20%, things become much more attractive. I can see how people can go either way on a 5% increase if there are any negative consequences (such as increased build time).

Alifatisk2• 1 year ago

I wonder if there is a way to install the npm packages without the crap they come included with (like docs, tests, readme etc).

rafaelmn2• 1 year ago

I wonder if you could get better results if you built a dictionary over entire npm. I suspect most common words could easily be reduced to 16k word index. Would be much faster, dictionary would probably fit in cache, can even optimize it in memory for cache prefetch.

zahlman5• 1 year ago

This seems like a non-starter to me - new packages are added to npm all the time, and will alter the word frequency distribution. If you aren't prepared to re-build constantly and accept that the dictionary isn't optimal, then it's hard to imagine it being significantly better than what you build with a more naive approach. Basically - why try to fine-tune to a moving target?

arccy3• 1 year ago

would it really change that quickly? you might get significant savings from just having keywords, common variable names, standard library functions

rafaelmn2• 1 year ago

But is it really moving that fast ? I suspect most fundamental terms in programming and the variations do not change often. You will always have keywords, built ins and the most popular concepts from libs/frameworks.

So it is basically downloading a few hundred kb dictionary every year ?

hinkley7• 1 year ago

Pulling on this thread, there are a few people who have looked at the ways zopfli is inefficient. Including this guy who forked it, and tried to contribute a couple improvements back to master:

https://github.com/fhanau/Efficient-Compression-Tool

These days if you’re going to iterate on a solution you’d better make it multithreaded. We have laptops where sequential code uses 8% of the available cpu.

vardump• 1 year ago

These days if you’re going to iterate on a solution you’d better make it multithreaded.

Repetition eliminating compression tends to be inherently sequential. You'd probably need to change the file format to support chunks (or multiple streams) to do so.

Because of LZ back references, you can't LZ compress different chunks separately on different cores and have only one compression stream.

Statistics acquisition (histograms) and entropy coding could be parallel I guess.

(Not a compression guru, so take above with a pinch of salt.)

hinkley7• 1 year ago

There are gzip variants that break the file into blocks and run in parallel. They lose a couple of % by truncating the available history.

But zopfli appears to do a lot of backtracking to find the best permutations for matching runs that have several different solutions. There’s a couple of ways you could run those in parallel. Some with a lot of coordination overhead, others with a lot of redundant calculation.

frabjoused• 1 year ago

This reminds me of a time I lost an argument with John-David Dalton about cleaning up/minifying lodash as an npm dependency, because when including the readme and license for every sub-library, a lodash import came to ~2.5MB at the time. This also took a lot of seeking time for disks because there were so many individual files.

The conversation started and ended at the word cache.

MBCook5• 1 year ago

This also took a lot of seeking time for disks because there were so many individual files.

The fact NPM keeps things in node_modules unzipped seems wild to me. Filesystems are not great at hundreds of thousands of little files. Some are bad, others are terrible.

Zip files are easier to store, take up less space, and CPUs are faster than disks so the decompression in memory is probably faster reading the unzipped files.

That was one of my favorite features of Yarn when I tried it - pnp mode. But since it’s not what NPM does it requires a shim that doesn’t work with all ps mage’s. Or at least didn’t a few years ago.

zahlman5• 1 year ago

I'd love to see an effort like like this succeed in the Python ecosystem. Right now, PyPI is dependent upon Fastly to serve files, on the order of >1 petabyte per day. That's a truly massive in-kind donation, compared to the PSF's operating budget (only a few million dollars per year - far smaller than Linux or Mozilla).

cozzyd2• 1 year ago

No problem, I'm sure if Fastly stopped doing it JiaTanCo would step up

JoeAltmaier• 1 year ago

These days technology moves so fast it's hard to keep up. The slowest link in the system is the human being.

That's a strong argument that 'if it isn't broke, don't fix it."

LOts of numbers being thrown around, you add up tiny things enough times you can get a big number. But is npm package download the thing that's tanking the internet? No? Then this is a second- or thirt-order optimization.

luzifer42• 1 year ago

I created once a maven plugin to recompress Java artefacts with zopfli. I rewrote it in Java and runs entirely in the JVM. This means, the speed is worse and may contain bugs:

https://luccappellaro.github.io/2015/03/01/ZopfliMaven.html

Julien_r22• 1 year ago

A pro could have been an extra narrative about carbon footprint savings.

I'm surprised it hasn't been raised when talking about saving 2Tb/year only for React. It represents costs, which doesn't seem to be an issue, but also computing power and storage. (Event with a higher/longer computing power due to slower compression, it's done once per version, which isn't really comprable to the amount of downloads anyway)

Hard to calculate the exact saving, but it would represent a smaller CO2 footprint..

wallunit• 1 year ago

I used zopflipng in the past to optimize PNG images. It made sense since there was no better alternative to store lossless image data than the PNG format at the given time in the given environment. Zopfli is awesome when you are locked in on deflate compression. I feel like if the npm folks would want to optimize for smaller package size a better strategy would be switching to some more effective text compression (e.g. bzip2, xz). That would result into a larger file size reduction than 5% for a smaller CPU time increase compared to Zopfli. You would need to come up with some migration strategy though as this change isn't per-se backwards compatible, but that seems manageable if you are in control of the tooling and infrastructure.

vivzkestrel• 1 year ago

I ll give you an even better idea but it ll need atleast a 100 volunteers, maybe a 1000. Take each package and rewrite it without external dependencies. That will cut tech debt for that package significantly. Just like how we have a @types/xyz where some dude named DefinitelyTyped is busy making typescript packages for everything, lets make a namespace like @efficient/cors @efficient/jsdom @efficient/jest etc and eliminate all external dependencies completely for every library on npm

DannyBee5• 1 year ago

zopfli is the wrong thing to use here.

If you want an example of where these things go badly:

the standard compression level for rpms on redhat distros is zstd level 19.

This has all the downsides of other algorithms - it's super slow - often 75-100x slower than the default level of zstd[1]. It achieves a few percent more compression for that speed. compared to even level 10, it's like 0-1% higher compression, but 20x slower.

This is bad enough - the kicker is that at this level, it's slower than xz for ~all cases, and xz is 10% smaller. The only reason to use zstd is because you want fairly good compression, but fast.

So here, they've chosen to use it in a way that compresses really slowly, but gives you none of the benefit of compressing really slowly.

Now, unlike the npm case, there was no good reason to choose level 19 - there were no backwards compatibility constraints driving it, etc. I went through the PR history on this change, it was not particularly illuminating (it does not seem lots of thought was given to the level choice).

I mention all this because it has a real effect on the experience of building rpms - this is why it takes eons to make kernel debuginfo rpms on fedora. Or any large RPM. Almost all time is spent compressing it with zstd at level 19. On my computer this takes many minutes. If you switch it to use even xz, it will do it about 15-20x faster (single threaded. if you thread both of them, xz will win by even more, because of how slow the setting is for zstd. If you use reasonable settings for zstd, obviously, it achieves gigabytes/second in parallel mode)

Using zopfli would be like choosing level 19 zstd for npm. While backwards compatibility is certainly painful to deal with here, zopfli is not likely better than doing nothing. You will make certain cases just insanely slow. You will save someone's bandwidth, but in exchange you will burn insane amounts of developer CPU.

zopfli is worse than level 19, it can often be 100x-200x slower than gzip -9.

Doesn't npm support insanely relaxed/etc scripting hooks anyway?

If so, if backwards compatibility is your main constraint, you would be "better off" double compressing (IE embed xz or whatever + a bootstrap decompressor and using the hooks to decompress it on old versions of npm). Or just shipping .tar.gz's than, when run, fetch the .xz and decompress it on older npm.

Or you know, fish in the right pond - you would almost certainly achieve much higher reductions by enforcing cleaner shipping packages (IE not including random garbage, etc) than by compressing the garbage more.

[1] on my computer, single threaded level 19 does 7meg/second, the default does 500meg/second. Level 10 does about 130meg/second.

Someone2• 1 year ago

the standard compression level for rpms on redhat distros is zstd level 19
The only reason to use zstd is because you want fairly good compression, but fast

I would think having fast decompression is desirable, too, especially for rpms on redhat distros, which get decompressed a lot more often than they get compressed, and where the CPUs doing decompression may be a lot slower than the CPUs doing the compression.

And zstd beats xz in decompression times.

canucker20164• 1 year ago

Here's the Fedora page relating to changing RPM from xz level 2 to zstd level 19.

https://fedoraproject.org/wiki/Changes/Switch_RPMs_to_zstd_c...

Just two tables for comparison, first one shows only the decompression for Firefox RPM, second shows compression time, compressed size, and decompression for a large RPM.

You'd think there'd be more data.

DannyBee5• 1 year ago

Let me start by reiterating - zstd is a great option. I think zstd level 5-10 would have been an awesome choice. I love zstd - it is a great algorithm that really hits the sweet spot for most users between really fast and good compression, and very very fast decompression. I use it all the time.

In this case, yes, zstd has faster decompression , but xz decompression speed is quite fast, even before you start using threads. I have no idea why their data found it so slow.

Here's an example of this: https://web.archive.org/web/20231218003530/https://catchchal...

Even on large compressed archives, xz decompression times does not go into minutes - even on a mid range 16 year old intel CPU like being used in this test. Assuming the redhat data on this single rpm is correct, i would bet it's more related to some weird buffering or chunking issue in the rpm's compressor library usage than actual xz decompression times. But nobody seems to have bothered look at why the times seemed ridiculous, at all, they just sort of accepted them as is.

They also based this particular change on the idea that they would get similar compression ratio to xz - they don't, as i showed.

Anyway, my point really wasn't "use xz", but that choosing zstd level 19 is probably the wrong choice no matter what.

There own table, which gives you data on 1 whole rpm, shows that zstd level 15 gave them compression comparable to xz (on that RPM, it's wrong in general), at compression speed similar to xz (also wrong in general, it's much slower than that).

It also showed that level 19 was 3x slower than that for no benefit.

Result: Let's use level 19.

Further the claim "Users that build their packages will experience slightly longer build times." is total nonsense - their own table shows this. If you had an RPM that was 1.6gb, but took 5 minutes to build (not uncommon, even for that size, since it's usually assets of some sort), you are now taking 30 minutes, and spending 24 of it on compression.

Before it took ... 3 minutes to do compression.

Calling this "slightly longer build times" is hilarious at best.

I'll make it concrete: their claim is based on building Firefox and compressing the result, and amusingly, even there it's still wrong. Firefox RPM build times on my machine are about 10-15 minutes. Before it took 3 minutes to compress the RPM. Now it takes 24.

This is not "slightly longer build times". Before it took 30% of the build time to compress the RPM.

Now it takes 24 minutes, or 2.5x the entire build time.

That is many things, but it is not a "slightly longer build time".

I'll just twist the knife a little more:

RPM supports using threading for the compressors, which is quite nice. It even supports basing it on the number of cpus you have set to use for builds. They give examples of how to do it, including for level 19:

  /usr/lib/rpm/macros:
  #                "w19T8.zstdio"  zstd level 19 using 8 threads
  #               "w7T0.zstdio"   zstd level 7 using %{getncpus} threads

The table with this single rpm even tested it with threads!

Despite this - they did not turn on threads in the result...

  /usr/lib/rpm/redhat/macros:%_binary_payload w19.zstdio

So they are doing all this single threaded for no particular reason - as far as i can tell, this is a bug in this well thought out change.

All this to say - i support NPM in being careful about this sort of change, because i've seen what happens when people aren't.

kavenkanum• 1 year ago

If OP wanted to shrink nom packages then npm could introduce two types of npm package - the build package and source one. This way a lot of packages would be smaller, because package code could be safely distributed through a separate package and not kept in the build package. There are a lot npm packages that explicitly include whole git repo in the package and they do so because there's only one type of package they can use

franga2000• 1 year ago

We wouldn't have to worry about over-the-wire package size if the modern DevOps approach wasn't "nuke everything, download from the Internet" every build.

Back in my Java days, most even small-time dev shops had a local Maven registry that would pass through and cache the big ones. A CI job, even if the "container" was nuked before each build, would create maybe a few kilobytes of Internet traffic, possibly none at all.

Now your average CI job spins up a fresh VM or container, pulls a Docker base image, apt installs a bunch of system dependencies, pip/npm/... installs a bunch of project dependencies, packages things up and pushes the image to the Docker registry. No Docker layer caching because it's fresh VM, no package manager caching because it's a fresh container, no object caching because...you get the idea....

Even if we accept that the benefits of the "clean slate every time" approach outweigh the gross inefficiency, why aren't we at least doing basic HTTP caching? I guess ingress is cheap and the egress on the other side is "someone else's money".

Terr_• 1 year ago

Yeah, I would also note that in addition to speed/transfer-costs, having an organizational package proxy is useful for reproducibility and security.

robertlagrant• 1 year ago

Lots of places use a cache like Artifactory so they don't get slammed with costs, and are resilient to network outages and dependency builds vanishing.

alimbada• 1 year ago

After reading the article, this comment and the comment thread further down on pnpm[1], it feels to me like the NPM team are doing everyone a disservice by ignoring the inefficiencies in the packaging system. It may not be deliberate or malicious but they could easily have provided better solutions than the one proposed in the article which, in my opinion is a band-aid solution at best. The real fix would be to implement what you mention here: local registry and caching, and/or symlinking a la pnpm.

[1] https://news.ycombinator.com/item?id=42841658

selfmodruntime• 1 year ago

In every org I've worked with, we had a local dependency mirror in the GitOps architecture.

Spivak• 1 year ago

I really don't want to go back to the old world where every part of your build is secretly stateful and fails in mysterious hard to reproduce ways.

You can and should have your own caching proxy for all your builds but local caches are evil.

animuchan• 1 year ago

This presents a great opportunity for alternative npm clients (such as bun, yarn). An almost free advantage over the mainline npm CLI.

jokethrowaway• 1 year ago

Nice exercise in bureaucracy.

- I don't see the problem with adding a C dependency to a node project, native modules are one of the nicest things about node.js

- Longer publishing time for huge packages (eg. typescript) is a bigger problem but I don't think it should impact a default. Just give the option to use gzip for the few outliers.

My failed attempt to shrink all NPM packages by 5%

Related Stories