Speeding up the JavaScript ecosystem – The barrel file debacle

https://marvinh.dev/blog/speeding-up-javascript-ecosystem-part-7/

by cdme • 3 years ago

149 45 3 years ago

shadowgovt• 3 years ago

Good write-up. Interesting to me that the author doesn't recommend minifying and packing the code as a solution though; even if you get rid of barrel files, not every imported symbol is always used and a pack-and-minify will leave unused code behind.

jraph• 3 years ago

Minifying and getting rid of unused code improve runtime performance, but it's work at compile time / during development. IIUC the author is mostly speaking about tooling performance (tooling that's used during development / compilation). I don't think minifying and dead code removal improve the issue the author writes about.

Sophistifunk2• 3 years ago

This is the kind of thing that should be fixed by improving tooling, rather than making your source less clear. Given the opportunity that was mjs, bun, deno, etc it's clear that file-based and url-based, and relative imports are things people don't want to move away from in the current JS ecosystem, which means "barrel files" (the name makes no sense to me either) are going to remain a useful part of trying to cut down on the noise.

skybrian• 3 years ago

A fundamental problem in the JavaScript ecosystem is mixing up source files and "binary" (optimized, minified) files. URL imports seem like a bad move because they can point to either.

Sourcemaps help somewhat, but even with Deno, when using an IDE, the source you want is often in another castle. You might get minimized code or type definitions.

Contrast with the Go system which rarely has this problem. Following imports goes to source.

Sophistifunk2• 3 years ago

Yeah this seems to be absolutely everywhere and entrenched, mainly because it's how npm did things :(

corbezzoli2• 3 years ago

Locally, never ever use barrel files. If you manage the entire project, just import the specific file you need.

However when exporting a public/published module it can become onerous to ask the user to import 7 files instead of 7 items from one module.

In a perfect world, this would not be much of an issue because tools would cache these resolutions/imports and their treeshaken code. In the current world, you could prepackage your project even on Node, so that there’s only one file to load, and no extra unused code.

akira2501• 3 years ago

However when exporting a public/published module it can become onerous to ask the user to import 7 files instead of 7 items from one module.

Plenty of large single units in almost any language are _developed_ this way, but is there any particular reason they are _deployed_ that way? It really seems like a distribution packaging issue.

corbezzoli2• 3 years ago

npm will take in anything that has a package.json file with two fields, so it’s un/fortunately rather dumb.

Nothing stops you from prepacking your files, but a lot of node developers are ironically allergic to build tools.

Vinnl• 3 years ago

The author says it's fine to add a barrel file for a public package; they're just saying you shouldn't add a barrel file for every folder: https://infosec.exchange/@marvinh/111203114104449172

keepamovin• 3 years ago

Jesus...50000 modules creates 40 seconds in overhead!!!

daliusd• 3 years ago

According to article 20 minutes

chrismorgan2• 3 years ago

That’s for running it 100 times, spread across four cores, i.e. multiplied by 25. About 48 seconds is the actual claimed figure.

TheSisb2• 3 years ago

In between barrel files and no barrel files, are unbarrelled files. This is where you have one installable package (i.e.: @cool-lib/core) and create imports from sub-paths like this:

import Button from "@cool-lib/core/button";

import Link from "@cool-lib/core/link";

import Widget from "@cool-lib/core/widget";

This is similar to what lodash does too. Still have more imports, but only one package.json dependency! And the same perf gains.

Jarred• 3 years ago

One of the optimizations we do in Bun.build() is to automatically rewrite imports/requires of CommonJS modules shaped like this

    if (process.env.NODE_ENV === "development") {
      module.exports = require("./another-file.dev.js")
    } else {
      module.exports = require("./another-file.js")  
    }

Into the destination path, removing this module entirely

I tried to make this work for ESM too, but there were too many edgecases due to module namespace objects being observable (`import * as `)

A similar thing could be done for barrel files with `"sideEffects": false`. We don't currently follow nested default exports which makes this harder to do as effectively (we do support "sideEffects": false)

crabmusket2• 3 years ago

I really like this article series, but this entry seems quite light on the usual rigorous approach.

Instead of detailed analysis of the barrel files in a specific project+dependencies, there are vague exaggerated comparisons like "What's quicker? Having to load 30k files or just 10? Probably only loading 10 files is faster." Of course 10 modules is faster. But are there 29,990 barrel modules in Webpack?

Also, why is running esbuild+node faster than just node? Does that suggest the problem is not just lots of dusk reads, but something inefficient in the way node processes the module graph? Could that be improved?

Anyway this is an interesting direction of inquiry, and definitely worth considering!

mhagemeister3• 3 years ago

Author here.

That's fair criticism. Whilst the setup shown in the article is synthetic, the experience is from working on real world projects. I've worked on a bunch of projects which had about 3000-6000 barrel files easily, and these then lead to every file in the project always being included. The total amount of files being loaded was somewhat around 30-60k, depending on the project.

I did the exact same thing as described in the article and wrote a custom test runner for jest that bundled the test files with esbuild before executing them. Even though bundling introduces a costly overhead on its own, it was still about 60% faster to bundle + run the bundled code vs not doing it. Bundling can also be amortized because it only needs to be done once per test run vs once per file. That's where the real gains came from, because in jest every single test file constructs the whole module graph from scratch.

I haven't checked myself, but there is likely improvements to be made in node (or other runtimes) to address this. Hopefully, some folks get inspired by this article to look at that.

crabmusket2• 3 years ago

Thank you so much for this article series! It's much needed and I love reading each installment.

can3p• 3 years ago

I think a fair question to explore is why is it not a problem in other languages (maybe it is)

Direct paths also make file structure a part of api and that may not be the desired effect. In addition the bigger problem is not barrel files in your project but rather those in all third-party dependencies

Kiro• 3 years ago

Excited about finishing your feature, you run the code and realize that it takes an awfully long time to complete.

How do you run your project where this happens? I thought everyone complied it separately and then ran it, not both at the same time. Shouldn't your tooling tell you immediately that it's the compilation being slow and not the code?

mhagemeister3• 3 years ago

Author here.

Compilation isn't the problem, loading and instantiating the module graph is. The timings for that are currently not exposed in any runtime afaik.

chrismorgan2• 3 years ago

The “cost of loading modules” diagram shows non-linear behaviour (though you should largely ignore the curve visible in the diagram because the x axis is way off linear):

• 0.15s ÷ 500 = 0.3ms

• 0.31s ÷ 1000 = 0.31ms

• 3.12s ÷ 10000 = 0.312ms

• 16.81s ÷ 25000 = 0.6724ms

• 48.44s ÷ 50000 = 0.9688ms

My own observation on a Surface Book six years ago was that in Node.js under Windows, each module had about 1ms of overhead when there was warm file system cache—that is, simply bundling with Rollup saved 1ms per file. If this sort of thing interests you, quite a lot of useful stuff came out of https://github.com/stylelint/stylelint/issues/2454 which I filed because I was unhappy with stylelint taking over a second to import. And that must have been only in the order of one or two thousand modules, when the behaviour is still close enough to linear.

TheFragenTaken• 3 years ago

There are typically useful ways (tools) in most bundlers to tree shake barrel files. Libraries also differ in how tree shakable they are.

Personally I think the readability of having few import lines from shared barrels, compared to a forest of lines, are very useful.

padjo• 3 years ago

I find that I essentially never look at the imports anymore, it’s all handled automatically for me. So there could be one line or a hundred lines I don’t care and just scroll past. I’m sure I could get an editor plugin to auto fold imports too

jacobr• 3 years ago

It’s built in to VS Code, see editor.foldingImportsByDefault.

Tade02• 3 years ago

In the popular jest test runner, each test file is executed in its own child process.

Is that confirmed?

I've been following this issue:

https://github.com/jestjs/jest/issues/6957

And what Jest actually does is still kind of muddy.

In contrast to that, other test runners like AVA have a clear description what happens when:

https://github.com/avajs/ava/blob/main/docs/01-writing-tests...

mhagemeister3• 3 years ago

It's not a secret what jest does. There are a few abstractions to jump through, but I find jest's code very readable overall. By default the `workerThreads` option is set to `false`. Then we reach their `WorkerPool` class which pools the test runs. The "Worker" in the name is not to be confused with WebWorkers or node's worker_threads. It's merely the term the jest team has chosen. Inside that and because workerThreads is set to false, we instantiate a child process here

https://github.com/jestjs/jest/blob/00ef0ed0a03764f24ff568bc...

Tade02• 3 years ago

This doesn't fully answer my question, because in the issue I linked it's explained that Jest might to decide to implicitly apply `runInBand` when... it feels like apparently, because that's not well documented.

I see it as a major design flaw, because I have had mysterious, impossible to debug errors happen due to this in just about every project that used Jest that I've been in.

silverwind• 3 years ago

I think it's the next step for package manager to bundle npm dependencies on installation and therefor optimize the import speed. With fast bundlers like esbuild, it's feasible.

mmis1000• 3 years ago

The problem here is that current gen bundler has a linking step that remove unused code if possible (called tree-shake in this context). This is conflicting with pre-bundle because it would cause anything to be bundled regardless of whether it is used. Or you will need to repackage the dependency when import changed. It's probably ok if you are in dev environment. But big package size isn't suitable for prod environment especially in a web page.

l5870uoo9y• 3 years ago

While opening up a index.js file only to discover that it exports nothing but variables and functions from other files is disappointing, the pattern that a folder has a formalized entry point does make it easier to navigate and comprehend projects.

h1fra• 3 years ago

Unless you are working in a single process script, barrel file is almost unavoidable. A typical nodejs app will often be split in many packages, requiring barrel file anyway.

agos• 3 years ago

I've never really used barrel files, I don't see why they are unavoidable

c-hendricks• 3 years ago

After years of writing JS/TS apps, barrel files are absolutely avoidable. You just write your functions / classes / variables, and import them wherever needed. No intermediary step of re-exporting them is needed.

rendall3• 3 years ago

I asked ChatGPT about the name "barrel files"

The term "barrel" is a metaphor. Just like a physical barrel holds multiple items in one container, a barrel file holds multiple exports in a single file, making it easier to manage and import.

https://chat.openai.com/share/93cb04ca-fd6f-46f0-991f-2d7f4c...

pwdisswordfishc• 3 years ago

"I asked ChatGPT" is the new Wikipedia link, it seems.

jeroenhd• 3 years ago

I generally assume the accounts posting "I asked ChatGPT about $topic" are karma farming bots. At least Wikipedia is checked for accuracy occasionally, ChatGPT links are the modern equivalent of lmgtfy

In this case the term looked up is literally explained in the linked article. Anyone who read the article shouldn't even be Googling for basic information, unless they need to know specifics that are out of scope for the article's subject.

CafeRacer3• 3 years ago

I like react, typescript, eslint, prettier, tailwind, react native, remix, etc. Libraries are good, nice, make total sense.

However, the ecosystem tooling is a total piece of crap.

Just the other day I was trying to update eslint dependencies in monorepo only to have builds broken, tailwind stop working on next.js project. Then little tailwind prettier plugin fell off. Some things support ESM, others not yet.

I seriously hope that something like bun will become a one single tool to use. One tool that works with TS, supports linting, testing, tailwind, etc, etc, etc. One single fucking thing that works.

I'd rather use someone elses opinionated approach and never look at the configs again than waste 3 days trying to configure a single fucking eslint config for monorepo.

Now the barrel fucking files is the problem. What happened to cache? Like it's 2023. Can't we just load everything from cache?

danwee• 3 years ago

Just the other day I was trying to update eslint dependencies in monorepo only to have builds broken, tailwind stop working on next.js project. Then little tailwind prettier plugin fell off. Some things support ESM, others not yet.

Perhaps that's the problem. Tons of moving parts => more chances for something to go wrong.

I seriously hope that something like bun will become a one single tool to use. One tool that works with TS, supports linting, testing, tailwind, etc, etc, etc. One single fucking thing that works.

I wonder why Node doesn't do that. They have an advantage point of being THE platform for JS development: `node lint`, `node build`, `node foo`, etc.

DanielHB• 3 years ago

it was a deliberate decision from the Node foundation to let the "ecosystem flourish". If they built-in those tools there would be a lot less evolution on what the tools could do.

Arguably this was a good decision in ~2010, but today it makes much less sense, tools haven't been evolving as fast anymore in the JS ecosystem and people seem to be standardising more and more on what works and what doesn't.

CafeRacer3• 3 years ago

Perhaps that's the problem. Tons of moving parts => more chances for something to go wrong.

There are too many changes and too many moving parts.

This reminds me of an old sketch. Don't remember the details but it's something like:

Oh, there are 6 popular messenger apps, let us build one to rule them all some time later... now there are 7 popular messenger apps

conaclos• 3 years ago

Biome [0] aims to be a linter and formatter for the web techs. It already supports JavaScript, TypeScript, JSX, and soon CSS.

[0] https://github.com/biomejs/biome

Speeding up the JavaScript ecosystem – The barrel file debacle

Related Stories