Developing in a Monorepo While Still Using Webpack

https://codeascraft.com/2020/04/06/developing-in-a-monorepo-while-still-using-webpack/

by fagnerbrack • 6 years ago

41 15 6 years ago

maccard3• 6 years ago

I wonder what the appeal of having the build tools written in JS is, as performance is key here. Spoilers on the article:

Rebuilding our seller tools, which previously took almost a minute on every request, now takes under 30 seconds when we’re starting a fresh compiler, and subsequent requests take only a second or two.

According to [0], esbuild is orders of magnitude faster than we pack. The speedup advertised is from 60s to hundreds of ms. The article talks about 12k js files, while according to [1], 4 year old hardware could handle an out-of-date check in less than a second on the entire code base if using a tool like make.

Im coming at this from an outside perspective, but even the results sound like they could be substantially faster? That said, it's an interesting article and I enjoyed reading it!

[0] https://esbuild.github.io/.

[1] https://david.rothlis.net/ninja-benchmark/

afavour• 6 years ago

I think the biggest reason is simply that esbuild hasn’t been around for that long and isn’t as customisable as Webpack.

The latter point is important: if esbuild wants to introduce more customisability it’ll need to use more JS, and be slower.

maccard3• 6 years ago

Customisable is great and all, but at the expense of 100x the performance, is it worth it? There's likely no need for the majority of the customisation in webpack, but removing it isn't an option either. The real issue is that webpack has gotten to the position it has now and nobody has stopped and asked "why"!

curlypaul924• 6 years ago

Our system used to use grunt and then gulp and later webpack, and I was always annoyed at how slow the builds were. The startup time alone was several seconds, and it never used more than 1 core (our dev machines have 64 cores).

I ended up rewriting the whole thing as rake tasks that call out to rollup and other tools. Now it's fast, and as a bonus I can actually understand what it's doing and why.

kaycebasques• 6 years ago

Tooling Report [1] also goes into detail about which build tools provide the most bang for your buck, in terms of features for optimizing your website code.

So, to circle back to your comment:

I wonder what the appeal of having the build tools written in JS is, as performance is key here.

Yes, performance is key. But we're presumably talking about two types of performance: performance of the build tool itself, and performance optimization of the website (enabled by the build tool). That could be a key tradeoff. Theoretically there's probably no reason why a feature-rich build tool for the web could not be written in something other than JS, but realistically does it exist right now? (I don't know much about this space so it's a genuine question.)

Also, it seems natural to me that, if you're writing a tool for web developers, you're more likely to be able to build a community around that tool if you write it in a language that web developers are familiar with. Since web developers have the most skin in the game to make that tool great. That's probably a trite/obvious argument, but it's worth mentioning nonetheless.

[1] https://bundlers.tooling.report/

Disclaimer: I'm on the same team as the Tooling Report people.

P.S. if any of the esbuild people are here, I believe Tooling Report is open to getting other build tools on the site (you would have to write all the tests). I could be wrong but it's worth reaching out to them.

maccard3• 6 years ago

But we're presumably talking about two types of performance:

Agreed, and you only need to look at C++ linkers for proof this. LTCG can take minutes (or even hours) for marginal gains.

That said, this entire article is about migrating the development builds and still being painfully slow. The links I shared to esbuild point to building release-able bundles, I.E. "Optimised". And let's be honest, the optimised bundles here (i.e. the output of webpack) aren't performing inlining, constant folding, expression simplification, dead code analysis, or anything of the sort, they're doing some fairly rudimentary file IO. Only the most invasive optimisations (whole program optimisation) should explain the orders of magnitude of a difference.

The reason they're slow is because they're written in JS, single threaded, with no considerations for performance. As esbuild has proven, rewriting in go with consideration for perf can provide a 100x speedup!

but realistically does it exist right now?

Yes, it does it's esbuild and I linked to it in my comment above!

That's probably a trite/obvious argument, but it's worth mentioning nonetheless.

Honestly it's probably the truth but it's insane how we've managed to get here!

emilsedgh• 6 years ago

What makes webpack slow is babel. You can remove babel and use esbuild through esbuild-loader to make webpack fast.

wildpeaks• 6 years ago

Exactly.

Babel used to be useful to transition to ES6 back in the day, but I'm very glad I removed it from my Webpack pipelines some years ago, Typescript already covers everything I was using Babel for and gets me Intellisense in editors on top of it.

Furthemore, I found out yesterday that Webpack 5 has Assets Modules that can detect non-script imports using "new URL()" syntax as well (https://webpack.js.org/guides/asset-modules/#url-assets), a feature I haven't seen in other bundlers.

uranusjr• 6 years ago

One important consideration is how you’d get contributors. The vast majority of a language tool’s users would be most proficient in that particular language, and using that same language maximises the project’s sustainability. Tools implemented with a foreign toolchain require a much longer initial runway, and may eventually still die simply due to the small intersection between people who care about the project and and who understand enough to maintain it.

Uehreka• 6 years ago

I'll be honest, until just now, I had no idea that esbuild even existed. Upon reading this comment, I immediately put down my iPad, ran (not walked) to my desktop, swapped babel-loader for esbuild-loader in a work project and a spare time project and benchmarked the results. For full production builds with minification etc., the work project went from 40 seconds to 20 and the spare time project went from 64 seconds to 20. Merry Christmas indeed.

I really don't care if my compilers are written in JS, Go, C, Dart or INTERCAL. If they do what I need them to do I'm going with the fastest one. And unless further testing reveals a huge drawback, that means esbuild.

jolux• 6 years ago

Does Bazel not support JavaScript well?

anonymousCar• 6 years ago

There's some rules here that you can use in bazel. It takes a bit of finesse to get some packages working with it. https://github.com/bazelbuild/rules_nodejs

pcmaffey• 6 years ago

Related: I learned recently that Lerna is intended specifically for publishing packages within a monorepo, and is entirely unnecessary if not publishing individual packages. Yarn workspaces is all you need to manage dependencies between packages. Not sure where this common misconception came from, but I definitely fell for it when starting out with monorepos.

weeboid• 6 years ago

The problem with naming your code "Kevin" is that any Kevins in the company near you will now be targets of derision, and will think you guys are lame/dumb.

It's like naming your dog a human name, someone with that name comes to your place and now hates you.

mewpmewp2• 6 years ago

Imagine someone giving a human name to a popular robotic assistant device. And the results of that for people with said name. Their lives would never be the same again. This is sad.

Developing in a Monorepo While Still Using Webpack

Related Stories