Rebuilding Babel: The Tokenizer

https://www.nan.fyi/tokenizer

by pgayed • 4 years ago

36 5 4 years ago

This is way more in depth of an evaluator than I've ever written, but it's a fun project. I'm a largely self taught dev and build an evaluator was an illuminating project for me. Mine was just a math parser but this is a cool project!

dilap2• 4 years ago

FWIW, a tokenizer is not really necessary. It (arguably) makes things cleaner / easier, but you can also just write a parser that operates directly on a byte stream, if you feel so inclined, and it turns out fine.

kccqzy• 4 years ago

Yes I agree. The article actually says the purpose is

To separate the logic that handles a language's microsyntax from the logic that handles regular syntax.

But everything is just syntax. This distinction between microsyntax and regular syntax is really just for convenience; and sometimes it's not that much convenience.

Waterluvian• 4 years ago

And isn’t this developer convenience at the cost of performance in a piece of software where performance is very important?

Today as my “end of the week fun experiment” I replaced Babel with esbuild on a years old 50k sloc production web application. It took 30 mins and builds now take 500ms instead of 63,000. Not quite apples to apples yet. But it’s such a compelling change that I can’t ignore it. I’m excited to see how Babel steps it up in the years ahead.

Rebuilding Babel: The Tokenizer

Related Stories