Lezer: A parsing system for CodeMirror, inspired by Tree-sitter

https://marijnhaverbeke.nl/blog/lezer.html

by goranmoomin • 2 years ago

166 42 2 years ago

lewisjoe2• 2 years ago

I'm always amazed by Marijn's brilliance (Author of Lezer, CodeMirror and the awesome ProseMirror toolkit)

The level of depth he dives into to perfect his projects is insane. For example, lezer is a parser generator( which by itself is not a trivial feat with novel ideas like incremental computations applied to parsing) to power his mainstream project which is CodeMirror.

Like this Prosemirror too has some insane levels of engineering underneath married with thoughtful architectural decisions.

Apparently, a big fan of his works.

AlexErrant3• 2 years ago

He's incredibly responsive too. He just answered my question about an hour after it was asked... on a Sunday evening.

Slightly off topic; is anyone aware of any Lezer grammar for regex? I've not been able to find any in the FOSS world. I suspect Regex101 has one, bit it's sadly closed source.

lewisjoe2• 2 years ago

Yeah, this happened to us too a couple of times. Once, we asked for a clarification and he took it as a sensible requirement, made the changes and took it to master in just over an evening.

The next day we just had to update our package to the latest version and marvel at his response time.

ananthakumaran• 2 years ago

Have you checked the js grammar? It has a regex grammar, though not sure if that's what you are looking for

simonw• 2 years ago

He's also the author of a really great JavaScript book: https://eloquentjavascript.net/ - which he's been intermittently updating since 2007!

RyanHamilton• 2 years ago

Massively agree. Code mirror 5 was excellent. Code mirror 6 was a big enough improvement to justify the upgrade. I've used it as part of 2 large projects and it's handled every expanding use case I've needed it to. It supports themes, sql,js and 2 weeks ago I used its diff functionality. Really great library I can fully recommend.

umvi• 2 years ago

Code mirror 6 is super hard to use though unless you are a front end expert. Code mirror 5 was basically plug and play. Code mirror 6 is a box of Legos and you are responsible to build and bundle your own editor from scratch.

bpev• 2 years ago

I jusy used this recently for making Traindown syntax highlighting! It was pretty intuitive, even though I haven't done this kind of thing before!

https://github.com/inro-digital/lang-traindown

https://traindown.com/

I did notice that it seems (rightfully) very focused on the exact usecase of syntax highlighting. Do people also use this kind of system for quickly building ways to parse data from arbitrary text syntaxes?

timenova• 2 years ago

(2019)

jitl3• 2 years ago

On one hand, it’s a nice framework. I customized the Typescript one a bunch for a lil side project and enjoyed myself. On the other hand, it would be great if CodeMirror could just work with Tree-sitter or similar. There’s a lot of ecosystem around other parsing systems, and needing to figure out Lezer stuff is a big friction for adopting CodeMirror 6 for me. There are not a lot of language packages listed: https://codemirror.net/docs/community/

There’s a kind of importer thingy here but it doesn’t work well for complex grammars: https://github.com/lezer-parser/import-tree-sitter

AlexErrant3• 2 years ago

Unfortunately, tree-sitter is written in C, which is still awkward to run in the browser (and CodeMirrror targets non-WASM browsers). It also generates very hefty grammar files because it makes the size/speed trade-off in a different way than a web system would.

I would be curious if there's been an effort to get tree-sitter working on the web.

gushogg-blake2• 2 years ago

Tree-sitter does run on the web. I got it working for my editor, but it did involve several days' worth of effort and getting into the weeds with emscripten. Details here - https://gushogg-blake.com/p/emscripten-web-modules/.

jitl3• 2 years ago

I learned from a google search that these days upstream tree-sitter provides WebAssembly bindings.

Source: https://github.com/tree-sitter/tree-sitter/tree/master/lib/b...

NPM: https://www.npmjs.com/package/web-tree-sitter

Download from the latest Github release: js file (https://github.com/tree-sitter/tree-sitter/releases/download...) and wasm file (https://github.com/tree-sitter/tree-sitter/releases/download...)

It's great to see upstream maintaining the bindings. I maintain a Typescript/Empscripten/WebAssembtly binding for quickjs and it's more involved than I would like.

Besides providing a WASM build, Emscripten can build a lot of C code to straight JS, which you can use in non-WebAssembly browsers, or in runtimes like Cloudflare or Vercel Edge that support WebAssembly but it's a pain in the behind. Unfortunately the tree-sitter upstream doesn't output such a build but given they already maintain the bindings it wouldn't be a lot of effort to pass the option to emcc (set -s WASM=0 -s SINGLE_FILE=1 instead of -s WASM=1)

conartist6• 2 years ago

I will probably be able to run tree-sitter grammars in plain JS (no WASM) soonish (within the next six months, say) on the BABLR VM

timhh3• 2 years ago

I attempted to use this but was disheartened but the fact that it doesn't statically type node names. Tree Sitter doesn't either but it has much more of an excuse given that it targets C.

https://github.com/lezer-parser/lezer/issues/8

The dev seems mildly hostile to outside involvement too, so I moved on. These days I use Chumsky which is Rust rather than Typescript, but also way more awesome, if you can deal with the often incomprehensible compilation errors at least!

https://github.com/zesterer/chumsky

stevemk14ebr• 2 years ago

None of those responses were at all hostile

lukan7• 2 years ago

He probably means this:

"It's rarely a good idea to jump in and start a big refactor of someone else's library without first discussing the direction you're going in."

But I agree, not hostile, just unwilling to do something, he does not want to do. Even if the other one already put work in it.

jonrosner• 2 years ago

I've been using both codemirror and lezer in Yaade (https://github.com/EsperoTech/yaade). Thanks to lezer I was able to write a JSON extension language that supports Yaade environment variables. Pretty cool project and very nicely documented! I love building OSS on top of OSS.

josh11b• 2 years ago

(2019)

amluto• 2 years ago

Silly question. The OP says:

    if (true) something()
    // Comment
    otherStatement()

> ... the comment should not be part of the if statement's node.

Why not? A comment seems like it isn’t functionally of the syntax tree, but, to the extent that one treats it like it is, it does seem like it’s in the middle of an if statement.

lioeters• 2 years ago

The first line is a complete statement that finishes at the end of the line - unless there's an else clause in the following line (not counting comments or empty lines). Expressed another way:

  if (true) {
    something()
  }

  // Comment

  otherStatement()

If this last line started with "else", then the comment would be within the if statement.

habitue• 2 years ago

The discussion of the error recovery strategy is pretty interesting. It seems very similar to me to monte carlo tree search, which kind of suggests using a value network to estimate which branches to prune

Lezer: A parsing system for CodeMirror, inspired by Tree-sitter

Related Stories