Incremental Parsing in Go

https://dev-nonsense.com/posts/incremental-parsing-in-go/

by willdaly • 4 years ago

180 74 4 years ago

xyzzy47474• 4 years ago

For max optimization, wouldn’t it be better to create a Rust or C library for parsing that Go links into? I personally don’t see the usefulness of trying to optimize Go itself too much as it’s handicapped by the runtime and garbage collection.

tester7562• 4 years ago

C library for parsing?

isn't it dangerous from security perspective?

xyzzy47474• 4 years ago

It's just an example. The options are really Rust (what I'd prefer), C++, C, or perhaps something like Nim that compiles to C.

If you’re trying to make an unoptimized parser, then use whatever you want.

pharmakom2• 4 years ago

For some reason we insist that language parsers are implemented in the language itself, even when the language isn’t great for parsers.

37ef_ced3• 4 years ago

You're in for a big surprise. Try using the language.

Spend some time using Go, and you will be impressed by its performance.

You'll wonder, "Were all those haters on Hacker News misinformed?"

xyzzy47474• 4 years ago

If you're making something requiring CPU optimization as a core feature, you might as well go with one of the fastest languages instead of handicapping your project from Day 1. Go is not considered one of the fastest. It's better for network or filesystem logic that is I/O limited.

bugfix-66• 4 years ago

Correct, Go is fast, very close to C.

And just like in C, if you want to avoid memory management overhead you can use a slice of structs, integers instead of pointers, and a freelist (if needed). For example, here is a pointerless sparse bit vector:

https://bugfix-66.com/7256e0772dc3b02d72abf15b171731c933fd44...

The article is storing parses in a balanced binary tree, like a packrat memoizing parser.

Here is the fastest balanced search tree in Go. It allocates (and uses Go's garbage collector) but you can easily use a slice of structs with integer index pointers and a freelist instead:

https://bugfix-66.com/c93e950965804eba90a34e0055985b1c42d5a1...

The above code will perform very similarly to C.

rollcat• 4 years ago

One of Go's primary goals has always been compilation speed.

Go started out in C, and was later (post-1.0) incrementally rewritten to be self-hosting. One of the authors (Ken Thompson) is also one of the co-creators of C. I would argue these guys know what they are doing.

kaba011• 4 years ago

I don’t know, not implementing generics when it was pretty obviously needed was a huge oversight, so I’m not sure.

Also, the reason for compiler bootstrapping is more of a “beauty thing”, then practicality. It would definitely be faster in a low-level language, but I doubt it would matter as an end user.

Thaxll• 4 years ago

I've seen some real world example where Go was as fast or faster than Rust for CPU / io intensive task.

Go is a fast language even with a GC.

https://github.com/boyter/scc/#performance

akira2501• 4 years ago

For maximum return on investment, wouldn't it better to focus on something other than raw speed? I personally don't see the usefulness of trying to make everything in Rust itself too much as it's handicapped by it's compiler and lack of specification.

frou_dh• 4 years ago

None of your ideas in this thread mention involving a profiler, so I take it you're from the Wild-Ass-Guess school of optimization?

binwiederhier• 4 years ago

Interesting read. Thank you for sharing. I always found parsers fascinating and mystical. It seems like these parser functions (which i think are analogous to what Rob Pike calls state functions) are a common way to do parsing, though i know very little about it. I especially found the combinators intriguing, though I don't care much for the functional programming syntax in a language like Go.

Anyway, thanks for sharing.

Tangentially, I wrote a little mini parser [0] of my own for my side project. It is inspired by Rob Pike's talk on parsers [1]. It doesn't use state functions, but instead just uses the call stack to keep track of where we are.

[0] https://github.com/binwiederhier/ntfy/blob/main/server/actio...

[1] https://www.youtube.com/watch?v=HxaD_trXwRE and https://go.dev/src/text/template/parse/lex.go

skohan• 4 years ago

Yeah parsers are fun! We did a recursive descent parser for a toy language in uni and I think it was one of the most illuminating and fun projects we did at school.

Lately I've been working on a tool to make it easy to implement a parser, and I end up using it for everything, because DSL's are so nice to work with.

tester7562• 4 years ago

Difference in complexity of IDE's parser and Compiler's parser feels like order of magnitude

IDE's you want to be very fast, so you use techniques like partial tree reparse and now when I think about it, then you also may need to update other places

like you use type that is defined somewhere below

and at first parse that type definition doesn't compile, so type usage above shows an error

and when you change type definition so it compiles, then if you only update that part of the tree, then previous would still scream about the error

it's really tricky

the theory behind how to deal with all of this problems seems to be easy

but when you actually get to the coding, then you have to be really thoughtful, careful and experienced in order to get the modeling of right

https://learn.microsoft.com/en-us/shows/seth-juarez/anders-h...

________

I have some experience with simpler and more complex parsers

and for me no other type of software requires this much careful thought as parsers do if you want to address all those things like

correctness, speed and recovery on broken code fragments, good error messages, good code, maybe concurrency

bbkane2• 4 years ago

You would like https://rust-analyzer.github.io/blog/2020/07/20/three-archit...

pharmakom2• 4 years ago

a successful parse consumes at least one rune

This avoids infinite loops?

Jtsummers2• 4 years ago

Yes, there would be two outcomes for an attempted parsing. Either it succeeds and makes progress (and eventually terminates) or it fails and consumes nothing (and terminates because eventually you run out of parsers to try).

throwaway290• 4 years ago

Off-topic but are there any aretext users? How does it fare?

derek8bai• 4 years ago

cool stuff

hk__2• 4 years ago

If you want a hard/interesting parsing challenge, try Clojure’s #_ reader macro. It’s a powerful construct that allows you to comment the next form. If you’re not used to Clojure, it’s like writing #_ in front of anything --a function, an array, a keyword, etc-- to comment it, even if it’s on multiple lines.

For example:

    #_ (defn foo
         [x y]
         (println x y))

This is equivalent to commenting the three lines. Things become even harder when you learn that these thing "swallow" the next form and can be used anywhere:

    (let [a #_ b 43] #_ #_ hello (H N) (print a))

The code above is equivalent to the following:

    (let [a 43] (print a))

All the rest is comments.

The hard/interesting bit is that to tokenize you must construct a syntax tree in order to correctly parse the next form, but in order to construct a syntax tree you first need to tokenize the code.

diffxx4• 4 years ago

The parsers produce a sequence to tokens, not a full syntax tree. Writing a tokenizer is much easier than parsing full syntax trees...Most other editors don’t construct the full syntax tree.

Syntax highlighting is nice, but fast semantic analysis is the real holy grail.

I have come to think that the best way to develop a new language would be to implement a text editor in that language before releasing the language. The editor should (ideally) have emacs and vim key bindings, though it would surely at least have the bindings that the language author uses. The compiler/interpreter for the language would embedded in the editor. This would allow for a much richer editing experience that goes beyond syntax highlighting. Indeed, the source code would become like a living document in the editor where they editor could display inline information about both syntax _and_ semantics. The editor need not be fancy. It could be written as a terminal application, kind of like a language specific nano or vim.

If the language/editor author is careful in how they design the editor, all of the syntactical and semantic tooling should be exportable into packages that can be consumed by other editors with plugin systems/lsp like vscode or neovim. Then the rich editing experience can be relatively easily exported to any other text editor. Tool authors would also then be able to write static analysis/linting/formatting/whatever tools on top of the semantic tooling that the language supports.

In some ways what I am describing is a more minimalist version of what one would get out of an image/IDE based language like smalltalk but the code representation would still remain as text files. The editor then becomes like a REPL except rather than being an ephemeral process, the REPL state is continually being written to disk and is resumable at any time from any computer capable of running the editor.

maxbond5• 4 years ago

I think it would be sufficient & more valuable to implement a language server for your new language, which keeps you focused on the parts related to your language rather than the struggles of implementing an editor, and then you'll be able to drop into VSCode, neovim, etc.

thechao• 4 years ago

Does anyone have an good intro tutorial for writing a language server in, say, C for some simple language? I find most of the docs a little too inscrutable to follow for just a bit of dabbling.

diffxx4• 4 years ago

If you write your own editor with language server features, writing a language server implementation _should_ (dangerous word) be relatively easy. By writing the editor in your language, you prove that your language is actually useful for a real world problem. You also have the flexibility to add any functionality that you wish without contorting your thinking to whatever may be imposed on you by the lsp model.

It isn't necessarily clear to me that writing a language server implementation is that much easier than writing an editor. To be fair, I have personally done neither (though I have written REPLs) so perhaps I am wildly off in my estimation.

Gibbon1• 4 years ago

I agree with the parent. The big advantage of what he's suggesting is code objects exist as first class objects. They don't have to creased by parsing a text file. Which means the editor can directly manipulate them.

If you've ever used CAD programs especially Schematic Capture and PCB Layout programs you'll get the idea. Everything displayed on the screen is an object in a database.

Instead of a parser blindly parsing a text file and coming across a struct definition, a struct is object created by the programmer. Which means you can have hard links between the objects that make up the program. And those can be directly manipulated with manually or programmatically.

The big advantage comes from maintenance and refactoring. Change the name of a field? Happens in exactly one place in the program. So instead of a diff with 1537 files changes you have 'renamed struct fobar to struct foobar'. Change a comment? Well it's just a changed comment.

bbkane2• 4 years ago

I know of two languages trying something similar to this:

- https://dion.systems/faq.html - https://www.roc-lang.org/

They're both experimental, but fun to read about!

theta_d• 4 years ago

So smalltalk?

sesm2• 4 years ago

This article uses the word 'rune' extensively. From the context I assume it means 'lexem' or 'token' (i.e. the unit the lexer/scanner produces and feeds to parser). But then the article uses the word 'token' to mean the output of a parser ('keyword token'), while the usual terminology is that parser output is called a 'parse tree'.

So, in this terminology, the parser consumes 'runes' and outputs 'tokens', while the usual terminology is that parser consumes 'tokens'/'lexems' and outputs 'parse tree'.

pgwhalen• 4 years ago

Rune is a type alias in go, which more or less maps to the more common words “character” or “code point”.

https://go.dev/blog/strings

sesm2• 4 years ago

Thanks! I don't know Go, so this terminology was surprising to me.

sjansen2• 4 years ago

In Go, `rune` is an alias for `int32` and is used to indicate the value is a Unicode "code point".

For characters in the ASCII range, that means it's just a character encoded using more bits. If you need to worry about the full Unicode range then it's important to understand Unicode Normalization Forms.

https://go.dev/blog/strings

https://en.wikipedia.org/wiki/Unicode_equivalence

Incremental Parsing in Go

Related Stories