HN user

ckok

105 karma

Carlo Kok

Posts3
Comments78
View on HN

I believe the proper term for what i am describing is a recursive descent parser. With which it is also quite doable to generate proper error handling and even recovery. Some form of this is used in almost every production language I think.

It has been years since I've written a proper parser but before that every time I had to write one I tried the latest and greatest first. ANTLR, coco/r, combinators. All the generated ones seemed to have a fatal flaw that hand writing didnt have. For example good error handling seemed almost impossible, very slow due to Infinite look ahead or they were almost impossible to debug to find an error in the input schema.

In the end hand crafting seems to be faster and simpler. Ymmv.

My point about the article was mostly that all the formal theory is nice but all it does is scare away people, while parsing is probably the simplest thing about writing a compiler.

I think this makes it sound a lot more difficult than it has to be, with the formal theory.

When it's really one of the most simple things if you divide it in parts and look at it from a tokenizer (string to list of tokens) and parser on top. Where the tokenizer can usually be very simple: a loop, large switch on the current character, where a choice is made on "what can this be", and making it into a formal token or error. Then a simple recursive parser that can almost be a 1 to 1 copy of the (E)BNF.

This has been said about pretty much every subject. Writing your own Browsers, compilers, cryptography, etc. But at least for me even if nothing comes of it just knowing how it really works, What steps are involved are part of using things properly. Some people are perfectly happy using a black box, but without kowning how its made, how do we know the limits? How will the next generation of llms happen if nobody can get excited about the internal workings?

Sso should be the default really. And as a starting point every language has openid connect implemented. Yes it needs work but surely better than the average password reuse?

For large and small companies surely the best way to maintain their users

JSONB has landed 3 years ago

Does anyone know if SQLite has a public roadmap? This combined with a gin or gin like index would be a killer combination

I couldn't find it in the docs but does this also support working in a htmx like mode? Where the server renders pages and it does partial updates of parts? Or what some frameworks now call server components?

Or is everything more like a initial render on the server and pure wasm from that time on?

Parser generators generally suffer from at least speed, recovery, error handling and lack of debuggability. They also require you to learn a new language, the parser generators language.

Last but not least, what you actually end up wanting to build is an ast, as at some point you want to do something with the input, for most parsers you then have to implement even more code to build up the ast.

It is much easier to hand write it. In the end its faster to write and usually faster to run.

Every few years I evaluate the new options for the languages I use (c#, pascal), every time so far I am disappointed with the tooling. Maybe one year.

But that's the right way to do it really... I remember an obscure thing in c# when var was introduced that if you have a type called 'var' in scope, type inference doesn't work and it uses that type instead. This is the way new features should be introduced.

I can't help but get the feeling that plan with Swift is to be the only language you can effectively use for Apple platforms, but also is probably going to be the language that will only be used on Apple platforms because of Apple's whims like the function builders.

No duplication. There's a single file where every type reference or method reference exists once. Note that it's not a library file. The compiled dll or exe has no relationship to the original way it was stored in .cs file. Instead it's a set of tables which hold all types, all members.

Swift on Windows 6 years ago

Sort of. I can't comment on Kotlin, but Swift is a different beast though. Swift's generics are very different from say c#, java or even c++ templates, especially when you mix them with protocol associated types, which are also sort of, but not exactly generics.

In Elements we spent a lot of time on getting it right, and I think we have most edge cases covered for user code. The way generics are used in the Swift runtime library seems impossible to map perfectly on top of the concept of generics as C# has them though.

In a way I'm happy rust does not have a stable abi. Swift does, but the stability is 'whatever apple swift emit'. Very little documentation, that what's there is out of date, so the only practical language that can interact with swift is swift. To be able to interact from another language, one would have to parse swift, make the semantics of all types and generics match exactly to be able to do the simplest things. (For example array and string, two core types are stock full of generics and protocols)

I'd hate to have the same happening for rust.

I tried this. Aot compiling with the .net runtime just isn't very practical. I know mono does this, with some guided info for reflection based classes. But the way the structure is setup, is that string pulls in globalization. And enumeration. And comparable and equality which both work by reflection to pick the right default implementation. Reflection is expected to work, which pulls in all methods and their dependencies. By the time you have a working executable for hello world, you're a few MB ahead. The class library just isn't setup for this kind of use.

Note that C#, Pascal don't "know" about known types during parsing like parsing C/C++ requires, it knows how a type is formatted, what it's looking for is: '<' ValidType (',' ValidType)+ '>'

If that's there, it's a generic, else it's an identifier, and somewhere up the parsing chain will take that up as LessThan.

Also note that Apple AArch64 (ie arm64) has a slightly different calling convention than Linux aarch64, which is going to influence pretty much any compiler/jit/runtime here.

Every few years I look at the latest parser combinators, parser generators, compiler compilers. And every time they seem to lack in some huge vital way (not always all of them, but always at least 1):

* Error handling always ends up being non-existent or of the quality of "begin, for, if, while, repeat, identifier, number, float expected" with no good way to override what happens

* Recovery is usually impossible

* Parser generator generates a full model that doesn't match what we need

* Working around the quirks of the input language ends up being more tricky than hand writing (almost every language has some ambigiuty)

* Slow: With ANTLR it's really easy to make it do gigantic amount of look aheads in complex languages, which isn't even really needed

I always end up going back to a simple hand crafted parser which is easier to read and write.

GitHub Codespaces 6 years ago

Gitlab looks nicer and has a lot more features even in the open source edition. (Though some paid features seem to be arbitrary paid vs free, like scoped tags, a feature everyone could use is paid, while a really Enterprise feature like kubernetes support is free)

I fully agree. I wrote my elements compilers this way. Slowly from a tokenizer, parser up to .net backend, Java backend. Bit by bit. The only way to learn is by doing.

I'd even go as far as saying the literature on the subject is overrated or out of date.

And not everything is set in stone. Not all compilers have a single symbol table. Sometimes it makes sense to have a type/global table and local table, and check the current class scope on the fly. Sometimes not. It doesn't really.

Abstraction doesn't make sense until you actually have multiple implementations. And you might have rewritten it four times by then.

My advice like above. Just start writing

I'm trying to write a drop in LLVM codegen replacement, ie something that takes bitcode (Which I already have) and generates x86_64, arm, object files etc. Back story: I've done compilers for most of my professional life but never did the actual native code generation myself, always using .net, java or llvm to do that part.

As a fun project, as I already had code to generate llvm bitcode from .NET, I now do mem2reg (convert stack spots to SSA registers), dead code elimination, constant folding and other small optimizations. That part now works, and I managed to create a simple x86_64 coff object file (with everything needed to link to it, including pdata, xdata) that returns the "max" value for a given integer.

That is about all that works for now, and I don't get to spend much time on it, but the end goal is to have a "good enough" codegenerator for non optimized cases, that could potentially be faster than llvm (to emit). The primary goal is to learn how to do this though :)

Right. To make RC work like a tracing GC (performant and without failing on circular references); it's going to be a combination. That said, there are a lot of interesting ideas in this that might be quite worth it. Like the article referenced, doesn't do a full heap scan, but makes a note of what is worth looking at.