Building a language server for Muon

https://nickmqb.github.io/2019/11/24/building-a-language-server-for-muon.html

by nickmqb • 7 years ago

101 54 7 years ago

nickmqb.github.io

jimws• 7 years ago

Lisp had this kind of interactive programming support for a long time using SLIME and Emacs. Why isn't this a more common feature in the more recent languages?

susam• 7 years ago

Lisp had this kind of interactive programming support for a long time using SLIME and Emacs.

For those like me who have Vim burnt into their fingers and brain, there is Slimv[1] for Vim. Another alternative is Vlime[2] for Vim. Both these plugins are based on the same client-server architecture that SLIME is based on. In fact, these plugins rely on Swank TCP server (the same thing that SLIME also relies on). Swank receives SLIME commands from Slimv or Vlime and executes them. Of these two Vim plugins, I prefer Slimv personally because apart from supporting Common Lisp, it supports MIT Scheme and Clojure too. Vlime supports Common Lisp only at this time.

[1] https://github.com/kovisoft/slimv

[2] https://github.com/l04m33/vlime

dunefox• 7 years ago

Or Spacemacs with Slime.

giancarlostoro2• 7 years ago

For that matter, VB, C#, Java and even Delphi all had a RAD approach to developing UIs. Why don't more modern languages work on even a basic UI as part of their standard libraries. What kind of system doesn't have a UI these days? Outside of servers obviously. The closest Go and Rust (modern languages...) are getting to a UI seems to be WebAssembly, which for now points towards an Electron UI solution.

I'm not super familiar with SLIME, but that's the goals of LangServ is to consolidate everything into a known protocol. There's also a spec for debugging.

Oddly enough the Wikipedia page for SLIME doesn't seem to mention Racket[0], but it does mention Clojure.

[0] https://en.wikipedia.org/wiki/SLIME

gmueckl2• 7 years ago

A flexible UI library that is up to modern expectations is complex and hard. It is even harder if it is supposed to be cross platform. So the only realistic option is to bind to a good existing library. However, these are all written in C++ and hard to bind to anything else. Gtk isn't sufficiently cross-platform to be attractive.

In the end, this is a project that is at least as much effort a creating a programming language.

oaiey4• 7 years ago

There are two aspects here: Interactive Development is incredible difficult. There is a talk of Anders Heijlsberg about that compiler building is no longer the same than 10 years ago. When you code, you need sub-100-ms reactions, the parser needs to be happy parsing incomplete, typically errorous code and then you need a code analysis engine dealing with that half-baked AST. Also traditional compilers are a strict pipeline and not a loop etc. Microsoft had to rewrite the C# compiler for that. The PHP LSP had to create an own parser engine because no suitable was available. It is essentially a lecture for future language creator: If you write a compiler, do not only built it for binary emitting but also for editors and analyzers.

And yes, the smart-asses of the 70/80s (aka LISP and Smalltalk) thought about that long time before anyone else. But hey, today's landscape is a bit more complex (think about CSS styles in a HTML embedded in a TypeScript enriched JavaScript JSX file).

The second aspect: LSP is great because it was a cross-vendor, cross-language initiative which had in its first year already two dozen languages on board in addition to half a dozen editors (VS Code, Emacs, Atom, VIM, ...). Emacs plugins could never deliver that because they were bound to Emacs.

TeMPOraL• 7 years ago

RE the second aspect: that's why I'm a fan of LSP and the related debugging server protocol. There's no way Emacs can have native IDE-like support for all the languages out there, and the small developer base makes it hard to compete with IDEs for popular languages, that have commercial interests backing them. LSP promises to level the playing field - it only takes one good language server to make IDE features suddenly available not just for IDEs, but also Emacs and Vim and others.

laumars3• 7 years ago

It's actually really common but the problem was that, until recently, it was an IDE specific feature and some of the best IDEs weren't free.

What this article is about is the new(ish) open protocol originally developed for Visual Studio Code that is being adopted by quite a few other editors. The language server protocol (LSP) enables developers to write the kind of tools you're describing and have them support any editor (which has LSP support) on any OS. It's pretty cool. However it's also a fairly involved process writing one so many of the smaller languages haven't yet had the bandwidth to writing their own language servers.

barrkel• 7 years ago

The problem is much easier (at least 10x easier) for a dynamic language with a REPL or playground environment (scratch in Emacs) than it is with a static language, or providing stateless insight based on cursor location rather than the set of forms you've evaluated.

With a REPL or playground, you've got all the libraries loaded, all the function definitions in a map in your interpreter, and if you have a partial identifier you can simply do a prefix search in the interpreter. The interpreter is live; define a new function, and it's added to the map, so it's available for subsequent completions. If the function definition is wrong, hey, it's a REPL, next line try again please.

In a stateless situation, or for a static language, you need to build that map. Ideally you use the compiler's front end, but then the front end needs to be hardened to recover properly from errors. It needs to be told about the cursor location, and when it hits the cursor location in the middle of an incomplete symbol, it needs to figure out what's appropriate to complete at that point based on the stack of scopes available, and jump back out to the editor (or use coroutines, or pause its thread, or whatever) with what it found. It's simply more work. And if you can't use the front end, because the front end wasn't built for it, and you can't modify it, well you're going to have to build what is effectively a new front end to solve the problem.

UglyToad5• 7 years ago

An interesting article.

I was minded to write an Ask HN related to this. Why do so many new languages launch without what I'd consider to be essential functionality for any language; autocomplete, good IDE integration, intellisense, auto refactoring and navigation support?

I get that all languages are just text at the end of the day but are people really holding the whole API surface and type arguments of standard libraries and 3rd party libraries in their head? Do they remember variable names across files and directories. I feel like my memory must be absolutely terrible because there's no way I could work in a language without decent autocomplete.

codetrotter• 7 years ago

Why do so many new languages launch without what I'd consider to be essential functionality for any language; autocomplete, good IDE integration, intellisense, auto refactoring and navigation support?

Probably because developing all of that stuff takes a lot of time still, even though it is probably easier now than it was before.

Everything gets done eventually if it’s important, but it may take a while until something gets done even though it is important because there is so much else to do as well.

oaiey4• 7 years ago

Because you either invest in language features or editor features.

I agree but most languages are built by single persons or very small teams. They have to select their battles. The last languages developed with that in mind were in order: TypeScript (essentially only for editors), Kotlin (developed by an IDE company) and C# (developed in a HUGE team including a huge IDE team next office door).

Quenty• 7 years ago

Notably both C# and TypeScript are developed in the same Microsoft division as Visual Studio and VSCode. :p

cinnamonheart• 7 years ago

It depends on the audience and goals for your language. If your language is intended to be a playground for new ideas, like a new type system or semantic model, you may only be concerned about getting that working and into other peoples' hands for experimentation, rather than usability.

UglyToad5• 7 years ago

I can certainly see why you'd not take the time to/need to do it if the purpose of the language is to explore concepts in language design.

But it seems to be missing from a lot of languages that launch with the intention of being used for large non-trivial multi-user codebases.

My question I guess was intended less as a criticism of language designers for not including these features and more to find out what I'm missing, that I'm not able to remember the entire (for example) Python standard library, or even the name of methods and classes I previously implemented.

skocznymroczny• 7 years ago

Many projects are open-source, developed by the community. It comes with several caveats:

1) many open-source developers use vim/emacs, maybe something like Sublime Text and don't have experience with IDE offerings for languages like Java/C#. As such, they don't really require much more than syntax highlighting and basic autocompletion.

2) many of the new languages boast powerful metaprogramming capabilities. Metaprogramming makes it very hard to implement working refactoring, because the IDE cannot reason about the code as well as in more 'static' languages like Java.

3) most new languages coming out are closer to the functional paradigm rather than OOP. I think OOP lends itself much more for refactoring features and things like autocomplete.

saagarjha2• 7 years ago

I have yet to find an actual Java autocomplete plug-in that’s not just “we’re running Eclipse in the background over your code”. Just because it’s highly static and regular doesn’t mean it has good tooling outside of IDEs!

UglyToad5• 7 years ago

I think part of my question was related to your part 1. Are people genuinely remembering the entire API surface of their application, their language's standard library and all 3rd party dependencies for their code when using a text-mode editor like Vim?

The only way I could possibly work like that is if my program wasn't bigger than a couple of files not exceeding 50 lines or if I spent all day in the API docs.

It has me worried I've just got incredibly early onset memory loss I guess.

pjmlp• 7 years ago

Lisp and Smalltalk practically invented the IDE concept, they have more than enough metaprogramming capabilities.

In fact the first refactoring tools were developed on Smalltalk.

ssivark• 7 years ago

Why do so many new languages launch without what I'd consider to be essential functionality for any language; autocomplete, good IDE integration, intellisense, auto refactoring and navigation support?

Because their target audience at launch time is not "end-users", but to create a community to share with the task of language building. And that is a very different kind of project compared to using a language to write applications.

It is not meant to be a functionally complete product, but a demo of what could be the kernel of a really cool product (say, v0.1) -- for purposes of recruiting a team/investment to help carry out that vision and actually ship a useful product (v1.0).

UglyToad5• 7 years ago

I can certainly see this is the case for new languages intended to introduce new and paradigm shifting concepts.

To move the goalposts somewhat I primarily had languages like say Python, Elm, maybe Rust or Go, in mind. I haven't done Python in ages so this is probably an unfair judgement in light of tools like PyCharm but it still seems its supported use-case is text-mode script writing and the ecosystem is still lacking when it comes to large-scale application development. Yet it seems to be far more popular (on here and from a developer happiness standpoint) for application development versus languages which provide tools specialized for this (Java, Kotlin, C#, etc).

daxfohl2• 7 years ago

You have to pick your battles. But I'll say that the Language Server Protocol is a game changer. Now at least the battle is communicating with a single well documented high level editor protocol that works everywhere. Prior you'd have to first choose an editor to integrate with, chug through far lower level details of how to color code, make dropdowns, etc, and do it in the editor's plugin language rather than your compiler's. My guess is you'll see more editor integration for new languages in the future if LSP stays relevant.

In my own little toy languages, integration with LSP is one of the first things I do after I have the basis for a hello world. So, before if statements or loops. It makes a nice feeling to be able to see your language get to look like a professional thing so rapidly. Also I think it helps in developing the compiler, that you know in advance what functionality it needs to have in order to provide good editor support. Also helps guide what unit tests should look like, and helps find bugs because they are apparent as you type.

So we could see an explosion of domain languages, since they are usually more about tooling, and with tooling a nice editor integration is very helpful.

nickmqb6• 7 years ago

To step back a bit, the way I see it, the main goal as a language designer is to ensure that users can be productive in the language. Many factors play a role here. Language features are an obvious one, but there's also (in no particular order): standard library, ease of integration with other code, documentation, community, tooling, etc. That's a lot, so as daxfohl says, picking your battles is key. Not everyone will agree on the relative priorities of these factors, and that's totally fine too. However, I personally think that tooling is (one of the) super important factors. "Fast & snappy tools" is actually one of the explicit design goals for Muon [1] (design principle #12). So releasing a language server so early on in the life of the language underscores that, yes, Muon is a language with a strong tooling focus, and I hope that resonates with people.

[1] https://github.com/nickmqb/muon

shakna• 7 years ago

I've spent three years working on/off on a language. It isn't ready for prime time yet, but I'm beginning to think about the editor side of things because I can now write significant sized programs in it.

But I can understand not wanting to.

I need to go learn an API, and learn how to express the grammar of my language in a new way because I probably can't just reuse the highly tuned recursive descent parser that's already part of the language:

Parts of my language aren't well expressed by some of the ways an LSP is expected to work, stuff inside blocks is only verified as valid, not parsed to check their types or construction. So even a "jump to definition" or "this is clearly an error" is going to involve a lot more work than the interpreter actually does.

Autocomplete itself could be a huge headache due to the highly dynamic nature of the language (variable names can be as dynamically created), without actually running the program, guessing what's inside the environment is going to be incomplete.

In short - editor support is either going to be half-assed (why do it), or more work than writing the language was (tedious).

annywhey• 7 years ago

Most of the effort in making a language a production language is on the tools and libraries end of things, and LSP is sort of the tip of the iceberg in terms of getting into that stuff. Before that, you probably want to have features like "good error messages" or "working string and math libraries".

For a long period in the past 15-odd years, new "Web" languages were getting plenty of adoption because the state of the tooling for that segment remained barebones everywhere, with a lot of functionality already in SQL or JS and anything in between being glue, and so competition on language features and syntax took precedent. It's probably in a consolidation phase now - things are getting more exciting in the lower layers of the stack instead.

otabdeveloper2• 7 years ago

Why do so many new languages launch without what I'd consider to be essential functionality for any language

Maybe because what 'you consider' is not a universal metric of utility?

I, personally, care not one whit about the useless stuff you mentioned. (Probably because I never programmed in Java and intend to keep it that way while I live.)

ktm5j2• 7 years ago

I thought this article was about "building a language server" like the title says, but this is really just a grammar set for an existing language server.. unless I'm missing something? I realize that this is being picky about wording, but it's a little misleading.

_bxg1• 7 years ago

My impression was that it's glue code that wraps around an existing compiler and turns it into a language server. There are piles of resources out there already on writing compilers

laumars3• 7 years ago

Is there any good resources on writing language servers? I've tried writing my own one but while LSP is well documented, it wasn't really beginner friendly (ie where do I start when writing one from scratch?).

nickmqb6• 7 years ago

I didn't look for any resources beyond the LSP spec and VS Code documentation, so I can't offer any recommendations here I'm afraid. That said, looking back I'm pretty happy with the choice of using VS Code as the initial editor to get things up and running. While not perfect (e.g. VS Code will silently ignore your JSON message if you miss a closing brace), the docs are pretty good and it offers a reasonable level of debugging functionality (such as an output window where you can see messages sent to the language server). The Muon VS Code extension [1] is pretty minimal in itself, and it's MIT licensed so feel free fork and use it as a starting point.

[1] https://github.com/nickmqb/vscode-muon

ahaferburg• 7 years ago

The Sublime Text LSP plugin has some debugging facilities, you can poke it however you like since it's written in Python, and the project is run by two of the most positive, supportive, and just plain nice people that I've ever dealt with in an open source project.

https://github.com/tomv564/LSP

nickmqb6• 7 years ago

Author here. I wrote both the language server and Muon compiler [1] from scratch. The language server reuses the compiler code, for parsing and type checking. I'm not sure which grammar you're referring to? The VS Code extension [2] does contain a grammar, but it's only used for syntax highlighting.

[1] https://github.com/nickmqb/muon/tree/master/compiler [2] https://github.com/nickmqb/vscode-muon

tasogare• 7 years ago

The properties of the language listed on the websites are compelling, but I wonder what are you using the language for except the compiler?

ktm5j2• 7 years ago

Sorry for the misunderstanding, I didn't read clearly.. my apologies. Very cool!

duckqlz• 7 years ago

Moun’s for loops make me angry.

all2• 7 years ago

Can you expand on this?

saagarjha2• 7 years ago

I’m also confused; Muon’s loop syntax and semantics looks nothing out of the ordinary except that it allows omitting expressions…

lioeters• 7 years ago

From "Muon by example" [0], the for loops look pretty natural/intuitive subjectively.

  for i := 1; i < n {
    temp := a
    a += b
    b = temp
  }

> `for` works mostly like in other imperative languages. For convenience, the 3rd term can be omitted; in that case, the index variable is incremented by 1 at the end of each iteration of the loop. The index variable is scoped to the loop body. The variable can be omitted if you want to use an existing index variable, e.g.: for ; i < n; i += 1 { ... }

With arrays:

  for x in arr {
    sum += x
  }

  for x, i in arr {
    sum += x
  }

  for arr {
    sum += it
  }

> A secondary loop variable may be specified which holds the index of the current element. If no loop variable is specified, the name `it` is used.

[0] https://github.com/nickmqb/muon/blob/master/docs/muon_by_exa...

ngcc_hk• 7 years ago

Good article. In fact the muon is very doc. Still, one of the curse when coping segment of python code is to handle those tab and white space. The muon help to have a feature (first line say tab or space). Sigh muon opt for that turn me off.

nickmqb6• 7 years ago

I agree that Muon's current handling of whitespace is not ideal. I have plans to allow users to customize it. For now, you can use the //tab_size=N directive as documented here: https://github.com/nickmqb/muon/blob/master/docs/muon_by_exa...

Building a language server for Muon

Related Stories