kanaka

I wouldn't say it was entirely unintentional :-). I was definitely aware that "mal" could mean "evil" when I named it. It was a bit more apropos when mal only had a single implementation in GNU Make macro language.

Mal uses a regex for lexing/tokenizing. I didn't want people to get hung up on the lexing step (my university compilers class spent 1/3rd of the semester just on lexing). It's certainly a worthwhile area to study but not the focus of mal/make-a-lisp.

It's a long regex, but it's just whitespace followed by an alternation with 5 different types of data: split-unquote, special characters, strings, comments, symbols. The string tokenizing branch is a bit complicated because it has to allow internal escaping of quotes. Early iterations of the guide didn't explain the regex in detail but the section now describes each of the regex components.

There are online tools to help visualize regex's. Here is a recent tweet including a visualization of mal's tokenizer regex: https://twitter.com/Mehulwastaken/status/1382292764834996230

- There are several reasons for the weird way that the output and expected messages are printed. One of the main reasons is that in order to enable more flexible test case results, the expected output is a regex rather than a plain string, while the output is just a plain string. There is probably a way this could be made clearer. I'll add it to my TODO list to look at.

- Do you have some specific examples? The test cases are marked as either deferrable or optional. You shouldn't ever have to implement optional (and if so that's a bug in the tests or the guide). I think the deferrable items are marked pretty clearly in the guide where they become mandatory in later steps. If it's not clear, then that's a bug.

- This is one of the tensions that exists with trying to make the guide incremental; later features may require re-work of earlier functionality. I do try and minimize that as much as possible although I've found it can really vary depending on the nature of the target language. Note that the primary goal of mal/make-a-lisp is pedagogical (as opposed to say "the easiest way to make your own Lisp"). So sometimes the need to go back and re-work something is in line with that goal.

If you have any concrete guide text or test driver improvements (especially that further the pedagogic goals of mal), I'm always happy to review pull requests! :-)

The steps and main files are the same for every implementation (that's part of the requirements for merging into the main tree). Some implementations have additional files like readline, utility routines, etc. But for the most part the general structure and file divisions are very similar.

Every implementation has a stats target that gives byte counts, LOCs, and comments that is specific to that implementation. I.e. from the top-level you can run the following to get stats for every language:

  make stats

  make stats | egrep "Running|total|comments"

It started with the question "Could you implement a Lisp using just GNU Make macros"? (Hint: Yes) Bit of trivia: Mal originally stood for MAke-Lisp. Conveniently the acronym didn't need to be changed for "Make-A-Lisp"

It then grew into a personal learning tool for me for learning new languages. As I implemented more languages, I refactored the structure to make it more portable and incremental. At some point I realized that it might be an interesting learning tool for others so I wrote an early version of the guide and had a friend work through it and give feedback. He liked it enough that he immediately did a second implementation.

At some point the project got some twitter/HN attention and other people began contributing implementations and feedback for the guide.

Mal also serves as programming language "Rosetta Stone". A bit like rosettacode.org but using a full working project.

Implementation size is hard to compare accurately across languages (bytes? lines of code? exclude comments? excluding leading spaces?). Also, the mal implementations were created by many different people so individual style plays a large role in concision/verbosity.

However, that being said, the following implementations are "smallish" (in both lines of code and bytes): mal itself (self-hosted), Clojure, factor, Perl 6, Ruby. The following are "largish": Awk, PL-pgSQL, C, PL-SQL, Chuck, Swift 2, Ada.

OCaml is in the "smallest" third of the implementations.

BYOL is targeted at implementing Lisp in C and focuses on learning C. There is a fair bit more hand-holding and example code in C for most of the steps than in the mal guide. BYOL is more polished and written in the form of a short book whereas the mal guide intentionally tries to be more concise and informal in style. I would read through the first couple of sections of both and decide based on your own goals and preferences.

I look forwards to your pull request! One challenge you'll run into is that (as I recall) the Boehm GC linkage no longer works with newer versions of Boehm.

I made a quick gist of the current LOC/file size stats: https://gist.github.com/kanaka/fdef20f5d0c2e04b97d9106b2f2f2...

Note that this should obviously be taken with a grain of salt: the implementations were created by many different people each with their own style and differing experience levels with the target language. That being said, from my own experience, I think the concision (size) of the implementation often reflects how "Lispy" a language is. Most of the Lisps are in the top half of the list (fewest LOCs). Ruby is often described as very Lisp-like. Factor is one that stands out for me (second after mal itself). Factor also happens to be one of the fastest implementations (for certain microbenchmarks).

The problem with that command is that it also counts the Makefile, Dockerfile and other supporting files. It also counts every implementation step (so lots of repeats in the LOC). However, each implementation does have a "stats" rule. So the following command at the top-level can be used to a get a nice summary:

    for i in $(make print-IMPLS); do \
      s=$(make stats^$i | grep total); \
      printf "%-9s %s\n" "$i" "$s"; \
    done | sort -n -k2

Clojurescript.net is running an online version of the bootstrapped REPL. It works on at least some mobile devices I've checked.

pcwalton: thanks to Alex Crichton, the implementation has been cleaned up and updated to Rust 1.0.0 nightly. Yay! The feedback you noted has been applied to the code base. The performance has definitely improved some. Alex did some performance profiling and it looks like that hashmap performance may be the main bottleneck. The micro-benchmarks (and yes I admit they aren't good tests) don't really test strings, so it's probably the newer Rust version and the other cleanup that Alex did.

But if you want to take another pass through the code now that it's not based on an ancient Rust version, I'm always happy to take any advice from the master. :-)

Yes, it was intended to be somewhat in jest. "The expert" was a subtle (perhaps too subtle) acknowledgement of his role. Although, I am serious that if somebody with better Rust chops than I wants to fixup the code so that it represents Rust better, then I AM quite happy to pull those in and improve the implementation.

I'm rather hesitant to post any sort of benchmarks (since the existing ones suck so bad), but the using these, the lua 5.1.5 equivalents would be: 1, 1, 293

Lua does seem to be an odd one in that the short tests run quickly but this does not translate into iterations for the longer 10 second test. Perhaps the mal implementation is triggering bad GC behavior or something and that drags down the longer running tests. That's just speculation though. Again, please take the numbers with a mountain size grain of salt.

I'm quite happy to take PRs from an expert to address the issues and represent Rust better. :-)

UPDATE: I will point out that the README is pretty clear that this is rust 0.13. Doesn't mean it's a good representation of rust 0.13 either of course, but it clearly isn't based on a recent version of Rust.

Thanks for the feedback.

That did seem rather inefficient at the time but I wasn't able to discern the more efficient method at the time. I've kind of been waiting for Rust 1 to cycle back around. But I'll see if I might be able to address some of those and bump to Rust 1.0 alpha in the next few days.

Note the conversation about performance is kind of unfortunate. Those numbers should be considered VERY rough (they were just a personal notes file of mine). Also, with --release, the numbers place rust in the same range as other compiled languages.

Well, I'm not going to remove it. But I will see if I can find some time to improve it in the next few days. I've been meaning to cycle back around. Of course, fixes from somebody who's actually a Rust expert would have been preferred :-)

The reason it still uses the alternate pcre is because this still hasn't been fixed: https://github.com/rust-lang/regex/issues/28 I would love to get rid of that nastiness.

Yeah, the performance benchmarks currently suck. They are neither statistically valid, comprehensive, and often just run with default settings. That was really just a personal notes file that I shared with def- without clarifying that it really wasn't ready for publishing.

That chart (i.e. cloc) is actually missing 6 of the more interesting implementations:

- forth

- mal itself (e.g. self-hosting: all implementations can run this)

- miniMAL (Lisp interpreter in < 1024 bytes of JS)

- PostScript

- R

- Racket

The forth and OCaml implementations were written from scratch by chouser (Chris Houser of "Joy of Clojure"). The rest were written from scratch by yours truly (kanaka/Joel Martin). However, it looks like there are a number of other implementations happening by other people now that HN has made it more visible.

mal C implementation: https://github.com/kanaka/mal/tree/master/c

Caveat, no GC. Although that could probably be added easily using the Boehm GC. I plan to do so at some point assuming nobody beats me to it.

Good resource. Added.

Python is one of the oldest implementations and I ought to do a another pass through to clean it up sometime.

I'm happy to take pull requests that make an implementation shorter and/or more idiomatic as long as it doesn't change the overall order/flow (and doesn't break any of the tests of course).

"Guacamole has much better performance ... custom, better-performing protocol" -> What aspect of performance? Latency? Bandwidth? Server CPU and memory usage? Scaling? Do you have some actual performance benchmarks comparing the two? And does guacamole perform better when connecting to the same VNC server or is that comparing VNC server with an RDP server. RDP itself is a newer, more complex and more efficient protocol than VNC but VNC is supported pretty much everywhere. RDP could pretty easily be implemented in noVNC by somebody with the time (in fact it's been on the aspirational feature list for years).

With noVNC the protocol decode/encode is done in the browser (modern browsers are plenty fast enough to do this easily), whereas with Guacamole the burden of decode/encode for every client happens on the server where the proxy/client part is running. Even if you need to run websockify to proxy/bridge noVNC, the only thing it is doing is shuttling network traffic and the python implementation of websockify can easily handle lots of simultaneous clients without breaking a sweat.

noVNC was designed with Infrastructure as a Server (IaaS) providers in mind so minimizing server CPU, memory and bandwidth was a goal in the design.

Also, guacamole does use websockets when it is available.

The websocket support in QEMU allows you to connect with noVNC directly without using a websockify (proxy/bridge). You can still choose to connect to the regular VNC port using the proxy, but if you activate websockets then you can connect without it. The same is true now with libvncserver based VNC servers (e.g. x11vnc)

The essay isn't actually that recent.

And Clojure can target and interoperate with at least two of those stacks: Java (standard Clojure), Javascript (ClojureScript). And there is experimental support for C/C++ (clojurec, Ferret, etc). Not to mention in progress support for targeting CLR/C# (ClojureCLR) and python (clojure-py).

And hopefully "unofficial" CLJS-in-CLJS by Clojure West. :-)

I just pushed type/record/protocol/reify support. Next big thing is to fix runtime namespace creation. Then file I/O. Then all the bugs discovered while try to compile the compiler using the compiled compiler (i.e. self-hosting).