SBCL: The Assembly Code Breadboard

https://pvk.ca/Blog/2014/03/15/sbcl-the-ultimate-assembly-code-breadboard/

by mrh57 • 2 years ago

81 19 2 years ago

I used SBCL for an algorithmic trading framework for EXACTLY this reason.

More precisely, this was SBCL/ANSI C application where most of the application was implemented in Common Lisp, some OS / low level stuff was written in ANSI C and same things were written in Lisp to compile to machine code and inserted into "C world".

There would be no Common Lisp virtual machine on the critical, low latency path. Instead, the high level code would be used to configure and construct the low lever path. For example, I would have a bunch of Lisp code to create decision trees, then get them simplified and compiled to machine code, then machine code would be inserted into the path of oncoming market data.

Or Common Lisp code would be used to take a bunch of XML files with descriptions of binary messages in XDP format, but then those macros would be used to generate efficient machine code for parsing and serialising messages. I actually stole this idea wholesale from fantastic book Practical Common Lisp (thank you Peter Seibel!)

SBCL was wonderful to work and tinker with. Love cffi and vops and the macros that bring it all together.

The sad part is that it was impossible to get anybody else working on it and I had to rewrite it in Java (and destroy performance in the process).

Such is the curse of Lisp, unfortunately. And polyglot programming is not really for large corporations. It is hard enough to find people good at one specific programming language, it is almost impossible to find somebody who will be good at both extremely low level (like understand what the CPU does when it gets an instruction) as well as the highest level (complex abstractions with Lisp macros, etc.) for a project where pretty much everything is custom.

guenthert• 2 years ago

The sad part is that it was impossible to get anybody else working on it and I had to rewrite it in Java (and destroy performance in the process).

Java has a CFFI and while it ain't pretty, I don't see where the performance impact would come from, when machine code is to be injected and the critical path is in machine code / C.

extremely low level (like understand what the CPU does when it gets an instruction) as well as the highest level (complex abstractions with Lisp macros, etc.)

Those skills don't seem all that far apart. I'd rather think it'll be difficult to find someone willing/able to encode and maintain financial / legal safety limits in machine code :-}

reikonomusha3• 2 years ago

I can't speak for onetimeuse92304, but it's probably because the program which generates machine code was in Common Lisp and SBCL using the techniques FTA, and it would basically be impossible to port in any reasonable way. Experimenting with and deploying assembly code generation in SBCL is very organized and flexible—much better in Lisp than in C even.

reikonomusha3• 2 years ago

Very cool story; thanks for sharing!

I think a major fault in the industry is "find somebody who's good at X". It's not a necessary or even optimal requirement. Common Lisp doesn't have too many particularly odd language features; I'd say learning the Rust borrow checker or Scheme's call/cc is more difficult than anything Lisp has to offer.

What organizations need to do is prepare to let smart engineers self-train on the language as a part of on-boarding, instead of making it a hard job requirement to know the tools before walking through the door. In my experience, even for good programmers who are pretty green in their career, are entirely productive in just a few weeks—maybe a month total. That includes learning a new language, a new editor, and a part of a new code base.

Perhaps counterintuitively, the hardest people to teach Common Lisp to in my experience have been "senior software engineers" who transition into the project, who are so dead-set on their preferred languages and tools that they're entirely unwilling to even try something new, even if they're being paid to do it. Instead, sometimes, they'll find it preferable to do political maneuvering to avoid it, either by carving out a new module in their language, by petitioning to rewrite it, or by prioritizing other things entirely.

onetimeuse923043• 2 years ago

Perhaps counterintuitively, the hardest people to teach Common Lisp to in my experience have been "senior software engineers" who transition into the project, who are so dead-set on their preferred languages and tools

That's essentially my experience. There is something about developers... they learn a new language but most people never really learn a new way of thinking.

I worked with one guy who tried to learn Common Lisp and even after a year on the project his code was essentially Java written in Lisp.

But this isn't really a problem with Lisp, specifically. I had the same problem with pretty much every programming language. Just recently I had fun debating the usefulness of Kotlin as a programming language. In my experience, for all the features Kotlin has, every single Kotlin program I have ever seen looks like a Java program translated line by line to Kotlin. Which is to say none of the Kotlin features are actually used to solve any important problems like reducing complexity of the application.

Common Lisp doesn't have too many particularly odd language features;

I would say for most developers I worked with the biggest challenge was always getting the idea of macros. There is technically no reason to use macros. But if you do not use macros you are not really programming Lisp.

The point of macros is the reversal of the programming process. In a regular programming language, you get a bunch of tools and then you learn the tools and then you figure out how to write program using the tools.

In Lisp, you figure out how you want your program to be written, and then make tools that make it possible. It is quite a big shift in thinking that many people have trouble with.

Once I got this, I am now using similar technique in other programming languages. I am looking at my problem and extracting the core complexity and try to figure out what would be the best way to write that complexity down as code. Then I figure out all the machinery to make it happen. It not as elegant as in lisp but it still leads to way better and more readable code at least where it matters.

_19qg• 2 years ago

"Anyone could learn Lisp in one day, except that if they already knew Fortran, it would take three days." — Marvin Minsky

dan-robertson• 2 years ago

What does “destroy performance in the process” mean? Was the performance being destroyed the performance of something that needed to be fast?

duudhwbwka• 2 years ago

Bravo

chrsig2• 2 years ago

SBCL has definitely proven itself to be a good companion to explore the generation of domain-specific machine code. I don’t know of any other language implementation with that kind of support for interactive programming and machine code generation (and inspection). FWIW, I believe LuaJIT + dynasm will soon be comparable.

Julia! It's very lispy, actually, but it does a very good job of exposing different layers of parsing & compilation through llvm

see the interactive utilities documentation[0] -- there are a few macros and functions, prefixed with code_, that output the generated code

[0] https://docs.julialang.org/en/v1/stdlib/InteractiveUtils/#In...

epgui• 2 years ago

I really don't understand what would make someone say that Julia is lispy. People say that about R as well, but I don't see that either.

To me, lisp is many things, but the first thing it is is s-expressions. Most/many languages have macros. Lisp is just particularly good at those...

klyrs• 2 years ago

I didn't interpret OC as calling Julia lispy, as much as saying that Julia is matched to SBCL's interactive machine code generation and inspection. I don't have a dog in that particular fight; I've barely touched either language.

Except, macros. I have never seen macro wizardry come anywhere near SBCL without homoiconicity.

edit: pre-coffee reading comprehension, apparently. The word lispy is _right there_ in OC.

dlopezjr• 2 years ago

Julia[0] shares features with Dylan[1] which was an attempt at designing a Lisp-like language without s-expressions.

[0] https://en.wikipedia.org/wiki/Julia_(programming_language)#L...

[1] https://en.m.wikipedia.org/wiki/Dylan_(programming_language)

Archit3ch• 2 years ago

From the Julia side, Common Lisp [0] feels like writing Julia. Both the code itself and the development experience.

[0] https://youtu.be/y5Rn41fPIuQ

reikonomusha3• 2 years ago

Common Lisp has DISASSEMBLE built-in to the language standard. You can take any function, even standard library functions, and inspect their disassembly.

SBCL goes further with two cool features:

1. It annotates the disassembly with your Lisp code. Sort of like godbolt, you can see how every line/chunk of your Lisp function gets translated to machine code.

2. It annotates the disassembly with profiler results. You can see the exact instructions the CPU is stalled at/hitting the most, directly inline in the disassembly.

This (and more) is all interactive and built-in. No additional installs. No additional tooling. No "re-run in debug mode".

These kinds of things make SBCL a really appealing choice for even extremely low-level programming.

jlarocco• 2 years ago

A "real life" example of this can be found in the Ironclad cryptography library [0]. It uses this to add support for AES instructions and some other niche assembly instructions to generate fast (at least a lot faster than it would have been) cryptography code.

https://github.com/sharplispers/ironclad/blob/master/src/opt...

phoe-krk• 2 years ago

(2014)

SBCL: The Assembly Code Breadboard

Related Stories