Steel – An embeddable and extensible Scheme dialect

245 184 3 years ago

This is just perfection on first glance, exactly what I've been looking for to augment my Rust projects that need a command/configuration language. And with a working repl, too, apparently.

I hope the second and third glances will be good too.

_nalply• 3 years ago

Helix Editor plans to use Steel as a plugin language [0].

[0] https://github.com/helix-editor/helix/pull/8675

agumonkey2• 3 years ago

emacsers looking deep from afar

NeutralForest4• 3 years ago

I would use Helix in the terminal if it supported Emacs keybindings tbh but I don't want to relearn another set of keybindings. Still I'd be interested in what it becomes.

nikolay6• 3 years ago

That's ridiculous!

icommentedtoday• 3 years ago

bobbyskelton414• 3 years ago

I saw your comment on the thread.

A. Most WebAssembly implementations are bigger than Helix itself.

B. WebAssembly is immature.

C. Lisp is perfectly fine. Even if you don't like it, it's not the end of the world if you have to use it to configure Helix.

zozbot2346• 3 years ago

It's perfectly viable as a MVP. WebAssembly plugins for Helix had been proposed many times but without an actual implementation being worked on.

CooCooCaCha5• 3 years ago

Why? The properties of lisp make it suitable for both a config file, and an extension language. It can be easily embedded into existing languages and is extremely flexible.

baq• 3 years ago

Certainly less ridiculous than YAML pretending to be a declarative key-value file store

sirsuki• 3 years ago

When I tried to introduce s-expressions to a DSL my co-workers nearly lynched me. The parenthesis were so violently hated I sunk into a deep hole and still haven’t came back out of it.

capableweb2• 3 years ago

It is really fun (sad) to see this, I've seen it so many times myself too. You show how something would be with s-expressions, compared to something, and all they can focus on is how many parenthesis there are. But when you sit down and count, they're the same amount as the code was when it wasn't s-expressions, just in different locations. And when you remove the parenthesis, they can kind of understand the code, kind of.

fwip2• 3 years ago

One option that might be suitable for a DSL, is implicit parens based on whitespace. A newline opens a new paren, and the paren closes when it reaches another line with the same indentation e.g:

  (defun factorial (x)
    (if (zerop x)
      1
      (* x (factorial (- x 1)))))

could be rewritten as

  defun factorial (x)
    if (zerop x)
      1
      * x (factorial (- x 1))

mtlmtlmtlmtl6• 3 years ago

It really is kind of frustrating isn't it? Because anyone who's put any real effort into learning Lisp knows that the parentheses are not what's hard to understand, actually. You don't even have to read them, really.

So when people complain loudly about parentheses in Lisp that just tells me they probably never made any real effort to learn it, and are actively opposed to trying.

It's not a productive starting point for a discussion.

alex_lav6• 3 years ago

Counterpoint: why are people under any obligation to try?

brundolf3• 3 years ago

I did it once in an internal UI because we needed to rapidly expose some functionality that could be composed/piped in complex ways and it would take a while to implement a normal UI

Was met with skepticism, esp around the engineering/maintenance overhead, until I told them the parser took me a couple hours to write and was only a hundred lines of code or so

e12e3• 3 years ago

Maybe:

https://srfi.schemers.org/srfi-49/srfi-49.html

https://www.draketo.de/software/wisp

e12e3• 3 years ago

Actually, there's a srfi for wisp:

https://srfi.schemers.org/srfi-119/srfi-119.html

darthrupert4• 3 years ago

The things I'm looking for in language syntax these days are

1. That it's unambiguous enough that my editor can format it correctly every time (provided the code is correct, obviously). Indentation-based languages like Python fail this.

2. That its elements and keywords are distinct enough that my editor can apply colours on it. I sometimes feel that S-expression languages fail at this because of their small number of distinct keywords, but that might not be true.

Besides these two points, if the language is not a joke language, it's probably fine.

xigoi5• 3 years ago

What is ambiguous about Python’s syntax?

oldpersonintx• 3 years ago

I want this to succeed!! please do not let the word "srfi" ever appear in the packages list...naming libraries with obscure numbers no one remembers was a terrible terrible idea that all schemes seem to perpetuate

mattparas19• 3 years ago

At the moment I only have 1 SRFI package, and its just to check against compatibility :) - my plan is to wrap the SRFI packages with friendly names and just include the metadata in the package spec so someone searching for it can find it easily.

zelphirkalt• 3 years ago

It is a standard. What do you expect? But there are some or many Schemes that offer their own libs on top or below SRFI implementations, which then have readable names.

adastra226• 3 years ago

I expect people to willing not conform when nonconformance improves the art.

aag• 3 years ago

SRFI editor here. The numeric designations are there because, among other reasons, there is sometimes more than one SRFI for a particular general idea. But there is nothing stopping libraries from having more than one name, including a semantically meaningful one. In fact, there has been a SRFI standardizing how that is done since 2008 [1].

[1] https://srfi.schemers.org/srfi-97/srfi-97.html

mattparas19• 3 years ago

This is a surprise! Steel is my project, happy to answer any questions anyone might have

dataangel• 3 years ago

Any chance of seeing the things I miss the most from Racket in other langs?

1. Parameters and Syntax Parameters (Syntax parameters make macros more powerful)

2. Turing complete macros (not just syntax-case)

3. Typed Racket

I almost used https://gamelisp.rs/ for a project but the nightly feature it needs broke and it's no longer maintained, glad to see something similar arise! You might want to consider adopting their choice of VecDeque as a list replacement, I think it makes a lot more sense than naive linked lists on modern machines.

mattparas19• 3 years ago

To answer each of these:

1. I do support parameters now, syntax parameters not yet. I would like to! But Racket has a pretty hefty head start on me so it'll take some time.

2. Right now I have syntax-rules macros, I also have defmacro style macros that get used internally in the kernel during expansion, but haven't yet opened them up for user space yet. Syntax case will be coming soon hopefully.

3. The odds of me being able to come up with an implementation to match typed racket pound for pound is pretty low. I have toyed with using contracts as types (where possible), with medium/promising success in certain situations. I have a soft spot for racket and have been modeling behavior after it, however it will take time to be able to create a macro system powerful enough to match it. It wouldn't be impossible to create an alternative syntax to just lower to steel after type checking, but I haven't put time into that.

On the list type - the list in use currently is an unrolled linked list https://github.com/mattwparas/im-lists, which I've found to yield much better performance for iteration than the naive linked lists. When possible, the vm does some in place mutation on the lists as well when consing, which helps performance as well. I also can hot swap for a vlist, but at the moment have stuck with unrolled linked lists.

soegaard2• 3 years ago

2. Turing complete macros (not just syntax-case)

I assume, you mean `syntax-rules` here.

If I understand correctly, it's pretty easy to simulate a paper tape with `syntax-rules` macro - that is `syntax-rules` macros are already "Turing complete".

https://okmij.org/ftp/Scheme/macros.html#turing-completeness...

But ... I know what you mean - in practise `syntax-case` or `syntax-parse` would be lovely to have.

duttakapil2• 3 years ago

May I ask, what is your personal journey of learning to code? Did you also discover Lisp/Scheme through SICP? And what have you professionally used Scheme for?

I am currently going through SICP, and I am also interested in Rust, so this project is a great discovery! Maybe I can contribute to it also.

mattparas19• 3 years ago

I learned to code primarily through school - I had the privilege of studying at Northwestern where a lot of the Racket people teach, so my first programming class was in Racket. I have worked through some of SICP and How to Design Programs. After Racket I learned some C, C++, and C#. Then taught myself python just independently doing some projects, ended up back taking a few classes in Racket, then one in Agda that got me down the programming language rabbit hole. Took a class in Rust and that got me working on Steel.

I haven't _directly_ used scheme professionally except for some steel scripts for automating some work flows and some racket programs for spark query plan analysis. I'd like to work in scheme more in my professional work, but for now I'm quite happy just working on it for fun.

Contributions are welcome! Feel free to either join the discord and ask questions there if you want a more chat based place, or open a discussion on github if you'd like to learn more. I have it on my TODO list to set up a matrix chat, just haven't gotten around to it - so apologies for having discord as the only chatroom.

JonChesterfield• 3 years ago

What are the immutable hash maps based on? How do you use functions or other maps as keys in them?

mattparas19• 3 years ago

They're using hash array mapped tries. I don't have my own personal implementation, I have been using https://github.com/bodil/im-rs until I can get around to making my own implementation (not that I really need to, but it would be a fun exercise).

Functions generate a hash based on a unique id generated for the function, plus the hash of any captured variables, and a hash of the pointer address to the function). That is off the top of my head though so I could be missing some details.

Hashing maps is tricky! With a sufficiently deep hash map you can run into problems since that invokes an equality check as well - at least how I handle it, is that you just attempt to naively hash the keys and values of the hash map, to create a hash code for that object. If the equality check ends up with a sufficiently large depth, eq returns false so we don't stack overflow.

falcor84• 3 years ago

Rust is gradually overcoming C and Fortran and Greenspun's tenth rule [0] is apparently inviolable.

[0] https://en.wikipedia.org/wiki/Greenspun%27s_tenth_rule

ghc• 3 years ago

Should have named it Steele since it's dragging Rust users half-way to Lisp :).

bitwize2• 3 years ago

Why would you want to do that? Lisp is dynamically typed and has a GC.

Jtsummers• 3 years ago

It's a joke that you seem to have missed.

We were not out to win over the Lisp programmers; we were after the C++ programmers. We managed to drag a lot of them about halfway to Lisp.

Guy Steele on Java

trealira• 3 years ago

Cases when you might prefer a garbage collected programming language rather than Rust:

- When garbage collection is the optimal or near optimal memory allocation pattern for your program.

- When you care less about minimizing memory usage than minimizing implementation complexity, development speed, and compile time speed.

- When your program deals with cyclic data structures.

- When you want green threads (like Goroutines)

I can say that I would rather use Lisp to write any program that manipulates tree or graph structures simply because it has a garbage collector.

wolverine8763• 3 years ago

Yes! I just posted

https://news.ycombinator.com/item?id=38510054

CooCooCaCha5• 3 years ago

How does Steel handle garbage collection? Would it be possible to manually control the garbage collector? For example, a game that runs the GC at the end of every frame.

mattparas19• 3 years ago

Immutable values are reference counted, so for most code, things will be dropped when they exit scope. For captured mutable values, there is a fairly mundane mark and sweep collector. It is possible to manually control the garbage collector, however I have not optimized it for that kind of workload. If you were embedding Steel in a game, I don't think it would be explicitly necessary to tune the GC as long as you aren't using a ton of mutation. If you were using a lot of mutation and still wanted a relatively performant GC collection at the end of every frame, then the underlying GC implementation would have to be changed or swapped to a different one (which is not impossible - I just only have one GC implemented)

mst7• 3 years ago

You might find the Recycler algorithm interesting - nim's ORC collector is ARC + Recycler and seems to be working out rather well.

CooCooCaCha5• 3 years ago

Thanks, I think swapping and controlling the GC would be a very useful feature.

In the game example I gave performance is important, but what's also important is consistency. Interactive apps rely on a steady framerate so what you want to avoid is accumulating garbage across multiple frames, then doing a single large collection pass.

In other words, it's better to do a bit of GC every frame than a bunch at once and risk stuttering.

wolverine8763• 3 years ago

I wonder where the name came from. Being HN, here is my nitpicking imperative:

Names are important. Our inheritance is wit, self-awareness, and irony; names that puncture ego and power and that appeal to joy: C, C++, GNU, Rust, Google, Yahoo!, Vim, Git, awk, etc. Others are beautiful, evocative images, like Apple and Amazon. Names communicate our culture and ideals to each other and to the next generation.

Careless, thoughtless names like Microsoft, IBM, etc. (ok it's ironic and self-deprecating, but without self-awareness or wit!), etc. should be hated and banned. Egotistical BS, especially Tolkien plagiarists who assert they are supernatural, should be tarred and feathered and paraded around town (with wit and irony).

(Plenty of names fall in some middle ground.)

If Steel is just a derivative of 'Rust' [edit: it is not, see the response below], it misses the self-awareness. Someone naming their development product - designed to build great structures - 'Rust' is engaging in a little self-deprication, joy, and self-awareness. Naming the derivative project 'Steel' possibly misses all that; there's a reason the original wasn't named Iron or Steel or Carbon Fiber. But maybe there's more to the name.

pasabagi• 3 years ago

I don't think it's particularly careless: SBCL is short for 'Steel Bank Common Lisp', which is a reference to how the CMU university, named after Carnegie Mellon, could be respectively substituted with 'Steel' (Carnegie) and 'Bank' (Mellon).

girvo• 3 years ago

I, somehow, never put together the SB -> CM link! That's awesome, thanks for sharing

mattparas19• 3 years ago

So here is the history:

1. Guy Steele (along with Gerald Sussman) created scheme, and Steel is close to Steele, just drop the e.

2. Steel is a scheme, and I observed that scheme names have a tendency to be named things crime related: Scheme, Racket (racketeering), Larceny, etc - Not a scientific analysis at all, but I found it funny at the time that Steel sounds like "steal".

3. You made the observation, Steel sounds like something that would be associated with Rust.

Beyond that, I just liked the name. No SEO involved, and arguably probably should have picked something more searchable, but I didn't start making it with the intention of there being a lot of users, it started as a project for school.

wolverine8763• 3 years ago

Awesome! Thank you. Someone else pointed out the Steele/Steel relationship and I was hopeful:

https://news.ycombinator.com/item?id=38510060

mst7• 3 years ago

Triple pun score!

(I always try and cram as many simultaneous jokes as possible into project names - and have a soft spot for lisp - so I wish you much joy and whatever level of success is most fun for you)

dannyobrien• 3 years ago

No judgement (I think great puns bring great minds together so namespace clashes are inevitable), but this was similar to the thinking behind a name for "Guile Steel", the low-level Scheme being plotted about by some of the Guile Scheme folks here: https://dustycloud.org/blog/guile-steel-proposal/

brundolf3• 3 years ago

An extra cheeky name might have been "stainless" (steel that doesn't rust) ;)

Y_Y• 3 years ago

beautiful, evocative images, like Apple

Is this a joke?

I_am_tiberius2• 3 years ago

It sounds great but what is it exactly and what can I use it for? Is there an equivalent of this in other languages?

mattparas19• 3 years ago

Hopefully the linked README provides a general overview (I know I need to write some more documentation!), but Steel is an implementation of the scheme programming language (not entirely compliant yet, but aiming for R5RS and R7RS compliance). It can be used as a standalone language via the interpreter/repl (like Python or Racket), or it can be embedded inside applications, like Lua. There are hundreds (thousands, probably) of embeddable languages, each with their own flavor - see a list compiled here for example https://github.com/dbohdan/embedded-scripting-languages

Use cases are generally for either configuration, scripting, or plugins - so scripting in games, or adding extensions to your text editor without having to use FFI or RPC + serializing a bunch of data. The advantage it has over using dynamic libraries (in general) is it runs in the same process, and can access the internal data structures directly without a lot of ceremony involved. The downside is that it is typically not as fast as native code unless a JIT is involved.

Javascript is an example of an embedded scripting, where the browser is the host application.

I_am_tiberius2• 3 years ago

Understood. Thank you for the summary!

tucnak• 3 years ago

Is it possible to avoid rust completely when programming using Steel?

mattparas19• 3 years ago

Yep - you can write standalone steel code without interacting with Rust at all, just interacting with the interpreter. I've done the first few days of the advent of code in Steel without needing to touch any Rust. Now, I will say that Steel has gotten visibility faster than I've been able to keep up with, so you might find a native function missing, or something that you would like to use that isn't implemented yet - at which point you either need to implement it yourself or open up an issue for someone to get to :)

schemescape3• 3 years ago

Solving Advent of Code problems in a language you implemented yourself should earn you an extra good star or something. Nice work, and cool project!

nerdponx2• 3 years ago

Which Scheme is this implementing? R7RS? Any SRFIs? Is there a library reference document?

mattparas19• 3 years ago

Starting with R5RS compliance, then once that is achieved moving on to R7RS.

I borrowed the R5RS test suite from chibi here - https://github.com/mattwparas/steel/blob/master/cogs/r5rs.sc... - only a few of these tests don't yet pass. Something like 135 pass, 4 fail, 20 skipped since I haven't implemented the primitives yet.

I've only tested against a few SRFIs so far, but am also attempting to run the R7RS benchmark suite https://ecraven.github.io/r7rs-benchmarks/ - So far you can view the progress here https://github.com/mattwparas/steel/tree/master/r7rs-benchma...

There are some more that aren't yet checked in. I plan to get a document up with the exact state of compliance soon.

The biggest difference right now is that, like Racket, Steel lists are immutable, so there I need a compatibility layer when running portable scheme.

nerdponx2• 3 years ago

Thanks! Was this originally a "just because" project, or do you expect it to offer some improvements or advantages over existing embeddable Schemes like Chibi (as you mentioned)?

perrygeo• 3 years ago

Fantastic project! If you're looking to embed a DSL/full programming language into your application, this seems right on point. Rhai and RustPython are two other options if you don't dig parentheses.

SushiHippie5• 3 years ago

Can someone explain these scheme and lisp languages to me? Every time I look at these languages, I can't grasp what you can use them for. And why one would use them.

I always feel like I'm missing something. In the example scripts and code snippets, I can see that you can define functions, that you can use lists, mathematical operations, you can build some algorithms, you can print text, but it never goes further than that.

I've only used languages like Python, Rust, C, Java, JavaScript, and they all have a very similar vibe, you have a std lib, which can interact with many things, you can build UIs, networking libraries and all that. And I could probably start using any language that is "similar" to these.

But I could never use one of these scheme/lisp languages, as I can't really grasp them.

Sorry, this comment is all over the place, because I can't really explain what's going on in my head when I see languages like this. I'd call myself a proficient programmer, but every time I look at these languages, it feels like as I've never seen code once in my lifetime.

Any help or hint at what I'm missing, is appreciated.

femiagbabiaka• 3 years ago

Basically the differences are in the concepts you'll use to write code. Lisps themselves are very different from each other, but just like the languages you're used to, many lisp distributions have standard libraries that can be called, and those building blocks can be used to build applications or whatever else. In this case specifically, Steel provides the facility to call Rust functions within a Steel program: https://github.com/mattwparas/steel.

So, although I haven't used Steel, it looks like the advantage you'd get from using it is the opportunity to take advantage of features it provides like transducers and contracts, which are feature common to some other Lisps as well.

So, just like choosing any other language, it boils down to a series of tradeoffs.

tmtvl3• 3 years ago

My first introduction to Lisp-like programming languages were the SICP videos with Abelson and Sussman so to me Lisps are just programming languages like any other but with a nicer syntax and great support for interactive programming.

globular-toast• 3 years ago

Why not just program everything in assembly? High level languages give you powers of expression that lead to better programs or are simply more convenient for the particular task at hand. Lisp languages have several features that aren't found in most other programming languages. And a lot features that are found in other languages originated in Lisp.

Perhaps what you are missing is the practical part. For that you should look to the two types of Lisp in widespread usage: Common Lisp and Emacs Lisp. Common Lisp is a general-purpose language with a very large standard library and rich set of third-party libraries. Emacs Lisp is a complete language, but only really used to build text editing like stuff for Emacs. There's tons of real, effective code out there in these languages.

Or perhaps you are confused about the functional programming part. This is only strongly associated with Scheme as other Lisps support other paradigms like object-oriented programming. Functional programming is a thing that takes a while to understand, although my theory is it's actually more natural and it's only because you already learnt imperative programming, which is thoroughly unnatural, that you find it odd. Once grasped it will help you with other languages that support functional programming like JavaScript and Python.

They say learning Lisp will make you a better programmer even if you never use it again. I tend to agree with this.

SushiHippie5• 3 years ago

Perhaps what you are missing is the practical part.

I think that's it. Thanks, I'll take a look at common lisp, especially at projects written in it.

e12e3• 3 years ago

Maybe look at: https://github.com/arclanguage/anarki/blob/master/apps/news/...

SushiHippie5• 3 years ago

Thanks, good hint!

dkarras• 3 years ago

used to be on the same camp. now I think I get it.

the language is malleable. because it is homoiconic. so while developing software, you are simultaneously writing a domain specific language for your problem. because macros.

in the end, if you like your craft, you end up with a "language" that is very suitable for solving the problem you have at hand, with very little noise.

the downside is, probably most others will not understand your code so lisps have heavy bias towards "solo hackers". BUT... if that is important to you, you can code towards understandability. so much so that you can make it very hard for others to make mistakes when using the public api.

so with most programming languages, you program within the constraints of the syntactic rules of the language. with lisps, you define the language you want to approach the problem, that language comes out by itself iteratively. for some, that is a joy. others don't care for it.

dboreham3• 3 years ago

Homoiconicity is a gimmick and it's a cult.

neilv• 3 years ago

it feels like as I've never seen code once in my lifetime.

I think you must be fixating on syntax, or have been seeing some code that is advanced or (it exists) a very confusing example by some academic to demonstrate some curiosity.

Except for idiomatic recursion (which you don't have to use), Scheme semantics should initially look familiar to a Python or JavaScript programmer, like a subset of that, just with a different syntax. (Scheme nuances are much better designed, but to a new programmer semantics will look like a subset.)

And the Scheme syntax is one of the simplest ever, once you understand it.

What should instead be confusing is a language with very different semantics, like lazy evaluation, or an OOPL with complicating dispatch rules to reason about.

dboreham3• 3 years ago

You're not missing anything. You have two choices: a) either take it as true that there's nothing to see here and move on, b) or if you have some spare time learn lisp and then move on. (b) avoids that nagging feeling that you're somehow unworthy.

mattparas19• 3 years ago

At least for me, there is a certain appeal in building the world. With scheme you get a very small set of functionality, but you use that to implement the rest of the language, and build abstractions on top of abstractions. Seeing how the whole system can be built from a small set of functionality is pretty cool, and also very satisfying. There is a talk from Andy Wingo, one of the maintainers of Guile, where he describes working on Guile as akin to tending to a garden, and I think its an apt comparison. Something about it feels very organic and personal, which is part of the appeal.

The syntax itself doesn't _really_ matter, it just makes it easy to do so - functions and syntax visually look the same, so it makes it easy to build.

Its not for everyone, but I think its worth exploring for a little bit. Similarly I think its worth really learning any language just a bit, if not to just expand your tool kit. The parenthesis do disappear at a certain point and you learn to read it, but if its not your thing that is fine.

sparkie• 3 years ago

Would strongly recommend watching the SICP video lectures: https://ocw.mit.edu/courses/6-001-structure-and-interpretati...

SushiHippie5• 3 years ago

Ah, thanks, that's what @tmtvl was talking about

brundolf3• 3 years ago

Most of them use most of the same core idioms you're familiar with from other languages, they just look superficially different because of the syntax (especially the parenthesis placement)

Try moving the left parenthesis of each expression over to the right by one and it may become clearer

(my-fn 1 2)

my-fn (1 2)

trenchgun• 3 years ago

you have a std lib, which can interact with many things, you can build UIs, networking libraries and all that.

Racket has all of that: https://docs.racket-lang.org

Common Lisp too: https://common-lisp.net/libraries

Clojure too: https://www.clojure-toolbox.com

(Or you could even just build anything within Emacs with Emacs Lisp: https://en.wikipedia.org/wiki/Emacs_Lisp)

And I could probably start using any language that is "similar" to these.

Then you probably could start also using any of the Lisps mentioned. Most likely you are just hung up on the surface syntax, which does not take long to get used to.

silent_cal• 3 years ago

Maybe a dumb question but does embeddable mean embeddable in web browsers? If so it would be awesome to include this in a web-based version of SICP.

Steel – An embeddable and extensible Scheme dialect

Related Stories