Clasp – A Common Lisp with LLVM back end and interoperation with C++

http://drmeister.wordpress.com/2014/09/18/announcing-clasp/

by drmeister • 12 years ago

278 74 12 years ago

drmeister.wordpress.com

eudox4• 12 years ago

Programmers can expose C++ classes as well as functions and class methods with a single line of code that provides the Clasp name and a pointer to the function/method.

Hellooooo Qt without Smoke bindings.

pnathan• 12 years ago

It would be really shiny to have a cross-platform GUI library in CL without grief. I've prodded that space repeatedly in the last 6 years and simply have not come away happy about any of it except LispWork's library.

wtbob2• 12 years ago

This is extremely exciting, particularly if it can be made to work with emscripten. Common Lisp in the browser, here we come!

aidenn06• 12 years ago

Almost certainly can't. The closest you can get is using clicc or ECL or something similar to generate C code, and then compile that with emscripten.

This invokes llvm at runtime, and AFAIK emscripten isn't ported to emscripten.

FWIW, I've tried other lisps under emscripten:

Most lisps generate machine code, store them in RAM, and then execute that RAM. This is not possible under emscripten.

Clisp is a good candidate, since it's byte-code interpreted rather than generating machine code, but clisp makes so many assumptions about how the machine works (in particular it strongly wants a C style stack and does manual stack-pointer manipulation). I actually got fairly far into the bootstrap process under emscripten, but the minimal lisp interpreter it compiles generated bizarre errors.

drmeister9• 12 years ago

I disagree. Clasp could do this because it compiles Common Lisp to LLVM-IR bitcode files (using COMPILE-FILE) as well as directly to native code using LLVM's MCJIT engine (using COMPILE). emscripten (https://github.com/kripken/emscripten) says that it compiles LLVM-IR to JavaScript. I haven't used emscripten, but I believe everything I read on the internet :-) and thus Common Lisp --[Clasp]--> LLVM-IR bitcode files --[emscripten]--> run within browsers.

Igglyboo2• 12 years ago

Sorry to go offtopic but since the you mentioned emscripten I though I'd ask.

Can you explain what emscripten is used for? I'm fully aware of what it does, just not why someone would want to use it. How do you use things like jQuery or access window or document from C and then compile it to js? Or is emscripten specifically for js that doesn't interact with the DOM?

S4M• 12 years ago

Common Lisp in the browser, here we come!

You have Parenscript for that: http://common-lisp.net/project/parenscript/

(I haven't used it yer though.)

agentultra• 12 years ago

It works really well. It's a subset of CL that it can compile though so you'll miss out on CLOS, restarts, etc. However the subset does include macros so it's serviceable.

mateuszf3• 12 years ago

ClojureScript already exists, and even if it's not Common Lisp, its still Lisp with homoiconity, macros and what not.

mateuszf3• 12 years ago

Ok. Anyone can explain what do I get the downvotes for?

drmeister9• 12 years ago

I don't see why we couldn't do this. emscripten (https://github.com/kripken/emscripten) says it takes LLVM-IR to JavaScript. The Clasp (Common Lisp) function COMPILE-FILE compiles to LLVM-IR bitcode files. If you have some time and you would like to do this - hit me up with an email - I'd love to make this happen.

phs2501• 12 years ago

You'll probably run into all the same problems other Common Lisps run in to when they try to shake out the compiler from their runtime, namely that often you really do want to create and run new code at runtime (i.e. EVAL but much more often (compile nil '(lambda () ...)). A common example in CL is creating CLOS dispatch functions with specialized parameters to speed up generic dispatch, which I believe most CLOS implementations do at runtime (certainly SBCL/CMUCL's PCL does). You'll wind up needing to include your compiler, Emscripten itself, and all of their dependencies (LLVM libraries?) which will be quite a huge pile of JavaScript. The alternative is to include a CL interpreter but that has obvious speed problems.

arvid2• 12 years ago

https://github.com/davazp/jscl

mcmancini• 12 years ago

Exciting work. CL on the LLVM has been needed.

Is this project something that you're doing to support your own research goals in computational chemistry? If so, can you share how this project fits into the bigger picture?

phkahler2• 12 years ago

Sounded good until I saw this:

A faster Clasp compiler is coming soon – the current Clasp compiler generates slow native code that is about 100x slower than highly tuned Common Lisp compilers like Steel Bank Common Lisp.

No mention of the expected speedup, but 100x is a huge performance problem.

ephemeralgomi• 12 years ago

In his defense, SBCL is really fast...

eudox4• 12 years ago

http://log.irc.tymoon.eu/freenode/lisp?around=2014-09-25T18:...

Essentially, LLVM does tons of low-level optimizations but Clasp does few high-level, CL-specific ones. SBCL, on the other hand, mostly does CL-level optimizations and few low-level ones.

drmeister9• 12 years ago

Yes, LLVM is awesome. Clasp uses the inlining LLVM provides to inline C++ code within Common Lisp code by generating LLVM-IR bitcode files from C++ source and linking them with Clasp generated bitcode files and then running them through LLVM module and function optimization passes. I'm really excited about combining the CL specific optimizations with the ones that LLVM provides - stay tuned.

drmeister9• 12 years ago

The reason is that Clasp doesn't do Lisp language level optimizations like escape analysis yet. All bindings are stored on the heap and the stack/registers are underutilized. LLVM is a great library for implementing C and C++ but more work needs to be done to support Lisp features like closures and first-class functions. We are working on that now. The goal was first to "make it correct" and now we will "make it fast". Once we have a faster compiler (give us a couple of months) I don't see why it couldn't approach the speed of SBCL (a tall order).

phkahler2• 12 years ago

Glad to hear it. I've been wondering when new languages would be developed on top of LLVM and it's starting to happen (Julia, now this). C++ is getting a lot of more advanced features, but the syntax is so clunky compared to something like Python for example. I also like the notion of easy C++ library use for things like QT.

aidenn06• 12 years ago

I'm not too worried, since a) that makes it about as fast as cpython and b) there is a lot of low-hanging fruit. There is work being done on integrating the Cleavir compiler which does a number of tricks. In particular escape analysis and lambda lifting should both help a lot on certain types of code.

pjmlp3• 12 years ago

Any weekend compiler project can be faster than CPython.

krig2• 12 years ago

Considering that SBCL is among the fastest programming language implementations there is, 100x is actually not terrible for a first version.

sigil2• 12 years ago

Considering that SBCL is among the fastest programming language implementations there is...

Source for this? I'd love to find a really fast lisp or scheme. Hadn't heard SBCL was particularly fast, and the alioth benchmarks don't show anything special there.

http://benchmarksgame.alioth.debian.org/u64q/benchmark.php?t...

Edit: actually SBCL stacks up alright against languages like Go or Rust in the alioth benchmarks, so maybe that's what you had in mind.

yvdriess• 12 years ago

Finger crossed then that Azule can get a proper moving GC to work with LLVM then, until then a CL on LLVM (even Julia) are stuck with kinda bad GC.

I wonder why start from scratch for an LLVM backend, can't SBCL be used to generate LLVM code?

aidenn06• 12 years ago

It wasn't from scratch, it was from ECL. And did you mean Azul?

drmeister9• 12 years ago

SBCL doesn't generate LLVM code. My primary goal was Common Lisp with C++ interoperation. It seemed easier at the time to start from the ECL Common Lisp code base and write a new C++ core and a Common Lisp interpreter that always interoperated with C++. As I wrote the compiler and expanded the system I maintained C++ interoperation all the time. There were a hundred problems that I had to discover and solve along the way to maintain a Common Lisp that interoperated with C++. You can get some level of interoperation between ECL and C++ up in a weekend but it won't work with C++ exceptions and RAII and there are dozens of other problems. In retrospect I don't think I would have gotten here starting from ECL because I never really understood the ECL code.

ioddly2• 12 years ago

Is the compacting garbage collector mentioned in the article not a proper moving GC? I am not familiar with it.

drmeister9• 12 years ago

Yes, the Memory Pool System by Ravenbrook (https://www.ravenbrook.com/project/mps/) is a proper, moving garbage collector. It uses precise GC on the heap and conservative GC on the stack, as does the garbage collector in Steel Bank Common Lisp on x86 chips. I need it because I need my code to run on 100,000 CPU supercomputers with a controlled memory footprint to develop organic nano machines (seriously).

edwintorok2• 12 years ago

The name is already taken unfortunately: http://www.cs.uni-potsdam.de/clasp/

  Package: clasp
  Version: 3.1.0-1
  Description-en: conflict-driven nogood learning answer set solver
  clasp is an answer set solver for (extended) normal logic
 programs.

lispm5• 12 years ago

"Common Lisp Analytical Statistical Package" = CLASP, ca. 1990

http://www.cs.cmu.edu/afs/cs/Web/Groups/AI/lang/lisp/code/ma...

http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.138.... http://dl.acm.org/citation.cfm?id=896952

Later another CLASP: Common LISP as simulation program (CLASP)

https://github.com/openlisp/clasp

chc• 12 years ago

It was also already taken by a device for fastening things. And my name was already taken by at least three people before me. Somehow we all get along.

edwintorok2• 12 years ago

Package names in distributions have to be unique though, so this one will probably have to be called clasp-llvm or something like that.

616c• 12 years ago

Someone has been building Ocaml's opam on Arch I see.

andrewchambers2• 12 years ago

All names are already taken, that's why Wikipedia has the disambiguation pages.

masukomi• 12 years ago

I would point out that MOCL https://wukix.com/mocl pretty much provides this functionality only with full Common Lisp compatibility. It compiles the CL to C and adds methods to expose your functions to Obj-C / C code (if you want). Works just fine. Builds with Xcode, which as I understand, does everything on/via LLVM.

lispm5• 12 years ago

mocl is different from many other implementations: it is a batch compiler for whole programs to C. The generated code does contain a limited evaluator and no compiler. Some dynamic features are gone...

aidenn06• 12 years ago

MOCL specifically does not have full CL compatibility.

latiera• 12 years ago

MOCL is garbage, commercial garbage at that. No thanks.

tunnuz• 12 years ago

Maybe not-so-related, but "clasp" is also the name of a pretty popular Answer Set Programming (ASP) solver (link http://www.cs.uni-potsdam.de/clasp).

tempodox2• 12 years ago

Wow, this news almost makes me drool. I will keep my eye on that. I'm on SBCL right now, but I can't wait to see what Lisp can do with LLVM...

fithisux• 12 years ago

I hope it turns out a good implementation since there is a lot of C++ code out there and I do not know C++. But I can comprehend Lisp.

JabavuAdams• 12 years ago

Clasp exposes the Clang AST library and the Clang ASTMatcher library. This allows programmers to write tools in Common Lisp that automatically analyze and refactor C++ programs. Clasp can be used to automatically clean-up and refactor large C++ codebases!

Niiice. I want to program C++, but not in C++.

drmeister9• 12 years ago

Yes, I'll post more on this and provide example code in the coming weeks on the blog. It's really exciting and it's something that Clasp can do right now even though the Clasp compiler isn't a highly optimizing one yet. Google uses automated Clang refactoring to clean up the googleplex (100 megaLOC). Clasp puts those capabilities into everyones hands now.

Clasp – A Common Lisp with LLVM back end and interoperation with C++

Related Stories