ClojureC, a compiler for Clojure that targets C as a backend

by terhechte • 13 years ago

144 86 13 years ago

sadly, this doesn't seem to support any sort of multithreading. Even something as simple as swap! isn't thread-safe in this implementation. So that kills one of the main reasons to use Clojure in the first place.

swannodette3• 13 years ago

Lack of multithreading seems like a result of the implementation being heavily based on ClojureScript ;) (it's actually pretty cool to see how reusable core.cljs is IMO). I imagine ClojureC will have its uses like ClojureScript does when the JVM is not an option.

terhechte2• 13 years ago

It is quite interesting though as soon as you extend it with calls to C libraries. Like if you want to use Clojure for libavcodec, or many others.

uvtc3• 13 years ago

> sadly, this doesn't seem to support any sort of multithreading

Maybe multithreading is just not yet implemented?

> So that kills one of the main reasons to use Clojure in the first place.

I could see ClojureC being very useful for when you want:

  * small footprint
  * easy C-library interop
  * fast start-up time

and where you don't necessarily need multithreading.

(BTW, I'd be curious to hear what you think might be the differences in use cases between ClojureC and mjolnir/clojure-metal.)

densh• 13 years ago

Why C but not llvm? Structured code generation is always better than string-based one.

pat_punnu4• 13 years ago

Can you show a proof for that claim?

I believe it to be nonsense.

bratsche2• 13 years ago

I'm not sure if this is what the previous poster was talking about, but clang has some APIs that let you get access to the AST pretty easily. It's been a really long time since I've looked at it, and at the time I don't think it was exposed as a library API for general consumption. But for example: https://github.com/bratsche/clang/blob/gtkrewriter/tools/cla...

coldtea7• 13 years ago

Nonsense?!

Outputting text to be interpreted as code is far more low level and error prone than targeting an AST via an API like LLVMs.

And you loose a lot of high quality tooling that you could take advantage of.

mjn• 13 years ago

If you generate C, you can get a C AST from the result easily, which is a structured format...

rayiner• 13 years ago

Generating C is quicker to write and easier to debug (can more easily read intermediate output) and doesn't require accessing a C++ api.

rdmckenzie2• 13 years ago

I can't stress the C++ api part of this enough. At the moment I'm building a Pascal compiler in Clojure for course work and when the time came to do byte code generation the first thing I looked to was the LLVM infrastructure both for ease of use and because it would trivially provide me with the ability to target non-x86 platforms.

The reality of the matter is that interacting with non-java linked libraries is a real pain from every JVM language know of. Two years ago this was posted here [https://github.com/jasonjckn/llvm-clojure-bindings], but since then LLVM has gone through two major restructurings so it didn't work out of the box and I estimated that it would take less effort to build my own naive infrastructure than to patch this one & integrate it.

As a result I'm generating code in terms of lists of newline terminated assembly statement strings that I can just print or write to a file when I'm done. While I agree that C is a sub-optimal output format in that you have to compile the output, it is also the clear lingua franca for systems programming and assembly generation these days. Generating C gives you interesting options like linking to other C codebases or your own C code the same way that cljs gives you the option of interacting with "native" javascript libraries as well as clojurescript toolkits.

saejox• 13 years ago

Generated C can be copy pasted and readable by humans. Debugging IR code is a terrible experience.

swannodette3• 13 years ago

People have made this argument for ClojureScript as well. In the case of ClojureScript the benefits are considerably less clear as we rely on the ability of Google Closure to do fairly amazing optimizations on the generated JavaScript source.

That said I'd love to see a serious Clojure-in-Clojure targeting LLVM.

cliffbean• 13 years ago

Similarly, structured control flow (C's if, for, while, switch etc.) is better than unstructured (LLVM IR just has the equivalent of gotos (yes, LLVM IR has switch too, but it can go anywhere, so it still isn't inherently structured)). So it goes both ways.

Some other advantages of C over LLVM IR:

ABI compatibility with C automatically. In LLVM IR, being ABI-compatible with C is often a considerable headache and you have to do different things on each platform.

You can use C libraries by #including their header files (the way many C libraries were designed to be used), instead of hardwiring all that information (macro expansions, enum values, typedefs, inline functions, struct definitions, etc.) from their header files in your code generator.

You have the option of switching C compilers; you're not as locked into a single backend.

With a bit of care, you can make your output much more readable. LLVM IR demands either SSA form (most front-ends don't want to do this) or herds of allocas, loads, and stores everywhere. In C, you just say "int x;" to declare an int, and just "x" to refer to it.

You can use C features like bitfields, designated initializers, short-circuit operators (&&, ||), compound assignment (+=, -=, etc.), and so on. You can do all these things in LLVM IR, but you have to lower them yourself.

masklinn• 13 years ago

Generating C is much simpler than LLVM IR (and you can do structured code generation for C: generate and serialize C AST); it's infinitely easier to read and debug; it allows switching compiler; it provides C stdlib to piggyback on with very little overhead.

It's a fine strategy to start with, there are more important things than fucking around with LLVM IR at this point, they can switch to LLVM or a native generator if they get the thing effective and off-the ground.

akkartik• 13 years ago

"Before you can run anything make sure you have GLib 2 and the Boehm-Demers-Weiser garbage collector installed."

Wow, that's a pretty skimpy list of dependencies. But..

"Make sure you're using Leiningen 2."

..argh, installing that on ubuntu that requires 110 packages. All that just for a build system?

zhemao• 13 years ago

Wait, are you trying to install it as an APT package? It's probably easier to just download the script and install it that way. If you do that, it will install all the libraries to a folder in your home directory.

Tuna-Fish• 13 years ago

It's easier to get Lein as standalone. The Ubuntu package depends on half the build infrastructure of java, using really minimal parts of it here and there. If you just get the shell script and stuff it in your path, it can self-install a 10MB jar that has all it needs.

uvtc3• 13 years ago

> ..argh, installing that on ubuntu that requires 110 packages. All that just for a build system?

Don't install lein using the OS packaging system (apt, rpm, yum, etc.).

Instead, just grab the `lein` script (linked to from http://leiningen.org/ ), put it into your ~/bin, set it executable, and you're all set.

michaelochurch• 13 years ago

This. It's a lot easier to do it that way.

iso8859-1• 13 years ago

How does this compare to Chicken Scheme?

mullr2• 13 years ago

This is neat.

What worries me about this and similar efforts (like https://github.com/takeoutweight/clojure-scheme) is that clojure's standard library design assumes that the underlying runtime will be do some kind of polymorphic method inlining.

For example: the sequence library is all defined in terms of ISeq, which basically requires a "first" and "rest" to be defined for the data structure in question. These are polymorphic: there are different implementations of these for different data structures (list, vector, map, etc). So a dispatch step is required to choose the right one. In clojure-jvm, this is implemented using a java interface; this means the jvm will inline calls to said methods when they're being used in a tight loop. And if you use the standard library, calls to 'first' and 'rest' are going to be inside nearly all of your inner loops.

Compare this to a normal lisp or scheme: 'first' and 'rest' (or 'car' and 'cdr', whatever) are monomorphic. They only work on the linked-list data structure. So compiling these directly down to C functions makes perfect sense and incurs no performance penalty.

So in summary: clojure assumes theres a really smart JIT which is helping things along. This means it's not as suitable for alternate compilation targets as you might want it to be.

I wonder if there's something clever you could do here. Vtables could be reordered based on expected usage, certainly. Clojure can already do some measure of type inference, so this could be used for AOT inlining when it's available. Even if it's not, perhaps several versions of a call could be speculatively generated based on what the compiler does know already. The normal polymorphic inline caching technique could perhaps be abused to apply here. But it's hard to see how any of this can work in absence of a profile or heavy hinting.

(not a compiler writer, just interested in the problem)

vidarh9• 13 years ago

You can do polymorphic method caching/inlining with a hybrid ahead of time / JIT compiler targeting C reasonably easily. The code fragments required for caching at least will be small, and code to generate them at runtime is not a big deal. I'm playing with a Ruby compiler, and Ruby badly needs these type of optimizations to get fast, so I've spent a fair amount of time looking at it.

For a fair amount of cases you can do static analysis to get good guesses at likely types, even for cases where you can't be sure. E.g. speculatively even looking near call sites by method name to see if you can guess the type of objects that will get passed in looks to get you a reasonable chance at guessing at the top contenders to let you speculatively generate inlined versions without creating too much junk. But to get the most performance out of this you're likely to need to be prepared to do some very basic JIT.

mullr2• 13 years ago

>E.g. speculatively even looking near call sites by method name

That is devious and fantastic.

> But to get the most performance out of this you're likely to need to be prepared to do some very basic JIT.

Yeah. But the attractive targets here are places where you can't have a JIT: embedded systems and iOS.

saosebastiao4• 13 years ago

Yeah, I wish we could somehow persuade Mike Pall to work some magic here. LuaJIT's general design sounds like the perfect fit for something like this.

pjmlp8• 13 years ago

You can workaround it by making use of PGO (Profile Guided Optimization).

This requires you to execute the application with profiling turned on.

Then you compile the application a second time with the additional input of the profile results, this way the optimizer gets additional information that helps it to do decisions similar to a JIT.

It is all a matter of adding support for this.

nimrody• 13 years ago

Perhaps you could overcome some of these programs with whole-program-optimization. E.g., the "Stalin" Scheme compiler.

It might be interesting to translate Clojure into a subset of Scheme that is widely supported -- and use one of the mature Scheme -> C compilers (Gambit, Chicken, Bigloo) to generate the target executable.

terhechte2• 13 years ago

https://github.com/takeoutweight/clojure-scheme

lmkg• 13 years ago

Could the use of Reducers help alleviate this problem? I'm not completely familiar with the internals, but my naive understanding of how they work is that the reducer function would dispatch once on the sequence type, and then iteration stays entirely within that sequence's reducer method.

jgalt212• 13 years ago

As someone who doesn't use Clojure, but watches it fairly closely, I'd say the least appealing part of Clojure is its reliance on the JVM.

As such, I'd say efforts such as these are greatly welcomed.

cgag• 13 years ago

That's how I felt about it before I started using it. If it's more than just startup time holding you back, I'd give it a shot.

pjmlp8• 13 years ago

> As someone who doesn't use Clojure, but watches it fairly closely, I'd say the least appealing part of Clojure is its reliance on the JVM.

Like it or not, this is what made Clojure successful in the enterprise, at least when compared against other Lisps.

lispm• 13 years ago

Is that based on numbers or a guess? How many Clojure applications are there in comparison to Lisp or Scheme?

saosebastiao4• 13 years ago

So if I wanted to write CLI applications in Clojure, is this my best bet? Cause the JVM is about the least suitable platform I have ever worked with for CLI apps...which is most of what I do. I'm constantly in this pickle of wanting to use Clojure but defaulting to Ruby because the JVM is so terrible at it.

mambodog• 13 years ago

You could also use node.js with Clojurescript, if you're willing to accept the compromises of Clojurescript.

saosebastiao4• 13 years ago

I'm not familiar with the compromises of clojurescript nor running node.js as a CLI platform. Got any links that can help explain? Can it be compiled to a static executable?

bbq3• 13 years ago

If the startup slowness is your problem, look into Nailgun: http://www.martiansoftware.com/nailgun/

If that's not your problem with the JVM, what is?

saosebastiao4• 13 years ago

Nailgun is an additional dependency. And pretty insecure at that.

Rayne• 13 years ago

Nailgun is not a good solution to the problem. If the JVM startup speed is a problem, your best option is to use something not on the JVM and not use hacks that make it feel faster.

fiatmoney• 13 years ago

Depending on what you're doing, you can consider the REPL as your CLI and run everything in that process.

uvtc3• 13 years ago

The readme for ClojureC says it's still experimental at this point. I'm not sure how far along it is. You might poke around its tests to see what it does so far.

pjmlp8• 13 years ago

Get one of the many AOT compilers for Java. No more JVM startup time to worry about.

rhizome31• 13 years ago

The problem is that the slowness is not limited to running command line programs, the REPL is slow to start, compilation is slow and so is running any leiningen task. As suggested in another comment I gave a quick try at ClojureScript but the compilation process is also slow. Clojure is an appealing language but to me the general sluggishness of the environment is too irritating. The existence of things like Nailgun and Drip seem to confirm that there's a real problem there.

I suppose if you're coming from a C-based runtime like Ruby or Python, you have to adapt your workflow. I imagine that a lot of Clojure people come from Java and have already shaped their workflow around the JVM.

billsix• 13 years ago

For those interested in Lisp implementations which compile to C, providing cross-platform benefits and a nice FFI, gambit-c and chicken are performant, mature implementations of the Scheme programming language.

pjmlp8• 13 years ago

This is quite nice, on the other hand, one could just use one of the many Lisp compilers existing since years.

ClojureC, a compiler for Clojure that targets C as a backend

Related Stories