Challenging Clojure in Common Lisp (2014)

https://chriskohlhepp.wordpress.com/functional-programming-section/metacircular-adventures-in-functional-abstraction-challenging-clojure-in-common-lisp/

by krat0sprakhar • 10 years ago

124 75 10 years ago

chriskohlhepp.wordpress.com

raspasov4• 10 years ago

If the author reads this:

Can you elaborate more on what you mean by "lackluster performance"? What is the use case? If you're looking for top speed in terms of C/assembly performance - I'd say yes, probably the JVM will get in your way. However, I spent months building a database/key-value store in Clojure and it's quite doable to write very high performance code in Clojure as long as you put the right type hints everywhere. Tools like YourKit can help you identify the bottlenecks in your program. Again, depends on the use case, but Clojure makes it very, very idiomatic to write code that makes full use of a multicore system - something possible in the like of Java, C, et al but definitely not idiomatic at all.

pron6• 10 years ago

(not the author)

probably the JVM will get in your way

Not so much the JVM, but Clojure and how it uses the JVM (although, as you say, it's possible to get not too far from top JVM performance with Clojure). It's fairly easy to get C performance (and even beat it in concurrent code) for the same amount of effort on the JVM. Currently, the main handicap the JVM has is the lack of arrays-of-structs which may cause lots of cache-misses, and requires less-than-elegant code to overcome. This, thankfully, is being addressed by the addition of value types.

raspasov4• 10 years ago

Yup, the JVM is something that people (esp. startups & new-co's) are, all things being equal, vastly underusing or over-complaining about :).

Before I started doing Clojure the JVM ecosystem was this "scary" thing but 4 years later I've learned to appreciate it a lot - stable AND sound libraries, high performance out-of-the box, and very,very high performance if you really need it.

WildUtah2• 10 years ago

It's fairly easy to get C performance (and even beat it in concurrent code) for the same amount of effort on the JVM.

Ha.

That's what we were promised back in the mid-1990s. It's no more true today than it was then. Well written Java will always be about 2-4x slower than similar C code.

JVM applications have many compensating advantages, including the aforementioned ease of exploiting concurrency. That doesn't erase the reality of single thread performance where Java has never, in twenty years of strong expert effort, caught up.

agibsonccc5• 10 years ago

It can though. We are a java shop that does HPC and deep learning. Our work is open source. We've been around 2.5 going on 3 years now. Over the years we've surpassed bottleneck after bottleneck in the JVM.

It's a faster wrapper around c code for us. We maintain and created: https://github.com/bytedeco/javacpp

which wraps a lot of c++ components. This is also how we do GPUs and the like with our own memory management for our "numpy for java": https://github.com/deeplearning4j/nd4j/tree/master/nd4j-back...

Short of it "off heap and NO GC" matters a lot.

We have 1 c codebase we use that's meant to be controlled via JNI here: https://github.com/deeplearning4j/libnd4j

The speed gains we've seen are massive. Java can't compete with good ole simd and the like for numerical computing.

We are a big advocate of the JVM as a platform but let's be clear about its weaknesses. You need to use unsafe (see: netty,aeron, and any other low level database written in java or using java as the core language with some c++)and other tricks out of the box for real performance.

virtualwhys• 10 years ago

This, thankfully, is being addressed by the addition of value types.

Last I heard (JLS 2016 keynote by Brian Goetz) they have yet to commit/guarantee when value types will land. Java 10 is planned; if they're completed by then we're looking at probably 2020 given the delays with Java 9 release.

That's 3 years for other languages/platforms to evolve while the JVM unboxes itself.

krat0sprakhar• 10 years ago

as long as you put the right type hints everywhere.

I'm a novice Clojure programmer and haven't previously heard of this. Can you please elaborate?

ajss• 10 years ago

(defn [o n] (.write o n)) will have to look up the .write method by doing reflection on o & n at runtime, which is really slow.

(defn [^WriteInterface o ^long n] (.write o n)) can be much faster because the compiler knows the type of o & n.

moomin• 10 years ago

Short version: put :warn-on-reflection in your leiningen file, And then add type hints (google them) to reduce the warnings to zero.

There's no reason the average Clojure project ever need be using reflection for dispatch.

hkjgkjy• 10 years ago

You don't need to specify what type a function takes in Clojure, which is often nice. If you do though, the compiler can sometimes optimize, and make things much faster. Basically, the compiler does not have to

* look up the type of the thing * run the action on it

but can instead skip straight to step 2.

If you somewhere in your Clojure program (or repl) run (set! warn-on-reflection true) it will print to the console when you are doing reflection.

I think it's a good idea to put in the beginning of your core namespace - so that it is always on. Why not always wear the seatbelt when driving, you know?

fsloth2• 10 years ago

"something possible in the like of Java, C, et al but definitely not idiomatic at all."

What on earth is non-idiomatic multicore c code? You have to try really hard with Macro tomfoolery before c becomes non-idiomatic :) .

lvh7• 10 years ago

Isn't this just a convoluted way of arguing that there's no such thing as idiomatic multicore C? If everything's idiomatic, then surely nothing is.

klibertp2• 10 years ago

I'm not the author, but looking through the sibling comments I feel I should point out the obvious: startup time.

It can take long seconds for Clojure REPL to start up. No other dynamic language I had contact with was that bad on that front. Even Erlang and Elixir boot up faster!

Of course, I know about various ways of setting up a daemon process in the background, but then there is a need for reloading/hot-swapping code, which Clojure also fails to do well. Both Erlang and Common Lisp (also Emacs... even Emacs!) are better in this regard, both for different reasons.

Anyway, Clojure overall performance impression - how fast it feels for users - is bad and will remain bad unless the startup problem is fixed.

hellofunk5• 10 years ago

Somehow I could not shake the feeling that homoiconic macros were intimately tied to the very essence of functional programming

That's the first time I've heard anything like this. Lots of great FP languages exist that are not dynamic or offer such macros, why would they be a requirement for FP?

fnordsensei3• 10 years ago

It's not, he just couldn't shake shake the feeling. Kind of how you can't shake the feeling that there should be left-over pizza in the fridge, knowing very well that you already ate it all.

I think many of us probably tend to associate certain styles and features with functional programming that really are orthogonal to whether the program is functional or not.

As I've had it explained to me, if you write your program using functions whose output only depends on the arguments provided to that function, and nothing else, then you have yourself a functional program. But then again, people tend to treat the concept like a scale, with some programs (or languages) being more or less functional than others.

hellofunk5• 10 years ago

whose output only depends on the arguments provided to that function, and nothing else, then you have yourself a functional program

You can choose to do this in any language, but that does not make any language conducive to FP. The other main ingredient is that a function should be a primitive data type in the language, and can be stored, re-assigned at run-time, passed to other functions, and returned from functions. Without this, you definitely cannot have a functional language.

mhluongo• 10 years ago

That would make the function pure, not necessarily make what you're doing functional programming. First-class functions and their manipulation is also a requirement, IMO

nickbauman2• 10 years ago

I think the problem hovers around the idea of first class ness. In Clojure, I can use a function like '<' as a key in a hashmap. In Python, I can't. Because '<' isn't a function, it's an operator in Python. First classedness has to do with non-complectedness. Python has operators and statements in addition to expressions. Clojure, because it's a Lisp, gets by with just expressions.

minikomi• 10 years ago

The end result seems a little strange to me. Why doesn't it KNEAD and BAKE without being told, and yet DECORATE's just by adding sugar?

Also, this is a perfect task for core.async / core.match in clojure. Definitely not 6 days worth of work.

lvh7• 10 years ago

You mentioned a desire to get some visibility into the runtime effects of your code. You get that (ASM dumps) out of CL, but you also get JVM bytecode and associated ASM dumps out of Clojure, even if your code is doing FFI. I go into detail on this in my Conj talk: https://www.youtube.com/watch?v=Lf-M1ZH6KME

lvh7• 10 years ago

You mentioned that you wanted something non-hosted. Unless I missed it, the arguments are roughly performance (not really made) and visibility (not true, see [1]). Hosted versus non-hosted seems like a property without a particular intrinsic value judgement. I'm going to argue that for a Lisp, being hosted is a great idea.

Firstly, JS is a very credible and easy to use target for Clojure. It has source map support and produces aggressively optimized JS via the Google Closure compiler. (Parenscript, the CL competitor, can claim neither.) I've had plenty of code where I literally rename the "clj" extension to "cljc" and it magically runs in the browser.

Secondly, I get to use ample libraries in both cases because they already have thriving ecosystems that are completely independent from all of these parentheses. For example, today I'm writing AWS code, and I get to use AWS's Java SDK painlessly. If I'm on CL, it appears I'm out of luck.

Being a hosted language, in and of itself, hence seems like a clear choice if it isn't for the (not made) performance argument. So, perhaps we should talk about that instead :)

[1] https://news.ycombinator.com/item?id=13349747

junke4• 10 years ago

See ABCL (http://abcl.org/) and JSCL (https://github.com/jscl-project/jscl)

lvh7• 10 years ago

JSCL is quite different from ClojureScript. ClojureScript has a self-hosted form (which, AFAICT, is the default for JSCL), but that's not what you want for most applications if you want to produce efficient minified code. My two specific claims about ClojureScript being good included source maps and aggressive optimizations (and all of the cool side effects from having Closure as your module system) -- neither applies to Parenscript (as I mentioned) or JSCL.

lvh7• 10 years ago

Some Java:

  public class Main {
      public int addTwoNumbers(int a, int b) {
          return a + b;
      }
  }

Calling it from ABCL:

  (defun void-function (param)
    (let* ((class (jclass "Main"))
           (intclass (jclass "int"))
           (method (jmethod class "addTwoNumbers" intclass intclass))
           (result (jcall method param 2 4)))
      (format t "in void-function, result of calling addTwoNumbers(2, 4): ~a~%" result)))

Calling it from Clojure:

  (defn f []
    (prn "calling addTwoNumbers from f" (.addTwoNumbers (Main.) 2 4)))

Note that the Clojure version creates the object, and the ABCL version doesn't do that for you; you still need to instantiate a Main yourself.

Also, my concrete claim was that I could use AWS easily. I can use both the SDK directly quite painlessly (see above) or use https://github.com/mcohen01/amazonica. Where is the equivalent CL library?

_halgari4• 10 years ago

Can we get some sort of edit to this title to show that it's from back in 2014? A ton of work has gone into Clojure since then and in some aspects the language has gotten orders of magnitude faster (not that it was really that slow to begin with).

sctb• 10 years ago

Thank you, we've updated the title.

_halgari4• 10 years ago

STM and concurrency...sure those are "aspects" of Clojure, Clojure's biggest strength is that it's data-centric. The vast majority of the language is focused around manipulating hashmaps, vectors, sequences, etc, and do that in an efficient way. All the concurrency stuff is just icing.

So I really have to sit back and shake my head when the author says he's going to be as good as Clojure, then goes off into the weeds with custom syntax, STM and actors. Really? Why actors?

This would have been a much better article if it just left Clojure out of the discussion since whenever the author talks about the language he's mostly wrong.

And as always, lies, damn lies, and benchmarks: https://benchmarksgame.alioth.debian.org/u64q/compare.php?la... If you're going to use a phrase like "faster", at least spend some time to define the word.

m0skit0• 10 years ago

Isn't a functional language supposed to be function centric?

_halgari4• 10 years ago

Sure, but what do those functions do? In clojure they manipulate data. That's pretty much the gist of the language: using pure functions to manipulate data.

Almost all the work I do these days as a software engineer is data transformation. Going from a HTTP request, which is data (even the header is a hashmap), and a HTTP body, which is data, into some business logic that eventually writes to a database in a different format.

Even the most complex systems I've built containing dozens of servers and multiple databases, queues, http servers, etc. All boil down to transforming data from format A to format B perhaps with conditional logic applied.

So yes, Clojure is a functional language, but functions are just a tool to be used to get the actual work done of transforming data.

macmac2• 10 years ago

Clojure's rich persistent data structures together with protocols facilitates functional programming in Clojure. When Clojure is called a data-centric I think most users mean that it encourages functional programming applied to rich immutable data.

freeduck• 10 years ago

Cl has fset for persisten collections

brudgers5• 10 years ago

I like Clojure. It's data-centricness is something that it shares with certain ways of programming in Common Lisp. In particular the use of lists in Common Lisp. For example, a list holding a schedule of appointments

  ;;; adapted from 
  ;;; https://www.ida.liu.se/ext/caisor/archive/1978/001/caisor-1978-001.pdf
  (((JAN 12 2014)
      ((9 15) (10 00) (SEE ANDERSON))
      ((10 45) (11 00) (SEE LUNDSTROM))
      ((13 15) (16 00) (ATTEND Y COMMITTEE MEETING)))
  (((JAN 13 2014)
    ((9 30) (10 00) (ATTEND NEW PRODUCTS PRESENTATION)))

A lot of what Clojure brings is more efficient abstractions. For example Common Lisp has two key-value stores a-list and hash tables that Clojure collapses into one thing and then rolls together with hashes and sets as much as possible.

_halgari4• 10 years ago

Same data in Clojure:

    [{:appt/start #inst "01-02-2014T9:15:00Z" :appt/end #inst "01-02-2014T9:15:00Z" :appt/description "See Anderson"}
     {:appt/start #inst "01-02-2014T10:45:00Z" :appt/end #inst "01-02-2014T11:00:00Z" :appt/description "See Lundstrom"}
     {:appt/start #inst "01-02-2014T13:15:00Z" :appt/end #inst "01-02-2014T16:00:00Z" :appt/description "Attend Y Committee Meeting"}
     ...]

Let's point out some important differences:

1) Clojure prefers maps over cons cells. This means that I always know exactly what I'm looking at. I don't have to guess that the second times are the end-times...I know because it' named :appt/end.

2) We don't overload data types. Have a date? Use a date type. In the CL example we see symbols and numbers sometimes used for descriptions, sometimes for times, etc. Here in Clojure we have #inst ... which creates an actual DateTime object. Now I no longer wonder what I'm looking at, I know it's a date.

3) Clojure prefers data over DSLs. What this calendar example shows is some sort of domain specific language that not only has to be parsed by a program, but it also has to be parsed by a human.

4) Got a meeting description, use a actual string....what's up with the use of symbols as strings (I never understood that about CL).

5) From the perspective of a data modeler...in the CL example if you want a meeting to last over midnight or for longer than a day, it looks like you're sunk.

But thanks for the example, it's fun to see how data modeling is done in other languages.

klibertp2• 10 years ago

Really? Why actors?

Probably because Common Lisp is already on a similar level of being data-centric, so it would be redundant to talk about it. Actually, the only difference between how Clojure and CL treat data is that (while both offer access to both kinds of data) Clojure strongly prefers immutable values, while CL doesn't care.

dsrguru• 10 years ago

Clojure doesn't just "strongly prefer" immutable data but syntactically ensures that all data is immutable unless governed by one of four concurrency mechanisms. Persistent data structures aren't just a popular convention in Clojure--the language is literally built around them.

igouy• 10 years ago

And as always, lies, damn lies, and benchmarks

Things change: not the same JVM as 2014, not the same Clojure.

We may quote to one another with a chuckle the words of the Wise Statesman, lies, damned lies and statistics, still there are some easy figures which the simplest must understand but the astutest cannot wriggle out of. 1895 Leonard Henry Courtney

doall• 10 years ago

To understand concurrent programming in Clojure, at least you should read [1]. In general, the symbolic STM (Refs) is not often used in Clojure programming. Atoms (a wrapper of the Java AtomicReference) is much more lightweight and is used in most cases.

[1] http://clojure-doc.org/articles/language/concurrency_and_par...

zengid• 10 years ago

I feel like the author didn't do this as a way to bash Clojure but to try and progress Common Lisp. Anecdotally, I remember Hickey saying something about a stagnation in the CL community, so if Clojure inspires cross-pollination of ideas then that's a good thing.

For instance, Simon Peyton Jones describes [1] how they took the ideas around STM from 'unsafe' languages and put it into Haskell, developed it further, and then later the 'unsafe' languages took the Haskell innovations and put it back into their languages.

Clojure isn't perfect, and Common Lisp isn't perfect, but hey, they're better than programming in assembly. We should be stoked about someone trying to innovate; why do these posts always turn into language wars?

[1] https://youtu.be/iSmkqocn0oQ around 3:50, but the video is short so it's worth watching the whole thing.

brudgers5• 10 years ago

I didn't take it as Clojure bashing either. On the other hand, it does seem to be a sort of back-footed defense of Common Lisp, to me. Partially because it seems to miss acknowledgement that part of the goal of Clojure was to bring the level of abstraction that Common Lisp affords to the JVM, for example Common Lisp has the abstraction of Seq(uences) and functions that operate upon them.

But the big elephant to me is that Common Lisp provides a very strong set of Von Neumann abstractions in addition to its functional abstractions. Lists are an abstraction that maps to sequential reads of a contiguous block of memory. Setf has pointer and offset semantics. And the much maligned `loop` is a brilliant abstraction over iteration that lets a program be explicit about the why of a loop.

That's not to say there are not advantages to functional programming. But the popularity of Go and Python etc. suggest that abstractions that run counter to functional programming mores is not a reason to find fault in Common Lisp. Like Clojure, it's a big language that tries to meet programmers where they are in the projects that they work on. Nobody faults Clojure for allowing Java's mutable vectors because they're there in the language for practical reasons.

macmac2• 10 years ago

"Object orientation falls short here in that it does readily permit mathematical composition."

Surely "does" is meant to be negated?

Challenging Clojure in Common Lisp (2014)

Related Stories