Clojure from a Schemer's Perspective

https://www.more-magic.net/posts/thoughts-on-clojure.html

by todsacerdoti • 5 years ago

149 85 5 years ago

kgwxd• 5 years ago

You might just be using a completely different underlying data structure than expected, depending on which operations you've performed.

I haven't had the pleasure of using Clojure in production, but I'm somewhat obsessed with the language and follow it fairly closely. Someone on r/Clojure[1] mentioned Specter[2] with a link to this video[3]. One of the first things the video covers is how the library deals with this concern.

[1] https://www.reddit.com/r/Clojure/comments/loz77v/just_came_a...

[2] https://github.com/redplanetlabs/specter

[3] https://www.youtube.com/watch?v=rh5J4vacG98

fulafel7• 5 years ago

Can anyone give spoilers about what this is about for the video reluctant among us? Maybe lazy sequences that are accessed the first time vs reccessing them after being realized?

dkersten10• 5 years ago

He starts by talking about how specter maintains the types of your structure as you transform it. For example, if you map (using Clojure's normal map function) over a vector, you get back a sequence. The equivalent in specter returns a map. However, specter does this at all levels of nesting, so your vectors stay vectors, your sets stay sets and so on.

Specter just does this for you, under the hood. In vanilla Clojure, the natural way to transform some nested data might be something like update-in and possibly some nested transformation functions and maintaining the types by hand can become quite cumbersome. In specter, you don't have to think about it. Its neat.

drcode2• 5 years ago

As a clojure fan, I think this is a really well balanced and fair assessment of clojure. It's a bit surprising they didn't get into the whole "hygenic macro" business, which seems like the most obvious differentiator between the two languages to me. I would argue the macro system in scheme is very complicated (even more so in the extensions found in many scheme implementations) and this somewhat undercuts the minimalism appeal of the scheme language. (Though the scheme macro system is still extremely cool from a computer science standpoint.)

Also, the post states that you can't have nested "tail-call optimized" loops in clojure, which is only partially correct: Using nested loops is commonly done in Clojure, the only thing you can't do is have the loops call each other mutually-recursively, which is a far less common use case (but still a feature I hope to see in Clojure some day).

xiaq• 5 years ago

Also, the post states that you can't have nested "tail-call optimized" loops in clojure, which is only partially correct: Using nested loops is commonly done in Clojure, the only thing you can't do is have the loops call each other mutually-recursively, which is a far less common use case (but still a feature I hope to see in Clojure some day).

I guess you are referring to recur (https://clojuredocs.org/clojure.core/recur), and I also think that it's a quite cheap way to get ~90% of the benefit of TCO without actually implementing TCO. It's also more explicit than TCO which is a plus over TCO.

For mutual recursions there is always trampoline (https://clojuredocs.org/clojure.core/trampoline), FWIW.

drcode2• 5 years ago

Yes, 'recur is most commonly used in conjunction with 'loop, thanks for clarifying.

sjamaan2• 5 years ago

As a clojure fan, I think this is a really well balanced and fair assessment of clojure.

Thanks for the kind words. I tried to keep my biases out of it as much as possible and tried to avoid making it too ranty. And on the whole, I like Clojure; it's just not really my first choice as far as Lisps go.

It's a bit surprising they didn't get into the whole "hygenic macro" business, which seems like the most obvious differentiator between the two languages to me.

I debated including a bit about macros, but in the end decided that this would be a bit too in-depth. Clojure uses namespaces in a somewhat clever way to work around the bulk of the hygiene issues you get in CL with defmacro, but the macro system itself is rather uninteresting. I think their custom syntax for gensym is a nice touch, even though it's yet more syntax.

Scheme's macro system is very advanced and relatively complicated indeed, and the R5RS/R7RS standard system is only pattern matching/rewriting based.

Maybe I'll do a separate piece on the macro system if people are really interested.

fiddlerwoaroof2• 5 years ago

I’m curious what you mean by “ Clojure uses namespaces in a somewhat clever way to work around the bulk of the hygiene issues you get in CL with defmacro”. CL and Clojure are, in my experience, basically equivalent here: CL’s solution to hygiene issues is a combination of packages (“namespaces”), gensym. The main difference is that, as a Lisp-2, hygiene issues tend to be easier to avoid because you can’t use LET to shadow functions (which, in my experience writing Clojure, is the most common problem: accidentally declaring a variable called “name”, or similar).

didibus2• 5 years ago

And on the whole, I like Clojure; it's just not really my first choice as far as Lisps go

... Yet ;)

I think reading your article, familiarity seemed to be the source of a lot of your pain points. I wonder if the more you use it, to the point it becomes as familiar as Scheme, if you'd change your choice. Maybe not, but I think that's a possibility.

samatman2• 5 years ago

Definitely interested!

I've wrapped my head around the CL/Clojure sort of macro but never really grasped the essence of Scheme's hygenic system, always happy to read more about Lisp macros.

brabel• 5 years ago

Hey, I also enjoyed the article... and would definitely be interested in reading more about macros.

But unrelated to that, I wanted to comment on this observation:

In my current project, it takes almost 30 seconds to boot up a development REPL. Half a minute!

I work mostly with JVM languages and the JVM startup nowadays is actually very fast for a VM, under 100ms for your code in main to be running from a cold start. 30 seconds to boot is completely unheard of in my experience. If whatever you're using is open source, I wouldn't mind having a look to figure out what the problem may be.

abecedarius• 5 years ago

I think what the OP meant about nested loops was that you can't directly write an inner loop that conditionally continues with the outer loop. In Scheme you'd write a named LET inside a named LET, with the inner one invoking the outer in a branch of an IF. With Clojure's loop/recur, as I understand it, you'd have to make the inner loop return a flag for the outer loop to test again about whether to recur, because the only construct you have for continuing a loop is 'recur'.

This is not a rare situation. Am I missing something about how Clojure does it?

My Lisp dialect (Cant) has a construct like the named LET, but with the name being optional (defaulting to 'loop'). This makes simple loops as concise as Clojure's, and fancier ones as flexible as Scheme's.

didibus2• 5 years ago

You got the right idea, but in my three years of using Clojure professionally I never ounce needed this, so I feel at least in some cases like mine, this is a pretty rare situation.

TacticalCoder3• 5 years ago

It's a bit surprising they didn't get into the whole "hygenic macro" business...

Slight typo but it's "hygienic", not "hygenic" AFAIK.

karmakaze• 5 years ago

Nice writeup and a good summary/tour.

The biggest advantage of Clojure over more succinct Lisps is that the extra verbage and built-ins promote a larger shared base and patterns on how things should be done. This is less likely to turn each project into its own language. Clojure skills are much more likely to transfer between companies.

The other thing about being in the JVM ecosystem is that it promotes sharing so there's less reinventing the parts of the wheel that you need for 'this' project that's unsuitable for other use-cases.

codemonkey-zeta• 5 years ago

Great article!

IIRC the necessity of loop/recur for TCO is a result of being hosted on the JVM and not a direct design decision for Clojure.

If I really wanted to play devil's advocate there is an argument to be made that needing to use loop/recur could make an novice programmer more conscientious about tail calls (since it won't work if the recur isn't actually in tail position), whereas in Scheme you get TCO for free, but might not be aware when you're getting it or not.

In any case I kind of enjoy the loop/recur syntax since I don't have to name an intermediate tail-recursive function in a let (this could be naivety WRT writing scheme on my part, but I assume that's what usually happens).

The fact loop/recur cannot be nested is something that I've run into a few times and is a fair criticism. I also find that I end up converting anonymous lambdas to fn all the time, and I'm not the biggest fan.

IMO the most ergonomic lambda syntax I've worked with is Scala. I miss it in every other language that has lambdas.

dkersten10• 5 years ago

IIRC the necessity of loop/recur for TCO is a result of being hosted on the JVM and not a direct design decision for Clojure.

and from the article:

I really started noticing mistakes that make additional features appear necessary: for example, there's a special macro called loop to make tail recursive calls. This uses a keyword recur to call back into the loop.

While it may be necessary due to the JVM, having them was also a conscious design choice and many people (myself included) actually prefer being explicit about TCO, because normally when its used, its not used as an optimization but as a required feature to make the semantics work correctly (eg a recursion-based loop), with "recur", the intent that it be tail-call "optimized" is clear and if it cannot be tail-call optimized, then its an error (recur won't compile if its in a non-tail position). So, I personally would use recur even if normal recursion did have TCO.

Also, TCO is more general than "recur" and I believe having recur was also somewhat meant to make the semantics clear. For example, recur can only recurse to the closest loop or function. Proper TCO can also handle cases like mutual recursion between two or more functions (ie f1 calls f2 which calls f1. If both calls are tail calls, then TCO can be used here. This is actually what the JVM has problems with: making TCO work only for the cases that recur covers is easily possible on JVM, but having recur makes it clear -- at least to me -- that its a specific feature with specific semantics rather than general TCO).

So anyway, OP calls it a mistake, I call it a feature that I like. There are a few other places where OP says something is unnecessary or wrong or a mistake, but me, someone who has used Common Lisp and Scheme, but never in earnest, but has been using Clojure for ten years, I find them to be nice features, or make the code look cleaner to my eyes, or are otherwise good design decisions to me.

madhadron• 5 years ago

there's a special macro called loop to make tail recursive calls. This uses a keyword recur to call back into the loop.

This isn't so much a language designer choice as a consequence of targeting the JVM. Tail recursion on the JVM has been a problem for decades.

hencq• 5 years ago

That's not true. The JVM doesn't optimize tail calls, so, yes, that is a limitation. However, tail recursion (a subset of tail calls) is easy to detect and can be turned into iteration. That's exactly what loop/recur does. That syntax is a choice though. They could have decided to have named lets and make calls in tail position do the same thing as loop/recur does now. Instead they decided to make the syntax more explicit, so the tail recursion is more obvious.

ARandomerDude2• 5 years ago

Huge Clojure fan here, been using it professionally for 9 years.

This is a good article, highlighting a couple of the warts, which I think is a missing element for most people who like a language.

I completely agree with the nil punning observation. One of the issues pointed out in the article that resonated most with me is functions in the standard library usually (but not always) nil pun. You get used to nil, so when the standard library does throw a runtime exception, you're surprised. For example:

(:foo nil) => nil

(int nil) => exception

Overall great article.

ballpark• 5 years ago

It can be indicative of the goodness of a language when you use it for 9 years and are still a huge fan. I'm in the same boat.

samatman2• 5 years ago

Is this a case of Java leaking through again?

As in, Clojure uses int rather than Integer, so it simply can't be nil aka null?

ARandomerDude2• 5 years ago

Yes, I think so. Given how other functions work though, I would expect nil in this case, since Clojure is a language, and not just a Java library.

casion2• 5 years ago

If you want to nil pun like that, it's not difficult to do it where necessary.

(some-> nil :foo)

(some-> nil int)

Of course this fails if your nil is actually a string, but that's a different issue.

claytn• 5 years ago

I've been writing side projects in clojure for over a year now, and really enjoyed this assessment/review. The line, "At those moments, you're basically programming Java with parentheses" hit home for me. I'll always brag about the interop capabilities of clojure, but I will avoid using it as much as possible.

elwell• 5 years ago

Anecdotally, I've been using Clojure professionally for about seven years and only very rarely need to interop with Java. Of course it depends on what you're trying to accomplish. On the other hand, interop with JS from ClojureScript is very common; shadow-cljs has made that experience more inviting.

cutler5• 5 years ago

My experience with Clojurescript has been the opposite. React, on which most idiomatic CLJS is based (Om, Reagent), is such a moving target that Clojurescript is now years behind the curve. There was a lot of initial momentum under David Nolen's stewardship but that seems to have faded. React now has hooks and form but Reagent is basically the same as when it first appeared.

dkersten10• 5 years ago

I'm also a longtime Clojure user and I barely use Java interop at all. Occasionally I do something like convert an instant to a java.sql.Timestamp, or I call Math/abs or something, but even these tend to be isolated cases. Its rare that I need deeper interop, unless I'm wrapping a Java library myself (which I haven't needed to do in a long time).

The biggest area where the JVM leaks through is exceptions and error messages.

loevborg2• 5 years ago

vectors can easily be extended (on either end!) functionally

It's true that you can extend a vector on either end

    user=> (def xs [2 3])
    #'user/xs
    user=> (conj xs 4)
    [2 3 4]
    user=> (into [1] xs)
    [1 2 3]

but only the first operation will be O(1). The second operation is O(n) and should be avoided in hot loops.

hawkice• 5 years ago

For future readers: if you want to do this, I would first just accumulate into the seq backwards and reverse it into a vec. If that wasn't enough, use transients. Not _that_ different from other languages -- if you want to do something, and the obvious way is slow, (reverse (accumulate (reverse (map reverse vecs)))) is the obvious workaround for asymmetric performance (although I've not had this issue personally).

loevborg2• 5 years ago

Not sure I understand your comment. Is `accumulate` a scheme function?

My comment was intended to say that Clojure vectors have some operations that are performant and some that aren't. It is part of Clojure's philosophy to encourage the operations that are natural for the respective data type. `conj` appends to the end when applied to a vector but inserts at the head when applied to a list.

So you want to pick the appropriate data structure for the job and make sure only natural operations are used. For a vector appending is natural whereas inserting at the head is not.

fnordsensei2• 5 years ago

Interestingly, im-rs for Rust offers better than linear performance when inserting at either end: https://docs.rs/im/15.0.0/im/

curryst• 5 years ago

The underlying design of Clojure's data structures must be different. It needs to efficiently support functional updates; you don't want to fully copy a hash table or vector whenever you add a new entry.

I've actually been curious about this myself. No one seems to scream that functional languages are slow, so I'm curious what's going on under the hood. Like the author, I would presume they don't fully copy the hash table every time I add an entry.

Is Clojure smart enough to know that the previous hash map is now out of scope and can be GCed, so it just mutates the hash map in place under the hood? In the tiny amount of Clojure I've written, that seems possible. On the other hand, it makes performance much harder to reason about since mutating a variable that stays in scope after the mutation presumably would still trigger a copy. Do they just implement some kind of a "fall-through"? I.e. if I add a key to a hashmap, it creates a new empty hashmap, adds that value and a reference to the original hashmap. And then when I retrieve values, it searches the new empty hashmap, and then the "parent" hashmap if it isn't found?

I'm curious because a lot of functional programming seems like the kind of thing that would thrash memory in a GCed language. It seems to an outsider like you can't do anything without allocating memory, which you're often done using almost immediately. It would seem like "y = x + 1" would always take more time than "x = x + 1" because of the allocation.

Maybe GC is just a lot better than I think. Or maybe the functional style, with it's less complicated scoping, makes GC trivial enough that it offsets the additional allocations. Does anyone have any idea, or maybe have a link handy? I'm not a language designer, nor a Java programmer, so I fear the source code may not be terribly useful to me. I'm also a terrible Clojure programmer, if Clojure is written in Clojure these days (although I'd like to get better one day, it seems like a really fun language).

newlisper• 5 years ago

so I'm curious what's going on under the hood

https://en.wikipedia.org/wiki/Persistent_data_structure

Maybe GC is just a lot better than I think.

Yes, Clojure relies heavily on the JVM's GC being very good. It trashes it like there is no tomorrow and it would be a lot of work and extremely hard for an implementation of Clojure from scratch to match the performance of Clojure in the JVM because of how good the JVM's GC is.

Having said that, I have move 3 projects (10k-20k LoC) to JS from Clojure and don't plan creating new ones in Clojure, the JS projects ended up being faster, shorter and easier to understand. Idiomatic Clojure is very slow, as soon as you want to squeeze any little performance out it your code base will get ugly really fast. Learning Clojure is nice for the insights but I'll will pick nodejs first any day for new projects. Even if I need the JVM, my first choice probably will be Kotlin and then Clojure.

Of course there are many more downsides to using Clojure. No ecosystem, the cognitive overhead of doing interop with over-abstracted over-engineered Java libraries(because of no ecosystem ;)), the horrible startup times, the cultist community and the interop is really not that good, sometimes you have to write a Java wrapper over the Java lib to make it usable from Clojure. The benefits over JS are minimal but the overhead and downsides are too much. Worth learning it but not worth using it for real production projects.

zdragnar• 5 years ago

Is there anything in particular that you had to give up to make the move (aside from thr jvm)? I am assuming that for node.js to be shorter and faster you avoided the bigger frameworks (next, typeorm etc) and had to stick pretty close to basic middleware ala express.

thom• 5 years ago

I’m intrigued by what kind of code you’re finding is shorter and faster in JS. I’ve had the opposite experience, even limiting Clojure to a single thread.

l_t• 5 years ago

Yep -- under the hood, "immutability" in Clojure is implemented with data structures that provide pretty good performance by sharing sections of their immutable structure with each other.

For example, if you have a vector of 100 items, and you "mutate" that by adding an item (actually creating a new vector), the language doesn't allocate a new 101-length vector. Instead, we can take advantage of the assumption of immutability to "share structure" between both vectors, and just allocate a new vector with two items (the new item, and a link to the old vector.) The same kind of idea can be used to share structure in associative data structures like hash-maps.

I'm no expert on this, so my explanation is pretty anemic and probably somewhat wrong. If you're curious, the book "Purely Functional Data Structures" [0] covers these concepts in concrete detail.

[0] https://www.amazon.com/Purely-Functional-Data-Structures-Oka...

wging• 5 years ago

Here's how the persistent vectors work (old article series, but good. hopefully someone will tell us if it's out of date): https://hypirion.com/musings/understanding-persistent-vector...

lukashrb• 5 years ago

I really enjoyed the read and think this is a really good, fair and down to earth perspective. Also reminds me how much there is to explore for me :)

bitwize• 5 years ago

This is a really well balanced look at Clojure, much more charitable than I would give it. :) I've always considered Clojure's greatest strength to be its JVM integration, and that doesn't really give you anything Scheme (Per Bothner's Kawa) or CL (ABCL) don't. Otherwise Clojure is just weird enough to frustrate me without a compelling must-migrate payoff to the weirdness.

But again, that's just me.

cutler5• 5 years ago

It depends where you're coming from. If it's Python, Ruby or JS you'll feel right at home with Clojure's maps and vectors.

th0ma5• 5 years ago

I came to Clojure after Kawa and while I appreciate the simplicity of Kawa, the more advanced Java integration and build system in Clojure is really great.

lenkite5• 5 years ago

If Clojure offered a direct compile-to-native option for any platform (desktop/mobile/WASM) with no JVM in between, I suspect it would explode in popularity.

fnordsensei2• 5 years ago

It does, via GraalVM. Babashka, for example, leverages this: https://github.com/babashka/babashka

yen223• 5 years ago

If this were true, wouldn't something like racket or common lisp be more popular than it currently is?

lenkite5• 5 years ago

Can you point me to a guide that shows how to use racket or common lisp on mobile operating systems like android/ios or run them as WASM web-apps ?

fulafel7• 5 years ago

What advantages do you feel WebAssemlby would have vs the current JS as a compile target?

lenkite5• 5 years ago

Size and Performance for large web-apps presumably. For example, Figma which is currently wiping Sketch's face on the floor is written in C++ and compiled to WASM. https://medium.com/javascript-in-plain-english/made-with-web...

sometimesweenie• 5 years ago

The -> macro is a clever hack which allows you to "thread" values through expressions. It implicitly adds the value as the first argument to the first forms, the result of that form as the first argument for the next, etc. Because the core library is quite well-designed, this works 90% of the time. Then the other 10% you need ->> which does the same but adds the implicit argument at the end of the forms. And that's not always enough either, so they decided to add a generic version called as-> which binds the value to a name so you can put it at any place in the forms. These macros also don't compose well.

This is the biggest criticism of clojure I agree with the author on. Trying to compose threading macros is extremely annoying even though it’s not that difficult to refactor. Somewhere in the thread, you’ll want to put the return value of the previous function in a different argument place. I am not sure the reasoning behind designing the macros this way. With how well the language is designed, this annoyance has always seemed out of place and somewhat of an after thought. I made my own macro [0] as an experiment to circumvent this annoyance... which was an interesting experience (both writing and using the macro). Still, I don’t think it’s a good enough solution so I’m still trying to think of better ways to implement thread macros in clojure because I love the idea.

[0] https://github.com/cj-price/pipe.dream

Straw• 5 years ago

I've found -<> and -<>> very useful- they by default put things in first and last place respectively, but let you override with <>:

(-<> x (foo bar) (map baz <>))

Found here: https://github.com/rplevy/swiss-arrows

simongray• 5 years ago

In that case, why not go all the way and use the the exciting new ->->><?as->cond->! macro demonstrated in Every Clojure Talk Ever? ;-)

https://youtu.be/jlPaby7suOc

temp00345• 5 years ago

There's a simple rule I use: -> is for maps and ->> is for collections.

Functions that take a map should take it as first argument, functions that take collections should take the collection as last argument. This is how the clojure core functions work, eg. -> `assoc`, `conj`, `dissoc` vs ->> `map`, `filter`, `reduce`, `some`.

If you follow this rule in your code, threading is nice and looks good. If you want to force different argument position, use a lambda, like so (->> coll (#(my-fn %))).

However, I recommend against it. Use a binding and then use the other threading macro if necessary.

casion2• 5 years ago

For example, sometimes you need a let in a -> chain to have a temporary binding. That doesn't work because you can't randomly insert forms into let, so you have to split things up again.

Aside from the rest of the article, I find this one weird.

Of course you can do that, you just supply a function which has whatever you need in it. That's how you'd ideally be operating in Scheme as well.

-> is effectively _function_ chaining, not "do a bunch of things in this block" chaining.

valzam2• 5 years ago

Do people actually build large codebases with the 'hash map first' approach? It works for scripts but I always find it very difficult to follow code that transforms a bag of stuff into another bag of stuff.

TacticalCoder3• 5 years ago

Do people actually build large codebases with the 'hash map first' approach?

There are a few large projects written in Clojure. I'd say it's a "data first" approach, not uniquely "map first".

There's this famous 4 minutes rant by Rich Hickey (the creator of Clojure) where he talks about maps and about how Java does OOP (and how maps are served through getters but you can't see easily that they're maps), it's golden:

https://youtu.be/aSEQfqNYNAc

cutler5• 5 years ago

I think every Java (or C# or PHP ...) programmer should be made to watch this video every day before breakfast. Then and only then will software change.

valzam2• 5 years ago

I totally agree that the getter/setter scourge in Java is terrible but better languages like Scala or Python allow you to access fields with simple dot syntax and still inject getter/setter code.

My point was that without having a schema (which is essentially what a case class or ADT is) it makes to very hard to understand what objects are being passed around.

fulafel7• 5 years ago

Bigger code bases tend to use ways of imposing more structure while still using maps, like use of schemas (see malli & spec), standard patterns of passing down subtrees of the app's nested maps, using map destructuring in argument lists, etc.

Also good design and timely refactoring of said data structures to keep your functions from having to reach far into data.

You also decide what part of your tree should live in a database, supported by a query language. You can use SQL-based thins or Datomic and the many open source implementations of the same model. In the latter case it feels more like a "turtles all the way down" kind of thing.

dkersten10• 5 years ago

I've written a cryptocurrency trading bot in clojure that uses this approach very heavily. I typically send around a context variable (containing the database connection, logger, correlation-id and other ancillary stuff) as well as maps specific to the task I'm performing (orders, market data, etc).

I heavily use destructuring so that functions access the bits they care about, threading macros like (-> stuff (assoc :foo x) ...) and spec to make sure the maps have the structure and keys I expect and rely on non-existent keys returning nil by default. I also use the sequence functions heavily. Mapping, filtering and reducing over vectors of maps, group-by if I need them grouped and so on.

I personally find it to be a very pleasant way to program.

cutler5• 5 years ago

Life is too short to not be working with Clojure. "Ars longa, vita brevis".

Clojure from a Schemer's Perspective

Related Stories