Clojure: The JFDI language

https://docs.google.com/presentation/d/15-7qFy6URdE7Owi2LitkQI_OHBu1AFWPUwHxgBc-O4E/edit?usp=sharing

by michaelochurch • 13 years ago

197 128 13 years ago

docs.google.com

It's not really accurate to think of core.async as mutable state at all, because by default channels are only queues in the sense of implicit waiting go blocks forming a queue.

Unless you use a buffer implementation, channels have no "placeness", you're not mutating/adding to a queue by parking a go block.

I liked everything else.

michaelochurch8• 13 years ago

Yeah, a couple of other people pointed that out.

core.async has state (when buffers are used, in particular) but state is not exactly the same thing as mutability. Producers and consumers don't have unconstrained access to channels, in the way that users of a mutable cell (a Clojure atom, a ref cell, or just a variable in many languages) can clobber it at will.

Likewise (although I didn't get into this) state doesn't necessarily violate referential transparency. Memoization uses state as an optimization, but doesn't interfere with RT.

coolsunglasses3• 13 years ago

Likewise (although I didn't get into this) state doesn't necessarily violate referential transparency.

Boy do Haskell users ride that one hard.

ninetax2• 13 years ago

Are there really more Clojure jobs than programmers? As a clojure programmer looking for a job I haven't seen many...

Wish there were more though!

michaelochurch8• 13 years ago

I said the opposite: more programmers than jobs.

That might change. Clojure's been making some pretty impressive in-roads into the enterprise.

At this year's Conj, there were a fair number of companies hiring. Staples Innovation Labs is apparently hiring a lot of Clojure people.

What part of the world are you in?

ninetax2• 13 years ago

edit: well after actually reading your comment, I see now what you said :). Yes that seems like a good reason for companies to adopt Clojure, I hope they do.

Ah, yeah, I really wanted to go to Conj, but only found out about it last minute. I'm in San Francisco.

It's not that there are no companies using Clojure, it's more that the companies where I do want to work, don't want to use it.

When I mention it in an interview or something they kind of give me a funny look like they hope I am not one of those LISP people. I mean I can see where they're coming from. It's a big risk to let someone write in a language you don't know much about, and that you don't know if you'll be able to hire for in the future.

And at the places where they are feeling a bit adventurous, everyone seems to be leaning towards Go :(

Oh well, I can still use it for all my side projects until it gains more popularity I suppose. It really is such fun to use. I only started 2 months ago and I have been having the best time of it.

_xczx• 13 years ago

The misconception list basically summarizes the interactions I have with people who know I use Clojure. It's really interesting that these are still around, especially considering that lisp hasn't been new for a long time.

As an aside, I loved the chrono trigger reference! I was actually listening to some of its music as I viewed the slides.

I know this is old hat for the FP community, but it's worth mentioning that the dot product would be normally be written as (reduce + (map * xs ys)) (or something similar) for the sake of readers scared by the loop acc pattern.

michaelochurch8• 13 years ago

It's really interesting that these are still around, especially considering that lisp hasn't been new for a long time.

People want to believe the world is simple, and that learning one PL is enough. Once you know Java, why learn another one? Either they're all the same, or they're so radically different they must be weak in some critical way.

When you grok Lisp, and realize that it is actually practical, you realize that you've only scratched the surface of the PL problem... and that's scary for a lot of people, especially as there's a tendency to overestimate the barriers between languages. (The Internet runs on components written on all sorts of different languages.)

it's worth mentioning that the dot product would be normally be written as (reduce + (map xs ys))*

Yes, I had that thought, too.

In fact, that formulation is too slow for, say, large-scale ML. I've been getting a lot of mileage out of Prismatic's hiphip library... and I really look forward to, about 5 years from now, when Clojure is as mature for numeric programming and ML as Python is with Numpy, Scipy, Numba et al.

danenania3• 13 years ago

Thanks for this. It's a great overview that manages to be both detailed and easily accessible.

Your visual approach to explaining macros with arrows to demonstrate the substitutions also really helped me to grok them in a way I haven't been able to previously. They are actually quite simple aren't they? It's basically the same idea as html templating, but outputs code instead of markup.

I'd love to see you get more in depth on some of the headier topics--core.logic, monads, storm/hadoop interop, etc. You seem to have a gift for making difficult concepts approachable.

billsix• 13 years ago

you may enjoy On Lisp. http://www.paulgraham.com/onlisp.html

coolsunglasses3• 13 years ago

Structure-aware, fully programmable (you use your full programming language at macro-expansion time) HTML templates, sure.

JackMorgan• 13 years ago

I wrote a post about macros in Clojure going into some more detail if you're interested in learning a bit more.

http://deliberate-software.com/intro-to-macros/

brandonbloom2• 13 years ago

To be clear, that ` character is called "syntax-quote" (in Clojure; "quasi-quote" elsewhere) and it's the "templating" part. It's both possible and common to use macros without syntax-quote and vice versa.

In fact, common practice in Clojure is to define a your macros in terms of a backing function:

    (defn foo-fn [body]
      (fancy stuff here that probably uses syntax-quote somewhere))

    (defmacro foo [& body]
      (foo-fn body))

Clojure's style of macro system is known as a "procedural" macro system because the macros can be any arbitrary procedures which returns code as data. There are other types of macro systems, like Scheme's syntax-rules, which are more directly embrace the fact that most macros are tree rewrites.

chc3• 13 years ago

Syntax-quote is a little bit different from quasiquote. Clojure's macros look a lot like Common Lisp's unhygienic macros, but Clojure's syntax-quote offers a sort of "hygiene by default" by forcing namespaces on variables and making it hard to introduce non-gensym symbols accidentally.

joelgrus• 13 years ago

This is a really nice overview of Clojure, thanks.

saosebastiao2• 13 years ago

Interesting to see you mention Supply Chain Optimization. Other than myself, I know of only one other person that actively uses it for the task. Is there some hidden community that I'm not aware of?

dm35• 13 years ago

Curiosity took the best of me. Could you elaborate, please? Where are you actively using SCO?

saosebastiao2• 13 years ago

I use Clojure for a lot of analysis, modeling, protoyping and optimization for supply chains (I work in Amazon's Supply Chain). The presentation mentioned that Clojure was used for SCO, and I was curious jf there was some community of supply chain users of Clojure somewhere because AFAIK I'm one of two people that use it for SCO.

EdwardDiego• 13 years ago

Cheers for a great synopsis that I intend to inflict upon my coworkers at the first available opportunity. :D

vorg2• 13 years ago

Functional programming has two simple, elegant design pattern that don't suck: noun and verb.

I'd add adjectives (including adverbs), which is the metadata on the nouns and verbs.

dscrd• 13 years ago

Interesting point. I wonder if it would have an effect on how non-programmers view coding if programming language constructs were renamed like that.

_glass• 13 years ago

Larry Wall designed Perl from a linguistic perspective. But I think it didn't make so much of a difference.

lhnz• 13 years ago

I could teach a student with good language theory to program faster this way, but other students probably not...

michaelochurch8• 13 years ago

Adjectives, to me, are more unresolved but I think Clojure's protocols appear to be close to the right answer.

Adjectives are properties or classes of things, like Numeric or Closeable (with-open pattern) or Monad. The Haskell solution is type classes. The Java solution is OOP. The Erlang solution (fairly similar) is the actor model, which is the inspiration for OOP's principle of locally interpreted functions/methods (as opposed to global function definitions). OCaml's is functors, which are extremely powerful but hard for most people to use.

vorg2• 13 years ago

Perhaps protocols/interfaces are like restrictive adjectival phrases, and metadata like non-restrictive.

Just in case you don't know: Restrictive clauses limit the possible meaning of a preceding subject, whereas non-restrictive clauses tell something about a preceding subject, but do not limit its meaning.

Example of restrictive use: The suspect in the lineup who has red hair committed the crime.

Example of non-restrictive use: The suspect in the lineup, who owns a red car, committed the crime.

sandGorgon2• 13 years ago

Question - there was a recent post about how the dailymail group moved 140k lines of java code to 4k lines in clojure.

Since I have never programmed much in Java (I went from c++ to ruby!), could someone tell me where the gains are coming from ?

I know that eliminating the setters and getters (and the private/protected/etc system) would eliminate a lot of code, but I dont understand if it would be that much.

Possibly, I'm also trying to understand if they would have gotten similar improvements in kloc if they had switched to.. ruby/python for instance.

PurplePanda• 13 years ago

Some reduction in lines of code may have come if they had rewritten it in Java too, due to better understanding the problem space after having written the first version. Java has a fair amount of boilerplate type code other than getters and setters. For example forcing you to use "classes" for everything, even when you really want a procedural or functional API. Some people try to write semi functional style programs anyway using object oriented "patterns", which can sometimes lead to really just a lof more lines of code too. Also explicit type annotations take up space. And the Java style of using exceptions can be quite verbose. There may also be better designed libraries to do a lot of things for you automatically in clojure. Macros are very helpful too. But I'm just making all of this up, haven't actually seen the codebases to compare them.

cliftonc2• 13 years ago

Hi, I'm the CTO at the MailOnline so can comment. The biggest reduction in loc was due to the move from a relational DB to elastic search containing flatter representations of our content. This basically meant that we could delete thousands of lines of boilerplate and create a very simple processing pipeline to render content, and for this clojure really shines of course.

The REPL made the first week of exploring the problem and solution space really effective, and even as a node guy I was impressed with what could be done.

I can say that there is definitely a big productivity boost, but as others have said it isn't purely the language, but rather thinking about the problem in a completely new way that really did it. I do credit clojure though with giving our engineers the ability to think differently, as that is often the hardest part.

There was a comment I made in a speech recently about functional programming being frictionless for content management system design, this is what I was alluding to.

I don't want to start a flame war, but we built an earlier version in ruby (with one of the best OO guys I know leading it), and it was much more complex through attempting to model our home page as a set of inter related objects, using visitor patterns to traverse etc., than the clojure version.

sandGorgon2• 13 years ago

very interesting. I had a similar experience with elasticsearch in modelling a ecommerce catalog - however that was with jruby. In fact, it took me quite some convincing to not go with something like mongodb - elasticsearch served us quite well.

Could you comment a bit more on the interaction of functional programming to content management ? Intuitively, it feels to me that a lot of it has to do with the NOSQL-like data model of elasticsearch. Being a lisp beginner, I feel that a visitor pattern is very analogous to "map" in functional programming - again, without all the OO getter/setter functions (e.g. "accept", "visit").

T-R• 13 years ago

Java (at least prior to Java 7), has some severe limitations in how code can be abstracted. As an example, it doesn't allow you to pass functions as arguments to other functions directly. As a result, the canonical way to handle situations where you would do this (such as passing a comparison operator to a sort function), is to declare an interface, define the function in a class that implements that interface (command pattern, etc.), and then instantiate an instance of the class and pass that. Closures are imitated by explicitly passing data to class constructors and storing it in member variables, and currying imitated through patterns like the onerous abstract factory pattern.

Additionally, the abstraction features Java does offer are implemented with some heavy language syntax, and often can't be generated at runtime or pre-compile time, at least not easily, and so there's a limit to the abstraction, even taking into account Java Generics. That's something where homoiconic languages like Clojure have a particular advantage.

As a result, abstracting things in these ways (which in other languages is incredibly common) gets to be painfully verbose. Often, things that would be beautifully abstracted in these ways are just left unabstracted (as in-line boilerplate) - the verbosity introduced makes the code less readable, rather than more, and so there's a stigma against "over-abstracted" code. The result is that code that does, or could, make use of these things takes exceedingly more code in Java than in a language like Clojure. Of course, conventions in Java of, e.g., giving each curly brace its own line aren't helping its case in line counts, either.

thaumasiotes• 13 years ago

Of course, conventions in Java of, e.g., giving each curly brace its own line aren't helping its case in line counts, either.

My comment is basically a free-association from this.

In C and Java both (and, um, CSS), the people I've encountered seem to strongly prefer giving each matched pair of curly braces one line (that is, the opening { doesn't get its own line, but the closing } does).

In lisps (to my knowledge) it's more normal to have all your closing )s stuck together at the end of the previous line. Using a clump of )))))s to discern exactly which scope you're in afterwards is tricky (I'm pretty sure that's exactly the "problem" that the C/Java style is trying to solve), so you look at the indentation instead.

Python formalized the indentation-is-scope paradigm, which means a malicious programmer can't trick you into believing you're looking at a different scope than you are, but my Python code always ends up being a lot taller than I feel it should be, so I have a lingering sense that Python took it a little too far. It turns out that I occasionally appreciate the option you get with lisps to put a short bit of code all on one line and mark the parse tree with parens instead of with line-breaks-plus-indentation.

(All that said, I do like Python a lot.)

d0m• 13 years ago

I think it's also in the mentality as in it's perfectly fine in lisp to use simple lists as data structure, whereas we mostly use wrapper objects around primitive or collections in Java.

For instance, you'll often see a "Person" class definition with getters, setters around the .adds collection function. The idea is to encapsulate the collections as much as possible so that if it changes down the road, nothing else is affected but that simple method. In lisp, however, you'd probably just (let (people '("tom" "john"))) and be done with it. I'm obviously exaggerating, but hopefully you get my point. It's similar in Python where I use dicts all the time, but feel bad using it in Java or C++ without abstracting or encapsulating it.

It's not so much a syntax issue (even though clojure syntax is less verbose than Java) but mostly the mentality behind the idioms IMHO.

kazagistar2• 13 years ago

Its very possible they would have gotten significant (though possibly not equivalent) reduction if they reprogrammed it directly in Java. An old codebase often has a lot of useless cruft and over-abstractions and so on, and you do better each time you code something. The experimental control is not the old code base, it is translating the same code base back to itself with similar engineering effort. No wonder few people actually try to gather that sort of data.

benjaminwootton• 13 years ago

There is a lot of ceremony in Java. If I want a single polymorphic function I'll create an interface and a bunch of classes to implement it, instantiate those classes and pass them to where they need to be.

In a functional program, I'll just create the function and pass it around anonymously if needed. The function will in itself will likely be simpler than the Java equivalent and stateless.

There will also be less DTO type class code for carrying around dumb collections of data, replaced with simple records and lists.

I'm not at the stage where functional programming has really clicked for me, but after writing some Clojure and Haskell recently, the sheer amount of code you have to type in in Java to express yourself is really starting to bug me!

eklavya2• 13 years ago

I don't get it. Scala has all the features, it's fast AND it's easier to read (no matter how much the presentation tries to play down the () issue, it's there.)

Why not use Scala insead?

krat0sprakhar• 13 years ago

Can't agree with this more. All the plus pointed touts for FP are available in Scala without the necessary mental context-switch that is required to grok the parenthesis extravaganza. Why Clojure instead of Scala?

s_kilk• 13 years ago

Well, first of all Clojure is a substantially simpler language than Scala, which counts for something. Secondly, I believe that Clojure does a better job of presenting the developer with a new way of thinking about problems, which (as alluded to further up the thread) can be a big source of productivity gain and inspiration.

dm35• 13 years ago

The 'easier to read' part is valid maybe for the first week of working with a Lisp (commenting as a part of a team which switched from mostly-Java to only-Clojure several months ago).

Pros for Clojure:

* Faster code -> result cycle. No need for compilation (except for AOT-compiling releases).

* Absence of static typing helps in some of the domains, e.g. web applications. We have successfully used Prismatic's Schema[1] in places where a stable data interchange format was needed.

* Binary compatibility. No need to recompile libraries for different Clojure versions.

* Community oriented towards simplification. People mostly try to mimic Rich Hickey and Clojure.core development approach.

* Orientation towards small libraries. I haven't yet had a need to use a framework written in Clojure (although Pedestal[2] seems interesting).

[1] https://github.com/Prismatic/schema [2] http://pedestal.io/

mullr• 13 years ago

Clojure is dynamically typed, homoiconic, and places a lot of emphasis on immutability and managing state. Scala has different design goals. Idiomatic Clojure code is quite different, structurally, from idiomatic Scala. You should try it, and believe it when everyone says that the parentheses fade after about a week.

eklavya2• 13 years ago

Well I just might, for now Scala seems to be doing a pretty good job though :P

Is learning clojure enough if I would like to write lisp in future?

yummyfajitas• 13 years ago

There are great reasons to use Scala (specifically Akka, static typing), but syntax is a bullshit reason. A decent programmer can adapt to normal languages like ruby, scala or lisp in about a week of use. Wacky languages like perl or k might take a bit longer.

Also, syntactically Scala is a lot more complicated than lisp. It's a bit closer to Java, but that doesn't make it simpler. Just more familiar.

jamii• 13 years ago

I'm just going to copy-and-paste this old comment:

When evaluating language tradeoffs the scala community values power more and the clojure community values simplicity more.

Scala wants to provide you with the most powerful language possible. Clojure wants to make it easy for you to mold your own language.

Scala language tooling works at compile-time and analyses the language statically. Clojure language tooling connects to a running environment and uses introspection. Scala has much better static checking and clojure has much better introspection. While both languages have a repl, the clojure community puts much more emphasis on live, interactive programming.

Clojures syntactic extension mechanisms are much simpler, being based on sexps. Scalas are powerful (eg can access the lexical environment) but much more heavy-weight.

Clojure has run-time eval. This makes it possible to do eg domain-specific jit (eg compiling DSLs -> clojure code -> java bytecode on the fly). Scala has a run-time interpreter which I haven't used but is reputed to be slow.

I haven't yet explored non-jvm backends for scala but I suspect that clojure will eventually dominate on that front for two reasons: 1) when clojure-in-clojure is finished porting clojure to a new platform will be far easier than scala and 2) clojure was designed from the beginning to be a parasitic language and delegates many features to the underlying platform.

Scala breaks backwards compatibility much more often than clojure. Whether this is a pro or con is up for debate. There are lots of small niggling things in clojure I would like to change but on the other hand maintaining scala code across compiler updates is a pain in the ass.

Personally, I believe that the number one problem a language needs to address is managing complexity. The clojure community largely seems to get that and is driving for simple, orthogonal language features and libraries to an extent that I've not seen in any other community (eg http://blog.getprismatic.com/blog/2012/10/1/prismatics-graph...). In addition, the most exciting research I've seen on that front (eg http://www.vpri.org/pdf/tr2011004_steps11.pdf http://boom.cs.berkeley.edu/ ) and most of my favorite tools rely on powerful introspection and runtime code generation. I think Yegge describes this better than I can - http://steve-yegge.blogspot.co.uk/2007/01/pinocchio-problem....

michaelochurch8• 13 years ago

I got really into Scala a couple of years ago, and then worked on a big Scala project. One of the problems is that if you throw average Java developers into it, and don't give them the time to learn the language properly-- these were smart engineers, it was a lack-of-time problem rather than an ability issue-- you get a lot of Scala-in-Java. The code is just as unreadable, "enterprisey", and unattractive as it would be in Java.

I would use Scala for a personal project where I felt I needed compiler-time typing: I was writing something for the ages, or where bugs might be truly intolerable (say, high-frequency trading) but code would remain small. I wouldn't recommend it for a huge project, though. (Then again, I generally wouldn't recommend huge projects.)

Scala tends to fall down on large projects, especially if the developers aren't skilled. Cyclical dependencies (yes, I know they shouldn't exist, but they will if you have a team of 25+ engineers and a wide skill spectrum) defeat incremental compilation, which matters when your compile speed is ~500LoC/second (Scala's compiler is very powerful). Also, SBT is a nightmare. I haven't used Scala's macro system but I can't imagine it being as clean as Clojure's.

I like Scala on its terms alone, even if I've seen it go wrong on large codebases. (Java has the same issues, but it has tools that rescue you. It's an ugly hackish thing, to rely on tools to deal with code messes; but they're there.) I like static typing (at least, the theory that comes from it.) However, I tend to think that what most programmers really want when they say they want "static typing" is something more like Prismatic's Schema: runtime checks that can be turned on or off (performance) as needed.

I think Clojure's easier to read and to write. It just looks harder, because people exaggerate the parentheses problem, which is not much of one at all. If anything, the only thing that's somewhat hard to explain to newcomers is quoting (e.g. why '(1 2 3) returns (1 2 3) at REPL but (1 2 3) is an error). The quotes were the hardest thing for me to get right when I started in Lisp.

edoloughlin• 13 years ago

no matter how much the presentation tries to play down the () issue, it's there.

I'm getting tired of statements like this. I've been using Clojure for a couple of years now. I got over the () "issue" after a couple of weeks, once I discovered Counterclockwise for Eclipse (and, eventually, Emacs paredit or smartparens). You just don't notice them after a short period.

In general, editors that "understand" Lisp-like syntax are more powerful than other code editors. Being able to select forms - anything enclosed in a (), {} or [] - with a key combination and move them easily around the hierarchy (also see barfage & slurpage in the paredit cheatsheet: http://www.emacswiki.org/emacs/PareditCheatsheet) makes editing code faster than other languages.

As for the "easy to read" argument, I now find Clojure easier to read than, say, Java because there's generally less of it.

dkersten2• 13 years ago

Agreed. I remember hearing the exact same about significant whitespace in Python. Its just one of those things that, after a little bit of use, simply fades away into the background and stops being an issue. Many people even grow to like significant whitespace in Python and parens in Lisps.

bgruber• 13 years ago

frankly, in the world, people are using scala instead.

however, i have to differ with you completely regarding scala being easier to read. as someone who regularly works with scala code i find it incredibly difficult to read in comparison to clojure. Lisp syntax is different from what most people are used to but is very simple; scala looks more like the syntax that people are used to, but is deceptively quite different, and is quite complex.

davidp• 13 years ago

I enjoyed the presentation very much, until

    "Engineers 3-10x more productive"

Citation needed.

If this were generally true (of any technology X) the dollars would be flowing that way in an unstoppable torrent, and anyone using anything else would be the subject of ridicule and/or pity. It would be like writing custom websites in C.

Since that isn't happening, I have to take assertions like that with a sizable chunk of salt. The alternative would mean that there's a gigantic arbitrage opportunity that's being missed by a bunch of highly motivated people. So yeah, citation(s) please.

terranstyler• 13 years ago

IMO the problem with the unstoppable torrent is that the people who decide where company dollars flow to can hardly tell C from Clojure so they rely on the opinion of those whose job would then be to say "ah yeah right, me and my team have been using the wrong tools all the way" which doesn't happen for obvious reasons.

I started up with clj and I am trying kinda heavy arbitrage here :)

danenania3• 13 years ago

The reason is simply that more powerful tools require more skill and experience to use effectively. It's not just a question of switching from imperative language X to clojure and seeing a huge productivity boost. You must also understand why clojure is designed the way it is and how to use it effectively. For awhile, you'll likely be less productive as you learn how to approach problems functionally and wrap your head around the capabilities of lisp, but you'll also be building stronger, more reliable systems, and at the end of the learning process, you will indeed be significantly more productive.

Managing a transition like this is far from trivial for a company facing a variety of immediate demands and constraints, even if it promises to pay dividends in the long run.

kazagistar2• 13 years ago

Right, but just about every language makes this claim. Very few support it.

As far as I am concerned, they are all being deeply deceitful. If you make some "fact" up, and pawn it off as the truth, you are being dishonest.

coldtea3• 13 years ago

at the end of the learning process, you will indeed be significantly more productive.

That's standard process for any language.

The question is: why should we assume that Clojure pays more/special dividends in the long run?

craigching2• 13 years ago

It's not just a question of switching from imperative language X to clojure and seeing a huge productivity boost.

It depends on who you are. I recently used Clojure on a skunkworks feature for our product for which I had three days to complete. I am absolutely convinced that it was at least 3x smaller (LOC) and took me at least 1/3 of the time it would have taken in Java. I am an expert Java programmer and not a newbie to Clojure, but I still have to look up a lot of things.

The killer productivity feature of Clojure (LISPS and other languages) is the REPL. Trying things out, analyzing data structures, etc. is where I would have spent a ton of time in Java. Even if you're not an expert Clojure programmer, just using the REPL can save you tons of time.

So not necessarily disagreeing with you, but I also think that a non-expert in Clojure can still run circles around an expert Java programmer when considering productivity.

iSnow• 13 years ago

Confusing correlation and causation. Clojure coders might be 3x more productive, but they would be in any language. The chance is high that only the top 1% (or something like that) of all coders would dare to make the switch to this language, so you get a biased sample to begin with.

The problem with this of course is that when a company tries to switch their more mundane engineers to Clojure, the project will likely fail.

_pmf_3• 13 years ago

Confusing correlation and causation. Clojure coders might be 3x more productive, but they would be in any language. The chance is high that only the top 1% (or something like that) of all coders would dare to make the switch to this language

It's too bad that these top coders are so bad at mundane tasks like high-school-level statistics.

michaelochurch8• 13 years ago

I disagree. I think that top-1% programmers actually fare worse when thrown into a typical Java codebase. Sure, they're very productive on their own stuff (whether in Clojure or Java, although I still think those programmers will fare better with Clojure) but if you dropped a 1%er (or even a 10%er) into a pile of someone else's FactoryFactory garbage, he'd either rewrite it or fail completely to make any headway. Bad code is the Waterloo of good engineers.

I've seen companies end up in inversion, which means that the most apparently productive engineers are the worst ones. It's a common effect in "design pattern" Java shops with all their Visitor and Factory and Vibrator and AbstractSingletonFactory patterns. It's not pretty. Good people are either taken badly (because they keep agitating for rewrites) or become disengaged and eventually leave or are fired.

jwr• 13 years ago

While I agree the quoted assertion is bogus, I disagree with your reasoning. There are many arbitrage opportunities that are being missed, whenever technology, science, or anything that requires thinking is involved. This is due to human stupidity and/or unwillingness to learn new things.

calibraxis• 13 years ago

And there's many cultural forces at play. Many managers would pay a premium for a large disposable supply of programmers, rather than a smaller more skilled supply. Bargaining power becomes a factor; the smaller labor supply may demand more decisionmaking power than you culturally want to allow.

leokun• 13 years ago

Can you name a specific arbitrage opportunity because I wouldn't mind making some money.

mzl• 13 years ago

While I have no knowledge of whether or not Clojure programmers are more productive, a recently linked mail from Joe Armstrong on Erlang productivity was quite interesting: https://groups.google.com/forum/#!msg/erlang-programming/Oiy...

regularfry• 13 years ago

Here I wanted to cite the (moderately) well-known result that defect rate is proportional to lines of code, independent of language, but I can't actually find where it comes from.

Help?

pasbesoin• 13 years ago

I, as a single Perl (and regex, and VBA -- laugh all you want) programmer, have done things for which an EDS contract was ruled out by a major institution because it (and however much in terms of resources and expense it would entail) would be "too expensive". (Instead, they chose to continue "throwing people" -- full-time staff -- at the problem. Use that as a measure of just how expensive that prospective EDS work was considered to be.)

The language, or its aficionado -- either way, specific technologies have made me, in my own estimation, multiple factors more productive. Yes, there's the learning curve, and understanding things that are not explicitly stated line by line throughout the code. But then, there's getting shit done.

Unfortunately, a lot of people, processes, and institutions get hung up at the sentence before last of that previous paragraph.

I also turned other people onto some useful technologies, such as pointing a consultant at some reporting software that my department didn't want to spring for but for which -- of course -- the outside consultant had budget. I heard from her a month or two later that it had greatly eased things and turned her into a sort of reporting superhero.

Throughout my career, I've seen people who, both on their own as well as through dint of better technology awareness and selection, have been multiple factors more productive than their coworkers.

And yes, often they are fighting the trend and significant inertia, rather than being embraced.

jcromartie• 13 years ago

Yeah, I think he's just confusing the economy of LoC with productivity. I might actually be 2-3X faster with Clojure over Java, but I'm sure I could crank out a CRUD web app in Rails 2-3X faster than in Clojure at the moment..

espeed3• 13 years ago

Citation needed.

The line on slide 111, "Engineers 3-10x more productive, smaller teams", is probably in reference to...

Slide 16: "Code is typically 3-10x smaller when written in Clojure."

Slide 110: "3-10x less code. Limiting factor in programming speed is essential complexity of the problem, not typing."

I have to take assertions like that with a sizable chunk of salt. The alternative would mean that there's a gigantic arbitrage opportunity that's being missed by a bunch of highly motivated people.

That there is a "a gigantic arbitrage opportunity being missed" is essentially the thesis of the talk.

He states the reason why (Clojure is cloaked in 50 years of Lisp misconceptions) on slide 21: "Lisp is an old weird AI language no one uses anymore. It's so slow it requires a supercomputer. It's impossible to hire people who know how to use it. Functional programming has been proven impractical. Also, parentheses."

The body of the talk is designed to dispel these beliefs.

bonemachine2• 13 years ago

As in: the citation is a "reference" to two other assertions in the same slide set, themselves simply stated with no backing evidence.

_pmf_3• 13 years ago

I enjoyed the presentation very much, until "Engineers 3-10x more productive" Citation needed.

Oh, it's a perfectly plausible number. It increases the productivity of each developer by a factor of 3 to 10 and also increases the maintenance costs by 3^n_engineers or 10^n_engineers. This seems to be the tradeoff that startups make.

michaelochurch8• 13 years ago

3-10x is, of course, intended as a 95%-of-cases interval with some basic assumptions. Some contexts (e.g. type of problem, engineer, timeframe) will be below 3x and others will be above 10x. (The 99.99% interval is probably 0.2-100x.) Among those basic assumptions is that the engineers have the curiosity, drive, and ability to learn it. Ability is the least issue of the three; if they're not smart enough to learn Clojure, they're not smart enough to Java or C++ well, and they're mostly making messes.

Lisp has a bad reputation compared to what it is. People think it's impractical, that Lispers are arrogant (due to a few vocal but influential people in the Lisp community) and that it's archaic or weird. I've addressed those perceptions, but they're out there. Clojure is, in fact, a pretty damn practical language.

Now, the arbitrage argument: in larger companies, the concerns aren't engineer productivity so much as appearances. For a middle-manager to launch a meaningful Clojure/Lisp initiative would be to put himself out there in a major way, and since programmer productivity is only one piece of a much larger system, it might not have that big an impact on the company as a whole. It might not be worth it. (Would Google's market cap go up 3-10x if it replaced Java with Clojure? Almost certainly not. Maybe the fair value would improve by 1.05x on fundamentals, which would be drowned out by month-to-month volatility.) On the other hand, startups generally can't afford (or don't think they can afford) to experiment with newer, quirkier languages. VC-funded startups tend (surprisingly?) to have stodgy closed-allocation work cultures and boring tech stacks because they're expected to allocate 100% of the "risk allowance" to one department: business-model risk. That's not to say that there aren't other kinds of companies that can (and, in some cases, do) use more innovative tools; but the two dominant cultural forces (VC startups and large corporations) in technology don't favor it.

For the large companies, I'll admit that there's an Amdahl's Law sort of effect at play. Let's say that you improve productivity of engineers by 3-10x. Does that "3x" the whole company? As I alluded to for Google, probably not. It makes something else the limiting factor. For most firms, using the wrong programming languages is nowhere near their top problem. To use Google as the example, it might deserve a 5% bump on market cap if it included Clojure (a small gain compared to monthly volatility, so small that no one could prove causality and take credit) on the white-list but the fair value would go up 50-200% (again, on fundamentals, as I can't predict investor opinion) if it implemented open allocation. Programming language innovation just isn't the top priority for most companies. But for an individual developer, the advantage conferred by a powerful language can be pretty serious.

charlieflowers• 13 years ago

What do you mean by"open allocation"?

taude2• 13 years ago

I remember when Ruby on Rails developers were claiming this. Yet, I couldn't find one to deliver a website at 1/10th the price...or even 1/5th the price.

EDIT: (yes, being a little cheeky because as we know the physical time programming isn't everything in building a website)

wuschel• 13 years ago

(Disclaimer: I am a python that found a new home in the scheme world)

Clojure is a Lisp. Lisp based languages are great.

However:

One of the problems with clojure - or rather with its home or ecosystem - is the fact that it requires a very high activation energy to get going. Grasp the concept of functionial programming, cc/call, macros, totally different syntax than your previous imperative language of choice, and last but not the giant blob that is called Java platform with all the traps regarding project management, IDEs, building, compiling, deploying etc. I know that there are great tools such an Leiningen or LightTable, but they only make the whole process of transition more manageable.

Now, I know that Java is a huge benefit for all that want to make use of its gazillion tools. But if You are not into Java, then Clojure might give you some hickups.

In comparison to Clojure, Python also allows for some functional programming, gives the eager disciple an easy going start. Most of libraries are written in Python (or exist in a fast, a C-esque implementation) and are very easy to grasp. I can not say that for the Java enviroment Clojure has been born into.

I frown at java lingo error tracebacks in the REPL, and to bring a external Java library into play is even more problematic. Any thoughts on this how to remedy this problem without getting involved into Java?

I wish there would be some kind of Index of how to use certain Java libraries from Clojure, or an index of Clojure wrapping packages. It might make life a bit easier.

Finally, I claim that for now, I am quite happy with my choice to migrate from Clojure to Chicken Scheme/picolisp. You can get quite a bunch of Clojure data types from the good package repository. It compiles without setting your hair on fire. And it is really simple, as the language basics are small.

I will keep an eye on Clojure. but I must say that Julia is as appealing to me when it comes to the choice of future language.

jamesjporter• 13 years ago

Julia is a really cool language; I came to it from a python background and in a lot if ways it feels like "python done right"

minikomi2• 13 years ago

Please drop by the racket world sometime. The documentation is fantastic :)

Can you recommend a good picolisp resource?

john2x3• 13 years ago

Agree on the Java ecosystem. I deployed a toy app using Tomcat a couple of months ago, and now I have no idea how to update it. I lost the "tutorial" I found and can't remember what it was (stupid mistake), but still, the fact that I had to resort to a pretty unknown tutorial to deploy something shows how desperate I was.

I'm looking for another language with similar philosophy to Clojure's, but with a better ecosystem.

chc3• 13 years ago

FYI, you could have just used http-kit. If you don't want an enterprisey ecosystem, don't use the enterprisey stuff. That's how I use it.

yresnob• 13 years ago

Great overview..

from direct work experience in building complex systems in clojure....I agree with all the statements made in this document...

I am a better developer because of it and will look at things differently if I need to pick up java or scala again..

and I will also make sure to always "get in the hammock" before writing a line of code..

turar2• 13 years ago

Is there a web framework for Clojure similar to Play for Scala?

yogthos4• 13 years ago

Closer to Sinatra or Flask, but you can always check out http://www.luminusweb.net/

turar2• 13 years ago

Thanks! I found an interesting discussion on the topic here as well: http://grokbase.com/t/gg/clojure/131bp962q1/full-stack-cloju...

swah• 13 years ago

Related to this is the Erann Gat "How I lost my faith in Lisp" story[1]. He was also a googler but going the other way. Of course, this is quite old (before Clojure). But the points remains...

[1] https://groups.google.com/forum/#!msg/comp.lang.lisp/L5dZ-j6...

Clojure: The JFDI language

Related Stories