The Python Paradox is now the Scala Paradox

Esoteric languages are a lot of fun. But they're not very productive, exactly for the reasons mentioned: lousy IDE support, buggy compilers, few libraries, few bindings to existing libraries, and so on. When you work in a stable language, you can assume that when something goes wrong -you- messed up. In a new and cool language you don't have that luxury.

You'll be less productive, but you'll have more fun when you get to program in language. A good trade off? I don't think so. A few years down the road you'll have to maintain the same project. The honeymoon period is over, and the code base isn't quite as good as you'd like it to be (you designed the project back when you didn't know the language very well, and it shows). You know you could spend a few weeks refactoring, but that would probably introduce bugs and you'll drive the other people in your team crazy if you start renaming methods, and move stuff around just to make the code "nicer". And besides, you'd much rather be programming in the next-nifty-language, because it lets you extend the syntax at runtime and it has a debugger that will let you go back in time.

The bottom line is that when you pick a language that is on the cutting edge of cool right now will no longer be cool a few years from now, but it still won't have any libraries and other essentials. Languages too have to cross the chasm, and they all have to go through a period where they're neither new and exciting nor stable.

If you're serious about software, concentrate on writing something that is reliable as hell and easy to maintain and install. Want a challenge? Write your app in such a way that power can be lost at any point in time without losing any user data. Or (for desktop apps) make sure that the UI -never- hangs, even when the hard drive has to spin up. Software is never done, there are always things you can make better, more fault tolerant, faster. It's not as if we've run out of challenges.

Messing around with unstable languages is a lot of fun. It's educational too. But don't build your business around it. Python/Django and Ruby/Rails are only barely stable enough to work with. Thinking that using Haskell, Clean, IO or Lisp will be a net benefit to productivity is foolish.

(Of course there are obvious exceptions, and that Clojure and Scala use the JVM does make the whole library thing less of an issue. But it's still a big deal.)

shabda2• 17 years ago

> But don't build your business around it. Python/Django and Ruby/Rails are only barely stable enough to work with.

I agree with your basic premise but,

Python and ruby both have extremely good library support. Hell, python's stated aim is batteries included. Both of them have (al least semi-)mature Jvm implementations. Django has a stated policy of not breaking backward compatibility on point releases.

I agree you need to choose stable languages to work with, but if you are selling consulting services around Java god luck competing with BigCo. If you are building your products with Struts, good luck doing xml-situps.

gizmo13• 17 years ago

If you think Python and Ruby have extremely good libraries your standards are pretty low. CPython itself is stable and well documented, but it's frequently breaking backwards compatibility across versions. My production servers can't get anything newer than Python 2.4, and the servers are barely a year old. Python 2.6 libraries almost never work in 2.4, and when I do upgrade to 2.6 I know a lot of my own code is going to break. If I used Perl instead of Python this would be no issue, because Perl _is_ stable and reliable as hell.

In two years time most libraries will move to Python 3.0, and Django is unlikely to patch security issues for Python 2.4 by then. In other words, I either have to spend a week upgrading stuff that already works, or I'll have to use insecure libraries. I don't like those options at all. How can we write solid and stable software if we don't have a stable foundation to build on?

C++ code I've written a dozen years ago still compiles and runs fine. Executables still run when double-clicked. If I scpy a Django directory to a different server I'm __surprised__ if it just works. Doesn't that say enough?

Python exists 20 years now and the spec is still not stable. Any Python library written today won't run on the computers of tomorrow. That's a pretty big price to pay for a print() function.

gaius4• 17 years ago

I'm not sure how true this is. I was into Java in '95 when it had "lousy IDE support, buggy compilers, few libraries, few bindings to existing libraries, and so on". Java fell far short of its potential, and the ambitions Sun had for it, but still, that wasn't a bad call. People who got into Python early on made a similarly good call.

And also, some esoteric languages, like Haskell, the ML family, Erlang and so on have deep roots, they may not be mainstream but they aren't going away anytime soon. And sometimes something funny happens; anyone who has ML experience is going to feel right at home on F# and suddenly the library problem is solved, the IDE problem is solved, the bindings problem is solved...

gizmo13• 17 years ago

If you were a .NET guy, would you consider building your startup on F#?

masklinn6• 17 years ago

> some esoteric languages, like Haskell, the ML family, Erlang and so on have deep roots

None of Haskell, ML or Erlang is an esoteric language.

masklinn6• 17 years ago

> Esoteric languages are a lot of fun.

No esoteric language was mentioned in TFA.

> But they're not very productive

Esoteric languages aren't "not very productive", they're the most counter-productive langages on the planet. Productivity runs counter to the point of an esoteric language.

> Messing around with unstable languages is a lot of fun. It's educational too. But don't build your business around it. Python/Django and Ruby/Rails are only barely stable enough to work with. Thinking that using Haskell, Clean, IO or Lisp will be a net benefit to productivity is foolish.

Wow. Cocaine is one hell of a drug.

almost• 17 years ago

Now I don't agree with his points exactly either. But you and I know full well what he meant when he said "Esoteric Languages". Arguing with people by intentionally misinterpreting their statements just makes you look stupid and it lowers the tone of the whole discussion.

tomjen22• 17 years ago

I am not sure you will be that much less productive if you go from Java to Scala. You know most of the libraries, and so code completion is less of an issue and Scala removes many of the problems that are in Java.

plinkplonk5• 17 years ago

" lousy IDE support, "

This is valid in a sense, but it is also important to remember that a lot of software was built with emacs and vi or even more ancient editors.

"If you're serious about software, concentrate on writing something that is reliable as hell and easy to maintain and install. "

That is terrible advice, not so much the sentence itself as the "stay away from powerful languages" spin you give it, as in

"Messing around with unstable languages is a lot of fun. It's educational too. But don't build your business around it. "

There are profitable companies using Haskell and OCaml (and Lisp and J and .. )!

You make a decent point then spoil it by stretching it too far, to the point of farce.

One situation in which you should definitely not use advanced languages is when you know you have (or intend to hire) very dumb "programmers" who won't be able to warp their heads around the "exotic" language, which probably doesn't have a "Learn in 21 days" book to go with it.

Many companies doing outsourced software do hire these "middle of the curve" programmers and will do well to stay with "stable" languages. If you have a bunch of bright programmers doing something innovative, it is a completely different situation.

By your logic PG wouldn't have built Viaweb in Lisp, FlightCaster wouldn't be using Clojure and so on.

Taking just one example, Clojure is evolving very fast, which for you would be "unstable". I know at least 4 startups using Clojure, one using Scala and no one's yet complained of "instability" and lack of IDE support destroying their productivity. There are tradeoffs (IDE maturity being one, if you are an I-can't-program-productively-without-my-IDE kind of guy ) with any language selection, but the consequences aren't as dire as you make it out to be.

As I said, this is bad (or rather very one sided, which amounts to the same thing in practice) advice. There is an excellent rationale for building certain types of applications in powerful languages, what PG calls LFSP - languages for smart people.

PG and other people have made this argument more eloquently than I can, so I'll refrain.

"Python/Django and Ruby/Rails are only barely stable enough to work with."

heh heh! Sure, What do you use then? COBOL? Should be "stable" enough for you I guess.

Somebody should tell all the stupid programmers using RoR or Django to run companies that they are being foolish, using "barely stable" software. ;-)

gizmo13• 17 years ago

Go easy on the sarcasm and ridicule; it doesn't help you make your case.

PG built Viaweb in Lisp because there was no Ruby, Python or Perl back then. When there are no libraries to chose from, you pick a language that lets you write libraries quickly. Today we boldly go where many have gone before, and that means that we can benefit from technologies others have built. Standing on the shoulders of giants, and all that.

I'm arguing that if you want to run a business you should be pragmatic about the language you pick.

The second half of your post seems to consist of mere hyperbole and strawmen (COBOL? Right). If there is excellent rationale for picking an esoteric language, I'd love to hear it.

allending• 17 years ago

I've found that the language (new or old) doesn't matter as much as the experience and depth of the user when it comes to well designed maintainable code. Learning times may vary, but these users try to do things well in any language.

anonanon• 17 years ago

What library is it that you're missing in Haskell but have in Java? What libraries do you feel are unstable? Have you actually programmed anything in Haskell?

The Java language in its current form is from 2004. The Haskell language in its current form is from 1998. Both originally appeared in 1990. Lisp is much older.

jimbokun3• 17 years ago

Which shows a potential danger of using more "stable" languages. Maybe the biggest flaw of Common Lisp is that the standard lacks many features necessary for modern development, like networking, threads, and Unicode. I think that the C++ standard lacks a String class (please correct me if I'm wrong).

So sometimes "stable" can degenerate into "frozen solid." Not having new libraries is arguably as bad as not being stable.

jimbokun3• 17 years ago

I've been using Clojure to "sketch" solutions for a Java project, then translating the results back into Java when I'm happy with it.

This gives me a REPL I can use to type in expressions referencing my existing code base and immediately see the result. Clojure has macros that make typing in many Java expressions require even fewer parens than Java itself! PG came up with the sketching analogy to describe what its like trying things in an interactive environment, immediately seeing the results, then modifying little by little to eventually get the result you want.

When I switch back to Java, on the other hand, I feel like I'm chiseling my code in granite. I need to have all of the boiler plate just so before I can see any results at all, followed by a compile and run or debug cycle. Test driven development makes this a little better, but just a little.

plinkplonk5• 17 years ago

@jimbokun "I've been using Clojure to "sketch" solutions for a Java project," and "When I switch back to Java, on the other hand, I feel like I'm chiseling my code in granite." remind me of this bit from SICP

"Pascal is for building pyramids -- imposing, breathtaking, static structures built by armies pushing heavy blocks into place. Lisp is for building organisms -- imposing, breathtaking, dynamic structures built by squads fitting fluctuating myriads of simpler organisms into place."

It seems that some things never change!

petercooper• 17 years ago

Because it's down now, a vague summary from what I saw.

Python and Ruby are now basically "mainstream" languages with little of the all new sparkle they had several years ago - little early adopter aura around them. What languages have that 2004 shine in 2009? Author decides to try Scala for this reason and likes it. Author thinks Scala is the new Python because people are excited by it.

wheels• 17 years ago

I have to say that, even as someone fascinated with Scala this week, I find the argumentation here a little disingenuous. There are a lot of ways to filter developers and I don't think that the ability to learn a new language sets the bar high enough to actually select for the good ones and potentially means selecting the wrong tool for the job.

There are a large group of people that are effectively programming language hobbyists, frequently learning new programming languages for kicks, and while that set intersects the set of good developers, it is not a subset of it.

gaius4• 17 years ago

The initial hump in hiring is always the shortlist - you might get a hundred (or a thousand!) CVs and you can maybe first-interview 10 and second-interview 2-3. So the priority is always to get down to a manageable list by rejecting candidates who are obviously unsuitable.

So you are correct, but you could probably even trust your HR department armed with a suitable list of esoteric languages keywords (instead of their usual "Java" etc) to build you a pretty good first-interview shortlist, even if you were after Java developers.

econoplas• 17 years ago

I disagree somewhat about the "not very productive" part, at least in the particular case of scala.

I've developed several large projects in Python (specifically CPython 2.x) for $$ for about 6 years and I am learning scala right now. So I have a somewhat interesting perspective on both.

Although I suppose I could be termed a Pythonista since I "do python" for a living, I consider myself more of a true polyglot... I also use Perl, Ruby, C++, Java, Lisp, C#, whatever language I can get my hands on that fits for the project at hand. I don't subscribe to the Golden Hammer theory.

So far, scala does feel like it has that 5x productivity improvement as compared to using straight Java, on par with what I experienced when I started porting significant portions of the code base of a large application from C++ to CPython (embedding CPython in C++, or vice-versa as needed).

I personally see scala being somewhat in a different category from CPython for 2 main reasons: JVM (stability, maturity) & Java interoperability (vast library resources).

1. The JVM is arguably the most widely used, stable, and mature virtual machine ever developed. Not surprising since Sun and IBM have both invested millions (billions?) in JVM development over the last 15+ years. The latest generation JVMs sport multi-threaded garbage collection, profiler support, remote debugging, remote method invocation, compile-and-reload while debugging, and very advanced JIT. JVM runs in more places than any other VM to date. 'Nuf said. Yes CLR is probably on similar footing nowadays. By comparison Python, Ruby, Perl, scheme, or any other language that has its own home-grown VM is always going to be behind that curve... for example, I was just reading about how most of the Common Lisps (many pre-dating the JVM) are still working on adding multi-threaded garbage collection.

2. The other (main) reason I'm looking at scala is the HUGE advantage of JVM interoperability and the vast libraries available.

From my Python & Java & other language experience, I have learned that:

=> cool-language-features-productivity-multiplier = 5x to 10x productivity improvement, such as what Lisp and most of the dynamic interpreted languages (CPython, Ruby, Perl) claim and pretty much deliver

=> existing-library-productivity-multiplier = 100x or more, since I don't have to re-invent every wheel every time for file format X reader/writer, GUI widget, or other library du-jour. Let's face it, we have to interoperate and communicate with a lot of different gizmos & whatzits nowadays, right?

We all know that we as programmers are vastly more productive when we can get almost any library imaginable (usually open source) to do almost anything we want... GUIs, Charting, FFTs/Numerics, networking, 3D graphics, Neural Nets, , and can even leverage Eclipse environment and toolkits for programming in the large for GUI building, writing rich-client platforms (building entire IDE's), Web server frameworks, web services & SOA development, DSLs & modeling, reporting, DB persistence, and then write our glue logic in scala or other JVM language of choice.

We can even intermix them with other JVM languages in the same project (Groovy, Jython, Jruby, clojure, PNut, etc).

The good news is... if I pick scala, and it doesn't work out... I can always selectively replace out the pieces I don't want to keep one-at-a-time in Java or any other JVM language as time permits. But if it works, I'll just leave it alone. Nobody even needs to know my JAR is written in scala except that they have to put a couple of scala JARs on the classpath. Not so for CPython where the installation & deployment overhead is much steeper in my experience. Not everyone has CPython 2.6 installed on every machine for example, but most everyone has a JVM 1.5 or greater whether they know it or not :)

Wow.

boskone• 17 years ago

Scala is _not_ some esoteric MIT, CMU language to make a point exercise. Its a pragmatic and practical attempt to "fix" Java so it has all the capabilities of Java with the proper foundational coherence which allows for additional, some yet to be defined or added, capabilities. Java has long ago painted itself into a corner. It has reached its evolutionary dead end.

As someone who lead a team which wrote one of the largest retail websites in the earlier days of Java, and as someone who has spent the last year rewriting legacy business logic in Scala, there is no comparison.

Scala wins not because its the new hipster language, it's distinctly superior across the board. You can write better, clearer, far more robust, far more scalable and far more easier to maintain and modify business logic at higher levels of productivity. That and its ability to work with legacy Java code makes it ideal. This is from direct and in-depth experience.

My concern with gizmo's post is its complete lack of claimed experience. No context of based on my experience where ... so we ... and then did a POC comparison with ... and found .... and concluded ...

In fact it appears very much to be a direct quote of from the preamble from the "Enterprise IT Managers Survival Handbook For Those Without Experience Or Knowledge Of Software Development Fundamentals"

Privately I've played with all of them. I love the shear outrageous power and beauty of Scheme (PLTs impl) and Haskell, the base competency of SML, and I've even used them in the enterprise in isolated in very few situations where the situation demanded it.

However, never once did it cross my mind to even "think" to introduce any of them into the enterprise (well a brief fling with Python for our operations group, it was found to be too hard).

Scala deserves to win. It is decidedly superior, however, lots of deservedly good things, in fact most, never see the light of day in IT.

gizmo13• 17 years ago

Note that I primarily object to the "let's do a startup with language FOO, because then everybody can see how cool we are and every smart programmer will want to join us because the language is so awesome!" attitude.

It's fine to take a calculated risk (Scala) if the direct benefits are measurable and significant. I'm not claiming that people should dogmatically stick to languages from the 90ies because they're safe. I'm saying that you need a very good reason to use a language with an uncertain future.

Also note that I (deliberately) didn't put Scala in the list of foolishness that contains Clean and IO.

plinkplonk5• 17 years ago

"Note that I primarily object to the "let's do a startup with language FOO, because then everybody can see how cool we are and every smart programmer will want to join us because the language is so awesome!" attitude."

(a) no one ever diplayed such an attitude on this discussion or put that argument forward. Strawman.

(b) This is what you said.

"Messing around with unstable languages is a lot of fun. It's educational too. But don't build your business around it. Python/Django and Ruby/Rails are only barely stable enough to work with. "

which is different from reacting to the non existent "everybody can see how cool we are if use language foo" argument.

"I also note that I (deliberately) didn't put Scala in the list of foolishness that contains Clean and IO."

But you did say " Thinking that using Haskell, Clean, IO or Lisp will be a net benefit to productivity is foolish."

You never really showed where the "foolishness" is.

Also, from your original list that still leaves Haskell and Lisp as languages that don't "provide a net productivity gain", which is a very strange (and as pointed out by boskone above, unsubstantiated) claim. There are profitable businesses built on both these languages (lisp is a family of languages but still .. ), and very capable programmers being massively productive using both.

You said something stupid with nothing to back it up, got hauled over the coals and are now backtracking trying to make a less objectionable claim.

You then claimed Perl didn't exist in 1995 (!) and that's why PG used CL for ViaWeb (!).

quoting boskone "In fact it appears very much to be a direct quote of from the preamble from the "Enterprise IT Managers Survival Handbook For Those Without Experience Or Knowledge Of Software Development Fundamentals"

Indeed.

The Python Paradox is now the Scala Paradox

Related Stories