The speed, size and dependability of programming languages

http://gmarceau.qc.ca/blog/2009/05/speed-size-and-dependability-of.html

by trjordan • 17 years ago

176 56 17 years ago

gmarceau.qc.ca

Confusion• 17 years ago

Shows and compares graphs plotting performance vs. code size, based on the programming language benchmarks at http://shootout.alioth.debian.org/. Note that 'dependability' here means 'the consistency of the performance or code size'.

Few solid conclusions can be drawn: small performance benchmarks are hardly indicative for both the code size and the performance of most actual projects in a language.

petercooper• 17 years ago

The references to Ruby aren't entirely what they seem in here. They seem to be relating to Ruby on the older shootouts, rather than the recent ones. That is, the uber-slow Ruby 1.8, rather than the Perl-beating and PHP-matching Ruby 1.9 which is more closely related to "yarv" on this list.

DavidMcLaughlin• 17 years ago

> Perl-beating and PHP-matching

Which is interesting considering Perl beats PHP on both axis in the linked article.

bad_user• 17 years ago

Maybe that's because Ruby 1.9 isn't being used in production yet. I tried using it but many gems are still incompatible with it.

SwellJoe2• 17 years ago

rather than the Perl-beating and PHP-matching Ruby 1.9

Maybe I just can't read, but Ruby 1.9 appears to still be slower than Perl on more than half of the tests. I wouldn't call that "Perl-beating".

draegtun• 17 years ago

Its great to see Ruby so much faster with 1.9 because 1.8 speed was such a stumbling block in data intensive stuff that I pushed towards it.

However don't count your chickens just yet with regards to Ruby 1.9 being "Perl-beating" ;-) Don't you think its odd that Perl 5.10.0 sits only just above Ruby 1.8.7 in the current alioth shootout? Quick look at the tests and I can see a problem with a known 5.10.0 speed issue bug. Fixing this in a quick local test here makes all the difference ;-)

igouy2• 17 years ago

> Fixing this in a quick local test here makes all the difference ;-)

It makes all the difference to that one benchmark - not to the overall situation.

silentbicycle13• 17 years ago

Very good graphs. Aside from X (time) and Y (code size), the center of the star implies "typical" performance, and the relative spread of the star shows specific data points. Ruby, for example, shoots out from the bottom right - to me, this says, "This is quite expressive, and if you know what you're doing, you can get reasonably good performance". It doesn't show memory usage or sample size, however - Rebol probably has significantly fewer samples than Python, for example, but it only looks like half as many.

It's also interesting how the LuaJIT star has about the same shape as the Lua star, but with 1/3 the width (which is consistent with my experience); Psyco relates similarly to Python. Surprisingly, the Lua star's center appears to be at the same X as GHC's, but lower down, and LuaJIT's overlaps OCaml's. While I can accept well-written Lua being approximately as expressive as Haskell (depending on the problem), I don't think it's in quiiiite the same ballpark as OCaml, speed-wise.

The most interesting shape, to me, is Io's: While the default performance seems pretty poor (the right edge meaning 8x as slow as the left edge), there are bands shooting all the way to the left, without adding any real height. I don't have any experience with Io (I'm curious, but have never had success porting it to OpenBSD/amd64); can anybody here comment on this?

magv• 17 years ago

> The most interesting shape, to me, is Io's

There seems to be a mistake. If you compare Io to JRuby [1], you'll find that Io is always the slower one, even though the picture in the article suggests otherwise.

[1] http://shootout.alioth.debian.org/gp4/benchmark.php?test=all...

silentbicycle13• 17 years ago

Ah, ok. Also, it seems to be missing about half the benchmarks.

by• 17 years ago

"Ruby, for example, shoots out from the bottom right - to me, this says, ... if you know what you're doing, you can get reasonably good performance" Sorry, but it does not say you can get reasonably good performance, it says this version of Ruby is slow, one of the slowest languages in the Shootout. And the graph itself says nothing about the skill level required to obtain the exhibited performance.

silentbicycle13• 17 years ago

> And the graph itself says nothing about the skill level required to obtain the exhibited performance.

In all fairness, neither do most people promoting C++'s performance-at-any-cost design.

I think a large spread in the performance correlates with requiring skill level, though -- it means that some people using the language get good results, but that many also get very bad ones. Ruby is a really slow language ("reasonably good" < great), but it's still good enough for many purposes, provided your algorithms are reasonable.

carbon8• 17 years ago

"Ruby, for example, shoots out from the bottom right - to me, this says"

"yarv" is the graph that represents what is now Ruby 1.9, "ruby" only represents Ruby 1.8

mullr• 17 years ago

Io can do fast vector processing if you use its vector library. That's what I assume is going on.

bayareaguy2• 17 years ago

Languages which include functional features such as lambda, map, and tail call optimization are highlighted in green.

I found one minor inconsistency: Lua has all these but is still colored blueish.

chops• 17 years ago

It doesn't have a native map command, but that's a rather trivial implemention.

But I agree, it should be colored green in the "functional programming" section.

silentbicycle13• 17 years ago

Lua isn't usually associated with the functional languages. Besides mainly being promoted for embedded / extension use, I think the single biggest reason for this is that, while it has tail-call elimination, closures, first class functions, etc., you still have to say "return x" at the end of functions. I do a lot of FP-style stuff in Lua, and while the language doesn't actively resist it the way Python does, it's not the main direction its design flows.

It seems like a small thing syntactically, but languages that return the result of their last expression by default (I think of them as "expression" languages), and that require an explicit block for side-effects (begin in Scheme and OCaml, progn in Common Lisp, etc.), tend to get idiomatically used in a functional manner, while languages that default to a series of statements with side-effects and require an explicit return at the end ("statement" languages) typically do not. While you can use several of the latter languages for functional programming, only the former are functional "by default".

Saying "function(x) return x + 1 end" is a tiny bit more trouble than "fun x -> x + 1" or "(lambda (x) (+ x 1))", so while lambdas still get used where they're the best fit (as arguments for map or filter, for example), they're less common overall. Defaults influence a language's overall style at least as much as what's theoretically possible. Python's concept of what's "pythonic" seems to be the most explicit acknowledgement of this.

Incidentally, I would love to see a language that crossed Lua with the ML family: a small, portable, clean implementation that interfaces easily with C, but with type inference, ML's top-notch module system, and (ideally) Haskell-style type-classes. OCaml is such a good language that I'm willing to tolerate its warts, but I'd strongly prefer a minimal ML-like language. (FWIW, I haven't used SML, though that's just because I do most of my programming on OpenBSD/amd64 and the smlnj port is i386 only.) When standalone, Lua is best suited to small/medium projects. Its historical emphasis on embedding in an existing project has made a module system for standalone use a bit of an afterthought.* As a project gets larger, static checking and stabilizing the module interfaces becomes far more important, and this is one area the ML languages really excel.

* For a good summary, see "Almost Good Enough to Scale: A Lua Mail Handler and Spam Filter" (http://www.lua.org/wshop08.html#ramsey)

gruseom3• 17 years ago

you still have to say "return (val)" at the end of functions

We have a bunch of code that compiles to both Common Lisp and Javascript, and those nasty "return" statements are one of the two biggest mismatches between the two languages (the other being Javascript's weird semantics around null, false, 0, and ""). One of these days I intend to try inserting them all automatically.

nazgulnarsil• 17 years ago

dependability mostly is a stand in for a program making more assumptions for you no? In a language that makes a lot of assumptions its easier to create something where assumptions conflict somewhat invisibly.

herdrick• 17 years ago

This is great.

If you've looked at Peter Norvig's spell checker shootout you won't be surprised much. Ex., the story on Common Lisp and Scheme implementations is the same here: MzScheme is the mediocre best of an unimpressive lot. (I want to see Clojure and Arc on this.) Python looks great again. F# looks weak here but wins big in the spell checker with reasonable-looking code. Sounds interesting.

What's the deal with Squeak? I had thought that it very concise.

The meta-message here is best of all: people are measuring language conciseness.

silentbicycle13• 17 years ago

Well, arc's running on top of MzScheme, so it's likely to be a bit slower. Also, Stalin in the left column is a Scheme implementation, albeit one that is very restrictive for sake of raw speed.

Also, it's worth noting that the features in a language that make large systems' codebases manageable (such as good module systems) are different from the features that allow you to pare down small scripts further. It's hard to show features supporting conciseness-in-the-large in a one page benchmark, so IMHO the latter is often greatly overemphasized. Being able to knock 2 lines off 20 is often just a parlor trick, though, while 50kloc off of 200kloc can mean life or death for a project. The programming world would be a happier place if the size of every legacy C++ / Java codebase were cut by a double-digit percentage.

granular2• 17 years ago

> > (I want to see Clojure and Arc on this.)

> Well, arc's running on top of MzScheme, so it's likely to be a bit slower.

I'd be interested to see how Arc fares on the code-size axis.

avibryant2• 17 years ago

Just a guess, but: Squeak only uses text files as an export format, and that format is more verbose than file-based source code would normally be. If you only counted the code and not the metadata, it would probably do better.

igouy2• 17 years ago

Not much - Smalltalk code is written in sentences not squiggles.

The productivity comes from reading and re-using the code that has already been written rather than duplicating stuff.

stonemetal• 17 years ago

Also note that in the benchmark it is a gzip of the source so it is more about how well your source code compresses.

russell• 17 years ago

It would be a useful to include the language name where it is not obvious. For example, he mentions Haskel, but I cant tell which one it is.

silentbicycle13• 17 years ago

It's "ghc", in the chart.

unbannable• 17 years ago

For this sort of comparison, mostly-functional multi-paradigm languages like SBCL and Ocaml should be run through the benchmarks twice; once with imperative code, once in the functional idiom. The often-repeated complaint (of which I'm skeptical) about functional languages is that they're only efficient when ugly imperative features are used, and it would be good to confirm or dispel that suspicion.

silentbicycle13• 17 years ago

It's more complicated than "functional is slower than imperative". Functional data structures are sometimes less immediately efficient, but add persistence (since they're immutable, they share subsections of the data structure, and/or have access to snapshots of past versions of the structure being modified). Whether this gives you needed features or just means extra work and cache misses depends on what you're doing. If you have a functional data structure and you want to add "undo" or backtracking, you're already most of the way there, while adding this to an imperative data structure means extra work to keep track of state changes.

_Purely Functional Data Structures_ by Chris Okasaki is, by far, the best collected resource, if you want to read further. (His thesis (http://www.cs.cmu.edu/~rwh/theses/okasaki.pdf) is the kernel of the book.) Also, the fourth chapter of Developing Applications with Objective Caml (English translation: http://caml.inria.fr/pub/docs/oreilly-book/) discusses functional and imperative styles' strengths and weaknesses.

Reducing everything to runtime benchmarks obscures how, in practice, the trade-off is often more like "it was a huge pain in the ass to get working without bugs, but it's 2% faster" versus "it took twenty minutes and was correct on the first try". (Also, whether or not you use it in production, OCaml rules for prototyping complex data structures, in the same way Erlang does concurrency and C++ does linking errors.)

javert• 17 years ago

This is one of the best things I've seen on the Internet in a while. :-) Definitely worth taking a look.

ellyagg• 17 years ago

Interesting. Looks like Yegge was right in falling in love with ocaml awhile back.

tlrobinson2• 17 years ago

The benchmarks used for the article was from 2005. Make sure to check out the 2009 versions:

http://gmarceau.qc.ca/blog/uploaded_images/size-vs-speed-vs-...

V8 has propelled JavaScript into the coveted bottom-left corner, which is awesome. SquirrelFish Extreme should be right up there with it, if it were included.

SwellJoe2• 17 years ago

More reason to believe JavaScript is going to be huge on the server-side. As concise and (nearly as) powerful as Python, Perl, and Ruby, but getting faster at a much more rapid clip...and already required for the front-end, so everybody on the team is already familiar with it. Library coverage will take another couple of years to reach a reasonable level, but I'd bet on it happening.

kinghajj• 17 years ago

I agree. Once I learned how truly powerful Javascript really is, once you get past the horrible DOM, I knew it could one day become a general-purpose scripting language and not just a web one. As soon as someone makes a good command-line Javascript interpreter, web apps will come as soon as someone creates a CGI extension to that interpreter.

akkartik• 17 years ago

I'm amazed gcc is slower than java.

And it's more expressive than scala and f#? Something's gotta be wrong there..

fauigerzigerk• 17 years ago

You need to look at how many data points each language implementation actually produces. gcc runs all programs whereas java steady state (the one that looks faster than gcc) lacks a lot of data points (i.e programs). As you can see in the diagram gcc's star is pulled towards the right by one data point (at least it looks like one). If java steady state ran all programs its star might shift to the right as well. Or it might not. It's just not comparable as it is. You need to look at the actual numbers instead of the diagrams.

My own benchmarks that I run on our heavily algorithmic long running code consistently show Java about 10% to 20% slower than gcc (C++) and memory usage about twice that of the C++ implementation (excluding the JVM memory itself). C and C++ performance depends heavily on avoiding malloc/new. C/C++ code that gratuitously allocates and frees heap memory tends to be much slower than Java code.

JulianMorrison• 17 years ago

I wonder why mlton doesn't have a higher profile. Is it hard to use, or something?

jongraehl2• 17 years ago

I looked at SML and there seemed to be a few warts compared to Ocaml: http://www.mpi-sws.org/~rossberg/sml-vs-ocaml.html

mlton may not allow separate compilation (it does whole-program optimization at least), and it's pretty slow even on small stuff, but if you were really using just the Standard ML language, you could use a faster compiler for development. I'd probably want to use some extensions, and I wasn't sure if the intersection of mlton's extensions with some other implementation could satisfy me.

silentbicycle13• 17 years ago

mlton seems to be used for optimized compilation for releases, while general development is typically done in another SML environment, such as smlnj.

I don't have really concrete experience with SML, just OCaml, which also has its share of warts. It's a fantastic language in certain niches, though; generally, the sort of niches for which dynamic / "scripting" languages are usually a poor fit. (I wouldn't write a serious compiler in Python, for example.) They're very complementary.

I think the ML family has a lot of advantages, but the community is pretty closely tied to academia (much like Haskell, but without the buzz). While that in itself is probably neither good nor bad, in this case a lot of the terminology used is especially math-y; There are many features ML has that programmers would find tremendously useful, except they're usually discussed using Greek letters, which is insular. I got a copy of The Definition of Standard ML from Amazon, hoping to get further insight into the language's design, and found it completely impenetrable. (Then again, I'm not a grad student.)

Newer languages seem to be using type inference (finally!), but as far as I know, nobody has really ran with the design of its module system.

ck113• 17 years ago

I thought this was the most telling line of the article: "Ultimately the first factor of performance is the maturity of the implementation."

That supports a common conviction held by fans of functional programming: if all of the years of arduous optimization that have been poured into GCC had instead been poured into (say) GHC, then Haskell would be even faster today than C is.

That is, to many people functional programming languages seem to have more potential for performance than lower-level procedural languages, since they give the compiler so much more to work with, and in the long run a compiler can optimize much better than a programmer. But so much more work has been put into the C-style compilers that it's hard to make a fair comparison. It's still hard, but this experiment seems to give some solace to the FP camp.

jongraehl2• 17 years ago

Haskell may potentially be compiled to within epsilon of C when using strict evaluation + mutable state + unboxed types (look at the shootout implementations; the majority are in this style), but lazy evaluation and functional data structures have a real cost.

It's not merely that the ghc developers have been insufficiently clever and diligent.

The speed, size and dependability of programming languages

Related Stories