Haskell is 4.2x as good as Erlang at lightweight concurrency. Give it some love.

http://shootout.alioth.debian.org/u32q/benchmark.php?test=threadring&lang=all

by dfranke • 17 years ago

125 35 17 years ago

shootout.alioth.debian.org

davidw• 17 years ago

Go build something with it, like a phone switch, then we'll talk.

Kidding aside, I think the fact that Erlang got thrown into the production ring so early is perhaps a hindrance to its growth in some ways. They can't just go fiddling with the language at this point without some serious breakage, whereas newer languages like Scala and Clojure would probably not create major problems should they go through a few incompatible versions. Yes, people are using them in production, but not for things like phone switches!

gaius2• 17 years ago

Haskell has amazing potential... But it's hard to argue with Erlang (running Ericsson's 24/7 real-money business), OCaml (running Jane Street's 24/7 real-money business) and so on. Is there anyone with serious performance and reliability requirements that's bet the farm on Haskell (yet)?

olefoo• 17 years ago

http://www.galois.com/ is a mostly haskell shop.

It's definitely an interesting language; but I would see it as more competitive with C, C++ or Java as a language for building high reliability components.

jerf2• 17 years ago

Haskell needs an equivalent to OTP and cross-process/cross-server message passing before it can seriously replace Erlang.

I say this as someone who would like to see that happen (though I have no resources to do it myself). But until that happens, Erlang has a niche that Haskell can't fill; you can toss robust, multi-machine-scalable servers together in Erlang way faster than you can put them together in Haskell. It's not something intrinsic to the language, it's just Haskell doesn't have the libraries.

dfranke4• 17 years ago

There was an effort to do this, but it seems to have fizzled: http://www.macs.hw.ac.uk/~dsg/gdh/

felixmar• 17 years ago

Mature, consistent and simple to use libraries with good documentation is what Haskell lacks most. The foundation is great. Scalable servers could be a good niche for Haskell, but it would probably need backing by one or more companies to get there.

silentbicycle• 17 years ago

To my understanding, a major reason that Erlang was written was because they needed a concurrent system with soft real-time guarantees - in order to work for telephony, it had to operate within predictable, bounded amounts of time.

Conventional Prolog wasn't an option* , because the backtracking inherent in its execution model leads to significant variance for worst-case timing for individual operations. The same problem can happen with lazy evaluation, for both time and space - while the sophisticated compiler optimizations can allow for some really remarkable amortized performance, individual worst-case measurements can still have some serious outliers.

* Though Erlang was prototyped in SICStus Prolog.

With lazy evaluation, forcing a thunk for a seemingly minor operation can trigger a burst of postponed work. This can be a problem if it means a sudden drop in a game's framerate, for example, much like non-incremental garbage collectors' pauses have made people wary of using GC. Similarly, a JITting compiler can do more elaborate optimizations if it can afford to stall the program perceptibly while performing them, but that would disqualify the compiler for many real-world uses.

Real-time operation can impact performance, much like implementing full transactions slows down databases. Still, it's a fairly significant feature in its own right, and a language that does concurrency faster overall, but cannot prevent occasional erratic timing, is aiming for a different target than those (e.g. Erlang) that have such guarantees. It's not unusual for a program that juggles tens of thousands of connections to need to respond to them promptly, for example. Concurrency isn't just about performance.

(Also, that's why the Erlang movie has that "Declarative real-time programming NOW!" banner in the background. :) )

gdp• 17 years ago

Has anyone noticed that SML/MLton is gone from the newest round of benchmarks? It used to do quite nicely on the concurrency benchmarks, IIRC.

jhancock2• 17 years ago

also of note: Smalltalk VisualWorks 35.86 seconds vs JRuby 6 minutes

That's conceptual more of an apples-to-apples comparison.

cnlwsu• 17 years ago

What surprised me the most on this graph was how well java performed (bearing you buy enough memory ;))

jhancock2• 17 years ago

yeah, at over 8x the memory, it beat an aging Smalltalk VM by 4 secs ;)

neilc3• 17 years ago

I'm confused: the benchmark notes explicitly say:

Programs may use kernel threads, lightweight threads; but coroutines, cooperative threads and other programs with custom schedulers will be listed as interesting alternative implementations.

However, the Haskell docs for Control.Concurrent note that in Haskell,

Concurrency is "lightweight", which means that both thread creation and context switching overheads are extremely low. Scheduling of Haskell threads is done internally in the Haskell runtime system, and doesn't make use of any operating system-supplied thread packages.

Which makes the comparison with a C implementation that uses kernel threads pretty unfair: presumably a C implementation that uses green threads / fibers would be much more competitive with the Haskell implementation. Comparing apples to oranges isn't very informative.

As an aside, choosing a language based on how fast its green thread implementation happens to be at passing tokens around a ring is utterly silly.

jrockway5• 17 years ago

You can compile your app with the threaded runtime, in which case the threads are scheduled amongst the available CPU cores.

Edit: The benchmark in question does just this, with 5 cores. So while the docs you cut-n-pasted are technically accurate, they do not apply to this particular run of this particular benchmark.

neilc3• 17 years ago

Well, fair enough: presumably GHC is creating 5 kernel threads under the covers, whereas the C implementation creates 503 kernel threads. Therefore the C implementation incurs approximately 100x the context-switching overhead. Once again, apples to oranges.

the threads are scheduled amongst the available CPU cores.

Actually, the per-CPU idle stats suggest that the Haskell program ran entirely on a single CPU, so it wasn't actually utilizing all 4 cores anyway.

mr_luc4• 17 years ago

I have a question about the CPU column:

> 0% 0% 0% 100%

This was a Quad Core CPU, so my first impulse is to ask, does that mean it was running on only one core?

dfranke4• 17 years ago

Yes. It was compiled with the threaded runtime, but the nature of the problem is such that all but one thread should be blocked at any given time, so the Haskell runtime scheduled them all on the same core.

mr_luc4• 17 years ago

That's pretty cool! Thanks for the informative response.

Yeah, when I looked at the definition of the problem, it sort of made sense.

Le sigh ... dammit, I should give Haskell more of my attention.

I just hate camel casing and type errors :( And if I'm going to procrastinate about writing For Reals code in a language, various lisps have that covered already.

Although, I'm writing a For Reals app in Ruby (w/merb) at the moment, and it's turning into a bit of an uggo. Maybe I'll procrastinate about finishing it, and give Haskell another go.

gaius2• 17 years ago

You'll notice they all add up to 100%... In this (artificial) test, only one thread is doing work at any time no matter on which core that thread is scheduled. I don't know if it was deliberate (i.e. a clever optimization) but if you're going to use your threads like that it makes sense, avoids cache misses.

ryanpetrich• 17 years ago

I wonder what sort of optimization Scala is employing ;)

wglb• 17 years ago

You can tell by looking at the colums to the right--they show the load on each of the cores.

What is particularly interesting is that Haskell uses only one CPU to beat Java which is using four.

jubos• 17 years ago

I am curious what qualifies as true CPU concurrency vs. threading tricks/shortcuts in the implementation of a language. If you look at java-6, c++, pascal, c, and ocaml, those are the top performers that actually utilize all 4 cores, where as haskell and erlang use 100% of one CPU. Also, in the interesting alternative programs section, the top performer is java (by cheating that they mention Haskell probably does behind the scenes)

jerf2• 17 years ago

See dfranks' message. If he's right, in this context being 100% on one CPU is a feature, not a bug.

jrockway5• 17 years ago

Also worth nothing that what GHC does in 8 seconds, Ruby does in 48 minutes. This is why I always chuckle when people get super-excited about multi-core concurrency in langauges like Perl, Python, and Ruby -- at best, you will make your app 4-8x faster. But by moving to a different language, you can make your app 360x faster.

Something to keep in mind.

neilc3• 17 years ago

But by moving to a different language, you can make your app 360x faster.

If your "app" is a toy microbenchmark with an under-specified benchmarking methodology, then sure. You could also make your "app" 360x faster by switching to a userspace thread implementation.

jrockway5• 17 years ago

Lightweight threads are a much better fit for many applications. If you are writing one of those applications, this benchmark shows that you can get great performance using Haskell... with only a few lines of code.

I am sure C would do better with a proper userspace thread library, but nobody has submitted such a program to the shootout yet. "If it's so easy..."

thenduks• 17 years ago

I see your point but unfortunately using Haskell doesn't magically also make your database (/etc other bottlenecks) 360x faster, so... Ruby it is :)

jamesbritt• 17 years ago

And you can call Haskell from Ruby

http://github.com/mwotton/Hubris/tree/master

jrockway5• 17 years ago

I have written a lot of database-driven web applications in dynamic languages, and the database has never been the bottleneck. It has always been in the application code.

(Actually, not true, when I worked for $LARGE_ADVERTISING_COMPANY, we checked the state of the session in the database for every request. This put an enormous load on the database, and made every page load take at least two seconds waiting for locks. No, I did not design any part of this system, nor was I allowed to fix it; so it doesn't count :)

kscaldef• 17 years ago

It should probably be noted that what's actually going on here is that there's some issue in the Erlang runtime that's hurting it substantially on the quad-core box. On the single-core benchmarks, it speeds up substantially and is neck-and-neck with Haskell.

Partly there's an issue here that the priority for Erlang has always been more multi-box distributed systems rather thaan single-box concurrency, and partly there's an issue that Erlang's not-really-open-source nature seems to hold back progress in fixing issues like this. And, partly there's the fact that the Erlang core team doesn't really care at all about micro-benchmarks like this.

Haskell is 4.2x as good as Erlang at lightweight concurrency. Give it some love.

Related Stories