My experience with the computer language shootout

http://alexgaynor.net/2011/apr/03/my-experience-computer-language-shootout/

by kingkilr • 15 years ago

95 59 15 years ago

wriq• 15 years ago

It's a shame that it all seems to be up to one person and what he will allow considering how often the shootout is referenced in discussions. I wonder if Mike Pall's experience was similar when he posted the Lua/LuaJIT versions.

scythe2• 15 years ago

Lua still has no regex-dna because the standard string functions aren't proper regexes (no choice operator) and LPeg isn't available as a Debian package (which is the real problem).

'course, it'd be hard for LuaJIT to do much better in the shootout than it does now. It's beating C#.

rtaycher• 15 years ago

Better(actually faster on 1/3, smaller on a 1/3) then mono!!!/c# http://shootout.alioth.debian.org/u32/benchmark.php?test=all... , mostly slower then java -server http://shootout.alioth.debian.org/u32/benchmark.php?test=all... . Still impressive.

mikemike3• 15 years ago

If the submissions using low-level types would be included in the total score, LuaJIT would show at around 1.5x instead of 2.5x.

pygy_• 15 years ago

AFAIK, the LuaJIT entries use custom, optimized scripts.

igouy16• 15 years ago

They certainly use Mike Pall's expert skill as a Lua programmer and implementer of LuaJIT ;-)

But they are Lua programs measured on both the Lua interpreter and LuaJIT.

(Programs that rely on the LuaJIT only FFI library and won't work with the Lua interpreter are shown separately.)

mikemike3• 15 years ago

Same experience here with LuaJIT. My submissions using low-level types (byte arrays and such) were put into 'interesting alternative', too.

Almost all other languages can use byte arrays, when they are the appropriate data structure for the job. The C submissions make heavy use of GCC extensions, Haskell gets to use mutable (OMG!) byte arrays and Free Pascal has about as much in common with Wirth's Pascal as the name.

But Python and Lua are not allowed to do that? Apparently not all languages are treated equally. Dismissing submissions by resorting to a flawed definition of 'standard' and then suppressing further debate is really lame.

I contributed almost all of the Lua programs to the shootout, but I do not feel particularly encouraged to continue contributing any programs.

igouy16• 15 years ago

Here's how Alex Gaynor compared his program to your programs-

"It's not used to do any crazy hackery like the LuaJIT one, just to access the libc write() function."

igouy16• 15 years ago

Please show that there's some substance behind your "suppressing further debate" accusation.

Obviously your opinion wasn't suppressed on proggit.

Obviously your opinion wasn't suppressed here.

And nothings been done to stop you posting in the discussion forum or commenting in the tracker.

igouy16• 15 years ago

It's up to you!

Take the measurement scripts (download from the Help page), measure programs and publish your measurements.

glenjamin2• 15 years ago

I don't see why the same program has to run on PyPy, CPython and Python3. Isn't the idea to do different implementations for each language. I know technically they expose the same (or a similar) language, but they're dissimilar under the hood.

I'd imagine the implementation varies across the other languages by more than just syntax.

The point that worries me the most, is the amount of microoptimisation being applied to these "benchmark" programs, making the results more or less pointless for real-world use.

igouy16• 15 years ago

There's nothing to say that all of CPython, Python 3, and PyPy must be measured.

For the moment, they all are being measured, so it's interesting to see that programs written for CPython might perform badly with PyPy, and programs written for PyPy might perform badly with CPython.

glenjamin2• 15 years ago

I think in general, programs written in "Python" will perform better in PyPy then CPython, but the current submissions are hyper-optimised for the implementation details of CPython.

jerf• 15 years ago

It's true that it's arbitrary; the problem is, there's no non-arbitrary alternative, so it's a weak charge to make.

Unless you're going to allow every language to converge on simply blasting some bytes into RAM and executing optimized machine-language code and watch all the languages cluster to the exact same location give or take how long it takes to load the machine code, you have to draw some lines about what's a "real" implementation and what isn't. If I were running it, I'd be even more strict and insist that the solutions must be "idiomatic"... and what's idiomatic and what isn't would also be arbitrary, because there is no escape from the "arbitrary". Yet the end result is useful. Not totally determinative, but it's not anywhere near "useless" either.

akkartik12• 15 years ago

Combining your comment and DarkShikari's (http://news.ycombinator.com/item?id=2404278), it seems like a good shootout should simply compare the simplest, most idiomatic solution possible in each language.

stewbrew• 15 years ago

The problem with a comparison of idiomatic solutions is that they rely in different degrees on what's in their libraries. IIRC the shootout tries to avoid this by comparing similar solutions using the same algorithm that are actually implemented with the interpreter/compiler in question. IMHO that's the more sensible way to do this.

igouy16• 15 years ago

If that's what you'd like to see then please make the measurements and publish the programs for that comparison!

enneff• 15 years ago

The irony here is that the proposed PyPy benchmark is more Python than the previous CPython-oriented one.

sambe• 15 years ago

I like your standard but I think the real criticism being made is not that it's arbitrary but that it's arbitrary in different ways for different languages. This starts to look like intentional dishonesty. Another standard closer to the status quo would be "anything portable". Why does one language require portability and not another? What is that actual problem with the example given? There doesn't appear to be one.

igouy16• 15 years ago

> Why does one language require portability and not another?

Because more than one language implementation was measured for that language but only one language implementation was measured for the other languages.

If only one language implementation was shown for Ruby it would be Ruby 1.9 - not JRuby

If only one language implementation was shown for Lua it would be Lua 5.1.4 - not LuaJIT

If only one language implementation was shown for Python it would be Python 3 - not PyPy

PyPy and LuaJIT are being treated more favourably than other language implementations.

DarkShikari• 15 years ago

In my experience language shootouts in general are often terrible by design -- with these being just a prime example.

A lot of the problem is in the questions they're designed to answer versus the questions people use them to answer.

For example, if I'm comparing Python and C, I typically want to know "how much slower would my program be in Python?", not "how much slower is my program in Python if I spent so much time hyper-optimizing it that I might as well have written it in C?"

But the test cases usually try to answer the latter, not the former.

dkarl• 15 years ago

It might be more reasonable than it seems at first glance. It's true that it's good to know how fast typical code runs, but there's another important question: when I run into performance problems and need to optimize a bottleneck, how fast can I make it before I have to resort to non-portable code or C extensions that complicate my build process?

dagw2• 15 years ago

Anybody writing, for example, Python code to solve these sort of problems in the real world would instantly reach for numpy. Which, while not part of the core language distribution, is pretty close to being a standard library for most python programmers. I'm sure several of the other languages have similar libraries that are being ignored in these benchmarks. Without taking things like that into account, theses results don't say too many useful things about real world performance.

justincormack• 15 years ago

Languages/implementations also vary in how much overhead switching to C costs you, especially in loops say, which eg the pidigits benchmark does sort of measure.

kingkilr2• 15 years ago

Maybe there's a space for "if you spent an average amount of time optimizing" :) For example my PyPy optimizations took maybe 4 hours total, with 0 time spent looking at assembly.

davidw4• 15 years ago

> In my experience language shootouts in general are often terrible by design

My experience is that this is the sort of thing where everyone rags on it, but no one actually attempts to provide something "better".

gaius• 15 years ago

Because there's always a "for what" in there. An an analogy, consider MMA. On the surface, it seems to answer the question "what's the best martial art?". But really, it only answers the question of what is the best martial art for fighting a single opponent in an octogon shaped ring in front of an audience aiming for a submission? And the answer is of course Brazilian Ju Jitsu, exactly the answer the founders of UFC wanted...

Former cop Rory Miller writes about this in his book, the police experimented with BJJ and found it useless. Why? Because in BJJ you pin your opponent on his back because it makes a better show for the audience, but as a cop you always pin your opponent on his front so you can handcuff him!

akkartik12• 15 years ago

Why does somebody saying "I wouldn't use this." have to provide an alternative? If such a statement is backed by reasons I find it interesting to read. They're saving me trouble trying it out, just like any other review.

igouy16• 15 years ago

> the questions they're designed to answer

You'd think there'd be some kind of statement about that?

http://shootout.alioth.debian.org/help.php#why

> the questions people use them to answer

You'd think there'd be some kind of advice about that?

http://shootout.alioth.debian.org/dont-jump-to-conclusions.p...

> I typically want to know "how much slower would my program be in Python?"

And we should all know the answer - It depends on how you wrote your program in C and it depends on how you write your program in Python.

akkartik12• 15 years ago

The question marks seem to suggest that you're poking a hole in grandparent's argument, but it seems like you're both in agreement that the shootout is misused. What am I missing?

onan_barbarian3• 15 years ago

The Shootout's C implementations do occasionally turn into a festival of SSE intrinsics and the associated unrolling. Some benchmarks, anyway.

Still, not using SSE when it's available is dumb and perhaps all the other languages need to start playing, too. A little tricker in Python, of course....

kingkilr2• 15 years ago

Well, that's what the compiler is for! PyPy uses SSE under the hood for various float handing things.

onan_barbarian3• 15 years ago

Unfortunately, SSE - especially in integer-land, so I suppose we're talking SSE2 and beyond - often requires more specialist care and feeding than any automatic method (compiler or run-time) can provide.

Some cases aren't hard to pick up (e.g. bulk operations on big arrays) but others require trickery of the kind that compilers usually don't (or couldn't) have.

This isn't made easier by the notoriously non-orthogonal nature of the SSE integer operations and the rather limited number of ways that you can get in and out of SSE-land (to, say, affect a conditional or get something into a GPR).

igouy16• 15 years ago

Also "J2SE platform version 1.4.2 now uses SSE and SSE2 instruction sets for floating point computations".

Luyt• 15 years ago

I once wrote a function which uses MMX to convert a (possibly very large) string to lowercase. That was pretty much faster than the library's strlwr() function. http://codepad.org/BeDqS1Ws

onan_barbarian3• 15 years ago

Great stuff. I think there are some potential tricks here to reduce the number of comparisons - maybe do a parallel subtract by k to pull down 'A' to -128 (smallest possible byte) then do your comparison against (ord('Z')-k). Or maybe push up 'Z' to +127..?

That way you can get a single comparison and can replace a pcmpgtb and pand with a single subtract. Then switch it to SSE2, unroll and you're good to go.

Alternately, http://www.azillionmonkeys.com/qed/asmexample.html in section 11 ("Converting Uppercase") contains a brainsmashing version of this entirely in SWAR ("SIMD Within a Register"), which could be adapted with a certain amount of pain (largely due to the absence of a double-quadword bitshift in SSE2, which is retardlepated).

scythe2• 15 years ago

Why not at the least accept the first submission for the PyPy entries? It shouldn't matter that it doesn't run on CPython -- if it's idiomatic Python and it runs on PyPy it looks valid.

At the same time, I can see where the guy is coming from with ctypes.

robryan• 15 years ago

Maybe you should have to submit and have it ran against a secret program that covers many areas of an implementation, so it's unknown precisely what needs to be optimized to improve on the test. Have a limit for how often you can retest it to so the only real way to be the fastest is to optimize lots of stuff which is a net win for language users.

My experience with the computer language shootout

Related Stories