The Great Ruby Shootout (December 2008)

http://antoniocangiano.com/2008/12/09/the-great-ruby-shootout-december-2008/

by acangiano • 18 years ago

41 32 18 years ago

hxa7241• 18 years ago

By my reckoning, that makes Ruby 1.9/Yarv the same kind of speed as Python. (At the slow end of the scale, but nothing to be ashamed of any more.)

old-gregg6• 18 years ago

I can't stand Windows, but wanted to point something out to all blind Windows haters out there: look at Vista vs Ubuntu numbers. Windows beats Linux running MRI by 10-50% in nearly all tests [when they pass] which aligns perfectly with my own observations.

Certainly such big difference can't be explained by more effective system calls. I attribute this to Microsoft's superior C/C++ compiler. Back in my C++ loving days I played with various compilers regularly, and MSVC has always been the most optimizing, and kept advancing rapidly from version to version. It also compiles 2-3 times faster than GCC if I remember correctly... uhhh I miss it. :-)

Also his JRuby startup time is inaccurate. No way JVM starts in under a second: it's more like 2 full seconds on his hardware. More likely he had some of JVM in the filesystem cache. Vista/OSX will pre-fetch JVM modules even after you reboot (not sure about Linux) if you run Java software and especially if you have 8GB of RAM...

kingkongrevenge2• 18 years ago

There are other faster options on Linux than gcc, you know. Doesn't icc benchmark faster than vc++?

I'm not sure why you don't hear more about gcc vs icc & borland in the open source world. They are free for non-commercial use after all.

smanek2• 18 years ago

Because icc's performance isn't great on non-intel cpus? And it isn't particularly portable (only available for Windows, Linux, OS X).

Plus, the open source world got burned pretty badly by BitKeeper. Now they want to use free (as in speech) tools.

nanexcool2• 18 years ago

Why is it that, more often than not, whenever someone says something positive about a Microsoft product or the company itself, they have to start with:

- I can't stand Windows but... - I'm 100% GNU/Linux but... - Microsoft and/or BillG can burn in hell but... - etc.

Why the need for justification? We're all informed people here. Can we state positive facts about Microsoft without fear of backlash? Can I post this reply without fear of it being buried?

old-gregg6• 18 years ago

Why the need for justification? We're all informed people here. Can we state positive facts about Microsoft without fear of backlash?

This wasn't the first positive post about Microsoft tech I've ever done. The answer to your question is: no we can't. :-) Just recently I got beaten to death by OSX and Linux fanboys on reddit for mentioning superior threading/process switching on Windows.

mixmax• 18 years ago

Yes we're all informed people, but we're also mere mortals suspectible to the same psychology as everyone else. Including instinctively downvoting something that doesn't fit with our worldview.

jpedrosa• 18 years ago

He tested Ruby 1.8.7 on Ubuntu and Ruby 1.8.6 on Windows.

Ruby 1.8.7 has added some features found in Ruby 1.9.1 in order to provide a forward path for those wishing to start adopting the new Ruby 1.9.1 features. For whatever reason, Ruby 1.8.7 has become a little slower when compared to Ruby 1.8.6, in general, at least during some testing I have done. So there's more than reaches the eyes in those numbers of the benchmarks.

Also, while Microsoft's C and C++ compiler provide lots of cool optimizations, Ruby is often compiled with just -O2 on Unix compatible systems because it's often fast enough, because Ruby is not known for trying to extract maximum performance from the bare metal anyway, so at the end of the day, things tend to a certain normalization which kills performance.

For that reason, MinGW and even Cygwin can work way too well on Windows machines with GCC, with Ruby tasks in mind. Though there might be little incompatibilities here and there with things like threading, sockets and so on.

I personally regard JRuby a great Ruby for Windows machines, because Java has great support there. More than that, though, I am excited with the work of the IronRuby guys because during my tests IronRuby has started performing within decent expectations, though it lags behind other Ruby implementations and has stability problems at this point.

newt0311• 18 years ago

Understand that unless the C/C++ code being compiled is actually targeted for gcc optimization, you won't get more than ~2-5% speedup in code even with -O3 or higher. Infact, go high enough like -O9 and you may even get incorrect code. The only time that settings like -O6 help are in cases like XaoS which is targeted for gcc optimization.

Furthermore, MSVC is not the only compiler better than gcc. As of now, LLVM also produces significantly better code: http://news.ycombinator.com/item?id=286408

DarkShikari• 18 years ago

MSVC the "most advanced" compiler, despite:

1. Doesn't support C99.

2. Can't correctly align its stack, making SSE optimizations of any sort a complete nightmare and forcing us to disable a number of optimizations when compiling under MSVC.

3. Has the worst 32-bit libc I have ever seen in my life, with a memcpy that goes at less than 1 byte per clock cycle for small copies, a massive performance killer for large apps.

4. Completely incompatible with almost all features of the more widely used C compilers, e.g. AT&T-syntax inline assembly. ICC, on the other hand, supports nearly everything.

5. Loves to mangle its entire project file every time you change one small thing, resulting in enormous numbers of false deltas in version control.

Now, I'm not big on GCC either and can come up with dozens of regressions and miscompilation problems with GCC, along with who-knows-how-many cases of suboptimal compilation. But at least it works and doesn't force us to mangle our entire codebase just to get it to compile!

If you want to make an argument about compilers superior to GCC, pick one that is actually good, like ICC, rather a relic like MSVC. Merely what I have to do to keep supporting this late-80s-technology beast makes me want to gouge my eyes out. I mean, seriously, a compiler that doesn't even support variable declarations in the middle of a code block? It's 2008, folks.

look_lookatme• 18 years ago

http://groups.google.com/group/comp.lang.ruby/browse_thread/...

lneves• 18 years ago

No way JVM starts in under a second: it's more like 2 full seconds on his hardware.

My machine has lower hardware specs than what the article mentions. This is what i get.

$ java -version

java version "1.6.0_10"

Java(TM) SE Runtime Environment (build 1.6.0_10-b33)

Java HotSpot(TM) 64-Bit Server VM (build 11.0-b15, mixed mode)

$ uname -a

Linux lneves-t61p 2.6.27-9-generic #1 SMP Thu Nov 20 22:15:32 UTC 2008 x86_64 GNU/Linux

$ echo "public class Test{public static void main(String[] args) {System.out.println(\"Hello World\");}}" > Test.java

$ javac Test.java

$ time java Test

Hello World

real 0m0.174s

user 0m0.068s

sys 0m0.016s

It seems that you are wrong.

More likely he had some of JVM in the filesystem cache. Vista/OSX will pre-fetch JVM modules even after you reboot (not sure about Linux) if you run Java software and especially if you have 8GB of RAM...

Do you mean Class Data Sharing?

http://java.sun.com/j2se/1.5.0/docs/guide/vm/class-data-shar...

Lets disable it:

$ time java -Xshare:off Test

Hello World

real 0m0.165s

user 0m0.060s

sys 0m0.040s

Not much of a difference.

petercooper4• 18 years ago

Just repeated your experiment on a MBP 2.2GHz. Got 0.424s real for the first attempt, then ~0.180s for all subsequent. So even on a far slower machine with no caching whatsoever, it's still under half a second :)

ig1• 18 years ago

Now I might be being stupid here but as far as I can see from the graph of total time that Vista and Ubuntu are roughly equivalent:

http://antoniocangiano.com/images/shootout3/chart_total_time...

michaelneale• 18 years ago

>Also his JRuby startup time is inaccurate.

That comment is inaccurate ;)

jbert• 18 years ago

To me, one of the more interesting aspects of that was that the ruby version optimised for reduced memory usage got a 2x speedup.

That makes sense in terms of cpu cache misses. Has anyone run any numbers to see how gcc -Ospace compares with -O2 on recent hardware?

KirinDave• 18 years ago

I'm glad that we're starting to see realistic and absolute comparisons.

And I wonder how many people are surprised to learn that JRuby is doing so well right now.

davidw• 18 years ago

The JRuby guys have definitely been doing lots of good work, it's just impressive to see it pay off like that.

old-gregg6• 18 years ago

I don't like JVM: I hate the mandatory 2-second startup time and "warming up" time. And the standard set of libraries doesn't look particularly well engineered to me (comparing Java SE to .NET or Python's libs). Also, despite being enterprise-tested and good at memory management with it's "best in class" generational GC, Java programs continue to be memory hogs, based on personal experience.

But you can't beat a nice piece of software (JRuby) and the huge ecosystem of high quality Java libraries. Moreover, JRuby guys are awesome at listening and talking to their community. You can go and ask any question about JRuby internals on #jruby - someone will always be there.

I am definitely considering switching my projects to JRuby, just need to find time.

ohhmaagawd• 18 years ago

jruby startup is about <1 second on my box. They did a bunch of tweaks earlier this year to solve the issue.

petercooper4• 18 years ago

I hate the mandatory 2-second startup time and "warming up" time.

I got about 0.45s for a basic Java app on first start, then 0.2s subsequently. For a JRuby one-liner, I get 0.618s consistently (no first start to count). So while JRuby appears to add about 0.4s to startup, it's no 2 seconds.

catch23• 18 years ago

JRuby is great, the only issue with existing projects is trying to convert all the gem dependencies that use native code over to java. Popular ones like Hpricot have jruby equivalents, but unpopular ones like ruby-gsl or image_science have no hope.

samuel• 18 years ago

The most interesting about the results IMHO is JRuby's performance. It's now faster than CRuby1.8, and, for what I have read, solves also the scalability problems associated to it(threads, GC, etc...).

And there's a lot of room for improvement, specially with the advent of invokedynamic. I wasn't eager to learn Ruby because of their alleged performance and scalability problems, but it seems that JRuby is changing that.

(If I'm mistaken, and things aren't so bright, please, enlighten me).

acangiano• 18 years ago

Please read this important update: http://news.ycombinator.com/item?id=393988

The Great Ruby Shootout (December 2008)

Related Stories