Staring at the Sun: Dalvik vs. Asm.js vs. Native

https://blog.mozilla.org/javascript/2013/08/01/staring-at-the-sun-dalvik-vs-spidermonkey/

by evilpie • 13 years ago

170 75 13 years ago

blog.mozilla.org

On one hand, it's great that the author did all this work. On the other hand, the benchmark is quite dubious: aside from the Javascript-engines-are-heavily-optimized-for-SunSpider thingy (which, if you click the link at the end of the article, ends up often making plain JS faster than asm.js), the most obvious port of JS code is likely not the most obvious way to write Java/C++, let alone the most performant way.

Still, the fact that you can compare Javascript and C++ without needing a log scale is quite an achievement.

krallja• 13 years ago

The asm.js is not implemented by hand; it is compiled from C++ via Emscripten, so there's no "obvious port of JS code" - it's all done by the compiler.

duaneb4• 13 years ago

Did the author write the C++ code? This is all pretty useless without being able to reproduce it.

EDIT: I'll eat my hat, the code looks great.

EDIT2: Looks like he allocates memory in an inner loop, no wonder native is so slow... I wouldn't take the native benchmarks seriously at all. https://github.com/kannanvijayan/benchdalvik/blob/master/nsi...

chrisaycock• 13 years ago

The point about memory allocation is similar to Walter Bright's explanation of D compiler performance [1]. There, the issue was deallocation; Walter "cheated" by never deallocating any memory since the compiler is a short-lived process.

As a complement in this Mozilla test, Kannan Vijayan believes that Asm.js ran so fast in the Binary Trees test because there is a single large allocation at process startup, much like a memory pool. The moral of the story is always be aware of what malloc() and free() are doing to your code.

[1] https://news.ycombinator.com/item?id=6103883

tehwalrus2• 13 years ago

I once made a C python extension considerably faster by being more careful with malloc and free - I can recommend it as an easy-ish way to speed up a C/C++ program.

duaneb4• 13 years ago

Anyone who has implemented a memory allocator knows how expensive it can be. If you have a simpler algorithm that works with your data, allocate a huge chunk and manage it yourself.

EDIT: Looks like the author of the native benchmarks needs to learn this lesson: https://github.com/kannanvijayan/benchdalvik/blob/master/nsi...

Sharlin2• 13 years ago

Practically the first thing to do in any serious game development project is to pick an allocator other than malloc.

munificent2• 13 years ago

Or better: more than one. Most games I worked on had a few different allocators specialized for different uses.

ancarda2• 13 years ago

As someone who has never touched C/C++ or managed their own memory, can you explain how other allocators work and how the performance differs from malloc?

peterhunt6• 13 years ago

I've been watching a bunch of this JS performance stuff unfold over the past few months (including that huge rant a while ago). People lament the lack of JITing in UIWebView or that JS is inherently slow or whatever but if you're targeting mobile, there's usually only one thing that matters:

Can your JS rendering code consistently execute within 16ms?

JS (even without JIT) is certainly fast enough to do this if you offload anything intensive to workers (in fact, I recommend that you put your whole app except for the real time aspects in a worker if possible) and schedule long running tasks over several requestAnimationFrames.

Usually the only issue is that GC pauses can cause hiccups > 16ms and you're in no control of that. This has traditionally been seen as a deal-breaker.

That's why I'm excited about Asm.js (and LLJS) -- even on generic JS runtimes, it's my understanding they don't generate garbage, and can execute without GC pauses. So I'm looking forward to writing most of my app in traditional JS in a worker, and the realtime components in Asm.js.

munificent2• 13 years ago

if you're targeting mobile, there's usually only one thing that matters:

There are two things that matter, actually:

1. How quickly can you get your app up and running and responding to user input?

2. Once that's done, can you continue to maintain 60 fps?

peterhunt6• 13 years ago

Good point. I was mostly responding to common criticisms. I think JS boot time is usually pretty good since it's easy to incrementally load code in.

modeless3• 13 years ago

GC pauses are not the only issue. Image loading often introduces longer pauses than GC, and input latency is a huge problem too. I've written a benchmark that exposes these responsiveness issues: http://google.github.io/latency-benchmark

peterhunt6• 13 years ago

Yeah jpeg decoding is a big one, but I think it's less to do with raw performance and more to do with the fact that there aren't any hooks to help you control the user experience around it (i.e. there is no way to determine if you're done decoding unless you write the decoder yourself and draw to canvas, which increases your latency by a lot).

IIRC I think dropped frames from jpeg decoding is less of an issue in today's webkit, but I'm not sure. I seem to remember there being workarounds like adding a no-op CSS animation or iframe to coax the browser to do it off of the UI thread.

That's a nice demo you have there, btw, I'll have to check it out more.

randallu• 13 years ago

Nice benchmark! Hopefully the asynchronous lazy decoding in Clank lands in Chrome soon and improves image decode blocking.

general_failure• 13 years ago

asm.js is a way to write javascript to help the JIT. It's intended to be autogenerated by tools. It has nothing to do with GC. asm.js will accumulate garbage just like normal javascript.

peterhunt6• 13 years ago

I don't think that's correct, since I believe asm.js manages its own heap via a big typed array and does not allocate anything tracked by the runtime GC (at least that's what LLJS does and I believe that asm.js does the same thing)

spyder• 13 years ago

There is no garbage in asm.js:

"The asm.js model provides a model closer to C/C++ by eliminating dynamic type guards, boxed values, and garbage collection."

From the asm.js FAQ (http://asmjs.org/faq.html)

tyre• 13 years ago

This is a rather dubious comparison and seems more targeted at selling ASM.js than making a meaningful comparison. The author includes asm.js but not native JS. Well, he actually does test it, but removes it from the results because it did too well.

From the article: To be frank, I didn’t include the regular Javascript scores in the results because regular Javascript did far too well, and I felt that including those scores would actually confuse the analysis instead of help it.

https://blog.mozilla.org/javascript/files/2013/08/Dalvik-vs-...

kannanvijayan10• 13 years ago

Author here. The "regular JS scores" do REALLY well on several benchmarks (faster than native, even), for one primary reason: the transcendental math cache (which every JS engine uses).

This is a very specific optimization that expects that functions like "sin", "cos", and "tan" will be called repeatedly with the same inputs, and puts a cache in front of those functions.

In sunspider, this helps. In the real world, we call this "overoptimization for sunspider".

I've said this before, and I'll say it again: Sunspider is a poor benchmark to use to talk about JS engines. All of them game it - ALL of them.

If I _had_ included the regular JS sunspider scores, the comparison would be unfair since all the JS engines are specifically optimized for sunspider.

The reason these benchmarks are somewhat appropriate for Java and C++ is _because_ Java and C++ compilers and libraries have not been optimized with sunspider in mind, and things like the transcendental math cache don't skew numbers.

(Also, to be clear: OdinMonkey, the asm.js compiler in SpiderMonkey, does NOT use the transcendental math cache like the optimizer for regular JS code does. This is why "plain JS" is faster than asm.js in several of the benchmarks).

mraleph2• 13 years ago

for one primary reason: the transcendental math cache

If this is the primary reason then it implies that those benchmarks primarily measure performance of transcendental operations with repetitive arguments and thus nothing interesting.

Why do you even bother include them in this case?

Sunspider is a poor benchmark to use to talk about JS engines

If you think it is a poor benchmark why do you use it or remind outside world of its existence?

In my opinion SunSpider is indeed a poor benchmark in general to talk about any kind of adaptive JIT (which includes JVM). That is why I never use it for anything.

justinsb5• 13 years ago

Kudos to the author for including their code. Too many benchmarks don't. And it is amazing that JS is even in the same ballpark as Java.

That said, like all benchmarks, there are systematic biases. I looked at the binary trees benchmark. The obvious problem is that it uses far too few iterations (100), so the runtime was 60 milliseconds (on my laptop). That's really not enough time for JIT to kick in, although probably JS does JIT more eagerly than Java. I upped it to 10000 iterations: (OpenJDK) Java took 3.3s, JS (with node) took 7.8s, C++ took 15s. (C++ is really hurt by garbage collection vs alloc/free.) Switching C++ to use Google's TCMalloc brought it down to 10s.

When you see a benchmark that says that X is faster than Java, and X does not include the letter C, take it with a pinch of salt!

janjongboom• 13 years ago

But now you're comparing unoptimized javascript to java, instead of asm.js.

icebraining• 13 years ago

And he's comparing JavaScript with the JVM, not Dalvik. And on a laptop.

justinsb5• 13 years ago

Ah - fair point. How can I repeat the asm.js results on my laptop?

kannanvijayan10• 13 years ago

One thing to keep in mind is that OpenJDK is different from Dalvik, and their JITs are very different.

Dalvik optimizes more heavily in favour of memory space (one of the reasons for its tracing JIT approach as opposed to method JIT approach).

Indeed, on the desktop, Java does very well on a lot of these benchmarks. My goal with these measurements, though, was to get an idea of what kind of performance could be expected by people who were writing asm.js code for Firefox OS devices, and how it compares to performance of similar code on other mobile platforms.

cygx• 13 years ago

And it is amazing that JS is even in the same ballpark as Java

It's not. The comparison is between C++/native, C++/asm.js and Java/Dalvik.

Yes, one of the C++ benchmarks goes through an intermediate Javascript stage, but the whole point of asm.js is that we can do so without too much of a performance penalty.

slacka• 13 years ago

Yes, these benchmarks by Mozilla are biased, but I wouldn't be surprised if Asm.js actually performs as well as Dalvik for 2 reasons. First, the Dalvik VM has always been optimized for memory footprint and not speed. Secondly, Google has focused their recent performance optimizations on the NDK, the platform that everyone uses when performance is critical. I think Google feels Dalvik is good enough for everything else.

I've looked at every change log since Android 2.0 and the last time I saw any performance tweaks to Dalvik was with the addition of JIT in 2.2. Dalvik is about 3x slower than Oracle's Hotspot[1]. Until Google moves Lars Bak to the Android team or buys Myriad for their Dalvik Turbo, I don't see this changing anytime soon.

[1] http://dl.acm.org/citation.cfm?id=2388956

justinsb5• 13 years ago

The spectral norm test appears to be wrong. On C++ & Java it gets the math expression in the A function wrong, and produces NaN, which apparently has a huge performance cost. Taking the correct version from the Javascript (and converting to a double correctly) gives the correct result, and (OpenJDK) Java runs twice as fast.

justinsb5• 13 years ago

Kannan, if you want to fix it, the Java diff is:

- return ((double)1)/((i+j)(i+j+1)/((double)2+i+1));

+ return 1.0/((i+j)(i+j+1)/2+i+1);

That gives the same count as the JS. It'd be interesting to know whether Dalvik/ARM has the same performance hit on NaN!

kannanvijayan10• 13 years ago

Thanks!

I think the "/2" should be a "/((double)2)", since in C and Java it'll treat the former as a truncated int divide. I do see the issue with the precedence on the 2+i+1, though. Will fix up and re-run.

kannanvijayan10• 13 years ago

Hmm. Could you post the code for your corrected version?

I tried to find where a NaN would be introduced in the C++ and Java code, but I can't really spot it. The equation only has adds, multiplies, and divides, no negative numbers, and no divides by zero that I can see.

If you show me where the problem is, it wouldn't take me too long to re-run everything with the corrected code and update the results.

zbowling• 13 years ago

So we have a better NDK than Googles at Apportable. We use our own malloc and better C++ runtime (libc++ instead of libstdc++) and use Clang 3.4 instead of GCC.

One problem with your tests is that you are using the system malloc, and that is horribly slow (and dalvik gc will even obtain a global lock on it every now and then). Firefox does not use the system malloc (instead it uses jemalloc). This actually has a big time savings in tests that will call malloc at any point.

I would love to run your tests on our platform. Can publish the exact times you got? I want to spin it up and see if I can get better numbers on the native side.

kannanvijayan10• 13 years ago

Sure, here's a paste of a CSV file containing the data I recorded. This was taken on a Nexus 4 running Android 4.2.2. There are two columns for each bench - the right column containing nine individual scores, and the left column containing their aggregate stats:

http://paste.ubuntu.com/5960016/

azakai3• 13 years ago

All the code to the benchmarks is available, and all the tools use to compile them are open source, so please make that comparison, would be interesting to see!

MBCook• 13 years ago

Interesting. It's nice to see someone actually do analysis instead of just dumping numbers and saying "So X is the fastest, except when it's Y".

I'd like to see all these re-run on a desktop with a desktop JVM. I'm curious if the problems Dalvik showed on the binary tree bench are specific Dalvik specific or if Hotspot has optimizations that fix some of the issues identified.

kenster072• 13 years ago

It is quite misleading to write native code in the same way you write the javascript.

A wothwhile benchmark would test optimized code in all contestant languages.

kenster072• 13 years ago

Was this point downvoted for a good reason?

I will reiterate my point. There is no point in comparing suboptimal C++ to optimized asm.

Rather than downvoting me, the author if the study ought to rerun the benchmark with optimized C++.

lttlrck• 13 years ago

Its a shame the benchmarks chosen did not allow useful comparison between Asm.js and plain Javascript to be included. Plain JS is an important baseline.

fuckjavascript• 13 years ago

Well, it's good to see that the browser developers have finally conceded total defeat on Javascript as the basis for their platform, and are now simply constructing a nice little low-level virtual machine for general compilation. Granted, they're still hypocritically trying to keep up appearances by sharing as much machinery as possible with their Javascript VM, but nobody's perfect, I guess.

Staring at the Sun: Dalvik vs. Asm.js vs. Native

Related Stories