Vert.x (JVM async) vs Node.js Http benchmarks results

http://vertxproject.wordpress.com/2012/05/09/vert-x-vs-node-js-simple-http-benchmarks/

by purplefox • 14 years ago

151 118 14 years ago

vertxproject.wordpress.com

andrewvc3• 14 years ago

Interesting results.

While it's not surprising to me that the pure java implementation was far and ahead better since: 1.) Java's much faster than Javascript, and 2.) Netty (which Vert.x is based on), has been doing high performance async IO for years before node even existed in extremely high performance environments.

What is surprising however, is that javascript on the JVM w/ vert.x is faster than V8 with node. In both cases I would assume not much JS is executed, since in Vert.x most of the executing code is still Java, with a tiny piece of JS scripting on top, and in the Node.js case, I was under the impression that most of the executing code would be the C code that powers its HTTP server.

Does anyone with knowledge of the node internals know what's going on here? Is Node's HTTP parser just slower? Is its reactor not as efficient?

rapala• 14 years ago

Ah, but is Vert.x web scale?

mikeheath2• 14 years ago

I realize you're probably joking (at least, I hope so) but just to talk to the scalability point, I've had much better success scaling Netty (which Vert.x is built on) than Node. I've been able to scale Netty to more concurrent connections and Netty performs much better under load than Node. Just my experience.

wickedchicken• 14 years ago

> Ah, but is Vert.x web scale?

Better write a Mongo-backed fibonacci generator to check.

tferris3• 14 years ago

Very good point.

V8 does heavy code optimization and Node.js http sever code should be well optimized as well. And maybe the load balancer for the multicore Node test isn't optimized enough. Thus, these results feel a little shady. Anyway, Vert.x was the trigger that I am finally downloading the JVM (JDK) to try Vert.x (and maybe later Clojure).

But there's still one major drawback—the non-existant ecosystem. I know the hint to look for libs from the Java world but I need a concrete and precise guide how to do this. Let's say I want to plugin some Java lib for image manipulation. How? And who will guarantee that these libs will be concurrent and/or non-blocking as well? The lib developer or Vert.x? At the moment there is—except few out-of-the-box modules—nothing. No 3rd party libs, no module manager a la npm and no guide or documentation how to glue Java libs to this Vert.x thing. Nothing. Correct me if I'm wrong.

baudehlo4• 14 years ago

We know the java libraries are there for lots of things, but Node is all about async and handling thousands of connections. It does this by forcing the entire ecosystem to be async too (including things like database drivers).

By using a Java JDBC database driver you're completely losing any async support. Same presumably goes for redis or Mongo drivers. You can do some of the work with threads and pooling, but it's still not the same, and makes this another useless micro benchmark.

chubbard• 14 years ago

Non-existant ecosystem is a harsh assessment especially given the massive amount of java code our there. Any jar file can be used so the Java ImageIO package can easily be leveraged. It's early on for this tool chain, but all you have to do is get a 3rd party jar file in the class path of the server, and you can directly call Java methods from Javascript.

importPackage(java.io); var file = new File('/blah/blah/blah.txt');

Other languages would be equally easy as well. It's burried, but here is some relevant information from the docs on integrating 3rd party libs:

-cp <path> The path on which to search for the main and any other resources used by the verticle. This is ignored if you are running an installed module. This defaults to . (current directory). If your verticle references other scripts, classes or other resources (e.g. jar files) then make sure these are on this path. The path can contain multiple path entries separated by : (colon). Each path entry can be an absolute or relative path to a directory containing scripts, or absolute or relative filenames for jar or zip files. An example path might be -cp classes:lib/otherscripts:jars/myjar.jar:jars/otherjar.jar Always use the path to reference any resources that your verticle requires. Please, do not put them on the system classpath as this can cause isolation issues between deployed verticles.

sehugg2• 14 years ago

You don't need to necessarily have non-blocking libraries to take advantage of the async I/O. Netty uses worker threads to prevent blocking the server event loop and I assume Vert.X does as well. You can do blocking JDBC queries, Redis, etc.

As far as concurrency, most libraries state in the docs what is threadsafe and what is not, so this is more or less "caveat emptor" and RTFM.

Vert.x has its own module subsystem which appears to package up .jar files and some glue code for the Vert.x-specific interface, but I assume you can call any included Java classes directly: http://vertx.io/mods_manual.html

(Interestingly enough, Vert.x uses Gradle for its build/dependency manager, but I guess wanted something more Node-like for its module system)

phillmv• 14 years ago

> Let's say I want to plugin some Java lib for image manipulation. How? And who will guarantee that these libs will be concurrent and/or non-blocking as well?

What? What do you mean by non blocking in this context?

pron7• 14 years ago

It's funny, the path software technology takes. A Java clone of a fairly basic web engine (which is a fairly simple application type to begin with) convinces people to try the JVM which is the most performant managed runtime around and definitely one of the most advanced pieces of software ever created. I'm sometimes amazed that and people coming from easy-to-use web languages have never been exposed to it, and if they have they still have their doubts.

Dude, I've been developing real-time military applications and we've moved from C/C++ to Java and got performance gains (I'm not saying that C can't beat Java doing specific computations - of course it can - but when building a large application with millions of lines of code and a large team of developers, Java would be a safer bet than C++ if speed is your concern). In other words, no other environment can beat the JVM.

And, yeah, you should try Clojure. It's a cathartic experience.

EDIT: The only doubts I have about vert.x is that it can consistently beat "old" JVM servlet containers under heavy, real-world loads. That remains to be seen.

batista5• 14 years ago

>Let's say I want to plugin some Java lib for image manipulation. How? And who will guarantee that these libs will be concurrent and/or non-blocking as well?

Well, this is the JVM, not a single threaded javascript engine. As the other guy says below:

"You don't need to necessarily have non-blocking libraries to take advantage of the async I/O. Netty uses worker threads to prevent blocking the server event loop and I assume Vert.X does as well. You can do blocking JDBC queries, Redis, etc."

And threadsafe libs --of which are tons--, usually advertise it on "the box.

wh-uws2• 14 years ago

But be mindful though you always have to look very carefully at benchmarks.

I don't think these are speed tests.

Just number of connections that can be handled

mikeheath2• 14 years ago

I think you need to go back to the benchmarks and take a close look at the "Requests/Second" part.

These benchmarks are nothing but speed tests.

sehugg2• 14 years ago

I would also be interested in the internals. I'd expect JS + a small C event loop to outperform compiled JVM code + lots of async Java I/O libs (for a micro benchmark, at least).

But there's no question Netty is fast even on a non-benchmark. I'm using it for a long-polling server with upwards of 100K connections, and my CPU doesn't go much over 5%. Besides memory usage, the main limitation has been proper kernel configuration.

joncooper• 14 years ago

I was surprised. We built 'hellod' servers in numerous languages and with various runtime stacks[1]. The results are here and might surprise you:

https://github.com/carbonfive/hellod/blob/master/results.md

[1] C/libev, Clojure/Aleph, Clojure/Jetty, Erlang, Go, Java/Netty, Java/NIO, node.js, JRuby, MRI Ruby

cbsmith• 14 years ago

The polling loop and I/O interfaces with Vert.x is integrated in to the VM, whereas with Node.js, you'll calling out to libuv/libev/etc. That gives the JVM an advantage. Really, how much actual JavaScript code is V8 JIT'ing at all?

Really what you are comparing here is the efficiency of the paths between the two JavaScript engines and their polling I/O libraries, more than any JIT work (and let's face it, even if we were comparing JIT work, the Sun JVM has had a lot more time to tune and tweak than V8 has). In Node's case, it's talking through libuv, which means it has to transition in-and out of the V8 engine's runtime and in to a C runtime. Even if the C code is super zippy, that's costly. For Vert.x though, all the I/O and polling is integrated in to the runtime/VM/JIT. That's kind of nice.

An interesting way to test what I'm suggesting would be to do the same test with unix domain sockets/named pipes as the transport instead of TCP. The JVM doesn't have a native implementation, so it'd have to call out through JNI. I'd wager even odds it ends up slower than TCP on localhost.

courtneycouch3• 14 years ago

Using this benchmark: https://gist.github.com/2652991

nodejs beats Vert.x for example. People shouldn't be so quick to accept these half thought out microbenchmarks.

firefoxman1• 14 years ago

Quite interesting. Is the Vert.x API compatible with CommonJS and/or Node modules? If not, I think it will suffer from the same chicken/egg problem that WebOS had with apps.

manuscreationis3• 14 years ago

Seeing these numbers (although as the author states, the data needs to be taken with a grain of salt since he didn't try to set up a more "proper" test environment) really makes me want to take a serious look at vert.x, but what you mention here is my main concern.

Node is a joy to use because of the NPM, and all the available libraries. I'd hate to lose that.

That said, so long as it has a solid websocket api, and I can do routing in it like I would express, I'd give this framework a shot

eropple3• 14 years ago

> Node is a joy to use because of the NPM, and all the available libraries. I'd hate to lose that.

Maven's not bad at all (from a consumption perspective--from a dev perspective, it's a bit of a pain in the ass), and it looks like pretty much any Java library that doesn't do anything too insane should work just fine. (Disclaimer: never used vert.x myself, but I did a quick scan of the code.)

JustinJ70s• 14 years ago

Vert.x leverages the JVM so I don't think library support is really going to be much of a problem. Far from it.

purplefox11• 14 years ago

Agreed, it would be good to have CommonJS support.

However, it's unlikely that node modules will work as is with Vert.x, since the API is different. (Unless someone writes a translation layer)

jwingy• 14 years ago

With regards to Vert.x, it seems like really cool stuff.

On the blog post about version 1 being released it mentions being able to mix and match several programming languages. Does this mean you can use different libraries written in different languages in the same Vert.x server?

purplefox11• 14 years ago

Yes indeed. You can mix and match Java, JavaScript, Ruby and Groovy in the same app. We hope to support more languages going ahead (e.g. Java, Scala, ...)

eropple3• 14 years ago

Once you get Scala, I'll be heading your way. :-)

rabbitfang2• 14 years ago

+1 for Scala

Might be nice to also see a FAQ about the differences between vert.x and Play 2

exDM693• 14 years ago

Yet another meaningless micro benchmark. There is no point in measuring a hello world http request in one framework vs. another.

There should be a larger application that even remotely resembles some kind of real world usage. Maybe some day we'll have some kind of a "standard" for a web framework benchmark, an application that actually does something so it's worth benchmarking.

manuscreationis3• 14 years ago

I agree the "micro" benchmark isn't something people should look to as a definitive answer, but I don't think they should be outright dismissed either. If nothing else, they should be a jumping off point for real testing.

exDM693• 14 years ago

I agree that micro benchmarks are kind of useful in spotting anomalous, bad performance in the very simplest of cases.

huggyface• 14 years ago

These micro-benchmarks aren't even a good kicking off point for comparisons, as the things that yield a trivial benchmark win are often the things that yield significant performance troubles at scale.

andrewvc3• 14 years ago

Hardly meaningless. The problem is that the larger application benchmarks fall prey to accusations of "I wouldn't write it that way!". Their results are just as hotly disputed.

This micro-benchmark makes sense. It's benchmarking an HTTP server essentially. Any benchmark further than this would really be benchmarking the JVM and V8. While that would be interesting, I think in this case, this micro benchmark is OK as long as you know its limitations.

The reality is that most people's app code and databases are going to bottleneck long before either Vert.x or Node.js do. The main thing this benchmark clears up is that both of them are really, really, fast, and that if you do lots of simple to process responses you may want to go with Vert.x.

exDM693• 14 years ago

> This micro-benchmark makes sense. It's benchmarking an HTTP server essentially.

Kind-of true. But to give any results that can be applied to the real world, it would have to use a much larger payload than a hello world.

zafriedman2• 14 years ago

I'd upvote this more than once if I could. The beauty and efficiency of a web framework must be measured as a trade off between the amount of instructions executed to check for corner cases etc. Anyone could write a framework that executes hello world faster than a popular one if they just assume for instance that no one will ever make a POST request or use a query string. That wouldn't be a very useful framework, but it could definitely execute GET '/' fast.

mbq• 14 years ago

According to the code, it actually does a file read on every request -- this is certainly suspicious because some implicit caching may significantly change the results.

ww5205• 14 years ago

It includes a test with file streaming on Node.js, that bypass reading the file and write to socket via the Javascript loop completely.

Also file IO are heavily cached by the OS. You can bet that one file is read from memory most of the time. Disk IO is pretty much out of the equation.

jlouis• 14 years ago

Perhaps a dumb question: Are the headers the same? With so small files, the header layout is suddenly very important for your measurement.

I've seen my share of web servers which are very different in their compliance.

IsaacSchlueter5• 14 years ago

It's not a dumb question at all.

binarymax• 14 years ago

Don't serve static files with node. Use an nginx reverse proxy for your html/js/css assets.

trimbo2• 14 years ago

Came here to say this. While microbenchmarks are fun for all, in the real world you would have a completely different reason for choosing node.js that has nothing to do with this kind of performance. So just use the best tool for what is being benchmarked here: nginx.

zafriedman2• 14 years ago

I'm going to ask a dumb question as someone who is just learning node. What's wrong with serving assets out of public/ in an Express app? Why would someone not want to do this?

trimbo2• 14 years ago

For the vast majority of sites that ever hit the web, yeah, it will probably be fine. So go ahead and have fun learning node and do it however, you can always fix it later if the site gets overloaded.

At some volume though, you want to do what the grandparent recommends -- front your application server with nginx, apache, or something. Otherwise serving up static assets is taking away resources that should be spent on the application. Consider if you have a page with 5 images, 5 JS files and 5 CS files for each page created. That means you're serving 15 static assets for every 1 dynamic asset you'd really want to use Node for. While Node can still be "good enough", it is not spending its time doing the stuff you really want to use it for in the first place. A dedicated piece like nginx goes much further.

And also use a CDN :p

smagch• 14 years ago

I wanted to know the benchmarks including luvit. Vert.x vs luvit vs Node.js https://github.com/luvit/luvit

elliotlai• 14 years ago

luvit, love it!

pjmlp• 14 years ago

Just goes to show that when the node.js fad is over, the enterprise world will keep its high availability servers running on proven technologies.

deelowe• 14 years ago

Not really. It shows that this benchmark is crap (likely benchmarking disk io versus disk io + some caching). Read Isaac's comment for more detail. He sums it up pretty well. No profiling info, using a custom test, no analysis besides some pretty graphs.

I have a hard time believing the JVM is really 10x faster than v8 for such a simple server.

azakai• 14 years ago

> I have a hard time believing the JVM is really 10x faster than v8 for such a simple server.

Honest question, why?

rabbitfang2• 14 years ago

Why would it surprise you that a statically typed language like Java on the highly optimized JVM is faster than Javascript?

Any benchmark you care to look at should clearly shows JVM is much faster: http://shootout.alioth.debian.org/u64/benchmark.php?test=all...

fruchtose• 14 years ago

Node.js is not a fad. It represents the first workable JavaScript-based server with mass appeal. The real story is that JavaScript is here to stay. There have been other server-side JavaScript frameworks in the past, but none of them have taken off like Node.js. If Vert.x wins over the JavaScript crowd, then that's great, because coders will be able to write JavaScript.

batista5• 14 years ago

>Node.js is not a fad. It represents the first workable JavaScript-based server with mass appeal. The real story is that JavaScript is here to stay.

Javascript yes. Node.js, not so much.

>There have been other server-side JavaScript frameworks in the past, but none of them have taken off like Node.js.

Javascript was not a fast language in the past, nor was it much used for anything more than the most basic dynamic html stuff (rollovers etc)...

cageface• 14 years ago

I think it's actually unfortunate that the first popular server-side JS framework is so tied to the async model. I'd be more likely to try it if it weren't.

jroseattle• 14 years ago

I wouldn't dismiss the node.js stack quite so easily. Those "proven" technologies you mention had to go through their own lifecycle of continued improvement.

I remember a time when my colleagues who were steeped in C++ had a good laugh at my expense because I was building server-side web applications with a new framework and a hot, new language. It woefully under-performed similar C++ applications in benchmark tests.

It was 1998, and the language was Java. I could write my applications much faster and in a more maintainable way than they could, but they didn't care. Their technology was proven, and Java was simply a fad.

dap• 14 years ago

A rigorous benchmark must seek to explain the difference in performance, not just hand-wave saying "such-and-such is faster". Otherwise, you have no way of validating (for yourself, let alone demonstrating to others) that it's not a misconfiguration or a flaw in your benchmark.

halayli• 14 years ago

Why are they measuring requests/sec? Any server can accept connections at a high rate but what matters is responding in a timely manner.

I doubt the requests number too. Writing a dummy socket server (evented, threaded, ...) that just returns "HTTP/1.1 200 OK" will not get you anywhere close to 120k requests/sec. The system call becomes the bottleneck.

purplefox11• 14 years ago

It's labelled badly, what is actually measured is req/resp per second.

I.e from request to corresponding response and how many of those it can do per second.

If you doubt the numbers please feel free to run them yourselves, all code is in github

pohl• 14 years ago

Could you clarify that? Are you saying that if the response is sent within the same second that the request came in that it contributes to the metric?

Or would a response that is sent 30 seconds after the request came in contribute to the metric too?

ww5205• 14 years ago

Requests per second implies downloading the complete request for each request.

anuraj• 14 years ago

Very exciting initial results. The JVM simply is the most optimized runtime available right now. And Java is the dynamic language with best performance. Can't fail to notice Ruby is the slowest even on JVM. If you continue on this path and refine your APIs to be more user friendly, this would be the next big asynchronous server out there!

anonhacker• 14 years ago

Are there any easy cloud platforms or whatever that support easy deployment for Vert.x?

purplefox11• 14 years ago

People have already got Vert.x running on OpenShift and Heroku, and CloudFoundry support shouldn't be too much longer.

ww5205• 14 years ago

This is consistent with a benchmark I did a while back comparing Netty vs. Node.js. Not surprising since Netty powers Vert.x. Netty is pretty amazing. 500K concurrent connections and not batting an eye.

IsaacSchlueter5• 14 years ago

My comment: http://vertxproject.wordpress.com/2012/05/09/vert-x-vs-node-...

prospero• 14 years ago

Netty has support for zero-copy file responses, I imagine that's at least a large part of the performance discrepancy.

pron7• 14 years ago

I'm interested to know how vert.x compares to industrial-strength "traditional" servlet containers. My guess is that vert.x would outperform them under certain conditions, but all in all they would scale better.

I believe servlets are still the most scalable web stack out there.

simonw2• 14 years ago

I imagine you mean that servlets are the most performant web stack, not the most scalable web stack. Scalability != Performance.

pron7• 14 years ago

Well, I'm sure it's easy to beat servlets with a load that requires only one thread with a simple enough computation model. Servlets (and any truly multithreaded solution) trades single-thread performance with the ability to scale with the number of cores. That's what I meant.

So I would think that given a heavy load on a many-core machine with interesting enough computations, nothing could beat servlets' performance.

DTrejo• 14 years ago

The streaming node.js example he wrote uses a blocking call. This is not the node way, and would cause a definite slow-down.

If you're worried about your programs containing rogue & misbehaving code like this, I recommend you use https://github.com/isaacs/nosync

nfriedly2• 14 years ago

I had the impression that "Node.js (readfile)", and possibly "Node.js" meant the blocking call but that "Node.js (streams)" meant he was using something like fs.createReadStream(). But you're right, I don't see that anywhere in the posted source.

purplefox11• 14 years ago

I've tested several combinations of blocking, non blocking, readFile, streams (pipe) and using chunked transfer encoding.

Results vary a little but all way below the Vert.x results.

See blog post for the stats.

soc883• 14 years ago

JVM beating the shit out of Node.js? I guess that's not even slightly surprising to anyone outside the Node.js bubble.

thebluesky• 14 years ago

Java much faster than Javascript. News at 11...

alz• 14 years ago

In the test, the node code is actually calling an asynchronous function fs.readFile on every single request:

https://gist.github.com/2650401

Even with os caching there is still quite a bit of overhead there. Would be interesting to see the benchmarks run on the corrected code:

https://gist.github.com/2650401

courtneycouch3• 14 years ago

So a quick verification. The io is the difference between these two. The JVM is doing some caching somewhere, whereas the v8 engine is not. Making a small change to both (in order to ensure that both are using the exact same logic):

https://gist.github.com/2652991

and then I get the following results:

vert.x:

39890 Rate: count/sec: 3289.4736842105262 Average rate: 2958.1348708949613 42901 Rate: count/sec: 2656.924609764198 Average rate: 2936.994475653248 45952 Rate: count/sec: 3277.613897082924 Average rate: 2959.610027855153

node.js:

38439 Rate: count/sec: 4603.748766853009 Average rate: 4474.62212856734 41469 Rate: count/sec: 4620.4620462046205 Average rate: 4485.278159589091 44469 Rate: count/sec: 4666.666666666667 Average rate: 4497.515122894601

Making that change so they both store the file in memory and nodejs is 50% faster than vert.x.

This is using an m1.small instance on EC2, and both vert.x and nodejs only using a single core.

tipiirai• 14 years ago

Yea. There is no way java/rhino could beat c/v8.

courtneycouch3• 14 years ago

In any case even if it was faster, it wouldn't be an order of magnitude faster. That kind of a difference should be an indication that something is wrong with the benchmark.

purplefox11• 14 years ago

The JVM does not doing any caching.

And artificially crippling Vert.x to a single core does not prove anything. Anybody who cares about performance will be using more than one core.

VeejayRampay• 14 years ago

Testing A against B. A is found to be faster.

People in the B community come out in outrage saying that the testing is flawed, that microbenchmarking is useless, that this and that. Rinse and repeat.

Vert.x (JVM async) vs Node.js Http benchmarks results

Related Stories