Debunking the Java Performance Myth

https://medium.com/@elliot_f/debunking-the-java-performance-myth-29b842955a24

by emforce • 9 years ago

82 97 9 years ago

sempron642• 9 years ago

I think an important element in comparing Go to Java for webserver apps is language design. While Java 8 and Go's performance may be similar in small single-threaded request handler benchmarks, Java's baseline design encourages the creation of lots of expensive runtime abstractions, and makes parallelism difficult. Go on the other hand encourages a minimum of abstractions and makes parallelism trivial. That means that the marginal cost of business logic is much higher in Java, in both response times (due to unnecessary serialization) and overall computation. Some of this can be averted by careful engineering, but Java definitely isn't a "pit of success" in this regard - the easiest thing to do is usually the wrong thing.

twic• 9 years ago

Java's baseline design encourages the creation of lots of expensive runtime abstractions

Abstractions yes, expensive, perhaps not. One of the nice things about Java is that the JIT can collapse a lot of abstraction at runtime. Not all of it, but a lot of it. I am skeptical about the idea that Java naturally guides programmers towards expensive abstractions.

and makes parallelism difficult

Presumably, you mean something like "makes lightweight concurrency difficult, because all the backend APIs for database access etc are blocking", which is unfortunately true.

sidlls• 9 years ago

I think perhaps it's more of a "pure OO" approach than the language itself. I've seen plenty of Java and C++ code where there is one interface class for every concrete class, even when just one concrete class is ever implemented, for example. I've also seen cases where some "factory" pattern is force-fit into the architecture, and the objects it creates are kinda sorta related, but ought not be in code. It gets exponentially worse as more interfaces and more levels in a hierarchy are allowed.

The compiler isn't always going to be able to collapse or elide these.

diebir2• 9 years ago

The thing about Go I do not get is why somebody would go back to essentially C to develop modern code? The tools are lacking, the language features are lacking, the libraries are lacking, so why?

Go is still garbage collected language, so in a sense it combines the most problematic features of C and Java.

flavor82• 9 years ago

I'm not sure that there's a myth to be debunked, is there? Java's known to be high performance as a web application server, if good architectural choices are made. The well referred to https://www.techempower.com/benchmarks/ reliably has some of the better Java frameworks ranking near the top along with C and C++ servers.

mikeash2• 9 years ago

Yeah, the title confuses me greatly. I assumed this was going to be an article about how Java is slow, debunking the myth that it’s really fast.

whois• 9 years ago

Really? You've never heard people joke about Java being slow? Half the programmers I've met have this cult like hate of java and cite performance as the main reason.

anon335dtzbvc• 9 years ago

yup me too

jackmott2• 9 years ago

Java desktop applications tend to be slow and bloated. Java server applications tend to be pretty ok if well designed.

For desktop applications you suffer some more from innate disadvantages of JITs and the common graphics libs. People experience Java when using eclipse and minecraft and so perceive it as slow. When a java server provided data fast to a wevsite, people don't know that is java.

sempron642• 9 years ago

"If good architectural choices are made" is a high bar, considering that many popular Java frameworks (not the ones high up on the list) have already made bad choices (performance-wise) for you.

flavor82• 9 years ago

Well, true - but the highest ranked Python(#1) framework is at 25%, with probably a dozen or so Java configurations above it in the list. If low latency and high throughput is a primary goal (i.e. you really need scale) then you're probably going to know about gemini, undertow, grizzly, etc if you're in the Java ecosystem.

(#1 mentioned since it's one of the two languages compared to Java in TFA).

TheCoelacanth• 9 years ago

I think Java being slow was the reality 20 years ago, a widely believed myth 10 years ago, and something that no one but dinosaurs believes now.

ssijak3• 9 years ago

One more dumb test not even giving the code and test environment fully. Btw, this tests are useless. Add some real usecase, a database, some number crunching, jobs execution and see python fail even more for example.

falcolas• 9 years ago

The JVM has indeed matured to the point where it's on par with other VM style languages, and better than interpreted languages. This wasn't always the case: early in Java's lifetime it was a slow, memory intensive hog. That stigma has stuck.

That said, it's still pretty heavyweight (both the language and the runtime).

It also wouldn't be my first choice for compute intensive tasks (for example, a non-toy ray tracer). It would do the job eventually, but at a cost. Of course, I wouldn't pick Go or Python either.

agumonkey• 9 years ago

Reminds me people blaming clojure slow boot on the JVM when it was in fact clojure tooling that took time.

Things aren't straightforward, we need to go deep before concluding. Profilers, profilers, profilers.

amaranth• 9 years ago

There are people in college or entering the workforce who weren't even alive during the "Java is slow" period. The HotSpot JIT has been around since 1999. I wonder if we'll ever be able to let this go.

Timothycquinn2• 9 years ago

Would be interested in seeing Elixir/Phoenix compared here. I've seen some amazing scalability figures from those folks: http://phoenixframework.org/blog/the-road-to-2-million-webso...

For posterity sake, throw in Ruby on Rails as well.

brightball2• 9 years ago

Elixir is more focused on concurrent performance than raw performance. It’s still very fast but favors consistency above all else. I remember seeing a great benchmark that compared Elixir to Python and Go where they charted the variability of response times. Python and Go were very spikey with Go producing the fastest overall time. Elixir was extremely steady. Makes it an ideal choice for real time ops, holding millions of web sockets, ensuring a heavy process doesn’t affect the experience of everything else that’s executing.

You’ll also get natural clustering ability and a level of fault tolerance that’s almost silly.

I’ve been trying to find the time to demonstrate this with the tech empower benchmarks honestly. I want to see what will happen if you run all of them at the same time for a particular platform.

rlander• 9 years ago

It baffles me how people give Elixir the credit for this. Everything you mentioned here have nothing to do with Elixir, but with how Erlang and its preemptive scheduler were optimized for soft real-time apps.

boggio• 9 years ago

Flask is synchronous and it shouldn't be labeleld "Python 2.7 or Python 3.6". Comparing that to async frameworks is not quite fair. (Also is SpringBoot async?)

ssijak3• 9 years ago

Spring Boot can be both reactive or ordinary thread pool based to serve requests or anything from the stack really. It is very very powerful framework and comparing something barebones to Boot is really dumb. Not to mention that author intentionally made java look like a monster when he could use some simple tweaks to reduce jvm memory usage by much and he hid what dependencies really imported. And also if he did not like Boot he could just fire barebons Jetty up and use 20mb heap tops.

chvid• 9 years ago

Did you publish the source code for the test servers (in the various languages)?

I am curious to what a "normal" Java microservice look like.

coldtea• 9 years ago

Not very different than a "normal" Node microservice.

It might even take fewer lines.

JepZ5• 9 years ago

Well, I have not taken a look at the implementation yet, but java is certainly known for delivering a decent performance even if the programmer is a little careless regarding perfromance.

In Go on the other hand, doing simple 'mistakes' like declaring a variable inside a loop can bring you to performance hell:

Declaration in a loop:

        var sum int
        for i := 0; i < n; i++ {
                x := i + i
                sum = x - i
        }

Declaration outside the loop:

        var sum int
        var x int
        for i := 0; i < n; i++ {
                x = i + i
                sum = x - i
        }

Benchmark results:

  BenchmarkInLoop-8       2000000000               0.40 ns/op
  BenchmarkOutLoop-8      2000000000               0.47 ns/op

And the same is certainly true for C/C++.

Src: https://nopaste.xyz/?68c6800acd9200d6#mDstI36uBPBU4Td8k//GNC...

Retric2• 9 years ago

This is a common compiler optimization. It says little about a language, just the compiler in use.

yoklov2• 9 years ago

Declaring a primitive variable inside vs outside a loop in c or c++ will have no effect when compiled with optimizations on by any major compiler. For non-primitive types it will depend on a number of factors, and the better choice for perf could go either way.

DannyBee2• 9 years ago

Yes, because if the loop is not always-executing, it may be cheaper to not initialize the complex object, even if it is safe to so.

anon946• 9 years ago

Any decent C/C++ compiler will give the same code. For example, this:

    int f1(int n) {
        int sum = 0;
        for (int i = 0; i < n; i++) {
            int x = i + i;
            sum = x - i;
        }
        return sum;
    }

    int f2(int n) {
        int sum = 0;
        int x;
        for (int i = 0; i < n; i++) {
            x = i + i;
            sum = x - i;
        }
        return sum;
    }

when compiled with 'gcc -O -S' gives:

            .file       "loop.c"
            .text
            .globl      f1
            .type       f1, @function
    f1:
    .LFB0:
            .cfi_startproc
            testl       %edi, %edi
            jle .L4
            movl        $0, %eax
    .L3:
            addl        $1, %eax
            cmpl        %eax, %edi
            jne .L3
            subl        $1, %eax
            ret
    .L4:
            movl        $0, %eax
            ret
            .cfi_endproc
    .LFE0:
            .size       f1, .-f1
            .globl      f2
            .type       f2, @function
    f2:
    .LFB1:
            .cfi_startproc
            testl       %edi, %edi
            jle .L9
            movl        $0, %eax
    .L8:
            addl        $1, %eax
            cmpl        %eax, %edi
            jne .L8
            subl        $1, %eax
            ret
    .L9:
            movl        $0, %eax
            ret
            .cfi_endproc
    .LFE1:
            .size       f2, .-f2
            .ident      "GCC: (GNU) 7.2.0"
            .section    .note.GNU-stack,"",@progbits

Even when compiled without optimization, the generated assembly was the same, though it was messier so I don't show it.

JepZ5• 9 years ago

Okay cool, so Go seems to be standing alone here without that optimization :-/

Can someone please call Rob Pike? We are discussing Golang 2.0 already and still don't have 'common' compiler optimizations... ;-) joke

helper• 9 years ago

What go compiler are you using? On my machine those run in exactly the same amount of time (go version go1.9 linux/amd64 ).

  $ ./tmp.test -test.bench .
  goos: linux
  goarch: amd64
  BenchmarkInLoop-4       2000000000               0.46 ns/op
  BenchmarkOutLoop-4      2000000000               0.46 ns/op
  PASS


  (pprof) disasm fast
  Total: 1.95s
  ROUTINE ======================== _/tmp.fast
       970ms      970ms (flat, cum) 49.74% of Total
           .          .     4ed7c0: MOVQ 0x8(SP), AX                             ;main_test.go:23
           .          .     4ed7c5: XORL CX, CX
           .          .     4ed7c7: MOVQ CX, DX
           .          .     4ed7ca: JMP 0x4ed7d6                             ;main_test.go:26
       660ms      660ms     4ed7cc: LEAQ 0x1(CX), BX                             ;_/tmp.fast main_test.go:26
       110ms      110ms     4ed7d0: MOVQ CX, DX
       170ms      170ms     4ed7d3: MOVQ BX, CX
        30ms       30ms     4ed7d6: CMPQ AX, CX
           .          .     4ed7d9: JL 0x4ed7cc                               ;main_test.go:26
           .          .     4ed7db: MOVQ DX, 0x10(SP)                           ;main_test.go:30
  (pprof) disasm slow
  Total: 1.95s
  ROUTINE ======================== _/tmp.slow
       960ms      960ms (flat, cum) 49.23% of Total
           .          .     4ed790: MOVQ 0x8(SP), AX                             ;main_test.go:14
           .          .     4ed795: XORL CX, CX
           .          .     4ed797: MOVQ CX, DX
           .          .     4ed79a: JMP 0x4ed7a6                             ;main_test.go:16
       690ms      690ms     4ed79c: LEAQ 0x1(CX), BX                             ;_/tmp.slow main_test.go:16
       100ms      100ms     4ed7a0: MOVQ CX, DX
       100ms      100ms     4ed7a3: MOVQ BX, CX
        70ms       70ms     4ed7a6: CMPQ AX, CX
           .          .     4ed7a9: JL 0x4ed79c                               ;main_test.go:16
           .          .     4ed7ab: MOVQ DX, 0x10(SP)                           ;main_test.go:20

JepZ5• 9 years ago

  $ go version
  go version go1.9.1 linux/amd64

JepZ5• 9 years ago

Well, the assembler looks the same to me too, but I can confirm different runtimes on two different machines (fast is always faster here).

  $ ./main.test -test.bench . -test.cpuprofile=test.profile
  goos: linux
  goarch: amd64
  BenchmarkInLoop-8       2000000000               0.42 ns/op
  BenchmarkOutLoop-8      2000000000               0.47 ns/op
  PASS

  (pprof) disasm fast 
  Total: 1.86s
  ROUTINE ======================== command-line-arguments.fast
       990ms      990ms (flat, cum) 53.23% of Total
           .          .     4ed7b0: MOVQ 0x8(SP), AX                             ;mem_test.go:16
           .          .     4ed7b5: XORL CX, CX
           .          .     4ed7b7: MOVQ CX, DX
           .          .     4ed7ba: JMP 0x4ed7c6                             ;mem_test.go:19
       650ms      650ms     4ed7bc: LEAQ 0x1(CX), BX                             ;command-line-arguments.fast mem_test.go:19
           .          .     4ed7c0: MOVQ CX, DX                               ;mem_test.go:19
        10ms       10ms     4ed7c3: MOVQ BX, CX                               ;command-line-arguments.fast mem_test.go:19
       330ms      330ms     4ed7c6: CMPQ AX, CX
           .          .     4ed7c9: JL 0x4ed7bc                               ;mem_test.go:19
           .          .     4ed7cb: MOVQ DX, 0x10(SP)                           ;mem_test.go:23
  (pprof) disasm slow
  Total: 1.86s
  ROUTINE ======================== command-line-arguments.slow
       870ms      870ms (flat, cum) 46.77% of Total
           .          .     4ed780: MOVQ 0x8(SP), AX                             ;mem_test.go:7
           .          .     4ed785: XORL CX, CX
           .          .     4ed787: MOVQ CX, DX
           .          .     4ed78a: JMP 0x4ed796                             ;mem_test.go:9
       560ms      560ms     4ed78c: LEAQ 0x1(CX), BX                             ;command-line-arguments.slow mem_test.go:9
           .          .     4ed790: MOVQ CX, DX                               ;mem_test.go:9
        20ms       20ms     4ed793: MOVQ BX, CX                               ;command-line-arguments.slow mem_test.go:9
       290ms      290ms     4ed796: CMPQ AX, CX
           .          .     4ed799: JL 0x4ed78c                               ;mem_test.go:9
           .          .     4ed79b: MOVQ DX, 0x10(SP)                           ;mem_test.go:13

traspler4• 9 years ago

Spring Boot really comes away awful in this article (and the one he links in the article). I have mostly worked on projects using Dropwizard and Spring Boot and have come to like some of the easy to use abstractions but I have only scratched the surface of these frameworks, I assume, and most of it I will probably never use. Most of the time it did not matter that much to give the Server a GB more memory but recently we had some real issues with CloudFoundry. To me it seems that it's not possible to somehow magically tweak Spring Boot to behave better. So I came to wonder, what other framework would strike a better balance? Maybe not offer the crazy specialised stuff Spring offers or maybe needing some more work to get something to run but then offer vastly better memory and speed charactersitics?

I guess one way would be to implement the things I actually use myself with less abstractions but that honsetly feels very daunting to me. The Java ecosystem is so vast and between Handling the Requests, DB access (with or without ORM), persistance, caching, security, I really have no clue on where to even start doing something like this myself. If someone more seasoned here has some input, I would be highly interested!

RhodesianHunter4• 9 years ago

You mentioned Dropwizard early on, in my experience I have found it to be the polar opposite of Spring.

Where Spring is an "everything but the kitchen sink" framework, Dropwizard is just a collection of best-in-class libraries that are easily swapped out if you prefer something else.

I don't get to do so often professionally, but if given my choice of tools ill generally opt for a combination of Dropwizard in Kotlin.

traspler4• 9 years ago

One big drawback of Dropwizard (at least for us) was that it was very problematic to deploy it as a war in a tomcat environment. There is wizard-in-a-box but there were a lot of issues with that as well.

cs02rm02• 9 years ago

For me dropwizard looks like what you'd get from a room if experienced Java guys picking the best of breed to do one thing and one thing well.

Spring boot feels like it was from a room full of people where there's a guy in the corner with a form of tourettes that makes him keep yelling "Spring! Take the Spring option!"

It just isn't always the best option and sure, you can often switch out for something else but that defeats the point of an opinionated collection of the best available choices.

matrix• 9 years ago

Agreed. Kotlin + Dropwizard is an interesting approach. How well do Kotlin and Jersey play together in your experience?

jacques_chester8• 9 years ago

Most of the time it did not matter that much to give the Server a GB more memory but recently we had some real issues with CloudFoundry.

A long-standing difficulty with the Java Buildpack was working out how much memory to give it -- the Garden container engine is particularly ruthless to any process going above the allocated memory limit. I believe it got a lot of attention in v4 (edit: see [0] and [1]).

My personal pet peeve is that the memory quota is identical for staging and runtime. Which means that memory-intensive staging operations force you to have wasteful runtime memory allocations. I mostly noticed this when using Rails apps that pull in Nokogiri -- it gets compiled during staging and often that causes a lot of memory usage.

Disclosure: I work for Pivotal, we do the majority of Cloud Foundry engineering. We also sponsor Spring.

[0] https://www.cloudfoundry.org/just-released-java-buildpack-4-...

[1] https://github.com/cloudfoundry/java-buildpack-memory-calcul...

traspler4• 9 years ago

Thanks for the links. It's good to see that there is ongoing development to improve the situation.

It would still be nice to be have a Spring Boot app work on a 512MB container but even pretty simple ones don't seem to.

msl09• 9 years ago

Pretty good article but I wish we had the source code so we could compare code complexity also.

barrkel• 9 years ago

The primary reasons for choosing a particular language for your application server usually come down to developer ergonomics, developer availability and library availability / integration for your specific domain. Performance rarely comes into it; performance while scaling into astronomical request numbers is important for top web sites, but most app servers don't see that kind of load.

Even if some requests needs crunchy CPU power, it's better to offload them into something asynchronous than try really hard to make them perform well enough to be synchronous. Typically, hefty CPU jobs come in varying sizes, they're rarely guaranteed to be runnable under the 200ms or so latency bar for synchronous requests.

tomelders2• 9 years ago

I find that laughably generous. In my experience, Java gets a pass because it's "enterprise", whatever the hell that means. I've never heared anyone make a compelling case for Java on technical merit. It's always "we're a Java shop, because we're enterprise".

ssijak3• 9 years ago

You say there is no technical merit to use Java ecosystem over something else?

RhodesianHunter4• 9 years ago

"I've never heared anyone make a compelling case for Java on technical merit."

IMO this is a sign that you should expand the company you keep rather than being indicative of any flaw with Java.

jlmorton• 9 years ago

If you're a business making a language choice based on the technical merits of the language itself, you are very much doing it wrong.

fgonzag• 9 years ago

Because you can find libraries for all the "Enterprise" technologies easily. Besides .net, not many ecosystems offer that kind of library support. From XML signatures, to pkcs11, to complete soap support (including ws-security), to many others.

Fact is, in the Enterprise you'll have to interface with many of the aforementioned technologies, and Java still has the most support for them. We find that even in .net we sometimes have to implement some things on our own from the horribly convoluted "Enterprise" technologies because a specific feature isn't supported.

Retric2• 9 years ago

1,000 requests per second for a hello world micro service is horribly slow. I don't know what architecture they are using bit ~100,000 / CPU for simple requests is completely reasonable making this test very dubious IMO.

tyingq• 9 years ago

Some of the stacks he tested had significant failure rates at 1000/sec though.

Bumping it up would have made the comparison less interesting.

Perhaps it was just a small VPS?

rsynnott2• 9 years ago

One issue here is that it’d really be more appropriate to run the test twice and take the second result. This should get most JITing out of the way.

It’s hard to be sure what’s happening given the lack of source code, but 1000 requests per second where the request doesn’t do much is a fairly minor load for a well-written java service on modest hardware. I would expect the ‘lightweight’ java example to do better, assuming it’s just jetty or something.

emforce• 9 years ago

Hi All, this has received an incredible amount of views and comments since I posted this and left the house!

I appreciate the feedback comments! I'm very much still playing around with different technologies and writing about them as I go, this is more a learning experience for me that I've documented and I would take what I'm saying with a pinch of salt.

I thought I'd also clarify that I'm using the code/docker images that I created in the previous article as the base from which I'm running my tests as that seems to have gotten missed by a few peeps!

jacques_chester8• 9 years ago

I definitely missed it. Maybe adding a link to this one as well would be helpful.

Edit: I see the code using the `com.sun.net.httpserver` package, but not the Spring Boot version.

ajnin• 9 years ago

How does an "Hello world" request on localhost take an average of 50~100ms using any framework ? 1ms would already be too much. Something seems way off with this benchmark.

pipio21• 9 years ago

The myth that java is slow is not a myth it is true for medium to big programs with lots of interconnections, because with the abstraction you loose control on the program behavior.

You gain in other things, like abstraction, but the penalty is enormous. We use very high level languages in house, but when we do it performance is not the priority.

Doing "Hello worlds" is not a valid debunking of anything because , guess what? printing a Hello world in a screen is a very simple operation.

We have millions of lines of code and we have done tests on just converting 20-30Ks of our c,c++ code to java and the result has been disaster: Hundreds or thousands of times slower. That's right.

In my opinion if other companies are deluded wanting to be lazy, much better for us as competitors, but children should not be intoxicated with bad advice.

My advice is do not believe me or anyone else, when in doubt just test on your own.

sk5t• 9 years ago

Like with any language, the level of abstraction is up to you... one could just as easily "lose control" with C++.

Hundreds or thousands of times slower Java? Have you considered that the conversion was done with a weak mastery of the language or runtime?

The HFT folks working in Java would be taken aback to learn the language doesn't work.

luord• 9 years ago

Damn, I will have to take a look at aiohttp. That's impressive, I'm glad that python, my favorite language, can have that kind of performance.

Debunking the Java Performance Myth

Related Stories