Scala vs. Go TCP Benchmark

http://eng.42go.com/scala-vs-go-tcp-benchmark/

by rck • 13 years ago

139 129 13 years ago

eng.42go.com

shin_lao5• 13 years ago

I think the JVM TCP stack is extremely mature, but I'm really surprised to see that Go is ten times slower.

It would have been interesting to have a C/C++ benchmark for reference.

skrebbel• 13 years ago

Your comment makes me wonder whether it matters on which JVM the benchmark is run. Are Oracle, GNU Classpath and Dalvik equally fast?

wyuenho2• 13 years ago

With Go using 20 times less memory, I'd take Go any day since I can just throw in 20 more processes and up my throughput 5-10x. But of course, scaling like this takes a bit of design and the processes have to share nothing.

shin_lao5• 13 years ago

I would take the memory measurement with a grain of salt. A 200 MiB large JVM doesn't mean all the 200 MiB are used. They could have been reserved (preemptive memory allocation).

trailfox4• 13 years ago

That doesn't even make sense. The Scala version can simply spin up more threads. It's not an issue of parallelism, the Scala version just happens to be faster, throwing more Go processes at the problem won't help. Go is already multithreaded. You don't need multiple processes to use the cores on the box. The JVM scales memory usage by default (heap size etc.) based on the amount of memory on the host. If you're going to worry about 100 MB RAM it's easy to constrain the JVM with -Xmx

vidarh2• 13 years ago

If the JVM and/or the Scala runtime does user-space buffering and Go forwards straight to the read/write syscalls, that alone would probably sufficient to explain the difference when reading/writing buffers this small.

If you don't do buffering in user space, the context switches in/out of the kernel will kill you when you do many small read or writes.

No idea if that's it, but that's the first place I tend to look when troubleshooting networking app performance as so many people just blindly use syscalls as if they're free.

jongraehl10• 13 years ago

The better performance of the Scala client+server if anything suggests less buffering, not more, since the next ping can't be written until the previous pong is received.

p.s. to parent: "the JVM TCP stack?" really?

damian2000• 13 years ago

Isn't this more like a comparison of the JVM's TCP library vs Go's TCP library... not so much Scala vs Go?

est2• 13 years ago

They are all wrappers around the OS TCP/IP stack.

willvarfar3• 13 years ago

The benefit of go is the cheap threads and CSP which make it scale well for complex servers.

I think we'll see the Go runtime gaining Single System Image distribution, performance improvements and libraries not services (e.g. groupcache) making it a very different world to develop in than Scala.

octo_t• 13 years ago

Scala has Actors, which are as pure CSP as you can get (Go's CSP has shared mutable state, Actors do not).

dschiptsov3• 13 years ago

... that the Go server had a memory footprint of only about 10 MB vs. Scala’s nearly 200 MB. Priceless!

200Mb for such a crappy server with only hundred connections.

So, it is not about whose wrapper around epoll is thinner, but about how the data are represented and copied.

ratherbefuddled• 13 years ago

You can't really come to a conclusion since no information is given about the configuration of the JVM. Most likely the minimum heap size was unset which often means it will default to 1/64 of the total system memory.

dschiptsov3• 13 years ago

I have my conclusions for quite long time.)

coldtea8• 13 years ago

200Mb for such a crappy server with only hundred connections.

Yes, and almost a megabyte of disk space for a "hello world" program. C++ is so inefficient, right?

That's not the way you judge the performance of a program and/or it's memory use -- and neither is comparing the 200MB of the JVM with the 10MB of Go here...

dschiptsov3• 13 years ago

No, it is almost 2Mb per connection.

Why so-called performance of a wrapper around OS syscalls is more important than resource costs of having a pending connection?)

est2• 13 years ago

The experiments where performed on a 2.7Ghz quad core MacBook Pro with both client and server running locally, so as to better measure pure processing overhead. The client would make 100 concurrent connections and send a total of 1 million pings to the server, evenly distributed over the connections. We measured the average round trip time.

Another let's rape localhost:8080 on a MacBook Pro™ benchmark

aa0• 13 years ago

Benchmarked? More like benchmucked.

rkangel2• 13 years ago

The way I see it, instead of basing a decision on what's currently fashionable, they've gone for some empirical data and have actually conducted an experiment.

Sure this is a fairly simplistic first step, but so should all first steps be as you gain understanding, and it's proved its worth by producing a surprising result in a very simple case, that's worth understanding before moving on to more complicated tests.

What you've got here is engineering, rather than just making it up as you go along.

jdub• 13 years ago

Please don't use the R word, which is so deeply emotive and meaningful, in such an inappropriate context.

fleitz• 13 years ago

Rape has meanings other than non-consensual sex.

GP: Please use the word rape to the full extent provided by the english language, and let those who don't like the word deal with it.

As for the benchmark, who writes their own load balancer? Isn't this generally a solved problem? If the point is to extract max performance from a simple ping-pong server then I'd go right to C and epoll/libevent/etc. I'm guessing that the team is somehow trying to extrapolate data from a ping-pong server to the actual problem they're trying to solve which is dare I say, stupid.

In general the best way to solve this problem is use whatever language the team / person writing the software likes because having them write it faster generally outweighs whatever server costs one will run into, whenever this is not the case the best answer is 99% of the time: write it in C.

andyhmltn2• 13 years ago

It's a word. I find it weird that people are so offended by a word rather than the context that's its used in. You really need to get a grip.

donquichotte• 13 years ago

But may I use the G word? http://www.youtube.com/watch?v=yh1RkRxnAOs

plainOldText• 13 years ago

Yeah, I'd be wary of benchmarks where the server and client run on the same machine.

laumars2• 13 years ago

I thought this. While I don't think that benchmark is entirely worthless, it's hardly conclusive enough to be basing a business decision on:

* What's the concurrency like between two servers? (you need to test both low and high concurrent loads)

* What's the difference between the two in creating new connections and/or threads?

* What's the performance like when you up the number of packets per connection? (a simple ping pong hiding buffering performance etc).

* What's the performance like on the host platforms? (I'm going to assume that they're not the same OS on their Macbooks as they will be on their load balancers).

Of course, arguing a case of using Scala because it's a language they're already proficient in is itself a perfectly valid reason. But my issue is they seemed more interested in picking either language based on the best performance without actually evaluating the performance of either in any detail.

madisp• 13 years ago

"The actual test code contained some functionality to deal with connection errors, omitted here for brevity."

Any way to see the actual test code?

anuraj• 13 years ago

It is the JVM rather than Scala - one of the fastest VMs implemented till date.

aaron42net• 13 years ago

If the eventual production app would run on Linux (which I'm only guessing based on the context), this benchmark should probably be run there. Darwin's surprisingly higher system call and context switch overhead can be deceptive for apps that are OS-bound.

cmccabe2• 13 years ago

EDIT. OK, so. I ran this benchmark myself on an 8-core Xeon running Linux. 2.13 GHz, CentOS 6.2. Kernel was 2.6.32-220.el6.x86_64. 50 gigs of RAM.

I got somewhere between 2.0 and 2.2 "milliseconds per ping" for Scala 2.9, and somewhere between 3.5 and 3.7 for Go 1.1. This is not the 10x difference that the authors reported, but it is something. The difference may be due in part to the different platform and hardware I am using.

Contrary to what I wrote earlier, I noticed that GOMAXPROCS=8 did seem to be slower than GOMAXPROCS=4 here. I got around 4 "milliseconds per ping" with GOMAXPROCS=8. Using a mutex and explicit condition variable shaved off maybe 0.2 milliseconds on average (very rough estimation).

Again contrary to what I wrote earlier, Nagle on versus off didn't seem to matter in the Go code. I still think you should always have it off for a test like this, but on my setup I did not see a difference.

I still don't think this benchmark is showing what they think it is. I have a hunch that this is more of a scheduler benchmark than a TCP benchmark at all. I think I'd have to haul out vtune to get any further, and I'm getting kind of tired (after midnight here).

BarkMore• 13 years ago

Nagle's algorithm is disabled by default in Go (http://golang.org/pkg/net/#TCPConn.SetNoDelay).

jongraehl10• 13 years ago

First of all, they're testing on MacOS, which is not going to be the platform they're actually using for the server code. The backends can be very different.

Unless you have evidence that Go has MacOS-only bugs, this is meaningless speculation (though I agree that it's possible that there's no problem on Linux and that people don't generally run mac servers, I'm not sure why we should privilege your hypothesis).

Agreed about GOMAXPROCS=4 - that seemed questionable to me (I don't know what it does, precisely, but I don't see a 4-anything limit in the Scala code).

jongraehl10• 13 years ago

Thanks to parent (after edit) for really testing. It turns out that he was right that Go on Mac is substantially worse than Linux - my bad. Maybe the Go lib authors didn't put much effort into reading the subtly different BSD/Darwin vs Linux syscall semantics.

To explain the Nagle algorithm's irrelevance to this case, we have to understand how it works. It doesn't delay any writes once you read. It only affects two consecutive small writes (my memory was fuzzy so I checked http://en.wikipedia.org/wiki/Nagle's_algorithm ). Odds of preemption between the client's write and its read seem small, so it shouldn't matter whether you Nagle or not.

trailfox4• 13 years ago

I got somewhere between 2.0 and 2.2 "milliseconds per ping" for Scala 2.9

The latest Scala is 2.10, which has a number of optimizations over 2.9...

dlsspy3• 13 years ago

Did you consider running the go client against the scala server and vice versa?

Also, that's kind of a lot of code. Here's my rewrite of the server: http://play.golang.org/p/hKztKKQf7v

It doesn't return the exact same result, but since you're not verifying the results, it is effectively the same (4 bytes in, 4 bytes back out). I did slightly better with a hand-crafted one.

A little cleanup on the client here: http://play.golang.org/p/vRNMzBFOs5

I'm guessing scala's hiding some magic, though.

I made a small change to the way the client is working, buffering reads and writes independently (can be observed later) and I get similar numbers (dropped my local runs from ~12 to .038). This is that version: http://play.golang.org/p/8fR6-y6EBy

Now, I don't know scala, but based on the constraints of the program, these actually all do the same thing. They time how long it takes to write 4 bytes * N and read 4 bytes * N. (my version adds error checking). The go version is reporting a bit more latency going in and out of the stack for individual syscalls.

I suspect the scala version isn't even making those, as it likely doesn't need to observe the answers.

You just get more options in a lower level language.

jongraehl10• 13 years ago

I think you're on the right track in supposing that there can't be a huge performance difference in such a simple task, given that both languages are compiled and reasonably low-level. The most plausible explanation would amount essentially to a misconfigured library, not a fundamental advantage due to say, advanced JVM JIT. Your suggestion to try server-{a,b} x client-{a,b} is also a good one.

Your modified Go server doesn't return "Pong" for "Ping". It returns "Ping". And the "a small change" version is nonsense. It's fundamentally different. - you're firing off all your requests before waiting for any replies, and so hiding the latency in the more common RPC style request-response chain, which is a real problem.

You speculate a lot ("hiding some magic" "likely doesn't need to observe the answers") when you haven't offered any insight.

EDIT: Nagle doesn't matter here - it doesn't delay any writes once you read (waiting server response). It only affects 2+ consecutive small writes (here I'm trusting http://en.wikipedia.org/wiki/Nagle's_algorithm - my own recollection was fuzzy). If Go sleeps client threads between the ping and the read-response call then I suppose it would matter (but only a little? and other comments say that Go defaults to no Nagle alg. anyway).

bsdetector3• 13 years ago

The most plausible explanation would amount essentially to a misconfigured library, not a fundamental advantage due to say, advanced JVM JIT.

Really, the most plausible explanation? I'd say the most plausible explanation is that M:N scheduling has always been bad at latency and fair scheduling. That's why everybody else abandoned it when that matters. It's basically only good for when fair and efficient scheduling doesn't matter, like maths for instance, which is why it's still used in Haskell and Rust. I wouldn't be surprised to see Rust at least abandon M:N soon though once they start really optimizing performance.

shin_lao5• 13 years ago

The most plausible explanation would amount essentially to a misconfigured library, not a fundamental advantage due to say, advanced JVM JIT.

Configuration rarely impacts such trivial cases. I would rather bet on a thread affinity or page locality.

dlsspy3• 13 years ago

Your modified Go server doesn't return "Pong" for "Ping".

The program doesn't read the result, so it doesn't matter. Returning Pong isn't harder, but why write all that code if it's going to be ignored anyway?

It's fundamentally different. - you're firing off all your requests before waiting for any replies, and so hiding the latency in the more common RPC style request-response chain, which is a real problem.

As I said, the program isn't correlating the responses with the requests in the first place -- or even validating it got one. I don't know scala, but I've done enough benchmarking to watch even less sophisticated compilers do weird things with ignored values.

I made a small change that produced semantically the same program (same validation, etc...). It had similar performance to the scala one. If you don't think that's helpful, then add further constraints.

luikore3• 13 years ago

So this Go server is slower than a single threaded, blocking TCP server in Ruby. And the memory? Almost the same:

require "socket" s = TCPServer.new 'localhost', 1201 loop do c = s.accept if c.read(4) c << 'Pong' end c.close end

dlsspy3• 13 years ago

It's not obvious to me what that does. Does that only service one of the 100 clients or one of their 10,000 pings each? Or both?

luikore3• 13 years ago

It's one at a time. But, since the server only performs a one-shot task without the "hang on and wait 30 seconds"-like long connections, and, the default socket backlog of TcpServer is > 100, so every client gets served within the delay of (0~99)*6ms.

In short, every round is fully served, and the concurrency level is 100.

jaekwon2• 13 years ago

hmm, i tried running that against the go client but the client wouldn't connect.

  dial tcp 127.0.0.1:1201: connection refused

i changed 'localhost' to '127.0.0.1' and now I see a whole lot of

  dial tcp 127.0.0.1:1201: connection reset by peer

some connections were successful though.

luikore3• 13 years ago

The first one may be your localhost not mapped to 127.0.0.1 ? The second one may be a problem of backlog size. On my system (ruby 2.1.0dev (2013-08-06) [x86_64-darwin12.4.0]) it's 128 and there's no reset. But you can manual change it with s.listen(1000)

smegel2• 13 years ago

Quick look at the language benchmark games, and Go is not 10x slower than Java for most tests. Java is often 3-5x slower than C/C++/etc (although maybe it didn't get enough time to warm up).

trailfox4• 13 years ago

A quick look at the language benchmark game, and Go varies between 7x slower than Java and approx. the same speed as Java (varies by test): http://benchmarksgame.alioth.debian.org/u64q/benchmark.php?t...

I fail to see how Go being slower than Java relates to C being faster than Java.

smegel2• 13 years ago

Well given both C and Go are native compiled languages, I would think C is a realistic (if distant) goal for Go performance, at least for algorithmic stuff where you're not bouncing in and out of the runtime. I was commenting on how far it has to go.

bad_user4• 13 years ago

So even if Scala is ahead in this (flawed) benchmark, that's not how you write a TCP server in Scala, because you want to do it non-blocking. Not doing it based on asynchronous I/O means that in a real-world scenario the server will choke under the weight of slow connections, not to mention be susceptible to really cheap DoS attacks like Slowloris [1].

Seriously, it goes beyond the underlying I/O API that you're using. If anywhere in the code you're reading from an InputStream or you're writing to an OutputStream that's connected to an open socket, then that's a blocking call that can crush your server. Right now, every Java Servlets container that's not compatible with the latest Servlets API 3.1 can be brought down with Slowloris, even if under the hood they are using NIO.

Option A for writing a server in Scala is Netty [2].

Option B for writing a server in Scala is the new I/O layer in Akka [3].

[1] http://ha.ckers.org/slowloris/

[2] http://netty.io/

[3] http://doc.akka.io/docs/akka/snapshot/scala/io.html

caniszczyk• 13 years ago

Also see Finagle (uses Netty): http://twitter.github.io/finagle/

geal• 13 years ago

Ok, so. When you need to write a load balancer and want to test different languages for the task, you don't do a benchmark like that one.

Writing a "ping-pong"? And not using the same client to test both servers?

It would not have been too hard to write a simple proxy in both languages. Not even worrying about parsing HTTP headers, just testing TCP, that is really easy.

Now, if you really want to test the performance, you have to implement it differently. Just two small features you would need in both:

* non blocking IO: right now, you're starting a new future in Scala, that's easy to write but not really efficient (it might work better with goroutines) * zero copy: if you're load balancing, you will spend your time moving bits, so you'd better make sure that you don't copy them too much. It is possible with Scala, but it looks like Go does not support it

Now, when you have reasonable testing grounds (that woudn't be more that a hundred lines in both languages), better get your statistics right.

"The client would make 100 concurrent connections and send a total of 1 million pings to the server, evenly distributed over the connections. We measured the average round trip time" -> that is NOT how you should test. Here, you would want to know what is the nominal pressure the balancer could handle, so you must measure a lot of metrics:

* RTT * bandwidth (per user and total) * time to warmup (the JVM optimizes a lot of things on the fly, you have to wait for it) * operational limits (what is the maximal bandwidth for which the performance crashes? same for number of users)

And then, you don't measure only the average values. You must measure the standard deviation. Because you could have a good average, but wildly varying data, and that is not good.

Last thing: the macbook may not be a good testing system.

jaekwon2• 13 years ago

Results from my Mac:

  Go server vs Go client: 10ms
  Scala server vs Scala client: 3ms
  Go server vs Scala client: 4ms
  Scala server vs Go client: ????

Scala server against the Go client is really slow (?). I reduced the ping count by a factor of 100, and extrapolating I think it would have reported around 670ms. What gives?

I don't know much about Scala Futures, but isn't Scala's client doing something completely different than the Go client? Scala's client with Future.sequence looks like it's calling each `ping` method sequentially.

Printing the connection identifier {0,100} on open & close shows that while it isn't completely sequential, only about a handful of connections are open at a time.

On the other hand, the Go client appears to switch amongst goroutines more frequently. All the connections open before any connection closes.

In other words, the I think the difference in performance is due to the difference in how randomly the connections are shuffled. The terrible performance time in the last case I think shows a bottleneck in the Scala server rather than the Go client.

corresation• 13 years ago

As a word of advice -- you are almost certainly wasting your time writing a load balancer. There are close to zero cases where someone can legitimately justify such an exercise.

In any case, your benchmark is flawed (as virtually all benchmarks are). The reason Go the client is slower is because it tries to pervasively "thread" via the M:N scheduler -- every wait check causes it to yield the actual thread and switch goroutine, creating a large amount of overhead. The Scala cases, on the other hand, is dramatically more limited and will not yield this overhead.

The Go server does not have this fault, and is likely top performance. And aren't we talking about a server anyways?

Now as to the client, while we could naively criticize M:N scheduling based upon this, try giving it a more realistic workload (unless you seriously plan on load balancing pongs): Instead of ping/pong, return larger lengths of data preferably over actual network connections (not localhost) - e.g. 32KB.

The Go client will catch up if not shoot into the lead. M:N scheduling is optimal for most real-world workloads, though it is less optimal for spin-off-a-million-goroutines that do nothing type tests.

This is not a test of TCP overhead, or a realistic test, but instead demonstrates the small overhead of goroutines when you give each a minuscule amount of wait work.

diakritikal3• 13 years ago

You're right about the Go client, I'm also scratching my head as to why it's doing the extra lookup instead of just using a dial function directly:

    tcpAddr, _ := net.ResolveTCPAddr("tcp4", "localhost:1201")
    conn, _ := net.DialTCP("tcp", nil, tcpAddr)

Vs.

    val socket = new Socket("localhost", 1201)

Edit: Go has e.g. net.Dial("tcp", "localhost:1201") and as someone pointed out elsewhere for a more accurate bench in both clients why not use the numerical address instead?

Oculus• 13 years ago

What would be the advantages of using a custom built load balancer vs. something off the shelf like Nginx?

voidlogic• 13 years ago

The experiments where performed on a 2.7Ghz quad core MacBook Pro with both client and server running locally

No no no. Assuming your production code runs on Linux THAT is where you need to do this test. It is extremely naive to assume that either the JVM or the Go runtime will perform system interfacing tasks even remotely similar between OSX and Linux. Linux is what you will use in production. Linux is almost always faster (more effort from devs both on the kernel TCP/IP side and the runtime/userspace side).

Write your Scala, Java, Go wherever you want, but please, benchmark it in a clone of your production environment!

P.S. In production I assume your client and server will not be local... don't do this, kernels do awesome/dirty optimizations over loop-back interfaces, sometimes even bypassing large parts of the TCP/IP stack, parts you want included in any meaningful benchmark.

msie• 13 years ago

Ugh, after reading all the comments here I wonder how a mere-mortal programmer gets multi-threaded, network programming done right. It's not clear to me if there is a clear winner between Go and Scala/JVM. Are the majority of programs out there crappy, memory-hogging and non-performant?

EDIT: Any good references out there? Thanks!

Scala vs. Go TCP Benchmark

Related Stories