Scaling to around 15K requests per second with Java

https://medium.com/inspiredbrilliance/journey-to-handling-15k-requests-per-second-with-java-part-1-fc405319c726

by singhalkul • 3 years ago

65 46 3 years ago

nhh7• 3 years ago

vert.x is a great framework. but in 2023 I would seriously explore regular spring boot with virtual threads and see how far that gets me first.

Reactive programming introduces complexity, one which may not be needed in the future due to improved core tech.

kapperchino• 3 years ago

+1 to this, reactive programming is a major pain in my ass. With virtual threads there’s no longer a need for them.

p2detar• 3 years ago

In general yes, but FWIW measurements point out that reactive vert.x seems to handle saturation much better. (1)

Also, vert.x are already exploring how to adapt to the new threads model. (2)

1 - https://gist.github.com/vietj/fe9f886d489853ab07111ba5715b13...

2 - https://vertx.io/blog/vertx-virtual-threads-incubator/

Lightbody• 3 years ago

Just curious -- have you personally confirmed this?

Not trying to cast doubt... just noting that I have exactly the same feelings (hopes?) but I have yet to personally confirm it, in large part because it's not yet fully GA so we haven't jumped in with our production workloads.

kitd2• 3 years ago

There's quite a few Java frameworks and libraries that will need to re-evaluate their existence (or at minimum, their core APIs) when green threads become popular.

kitd2• 3 years ago

Apologies replying to myself, but Netty, which underpins many of the popular Java backend frameworks, see backward compatibility as more important than supporting green threads.

https://github.com/netty/netty/issues/12816

It'll be interesting to see who (if anyone) picks up Netty's mantle in the Project Loom world.

sonicgear1• 3 years ago

Quarkus beats the crap out of Spring in every possible aspect, perf, DX, you name it.

willsmith72• 3 years ago

hard disagree on DX, which part of it do you find better with Quarkus?

inparen• 3 years ago

Virtual Threads are still in preview phase. People will wait for stable release or at least someone to try in prod and give lightning talk or publish paper. Seems little premature now.

no_wizard2• 3 years ago

Using Elixir for a similar problem and it scaled well past 15K without any real intervention on my part, and it was a similarly tiny instance.

Not that they shouldn't have used Java, I understand their application was already written in Java and it wouldn't make sense to use an unfamiliar language, but I do wonder about Javas' efficiency here.

Twirrim• 3 years ago

15k/rps is on the low side for Java, so I found the post to be a little strange. The netty library can handle 10x that without batting an eyelid. We do much more than 15k/s on API nodes with hardly any attention paid to optimisation around traffic.

Just to pick a random stack overflow post, as a little anecdata to go with anecdote https://stackoverflow.com/questions/44708704/netty-as-high-p... has someone who's complaining they were only hitting 50k POST/sec with Netty.

no_wizard2• 3 years ago

Maybe they wanted to share their experience with Vert.x and it is just poor form to focus on the 15K a second?

gmokki• 3 years ago

We did 10k rps almost 20 years ago on pair of Sun x4100 servers. Dual core Opteron@2,4GHz and Java 1.4. And that was with storing data to spinning rust (async) and business logic using custom scripting language.

coffeebeqn• 3 years ago

These benchmarks are a little pointless. 15k/s is either amazing or poor or average depending on what actual work the server does. If it’s just a glorified CDN as in many benchmarks then it’s not that great(?)

jerrygenser2• 3 years ago

Most of the ability to get throughput is based on not blocking a request on network or other IO and also not doing writes in band of request.

I think it's less about the web server choice and more about the architecture around pre-computing and caching certain results in memory and then asynchronously writing via a queue.

photonerd• 3 years ago

Given I’ve done ~8k with Node and not a ton of effort, I agree.

That said: it’s a good bit of writing & interesting

moomoo11• 3 years ago

I have a Go api that can handle 70-90k plus rps.

However in practice it’s more around 4-6k because it’s doing database operations. Some end points are only 1500 rps while some of my read ops are in the 10-25k.

bicijay• 3 years ago

Add a load balancer + 24 machines.

hn8305823• 3 years ago

Or write it in C on Linux/BSD and use a one 10 year old server

jacquesm3• 3 years ago

I wouldn't be so quick to say that. I've pretty spent the better part of my life writing C code, lots of it networked. And while you can get absolutely awesome performance out of it on modest hardware the JVM is no slouch at running networking stuff, I wouldn't be surprised if it is roughly on par with optimized C for that kind of application. Usually it's the kernel side that will be your bottleneck and it won't make much of a difference what the application is written in once you are that close to the theoretical max.

dabbz• 3 years ago

Writing it in C isn't some magic bullet. Java you're spending your time optimizing the tail end of efforts, but C you're spending your time optimizing the bulk of the effort that Java gives for free. (not that I'm defending Java, but C being low level doesn't make it instantly easy to do faster stuff with).

exabrial3• 3 years ago

Our ActiveMQ brokers running on a single-core, 1gb, Digital Ocean node easily push 5k messages/s through ~200 queues and another several thousand/s through a couple hundred topics. They do reach a cliff edge though on single cores. Dual cores pretty much quadruples the throughput to the point where SSD latency is the problem.

Our Tomcat servers don't see 15k requests/s, but our load testing has shown a pair of them is good up to 3k/s (each) on dual-core 2gb servers with 512mb heaps. We really didn't push it much farther because that exceeds our needed capacity.

This is pretty boring technology. I'm fairly certain we could hit 15k req/s per node with plain Tomcat if we actually tried. I do think if you wanted to stretch past 15k req/s you probably do need more exotic programming models like Vert.X, but that does come at complexity.

I think the ultimate in throughput would be to have Tomcat listen on a unix domain socket (This is actually supported in Tomcat(https://github.com/apache/tomcat/commit/a616bf385a350175a33a...) and have TCP terminated with HaProxy (https://bz.apache.org/bugzilla/show_bug.cgi?id=57830). We set this up and it is _really really really_ fast, but we'd be on our own for support since nobody else is really doing it. Tomcat also has some strange behavior with the sock file, you kinda have to manage it on your own which is strange, but it could be done within the startup scripts.

jacquesm3• 3 years ago

What kind of application is this and roughly what size messages?

exabrial3• 3 years ago

Multiple applications, not just a single on, powering call centers, not sexy SV stuff. Queues are used for two patterns: reliable data transport that we expect to be in near-realtime, and reliable event sourcing. The topics are mainly used non-reliable event signaling... for instance cache eviction (hey this row was updated, if you have it in your local cache, discard it or you'll get a conflict if you try to commit to the db). They also form the backbone for websocket broadcasting (sending messages to all of a users' app windows) Messages are all text/json or text/xml; I'd say 512b-8kb with an average around 1k. Data transmission messages are on the large end of that; signalling messages are on the small side of that.

grandinj• 3 years ago

Terminating the TCP with any kind of reverse-proxy and forwarding over a local TCP port to Tomcat would get you most of the benefit with very litte configuration and is a fairly standard setup.

It plays to the strengths of both becuase the reverse proxy handles long-lived but slow connections very well, and the Tomcat server gets to have fewer, but hotter threads.

exabrial3• 3 years ago

That's actually our prod setup; we do that now and it's a pretty robust. As a bonus is gives you a nice "wiretap point" you can use tcpdump for debugging.

In our limited tests with HaProxy -unixDomainSocket-> Tomcat, we saw a significant increase in throughput and decrease in latency (when it worked). It got rid of our wiretap point though which is pretty handy, and we didn't _really_ need the performance increase. If I were to speculate, HaProxy is a rocket and beats even the mature codebase of Tomcat it seems when parsing http, and probably takes advantage of kernel features that haven't made its way into Tomcat yet. It wasn't prod-worthy stable, sometimes execution threads seemed to hang.

l5870uoo9y• 3 years ago

To maintain consistency with the current infrastructure, we continued utilizing the existing setup, which includes a Kubernetes cluster on AWS EC2 instances.

Ironically this would likely be too expensive to deploy on AWS serverless infrastructure (AWS Lambda, API GateWay etc) even-though it touts endless scalability as it main feature.

berkle44554• 3 years ago

This was an impressive metric 20 years ago. Today you can achieve 15k/sec on a single $10/mo VPS running any number of Java, Go, JS, PHP, or Rust apps.

brunoborges2• 3 years ago

Specify the benchmark use case. Otherwise, nearly any runtime can do a "hello world" at that throughput. Including the JVM.

See TechEmpower benchmarks, or gRPC benchmarks as a starter. JVM workloads on those are always around top 5, at most top 10.

berkle44554• 3 years ago

Did my comment come across as anti-JVM to you? That was not my intention. My point is that 15k requests per second is child's play.

talldatethrow2• 3 years ago

I run a micro saas with 34 customers, with about 300 users online at any given moment at the peak.

It's written in php, running on a $4 a month shared hosting from namecheap.

Everyone says I'll need to move it over somewhere better if it grows. How soon? Even people with experience seem to have trouble explaining when. It always fascinates me and makes me wonder how many small webapps are on AWS for no real reason.

Personally, I need cpanel and phpmyadmin because I don't know anything else really. I still ftp files up manually from my windows computer. Going to a real server seems like it would make things 10x more complicated, and no one seems to be able to predict at what point I'll need to.

lionkor• 3 years ago

Going to a "real" server could just mean switching it up, like a $3-5 vps, sftp instead of ftp, ...? and then you can scale by just scaling up the vps. Your solution is quite good.

berkle44554• 3 years ago

are on AWS for no real reason.

One of the core offerings in AWS is EC2. Which are just shared (up to dedicated) virtual machines, and they start at a few dollars a month. I'm in no way suggesting a change to your setup, only pointing out that AWS isn't some mythical complex ecosystem.

Is namecheap running PHP8? That upgrade alone will net you a massive performance improvement over 5.x

hu3• 3 years ago

I remember a consulting gig where they were pushing 50k requests per second with PHP in a 2018 laptop by using this framework https://github.com/walkor/workerman

In a real application, with database connection pooling and auth sessions, it went down to 15k requests/s.

And that was PHP7. PHP8 introduced JIT so it's probably significantly faster these days and hopefully fully typed.

albertopv• 3 years ago

I think these comparisons are a bit silly, it really depends what you are doing for each request. Are you just proxying, message brokering or querying a database? It makes a lot of difference

Scaling to around 15K requests per second with Java

Related Stories