Techempower Web Framework Benchmarks Round 10

https://www.techempower.com/benchmarks/#section=data-r10

by pneumatics • 11 years ago

110 62 11 years ago

chrisan• 11 years ago

Come for that stats, stay for the comedy

The project returns with significant restructuring of the toolset and Travis CI integration. Fierce battles raged between the Compiled Empire and the Dynamic Rebellion and many requests died to bring us this data. Yes, there is some comic relief, but do not fear—the only jar-jars here are Java

Brilliant!

tolas• 11 years ago

Still no Elixir/Phoenix inclusion? I'd really love to see how it stacked up.

bhauer2• 11 years ago

Coming in Round 11 [1]! We'll aim to have Round 11 out much quicker than 10.

[1] https://github.com/TechEmpower/FrameworkBenchmarks/pull/1510

perishabledave• 11 years ago

Thanks! I'd like to see this as well.

saryant4• 11 years ago

The chart says Play Framework didn't complete but looking at the output, the logs say it did.

https://github.com/TechEmpower/TFB-Round-10/blob/master/peak...

What am I missing?

bhauer2• 11 years ago

Sorry, the links to logs were going to logs from a preview run. I've just changed the links to direct to the final logs. In the final run, play2-scala did not respond:

https://github.com/TechEmpower/TFB-Round-10/blob/master/peak...

saryant4• 11 years ago

Generally that error happens when Play tries to bind to a port that's already in use. Looking at the start scripts for the Scala projects, the RUNNING_PID file is just being removed if it exists but your script should probably kill that PID if the file exists before deleting the file.

https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...

wheaties2• 11 years ago

It's not that they didn't complete, they never even started. Definitely configured incorrectly for the tests.

kainsavage7• 11 years ago

Agreed. Our logging has undergone some solid improvements in the last week or two, and so round 11 will, if not resolve this issue completely, make the logged output more useful for tracking down issues like this during the preview runs.

virtualwhys• 11 years ago

Maybe they need to pass `-Dhttp.address=...` to the start script[1]; Play binds to `0.0.0.0` and port 9000 by default.

Here's the corresponding error from the log in Play[2]. Not sure what else it could be...

[1] https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...

[2] https://github.com/playframework/playframework/blob/2.2.x/fr...

richdougherty• 11 years ago

An error occurs, which is logged to stderr, but the benchmark logs don't capture stderr so it's hard to know what's happening. (Or maybe Play redirects stderr to a log file?)

The test passes in the preview runs, in the TechEmpower continuous integration tests and in the EC2 tests so it's probably some transient error that only occurred in the final bare metal test. Maybe there's a race condition in the Play 2 test scripts which only shows up sometimes.

I've spent a fair bit of time maintaining the Play 2 benchmark tests so it's very frustrating to get no result on the final test. Oh well!

saryant4• 11 years ago

Out of the box, the start script from "play stage" does not redirect stderr.

Though I didn't think to check the classpath when I was poking around the TechEmpower github repo. I wonder if a logback.xml slipped in somewhere that's siphoning off stderr to some unknown destination?

aikah2• 11 years ago

I'd like to see the memory benchmarks as well.It's fine to have something fast but the more memory the more expensive the boxes are.

kainsavage7• 11 years ago

We HAVE the memory data, but we have not distilled it into a consumable form to use on our website. Here, for example, is the cpu/ram usage report for ULib's json test: https://github.com/TechEmpower/TFB-Round-10/blob/master/peak...

WoodenChair• 11 years ago

Dart dominated the multiple queries test type.

Cyph0n3• 11 years ago

That is quite surprising. Anyone have an idea why that is?

kainsavage7• 11 years ago

Actually, not really. We checked the code to ensure that there was no gaming the system and it definitely APPEARS to be making separate database queries as we require in our rules. In fact, we had this same question in round 9 and had a number of people audit it. We cannot explain it other than it might be pretty darn fast.

wheaties2• 11 years ago

That's easy to answer, instead of using a relational database or even PostgreSQL's JSON data store, it's using MongoDB. I hate MongoDB but for tests like these where there's absolutely no writes what so ever, Mongo is going to fly. Hence, I read this as comparing MongoDB read performance to PostgreSQL, MySQL or SQL Server perf (or any of the other DB that are listed.)

That's why benchmarks like these have to be scrutinized. I like looking them over but in reality, they're not Apples to Apples.

vdaniuk3• 11 years ago

Okay, Nim is really fast in these benchmarks on ec2 servers, impressive.

Does anyone has experience with nim web stack? Is it ready for prime time? How much effort is required to create a simple CRUD json api?

On a sidenote I am really looking forward to comparing rust and elixir results in the next round of benchmarks.

DAddYE2• 11 years ago

See: https://github.com/dom96/jester

vdaniuk3• 11 years ago

thx for the suggestion

kainsavage7• 11 years ago

Rust and Elixir SHOULD be included in Round 11 - we have already accepted a pull request for the Phoenix framework (Elixir) and have had a pull request for Rust when it was in alpha. Hopefully, we will see another Rust pull request soon.

vdaniuk3• 11 years ago

Pleased to hear that, thanks for doing the benchmarks!

idlewan• 11 years ago

It's getting there fast. We still need some niceties like hot code reload for the server, but experiments like [0] are on the way. I'd love to see Rust and Elixir perform well too for the next rounds!

I recommend you to read the nim code used for the benchmark [1] [2] [3] [4], and if you like what you see, to start with the Nim tutorial [5].

[0] https://github.com/transfuturist/outlet

[1] https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast... [2]: https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast... [3]: https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast... [4]: https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...

[5] http://nim-lang.org/tut1.html

DAddYE2• 11 years ago

Few notes:

* Impressive Dart

* JRuby > MRI (I'd like to see JRuby 9k)

* Padrino that offers basically everything that Rails does performs impressively well. [Shameless Plug]

cheald• 11 years ago

My experience has been that JRuby 9k is slightly slower than MRI in a number of cases right now, mostly because its IR has been completely rewritten, and is still pending a performance pass.

That said, it's still a huge step up from the 1.7 series, and once the team starts knocking out performance problems it should be pretty magnificent.

redstripe• 11 years ago

There's something I don't understand in these comments. Why is everyone interested in language comparisons instead of the huge difference between EC2 and bare metal?

wmf• 11 years ago

Because people think they can't use bare metal and thus its advantages are irrelevant.

stefantalpalaru• 11 years ago

From what I know about the Xen virtualization used by EC2 the instance variability is too high to get consistent results over time and the overhead for this type of benchmarks is so big it's not even funny.

That's why the bare metal results are the only relevant ones. The playing field is fair and stable. Let the battle begin ;-)

buster• 11 years ago

Indeed.. I am wondering the same.. I fear that at some point the baseline for performance will be some virtualized server and people will forget just how fast real hardware can be.. At some point people will think it's normal to host a simple blog on 5 virtual servers ;)

Also, by its nature EC2 should be terrible for serious benchmarking, since you have no control whatsoever about the infrastructure.

sauere2• 11 years ago

Got to agree. Bare Metal performance isn't appreciated enough. I know a few companies that run fine with a mixed Metal/AWS combo. Metal handles 80% of the workload, and if that for some reason fails, EC2 instances are fired up to take over until it is fixed. This setup doesn't work for every scenario, but it is something to take into consideration.

ckluis2• 11 years ago

I remember being in a meeting talking about the tax companies using X-Metal servers for the whole year and scaling Y-cloud instances for the 3 months they have excessive usage.

sker• 11 years ago

I'd like to see some ASP.NET running on Owin. Perhaps I'll find the time to add it myself before round 11.

kainsavage7• 11 years ago

I am not familiar with Owin; how does it differ from Mono?

meragrin• 11 years ago

Owin is an interface between .NET web servers and web applications.

Mono is an implementation of the .NET runtime and framework.

ckluis2• 11 years ago

It’d be interesting to see on Nano too.

sauere2• 11 years ago

Bottle handling 5x more requests than Flask. Impressive, but overall Python framework performance ist still... meh.

Cyph0n3• 11 years ago

Falcon is where the performance is at according to the benchmarks.

trentnelson• 11 years ago

Or PyParallel, when they include Windows in the next round: https://speakerdeck.com/trent/pyparallel-pycon-2015-language...

Consistently orders of magnitude faster than everything else out there in the Python landscape.

Joeri• 11 years ago

Every time I'm struck by how slow the big PHP frameworks are compared to raw PHP. Either nobody cares about making those tests perform better, or something is very wrong in the architecture of those frameworks.

I expect the cause is that too much code is being loaded for every request. PHP tears down and rebuilds the world for every request, and the popular frameworks load a lot of code and instantiate a lot of objects for every request.

panopticon• 11 years ago

PHP frameworks generally rely on heavy amounts of caching at every level (database, bytecode, Varnish, etc) to make up for this.

Why this is the status quo is something I also question.

aikah2• 11 years ago

Either nobody cares about making those tests perform better

Very few do. Just look at Symfony's complexity , the devs managed to shove proxies into the IoC container to "lazy load services" so you could inject them into the controller class constructors without actually instantiating them until you call a route handler's controller instance method... Using complex class hierarchies in PHP has a big cost, while "raw PHP" barely executes the underlying C code. To be fair people using heavy frameworks come for the battery included first, not really for the speed of the router. And since devs want more batteries, it's unlikely it will get faster.

makeitsuckless• 11 years ago

Because we know these raw tests are bullshit since it's almost trivial to set up the frameworks and the infrastructure to make it perform perfectly fine.

And because of the share-nothing architecture it scales like a mofo before we have to start jumping through hoops (and 99% of us never reach that scale).

The lack of raw speed is a minor inconvenience that's easily fixed in practice.

Also, I doubt big PHP frameworks are the only ones with a disadvantage that only shows up in these kind of benchmarks. It's like testing cars solely for straight line speed.

merb• 11 years ago

Keep in mind most of these benchmarks won't happen in production. Especially not the netty and lwan ones.

MCRed• 11 years ago

Reading thru these tests they are measuring database performance as much as framework performance.

They are also single node which is great if you're entire system is only going to ever need one machine's worth of capacity (Eg: vertical scaling)

hamiltont3• 11 years ago

I've been working with this project for a while, here's some unorganized thoughts:

   1) Statistics
   2) Running Overhead
   3) Travis-CI
   4) Memory/Bandwidth/Other info
   5) Windows
   6) IRC
   7) Ease of Contributing

1) Currently, the TFB results are not statistically sound in any sense - for each round you're looking at one data point. EC2 has higher variability in performance, so that one data point is worth less than the bare metal data point. Re-running this round, I would expect to see at least 5 to 10% difference for each framework permutation. See point (2) to understand why we're not yet running 30 iterations and averaging (or something similar)

2) Running a benchmark "round" takes >24 hours, and still (sadly) a nontrivial amount of manpower. It's currently really tough to do lots of previews before an official round, and therefore tough to let framework contributors "optimize" their frameworks iteratively. I'm working on continuous benchmarking over at https://github.com/hamiltont/webjuice - it's a bit early for PRs, but open an issue if you want to chat

3) As you can imagine, our resource usage on Travis-CI is much higher than other open source projects. They have been nothing but amazing, and even reached out to chat about mutual solutions to potentially reduce our usage. Really great team

4) We do record a lot of this using the dstat tool. dstat outputs a huge amount of data, and no one has sent in a PR to help us aggregate that data into something easy to visualize. If you want this info, it's available in the results github in raw form.

5) Sadly windows support is struggling at the minute. We need something setup like Travis-CI but for our Windows system. CUrrently windows PRs have to be manually tested, and few of the contributors have either a) time to do it manually in a responsive manner or b) windows setups (a few do, but many of us dont). Any takers to help set something up? FYI, we have put a ton of work into keeping Mono support just so we can at least test that changes to C# tests at least run and pass verification, but naturally that isn't as nice as really having native windows support

6) join us on freenode at #techempower-fwbm - it's really fun meeting the brilliant people behind the frameworks

7) If I had to pick one big thing that's happened in between R9 and R10, it would be the drastically reduced barrier to entry. Running these benchmarks requires configuring three computers, which is much harder than something like pip install. Adding vagrant support that can setup a development environment in one command, or deploy to a benchmarking-ready AWS EC2 environment, has really reduced the barrier to getting involved. Adding Travis-CI made it better - it will automatically verify that your changes check out! Adding documentation at https://frameworkbenchmarks.readthedocs.org/en/latest/Projec... made it even easier. Having a stable IRC community is even better! Tons of changes have added to mean that it's now easier than ever for someone to get involved

skrowl• 11 years ago

Yeah, I clicked stats and tried to hide everything but IIS since that's the web server I use. No results. Closed page.

hamiltont3• 11 years ago

I'm open to recommendations for systems similar to Travis-CI but supporting Windows! Having some type of windows CI would really help bring windows uspport up to par

EDIT: Actually, let me just link everyone to github:

Here are the windows compatibility issues - https://github.com/TechEmpower/FrameworkBenchmarks/issues?q=...

Here is the specific issue asking for advice on what CI we should use to support windows: https://github.com/TechEmpower/FrameworkBenchmarks/issues/10...

cagenut2• 11 years ago

Since the "peak" hardware is a dual E5-2660v2 thats 32 threads, so a c3.8xlarge would be a much more comparable instance.

kainsavage7• 11 years ago

We aren't trying to measure each hardware set as apples-to-apples, but rather give the reader an idea of how performance characteristics for a chosen stack are affected by hosting environment. Specifically, we wanted the middle-of-the-road EC2 instances versus the extremely high-end Peak option to illustrate that difference.

cagenut2• 11 years ago

thanks for the response

I've noticed a weird trend where amazon created various slices of instance types a long time ago, and people have mentally gotten used to using larger ones far slower than moores law adds cores. So people will refer to something with 2 cores as "middle-of-the-road" and 32 as "extremely high-end" when in my brain thats "a cell phone" and "a 2 year old server".

dilatedmind• 11 years ago

what are the benefits of using this benchmark over using ab?

hamiltont3• 11 years ago

The main benefit is this allows rough comparison to a ton of other frameworks. Just running ab against your one server setup gives you one RPS/latency result on one hardware setup - that's good to know as an absolute metric, but tells you very little about your performance relative to other frameworks.

This project gives you RPS/latency metrics for many frameworks, on a few hardware setups. This enables a rough comparison of "how does my framework perform relative to all these other well-known or established frameworks". Naturally, the comparison is not perfect - there are a ton of reasons that measuring just requests/sec and latency doesn't allow complete comparison between two frameworks. However, once you accept that it is basically impossible to fully compare any two frameworks using just quantitative methods and these numbers should inform your choice of framework (instead of totally control your choice of framework), we can talk about why it's valuable.

Want to run a low-cost server in language X that you happen to love? This project can provide guidance about which frameworks written in language X are performing the best. Want to ensure your service can support 50k requests per second without loosing latency? This project can provide latency numbers for you to examine that let you know which frameworks appear to maintain acceptable latency even under high load.

If you wanted to, you could re-create this project by running ab against 100+ frameworks - that's the cornerstone of what is happening here. Granted, we currently use https://github.com/wg/wrk instead of ab, but the principle is the same - start up framework, run load generation, capture result data. Most of the codebase is dedicated to ensuring that these 100+ frameworks don't interfere with each other, setting up pseudo-production environments with separate server/database/load generation servers, and other concerns that have to be addressed.

Over time, this project has started collect more statistics than just requests/second and latency, which makes it more valuable than just running ab. As more metrics are added and more frameworks are added, this becomes a really valuable project for understanding how frameworks perform relative to one another.

vinceyuan• 11 years ago

Some frameworks which I never know performed very well. But looks like they are not mature. Which framework do you recommend? I used Node.js/Express, Rails and Sinatra but am not satisfied with them. I am learning Go.

Techempower Web Framework Benchmarks Round 10

Related Stories