pfalls

We use the cluster module to handle multiple workers (https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...), so far we've received feedback that this is a valid approach.

We use the cluster module to spin up multiple instances. We cap it at one instance per core.

The latest test that we have, dubbed "Fortunes", does server-side templating, so we (or a generous community member) could add a JSF implementation for that test.

Here is the issue for just that:

https://github.com/TechEmpower/FrameworkBenchmarks/issues/11...

If you're interested, we'd love to have some help getting this accomplished.

We would like to have a .NET test running on mono. We were hoping to get a pull request for round 4, but unfortunately we have yet to receive one.

Thanks for the kind words. We're very interested in adding additional tests, this round even includes a new test dubbed "Fortunes" which does in fact do server-side templating. We have an open github issue[1] asking for the community's input for just this sort of thing, and we'd love to have your feedback included.

[1] https://github.com/TechEmpower/FrameworkBenchmarks/issues/13...

I want to give a quick thank you to everyone who has contributed thus far. Being able to show frameworks that span this many different languages and platforms is really an amazing achievement that we only we able to accomplish with the help of the community. For the frameworks that are still missing, we're not done yet, continue submitting pull requests and we'll get them in.

Each test can be modified in isolation, if both tests see an improvement in that setup, we would like to include the changes for both though.

Correct, because we chose MySQL as the database backend, both Django and Flask are punished for not having a connection pool. We've received requests to do the test using Postgres which does have a connection pool, and we hope to eventually get to that.

There will most certainly be a Round 3, and 4 and on and on as long as we continue to have feedback from the community. I can guarantee that we'll have a Lua test in there, but if we receive a pull request, we'll include it.

We're very curious about PyPy as well, and have it on our list https://github.com/TechEmpower/FrameworkBenchmarks/issues/15

Yes, Github issues is a good place to ask for guidance.

We're looking into moving to unicorn as the community has suggested. If anyone is interested in setting up Puma or Goliath, we'd be interested in testing those out as well.

"I like these comparisons. It's unlikely that anyone is going to pick a framework for their project based on just this alone."

This is exactly right, performance is one of many items to consider when building your own website, and while we're aiming to have a very comprehensive and useful set of performance data, we would agree that other factors need to be taken into account.

The short answer is that at some point, we kept adding more and more tests on our own, which kept the blog from going out, and we finally decided to have a cutoff date and let the community continue adding tests that they were interested in. I think adding a database test to Netty (as well as Go for that matter) would be a great addition.

Great suggestions, we've thought about capturing other system data as well, but memory seems like a great choice to start with.

We've had a lot of requests Mono, and are actually very interested in trying this out. We've been hesitant to move forward only because we're unsure how stable the C# community believes Mono to be. We hope to have that test at some point though, but the fastest way to get a test included is to issue a pull request. So if anyone is interested in trying this out, let us know and we can help guide the process.

We have one pull request for Onion, which is c based. We're working with the author to work through some issues, but we're also excited to see how it will stack up.

Since we moved to a different benchmarking tool for round 2, we had to re-run all the tests, which is a time consuming process, especially on the EC2 hardware. Now that we've gotten that out of the way, we have an easier time running and processing results for individual tests. So stay tuned!

The only framework that uses MongoDB exclusively is the vert.x test, the nodejs tests have tests for both MySQL and MongoDB

Thanks for the kind words. Glad to hear that you've been introduced to some new frameworks, we feel the same way, and we're excited to see what else the community has up it's sleeve.

Thanks for the suggestion, we're still in the early stages of getting the latency information incorporated, and having the ability to sort by the various metrics makes a lot of sense.

Go 1.1 is very high on our list, we'd love to get that into the tests as soon as we can. Of course, anyone in the Go community is welcome to issue a pull request that updated the version to 1.1.

Edit: Just saw this pull request come in, so expect some Go 1.1 love in the next round. https://github.com/TechEmpower/FrameworkBenchmarks/pull/64

Thanks so much for the kind words. We're obviously having a great time working on this, and we too think there's a lot of value to this for the community at large.

We used Phusion Passenger, although we have plans to add additional servers (such as Unicorn). We tried to spend time with various server choices for all platforms, and for ruby, in our short test, Passenger won out against the others.

Our understanding is that when running Passenger, simply passing '-e production' to the command line is sufficient to run in production, but if that's incorrect, we'll gladly update the test.

for Vert.x, we specified the number of verticals in App.groovy, rather than on the command line, which we think is a valid way to specify it.

One of the next steps we'd like to take is to have a test that does cover a more typical web request, and is less database heavy than our 20 query test, just like you describe. Ultimately, we felt that these tests were a sufficient starting point.

We wanted to use an ORM in all cases, it was only recently that we started working on native mysql access. We hope to add native mongodb results as well.

This is great to know, we were hesitant to use non-stable versions (although we were forced to in certain cases), but knowing that it's what is common practice for production environments would change our minds.

EC2 was a hard platform to test on, only because our i7 hardware would give us results fairly instantaneously, but we became impatient when we had to wait upwards of 10x as long for the data on the EC2 large instances.

We're actually very interested in how the large/newer instances perform.