Revisiting 1M Writes per second

http://techblog.netflix.com/2014/07/revisiting-1-million-writes-per-second.html

by shifte • 12 years ago

119 38 12 years ago

techblog.netflix.com

jbellis• 12 years ago

This is running Cassandra 1.2.x which is over 18 months old.

Here's some performance notes on the latest (2.1rc4): http://www.datastax.com/dev/blog/cassandra-2-1-now-over-50-f...

walls• 12 years ago

Will 2.1 actually be usable? It seems like 2.0 has missed that mark in every single release.

threeseed2• 12 years ago

Are you saying 2.0 isn't usable ?

Because there are many companies (including ours) who would strongly disagree with you about that.

arielweisberg• 12 years ago

Would be great to know the exact data set size (not size in the database). I can't get an order of magnitude sense of what I am looking at without that.

I know I can divine it from the parameters to stress, but I have no idea if the row keys generated by different clients overlap and I don't know the default number of columns nor their size.

I think it's also important in this kind of benchmark to describe the distribution of access especially for a read intensive benchmark. Without that you really don't know what your are looking at. I am a fan of scrambled Zipfian.

jdf• 12 years ago

It seems like this info should be front in center in the test. 1M/s 100kb writes is much more impressive than 1M/s 16 byte writes.

That said, there's a previous benchmark linked to at the top of the post:

http://techblog.netflix.com/2011/11/benchmarking-cassandra-s...

The client is writing 10 columns per row key, row key randomly chosen from 27 million ids, each column has a key and 10 bytes of data. The total on disk size for each write including all overhead is about 400 bytes.

There are 3 replicas, so figure that in as well.

MoOmer• 12 years ago

I love using Cassandra; it's been a dream for analytics. Thank you Netflix et al. for not only moving the project forward, but providing research and commentary like this and others available at http://wiki.apache.org/cassandra/ArticlesAndPresentations

jxf• 12 years ago

Can you talk a little about how you're using it for analytics and why you like it over other similar choices?

rdtsc2• 12 years ago

My other favorite "scalability" study is from WhatsApp:

http://www.youtube.com/watch?v=c12cYAUTXXs

That's 'Billion' with a 'B': Scaling to the Next Level at WhatsApp

(that walk title was create before the acquisition and was mean to imply message count, after the acquisition it got a secondary meaning).

The one thing that is fascinating about it, is how small their team was compared to the volume and complexity of the operation.

dav-• 12 years ago

The thought of even touching systems at such scales terrifies me.

rdtsc2• 12 years ago

Agreed. Someone asked once how to you simulate load like that to test and he answered "we can't" we try to do gradual deployment and rolling upgrades.

turnip19792• 12 years ago

I wonder where the term sidecar originated from and what is the precise definition? Is this something invented by the netflix OSS people or does it predate that?

jedberg2• 12 years ago

It's from here: https://www.google.com/search?q=motorcycle+sidecar&source=ln...

turnip19792• 12 years ago

I was hoping for a more precise software concept. Maybe it is loosey goosey.

dnBldGVy• 12 years ago

So the test was run on a 285 node cluster with 60 clients. It would be nice to know how they arrived at those numbers. Was there some sort of formula used to calculate how large each group should be? How much trial and error was involved.

arrryarr2• 12 years ago

How many companies can possibly afford the cost and management pain of running a 285 node database? Few would have 285 servers of any type? So if this is what it takes to get 1M writes on Cassandra that is some poor ROI. 10K writes/sec is not impressive.

opendais2• 12 years ago

How would you build out at a metrics cluster that needs 1M writes/sec?

285 nodes that are easily automated to create/destroy/monitor doesn't seem like a management pain to me, personally. Just depends if you have the write internal tools built.

arrryarr2• 12 years ago

Ok, so you're super-admin. And about the ROI aspect? What's the cost for achieving the 1M write/sec metric in that mammoth cluster?

Retric3• 12 years ago

I have seen single servers do 250k writes/second. So it really depends on the data and access patterns more than some arbitrary one size fit's all solution.

arthursilva• 12 years ago

Well, it's not unreasonable (the maintenance part) with Cassandra.

threeseed2• 12 years ago

I don't understand your point.

The type of companies who could afford this are the types of companies who need 1M writes/second. Which are few and far between. And yes 10K is not that impressive but 1M is. And with Cassandra you could continue to improve that number just be rolling out more nodes.

rsynnott5• 12 years ago

They probably mean 10K writes/sec/node, which would be correct assuming a replication factor of three. It doesn't sound huge, but if it were sustained, with a key set vastly larger than memory, it's not too bad.

jandrewrogers3• 12 years ago

It is quite cheap to average a million writes per second. I've done it with five servers on AWS, and that was spatially indexing billions of GeoJSON polygons through storage while running queries against the index.

Many companies need far in excess of a million writes per second. Basically, most machine-generated data sources, whether it is personal location data or any other kind of telemetry. Many companies that do not generate that data themselves buy and consume it. I know of companies doing over a billion writes per second.

Cassandra is pretty good for this type of thing among open source software but it is not nearly as efficient as it could be in terms of write throughput. If the storage engine is correctly designed, you should be able to drive 10GbE all the way through storage -- call it 1 GB/sec per node. However, that does mean you can't do things like mmap()-ing files; those interfaces are slow due to poor scheduling by the OS when the throughput is very high.

larsmak• 12 years ago

The replication factor is set to 3, meaning that all data is stored on tree separate nodes - in different availability zones. So in practice the writes per sec is 3x.

JonoBB2• 12 years ago

Couldn't help but notice: $398.70 per hour = $9568.80 per day = ~$3.5m per annum. They obviously get a discount...but still. What kind of discount do guys like this get?

alex_sf• 12 years ago

I'd be surprised if anyone got discounts as deep as Netflix considering their usage.

For what it's worth, 3.5m/yr is about 0.08% their revenue.

JonoBB2• 12 years ago

Well, that puts things in perspective.

rsynnott5• 12 years ago

If they were doing this for real, they'd be using reservations; with one-year reservations, the i2.xlarge bit (the servers) cost $905k/year.

cpayne• 12 years ago

For this type of "report" the vendor will usually chuck in the hosting for free. It would have conditions like "can only be used for this test (not anything else" and "results must be published on your blog" etc. etc.

Microsoft do this all the time - its great publicity. It would be interesting to see if they (Microsoft) have something showing similar results...

Revisiting 1M Writes per second

Related Stories