Reducing Cassandra p99 latency by fixing OS page cache thrashing

https://blog.speedyio.com/do-you-care-about-p99-latency

by irondhoti • 6 months ago

15 4 6 months ago

blog.speedyio.com

I'm wondering if compaction can be isolated as a separate processing task and it's own memory budget that separates it's processing from the memory pages utilized by request serving

Because compaction is about generating a new set of better optimized stables and then discarding the old

Cassandra is was full of all these tasks limped into the same problematic single jfm heap, even though the data could be partitioned into hundreds of jvms with smaller /easier to to gc heaps

Iirc scylla utilizes this fact with cpu pinning across cpus and other tricks that the older Cassandra could not.

irondhoti2• 6 months ago

yea that's why a lot of enterprises have moved to scylladb from cassandra since it is written in cpp and doesnt have the performance bottlenecks jvm brings.

invertedohm• 6 months ago

Kernel 4.18 is _ancient_, weird to test on that. Also I wonder if the whole performance increase can't just be achieved by lowering the readahead size a bit. I see in their notes they have a decent SSD but set the blockdev readahead to 64. Depending on your workload it's much more performant to lower that to 16. As mentioned in TFA, fast storage is the most important factor here, and in my experience combining that with a smaller readahead pretty much fixes any read amplification issues you get with the larger readahead.

irondhoti2• 6 months ago

Take a look at this https://dl.acm.org/doi/pdf/10.1145/3341301.3359640 ; shows how newer kernels are slower. I have seen supercomputers like `summit` use linux 2.xx .

There are two performance problems presented here. One is the fact that the OS waits till a static memory bank water-level before triggering a cleanup.

The second is the one you pointed out; here again one thing to consider is cassandra's compression chunk size `chunk_length_in_kb`. readahead value less than chunk size makes cassandra slow in general. take a look at this https://thelastpickle.com/blog/2018/08/08/compression_perfor... for more info on it.

Reducing Cassandra p99 latency by fixing OS page cache thrashing

Related Stories