Transparent compression at IO speed for object storage

https://blog.min.io/transparent-data-compression/

by jtsymonds • 4 years ago

28 8 4 years ago

blog.min.io

The analysis should also cover zstd...

Zstd typically outperforms snappy, and has the benefit of a wide configurable window of compression ratio Vs speed - and that's important if you want to use a dynamic algorithm to select the compression ratio to maximize throughput.

OnlyMortal2• 4 years ago

Based on personal experience on enterprise storage solutions, LZO is the best generic compressor. Yeah, LZO.

rincebrain• 4 years ago

By which metrics?

Usually, when I see people claim things that are mutually inconsistent about compression, it turns out they have very different tradeoffs they want to do for decompression speed {avg,worst}/compression speed {avg,worst}/compression ratio.

londons_explore3• 4 years ago

This is a good chart to have in mind:

https://community.centminmod.com/data/attachment-files/2019/...

klauspost2• 4 years ago

(article author here)

Zstandard is good, no doubt. I didn't include compressor with entropy compression, except gzip which I added as a reference point for the ratios. After all I know it intimately after porting it to Go.

In overall terms, it does have an unavoidable bigger cost both for compression and decompression. So to have the smallest possible impact we have chosen not to go that direction (yet).

We might add it as a lifecycle option, where it compresses to zstd when the object reaches a certain age.

Furthermore the initial implementation was Snappy, so switching to S2 could be done seamless without requiring different decompressors and keep the implementation simple.

After all, the most effective way would be for the uploaded data to be compressed. The on-server compression is to catch the cases and compress them "reasonably" without taking too many resources.

Transparent compression at IO speed for object storage

Related Stories