HN user
Rust programmers have this "holier than you" attitude that is so toxic. It's essentially wokeism for programming. No wonder it originates from San Francisco, from all places.
The language itself features interesting ideas, many of them borrowed (pun intended) from Haskell, so not that new after all. But the community behavior proved consistently abysmal. A real put off.
you did not read the white paper then
This sounds extremely interesting. How do you generate the SDHC dictionary ?
Would it be interesting to compare your results with Zstandard using the same SDHC ? It seems to have a strong dictionary mode.
The formatting is just so bad, let's retry it here :
| Algo | compressed size | compression speed | decompression speed |
| --------- | --------------- | ----------------- | ------------------- |
| brotli -2 | 36223 | 220 MB/s | 670 MB/s |
| zstd -1 | 36655 | 480 MB/ | 1400 MB/s |
| brotli -1 | 38292 | 360 MB/s | 650 MB/s |
| brotli -0 | 41141 | 560 MB/s | 610 MB/s |Of course it is an unfair example because of the static dictionary that brotli uses
It is certainly a favorable ground for Brotli. Brotli claims an advantage in html compression, thanks to its integrated specialized dictionary. The real pb though is the suggested conclusion that these favorable results are broadly applicable everywhere else. That's a terrible suggestion. We need more examples, not just "html files" which happen to be Brotli's best case.
brotli 0.4.0 -11 compresses to 25413 bytes zstd 0.7.1 -22 compresses in 4.01 MB/s to 28363 bytes
Why you don't disclose the compression time of brotli ? Of course it does matter : everyone understand that an algorithm that spend 10x more cpu has the budget to compress more.
brotli 0.4.0 -0 compresses 783 MB/s and decompresses 809 MB/s zstd 0.7.1 -1 compresses 586 MB/s and decompresses 1691 MB/s
Here, you don't disclose the compression ratio of both algorithm, implying they are equal. By such standard, LZ4 is probably the best : it's so much faster ! Of course, they do not compress the same...
I was initially thrilled at your detailed answer, but now, quite frankly, I feel cheated. Grossly so.
This is really disappointing. I was so much vexed that I decided to run the tests myself.
Downloading and using __the same html file__, the same lzbench, same library versions, just a different computer and compiler, here is what it produced :
| Algo | compressed size | compression speed | decompression speed | | --------- | --------------- | ----------------- | ------------------- | | brotli -2 | 36223 | 220 MB/s | 670 MB/s | | zstd -1 | 36655 | 480 MB/ | 1400 MB/s | | brotli -1 | 38292 | 360 MB/s | 650 MB/s | | brotli -0 | 41141 | 560 MB/s | 610 MB/s |
__Conclusion__ : brotli -0 is indeed fast, faster than in my previous tests. It seems to be tuned to reach this objective, but throw away a lot of compression ratio to get there.
Consequently, brotli -0 is not comparable to zstd -1, it takes brotli -2 to produce an equivalent compressed size . By that time though, zstd is much, much faster.
Which is exactly the question I'm trying to get answers to : which algorithm compresses better for a given speed budget ? That's what matters, at least in our datacenter.
I'm not interested in ultra slow mode, but while at it, I wanted to complete the picture with the missing compression speed of brotli - 11. It produced : zstd -22 : 2.95 MB/s brotli -11 : 0.53 MB/s
So that's > 5x difference. It surely helps to reach better compression ratios.
I also wanted an answer to "by how much the dictionary helps ?".
Fortunately, TurboBench can help, thanks to a special mode which turns off dictionary compression. By using it on the very same sample, brotli -11 compressed size increases from 25413 to 26639 bytes. 5% larger, clearly not negligible. Still good, but it cuts the advertised size difference in half.
Anyway, clearly I feel disappointed to have to redo the tests myself, because some inconvenient results were intentionally undisclosed (or not produced). This really undermines my trust in future publications.
That learned me something : trust only benchmark done by yourself. And now, I should probably benchmark even more ...
Brotli's fastest compression is slightly faster than zstd's.
Come on, this is not serious.
Brotli's fastest compression algorithm is still significantly slower than zstd. And more importantly, it compresses _much worse_.
For a 3rd party evaluation, one can try [TurboBench](https://github.com/powturbo/TurboBench) or even [lzbench](https://github.com/inikep/lzbench) which are open-sourced. Squash introduces a wrapper layer with distortions which makes it less reliable, and more complex to use and install, quite a pity given the graphical presentation is very good. I'm interested in speed, and in this area, all benchmarks point in the same direction : for a given speed budget, Zstandard offers better ratio (and decompresses much faster).