Why you should always use gzip instead of deflate compression

http://stackoverflow.com/questions/388595/why-use-deflate-instead-of-gzip-for-text-files-served-by-apache/9856879#9856879

by tbassetto • 14 years ago

101 17 14 years ago

stackoverflow.com

DHowett• 14 years ago

The "CRC-32 is slow" sentiment feels like a bit of a straw-man argument against gzip. With today's computing power, the difference is nigh-unto negligible: checksumming is dwarfed by the actual decompression, or maybe even the network overhead/latency.

Gzip is, doubtless, better for the reasons laid out in the article, but why aren't we moving forward? Do any browsers support LZMA or bzip2? Would they be at all worth the effort? I assume not in the case of HTML/resources, but maybe in raw non-compressed binary streams.

seabee• 14 years ago

How many raw non-compressed binary streams do you need to deliver? Most assets a Web page requires are textual (CSS/JS) or already compressed: images, video, Flash content...

rmc• 14 years ago

The largest assets will be already compressed (eg images and/or video). Http compression is only really benefiting for textual content. Gzip gives you a massive space saving over non compressed for textual data, but after that you're in diminishing returns. Although bzip2 is better than gzip, its not a lot better.

burstlag• 14 years ago

bzip2 compresses significantly better than gzip (I typically see a delta of between 10% and 30% in filesize) but it comes at a significant cost: compressing/decompressing the same file generally takes many times longer. xz compresses even better but takes even longer.

Also, some web servers (or at least Apache) will serve gzipped static content automagically if it finds a matching gzipped file. For example, if you have /var/www/html/foobar.css.gz, it will serve the contents of that file directly when the client requests http://example.com/foobar.css. So if you're running a stock Apache configuration, there's no reason not to just run gzip on all of your static assets and get lower bandwidth bills right away.

keeperofdakeys2• 14 years ago

You need to consider that once you add something, you need to support it for the indefinite future. So we need to give serious consideration to things that are added. Does LZMA provide a sufficiently large compression ratio over gzip? Does the compression and decompression time introduce a lot of latency, especially on mobile devices? Does that increased latency mean plain text is faster?

defen• 14 years ago

That's why we have the Accept-Encoding header. Neither clients nor servers would need to support alternate compression schemes, either now or in the future. But if they both did, maybe there could be some benefit.

kzrdude• 14 years ago

It's not as simple as "forward" (and "backward" implied), the landscape has many more dimensions, cpu and memory and code size overhead etc.

justincormack• 14 years ago

Adler32 is very broken, but it doesnt really matter as you can use other checksums on top. http://www.leviathansecurity.com/blog/archives/16-Analysis-o...

Browsers should just stop supporting deflate, dont say they can in the request, and servers should drop it too. Ideally gzip support should be required not negotiable.

dfox• 14 years ago

On the other hand, quality of Adler32 is hardly an valid argument here, as HTTP without compression does not use any checksums whatsoever. The fact that it's not consistently implemented is.

By the way, Adler32 I don't think that it's right to call Adler32 broken, as there are much worse algorithms in common use (TCP checksum, for example), and limitations of Adler32 are widely known and are not too relevant for this application.

calloc• 14 years ago

Luckily once we all switch to IPv6 we will no longer have TCP checksums :-)

vasi• 14 years ago

Good post, but it would be great if "for HTTP" was added to the title. This has nothing to do with other use cases.

sciurus• 14 years ago

It's interesting to learn about the web browser and server compatibility problems.

In the general case, I recommend one of two compression algorithms:

* lzo, for when compression or decompression speed matters most * lzma, for when file size matters most

premchai21• 14 years ago

LZO, though, has the potential problem or benefit (depending on your situation) of not being readily incorporable into proprietary software without navigating Oberhumer's commercial-license maze, unless there's some form of clean-room reimplementation around that I don't know about. (“LZO Professional” at http://www.oberhumer.com/products/lzo-professional/ claims to be available only as a “binary-only evaluation library” under NDA; I don't know whether “evaluation” in this context would obstruct use in an actual product.) The zlib license makes it much easier to deploy. liblzma seems to be in the public domain.

wmf• 14 years ago

I think LZO has been obsoleted by Snappy anyway.

loupgaroublond• 14 years ago

lzma takes long enough to compress suitably large files that it's not always ideal for operations. While CPU time is generally cheaper than storage space, archiving systems don't always have the most powerful chips in them and tend to favor the cheapest storage possible.

Why you should always use gzip instead of deflate compression

Related Stories