Zlib-ng: a performance-oriented fork of zlib

by profquail • 11 years ago

100 45 11 years ago

r1ch• 11 years ago

One thing to be aware of, these replacement zlib libraries are not always "drop in" safe as some software such as nginx pre-allocate the zlib state buffers by hard coding in the original zlib structure sizes (the original zlib makes guarantees about memory usage that technically makes this behavior somewhat OK). Depending on how well the software is designed, the forked zlib versions may result in buffer overflows or other crashing. (nginx is fine - it spams your log file with errors and re-allocates).

legulere• 11 years ago

This makes me wonder what a good strategy would be for deprecating old architectures. Remove support for them in new versions and backport bugfixes to the last version supporting the old architectures indefinitely?

TheCondor• 11 years ago

It's a more difficult issue with something like OpenSSL as new features are added. Zlib is done though, there is zero interest in sdding deflate64 or a different algorithm, they could just put a version on ice, keep it around for distribution and make a more clean 2.0

gjm112• 11 years ago

It seems like this would be more compelling if accompanied by some benchmarks showing how much better the performance is than the original zlib's.

moreati3• 11 years ago

A very rough and ready, impromptu benchmark.

zlib latest took 26 seconds to compress 648 MB of text, zlib-ng latest took 16 seconds.

    alex@martha:~/src/zlib$ export TESTFILE=../enwiki-20141106-pages-articles-multistream-index.txt
    alex@martha:~/src/zlib$ du -h $TESTFILE
    648M	/home/alex/src/enwiki-20141106-pages-articles-multistream-index.txt
    alex@martha:~/src/zlib$ LD_LIBRARY_PATH=./ time ./minigzip <$TESTFILE >/dev/null
    23.75user 0.17system 0:26.05elapsed 91%CPU (0avgtext+0avgdata 1864maxresident)k
    56472inputs+0outputs (0major+144minor)pagefaults 0swaps
    alex@martha:~/src/zlib$ LD_LIBRARY_PATH=./ time ./minigzip <$TESTFILE >/dev/null
    24.29user 0.17system 0:26.80elapsed 91%CPU (0avgtext+0avgdata 1772maxresident)k
    0inputs+0outputs (0major+141minor)pagefaults 0swaps
    alex@martha:~/src/zlib$ cd ../zlib-ng
    alex@martha:~/src/zlib-ng$ LD_LIBRARY_PATH=./ time ./minigzip <$TESTFILE >/dev/null
    14.66user 0.11system 0:16.16elapsed 91%CPU (0avgtext+0avgdata 1604maxresident)k
    256inputs+0outputs (0major+140minor)pagefaults 0swaps
    alex@martha:~/src/zlib-ng$ LD_LIBRARY_PATH=./ time ./minigzip <$TESTFILE >/dev/null
    14.64user 0.12system 0:16.09elapsed 91%CPU (0avgtext+0avgdata 1840maxresident)k
    0inputs+0outputs (0major+144minor)pagefaults 0swaps

The machine is a MBP 11,2 with an i7-4750HQ CPU @ 2.00GHz, running Ubuntu 15.04 x64.

nly• 11 years ago

What about decompression? Are we already I/O bound? Compression is irrelevant in many cases, where you can precompress static content.

jsnell2• 11 years ago

Edit:

Results with zlib-ng and a newer version of the Cloudflare changes:

https://www.snellman.net/blog/archive/2015-06-05-updated-zli...

Original:

Here are some benchmarks of the Intel and Cloudflare forks that this project is based on:

https://www.snellman.net/blog/archive/2014-08-04-comparison-...

The speedups are not insignificant. (I'll see about updating the post to include results for zlib-ng).

Dead23• 11 years ago

As I mentioned in another comment, the minigzip used is very suboptimal due to reading and decompressing only 1 byte per inflate() call. Please retest with --zlib-compat or the fix that was merged today.

Dead23• 11 years ago

Btw, I always wondered.. Why didn't you test compression level 6, it is the default level, and used by for example stock apache/nginx settings.

Also #zlib-ng on freenode if you or anyone else want to discuss.

erikb2• 11 years ago

How to pronounce the "-ng"? "zetlibng" like "bang" without the "a" is kind of hard to produce for my mouth.

IshKebab• 11 years ago

zed-lib-enn-gee I presume.

TimWolla• 11 years ago

You spell it out. It's an acronym for “next generation”.

erikb2• 11 years ago

Ah that's what it stands for. Alright.

moreati3• 11 years ago

"zee lib en jee" was my first instinct

microtonal3• 11 years ago

The zlib code has to make numerous workarounds for old compilers that do not understand ANSI-C or to accommodate systems with limitations such as operating in a 16-bit environment.

This is true for many older open source libraries. I wonder:

- Are old platforms still tested as time passes? Is it really worth it maintaining extra cruft for old systems even if it is unknown whether it still works? Are there projects running CI for OS/2 Warp, MS-DOS, or A/UX?

- How many users of old or obscure platforms are there still around? Are they even updating such libraries?

- What platforms do we need to get rid of to get rid of autoconf/automake/configure and use e.g. CMake instead?

kukkukb• 11 years ago

Wasn't this one of main reasons that thenNeoVim project was started?

stefantalpalaru• 11 years ago

What platforms do we need to get rid of to get rid of autoconf/automake/configure and use e.g. CMake instead?

Why would you want that? Autotools is the devil we know. CMake has its own problems and from a statistical sample of one project implementing both, I came to the conclusion that CMake is more frustrating than Autotools when you need to do something unusual.

tanderson922• 11 years ago

Exactly. autotools is exceptionally well designed for dealing with more exotic use cases like cross compiling. When designing a cross-compiling linux distribution(http://exherbo.org/docs/multiarch.txt) we discovered that autotools is by far the easiest to deal with when it comes to arbitrary host & build targets, compilers, etc. I would be even bolder and say that:

Those who do not understand autotools are doomed to reinvent it, poorly.

microtonal3• 11 years ago

I came to the conclusion that CMake is more frustrating than Autotools when you need to do something unusual.

Unusual in terms of...? If I have learnt anything from Go, Java (per Maven), etc. it's that the far majority of project don't need to do something unusual.

Conversely, if you give people the room to do unusual things, they will.

astrodust• 11 years ago

CMake. Ugh. It solves some problems and creates many, many more in the process.

There's a certain simplicity from the perspective of the user that configure && make && make install works most of the time.

CMake requires a lot of effort to get running by comparison.

uxcn3• 11 years ago

A lot even limit their codebases to C89, which I don't understand. Are there really that many platforms/compilers that don't effectively support C99 or greater? Is sixteen years not long enough for a language feature set to stabilize?

Maybe there are tradeoffs or platforms I'm just not aware of, but I at least tend to use C99 as the LCD.

realityking• 11 years ago

That's mostly because of Microsoft's horribly outdated C-compiler

bodyfour2• 11 years ago

I'll be watching this with some interest. A couple years ago I tried using some of the x86_64 assembly versions of inflate_fast() and longest_match() but found them to be subtly buggy -- they'd both sometimes do reads outside of the allocated memory and could crash if that happened to land at the end of a mmap'ed segment. I emailed all of the relevant authors, but nobody seemed to be maintaining that stuff any more. Honestly, just having a zlib with a bug tracker is a huge improvement.

I am sad to see that a lot of the zlib-ng seem to be around code style updates, which will make it hard for things to travel back and forth between the codebases. I guess one branch or another will eventually have to "win", which is sad.

Also, is zlib-ng going to commit to keeping the same zlib license? Some of the zlib improvements floating around on the net have snuck in more restrictive licensing. It would be fantastic if zlib-ng could hold the line on license creep.

dalke• 11 years ago

According to the linked page, "Just remember that any code you submit must be your own and it must be zlib licensed."

tkhsu• 11 years ago

Intel optimized zlib: http://www.intel.com/content/www/us/en/intelligent-systems/i...

Github:https://github.com/jtkukunas/zlib

craftkiller• 11 years ago

For decompression miniz is already faster than zlib and with a more permissive license https://code.google.com/p/miniz/

And my post where I put times and code: http://fizz.buzz/posts/anything-you-zcat-i-zcat-faster-under...

ChickeNES• 11 years ago

Though it looks like there's been no updates since late 2013. I wonder if it makes sense for someone to step in and import it to Github

bcseeati• 11 years ago

miniz is already on GitHub: http://richg42.blogspot.com/2015/05/the-great-github-migrati...

sliverstorm• 11 years ago

If you want more performance out of zlib, step one is recompile.

I've seen flabbergasting speedup out of basic CPU-bound utilities simply by recompiling the source code for a fifteen-year-old binary using a modern compiler.

Which also brings up the point that if you want to benchmark zlib-ng, compile zlib with the same compiler.

Zlib-ng: a performance-oriented fork of zlib

Related Stories