Network file transfer with on-the-fly compression

http://www.codelathe.com/blog/index.php/2009/04/09/network-file-transfer-with-on-the-fly-compression/

by codemechanic • 17 years ago

9 6 17 years ago

codelathe.com

wmf2• 17 years ago

I'd like to see a comparison between this and scp with compression.

KirinDave• 17 years ago

That's the first thing I thought of too. The -C flag is there specifically for this very scenario.

Also, would "j" have worked better than "z" in the tar line? bzip2 is much slower than gzip, but does a better job.

wmf2• 17 years ago

Really we should have adaptive compression so that the CPU and network usage is balanced. Too bad it may require "rampant layering violation" to implement.

doosra• 17 years ago

scp -C does the same thing. Both methods use gzip for compression.

rarrrrrr• 17 years ago

But the tar method will generally be 5-15x faster for large collections of small files.

Handing it with tar makes operations on both ends asynchronous. scp (or rsync, etc.) of multiple files does file opens on each end in lockstep, resulting in a lot of idle wait time.

mwcremer• 17 years ago

Note that this is not likely to help with media files (mp3, mpeg, jpeg).

Network file transfer with on-the-fly compression

Related Stories