Scrap the SCP. How to copy data fast using pigz and nc

http://intermediatesql.com/linux/scrap-the-scp-how-to-copy-data-fast-using-pigz-and-nc/

by pmoriarty • 12 years ago

45 33 12 years ago

gpsarakis3• 12 years ago

Thumbs up for pv which shows the progress bar. Using parallel compression may have serious impact on CPU resources, so it needs to be balanced.

Have you also done any testing with ssh + gzip?

Also, as you note at the end, the security concerns are not trivial.

moe• 12 years ago

Looks like he reinvented scp -C?

gpsarakis3• 12 years ago

pigz does parallel compression, taking advantage of multiple cores. I am not really sure if you can achieve this with scp.

reidrac• 12 years ago

You can use pigz with SSH as you can pipe commands over SSH (google it). If nc is faster than scp, I guess encryption is a factor, but then they're not comparable solutions to the same problem.

otterley• 12 years ago

No -- compression isn't the bottleneck. ssh's tiny TCP buffer size is the issue.

huskyr• 12 years ago

I wonder how this would compare to rsync using the '-z' (compress) option.

inyourtenement2• 12 years ago

Speaking generally, rsync is faster than straight copying if the files partially exist in the destination first. If they don't exist, or they've changed completely, rsync is slower due to the checksumming on both sides.

buster• 12 years ago

There is no checksumming if the file doesnt exist..

drinchev2• 12 years ago

I don't get it.

    nc -l 8888

Doesn't make any encryption, right?

If the answer is yes, then why they are compared here? Anyone on public network might sniff what you are actually sending / receiving.

shernshiou• 12 years ago

yeh the title is misleading. They should compare with ftp.

teh_klev• 12 years ago

At the end:

"tar/pigz/nc transfer is not secure

Finally, there is one advantage that scp still holds: its transfer is secure while pigz/nc transfers data in clear text. So, if you are using unsecured networks, this option is probably not for you."

13throwaway• 12 years ago

You could try piping it through ssh, I don't know how that would effect the speed though.

tar -cf - /u02/databases/mydb/data_file-1.dbf | pigz | ssh user@destination "pigz -d | tar xf - -C /"

drinchev2• 12 years ago

Yeah, but it will need executing commands on both hosts. SSH works on top of SCP and only needs running sshd on the remote host and access to that machine.

All those solutions ( from the OP ) are faster, but you will need some setup, before you could use it from one host remotely and in the end you might end up using good old SMB or NFS which will give you speed ( http://www.linuxquestions.org/questions/linux-networking-3/s... )

bkeroack• 12 years ago

You can pipe through GPG with the -c (--symmetric) flag. Using a reasonably efficient cipher (AES256) has negligible impact on speed and minor CPU usage. I do this with automated DB backups.

nicois• 12 years ago

xz supports multiple threads, if CPU is no object. If you want to stream the results why not use ssh's -F option to create a secure tunnel, particularly if the required bandwidth is not high.

krzyk• 12 years ago

Which version of xz started supporting multiple threads?

My version latest from Debian testing (yes I know it is not cuting edge) says in man page: Multithreaded compression and decompression are not implemented yet, so this option has no effect for now.

zobzu4• 12 years ago

they have had it on the git repo for quite some time now (months i think), but distros generally dont have it yet.

inyourtenement2• 12 years ago

One good alternative I've used is socketpipe[0]. It handles logging into the remote server, setting up a listener, and then piping the data over.

But, that tunnel still isn't encrypted. So, either encrypt your data first, or setup some vpn tunnel. The inefficiency of scp isn't just due to the encryption, there's overhead in the ssh protocol.

0 http://freecode.com/projects/socketpipe

devonkim• 12 years ago

A VPN setup seems like overkill. You can run stunnel and run a daemon to loop back through the stunnel configured connection. It's not quite as easy to use offhand like scp, but it's certainly not as awkward to try to use as a random series of OpenSSL commands either and it'll be easier to understand what you tried to do later on the production machine.

throwed_me_away• 12 years ago

Please get in contact with me. I don't know how to message you privately, here, if even possible. Do you have a publicly-known email / PGP?

spydum• 12 years ago

Surprised to see NC slower than SSH on network throughput. All of the authors benefits came from compressing the payload first.. Curious if enabling compression at the highest levels for SSH would have worked, or as others suggested, piping pigs over the SSH connection itself.

aktau• 12 years ago

I used pigz for compressing and then rsync for transferring my (Postgre)SQL logs. The reason being that pigz has a built-in "rsyncable" mode, which allows rsync to avoid sending the whole log over always. The multicore compressing is just icing on the cake. gzip has a patch for an "rsyncable" mode, but it's no longer properly included in the distro's I use (debian).

At some point, the only way to copy data faster is to not copy it at all.

I wrote a blog post about it here: http://www.aktau.be/2014/10/23/pg-dump-and-pigz-easy-rsyncab...

adekok• 12 years ago

The problem is that SSH uses a terrible buffering algorithm. This fixes it:

http://www.psc.edu/index.php/hpn-ssh

It's able to consistently fill 1G ethernet links.

pmoriarty• 12 years ago

HPN looks interesting, but the objections to it here[1] give me pause.

bbcp,[2] which was mentioned in the original article, also looks promising. Particularly because it actually uses ssh, unlike pigz and nc.

[1] https://news.ycombinator.com/item?id=4710061

[2] http://www.slac.stanford.edu/~abh/bbcp/

dmm• 12 years ago

a terrible buffering algorithm

The main difference now it that hpn tweaks the tcp window to function better on high-latency links.

It's able to consistently fill 1G ethernet links.

Between two stock Debian sid laptops:

    dd if=/dev/zero bs=1048576 count=4096 | ssh <otherhost> dd of=/dev/null
    8388608+0 records in
    8388608+0 records out
    4294967296 bytes (4.3 GB) copied, 37.0848 s, 116 MB/s

That's 90% of the theoretical bandwith, right?

If you're really transferring huge file files across fat WAN links, why not use GridFTP like the supercomputing centers use? They transfer TB daily.

I wouldn't mess with patching openssh, especially with patches for old versions of openssh.

zobzu4• 12 years ago

xz + scp will still be faster than this. its not about the speed at which scp goes, is that the author doesn't compression before sending over scp, but does when he sends over nc and what not...

heck you can just use rsync over SSH for simplicity.

wyldfire2• 12 years ago

xz + scp will still be faster than this

That's not the case if the data you're sending is already compressed. But you're right that it's not a fair comparison. scp/ssh also has integrated compression with "-C".

zobzu4• 12 years ago

gzip: tar -cz /home/me/source/directory | ssh target tar -xz --directory /home/you/target/directory

xz: tar -cJ /home/me/source/directory | ssh target tar -xJ --directory /home/you/target/directory

ssh -C won't be as good. in particular with a recent xz that's threaded it will probably be faster than OP's with a good CPU

lxfontes• 12 years ago

Combine this with docker export / import and you got yourself an easy way to copy images between servers :)

If you still have the setup up and running, throw this in the mix:

  rsync -a -e 'ssh -c arcfour' source dest:destpath

and

  rsync -az -e 'ssh -c arcfour' source dest:destpath

still encrypted! The latter enables compression, but I've found it slower when dealing with several small files (ex: source code files)

zobzu4• 12 years ago

arcfour is arguably close enough to not encrypting these days ;)

wyldfire2• 12 years ago

scp's beauty is in its ubiquity. That host over there that I want to copy to? I need no extra configuration there because sshd is already running.

I agree that scp is dog slow and it can be pretty frustrating. As others have mentioned, cranking down the cipher provides a modest improvement. Ad-hoc netcat based solutions like this one are the way to go if you need throughput and not security.

kbar13• 12 years ago

Maybe you can set up stunnel between the two hosts for SSL goodness, point netcat to the stunnel port, and call it a day.

Scrap the SCP. How to copy data fast using pigz and nc

Related Stories