Comparing dd and cp when writing a file to a thumb drive

https://distrowatch.com/weekly-mobile.php?issue=20240122#qa

by vector_spaces • 2 years ago

58 40 2 years ago

distrowatch.com

bravetraveler2• 2 years ago

One of my favorite party tricks is using shell redirects to do this. Using coreutils ('cp') is a nice approach

I'm all for anything not 'dd' or cargo-culted like balenaEtcher

This all depends on a certain format of ISO that I can't recall

There's no magic. Checksum the drive and the file after

mort963• 2 years ago

Using shell redirects sucks though, you need to run the shell as root.

Using cp seems overkill? It's really really designed to copy files between file systems and has a lot of logic to handle different cases and different optimisations which don't matter when writing to block devices.

As you say, there's no magic. I want a program which just calls 'open' on a path I give it and then uses 'write' to write my data to it. cp does so, so much more related to file systems, shell redirects aren't a separate program I can run with sudo, but I can trust dd to do the job. And it has a (kinds bad) progress monitor to boot.

The default 512 byte block size is unfortunate though.

GuB-42• 2 years ago

As far as I know, you can't copy using shell redirects. At least not with a simple POSIX shell. Redirection only open file descriptors, but you need a command to actually make the copy from one to the other.

It can be "cat", which is often cargo culted in it own right. Or even "dd", which can work with stdin/out.

I usually prefer not to use shell redirects if there is a command that take filenames. That's because it gives more control to the app. The app knows how the file will be used, the shell doesn't, so it can open it the most appropriate way, avoid overwriting the output file if something goes wrong, output better error messages, etc... Now, if you don't trust the app (for example if you fear it will modify your input file), then shell redirects may be the better option.

Brian_K_White4• 2 years ago

You can absolutely read and write by just redirects. No cat or dd or anything needed.

bravetraveler2• 2 years ago

This works in ZSH but not BASH... I have a feeling you just helped me see why :D

js2• 2 years ago

You're right. You'd have to use the `read` built-in and `echo` or `printf` in a while loop combined with redirection, but the POSIX-specified `read` built-in is intended for text and isn't going work correctly with binary files.

With bash you could maybe use the non-standard `read -N` option but I'm not sure about null bytes.

Brian_K_White4• 2 years ago

bash builtins only, not even a subshell, binary-complete

while LANG=C IFS= read -d '' -r -n 1 x ;do printf '%c' "$x" ;done <dat1 >dat2

At least ksh and zsh have equivalent functionality though the details may differ.

Not dash or ancient sh.

I don't pretend it's practical, just possible.

I actually do have a neck beard this month.

tadfisher• 2 years ago

cp queries the preferred block size of the destination file in 'struct stat', and has specific tweaks for certain filesystems. As far as I can tell, dd does not do this as it calls through to 'write' directly.

In any case, the tests in ioblksize.h indicate that bs=4M is far too large and may perform worse than the default for cp/cat (128KiB). There is a script there that should clear things up for more modern systems.

The point about fdatasync is superfluous as you can run 'sync' yourself, or unmount the filesystem.

NoZebra120vClip3• 2 years ago

This example was performed on the block-special device, so there is no filesystem.

dd uses a default block size of 512 bytes, according to the manual page: calling write(2) directly means you need to choose a buffer of some size.

"bs=" sets both input and output block sizes, which probably isn't the best idea in this case.

Block sizes are a tricky subject:

https://utcc.utoronto.ca/~cks/space/blog/unix/StatfsPeculiar...

https://utcc.utoronto.ca/~cks/space/blog/tech/SSDsAnd4KSecto...

rollcat• 2 years ago

I've been re-implementing a bunch of coreutils as an exercise, and got stuck on dd input/output block sizes, AND disk/partition block sizes for a while. (As far as I understand it, for dd I need a ring buffer the size of max(ibs, obs), and then some moderately clever book-keeping to know when to trigger the next read/write, perhaps with code specific to ibs>obs, ibs<obs, etc; partitioning on the other hand is plainly stupid, there's decades of hardware and software just lying to each other and nothing makes sense.)

Thank you and everyone else in this thread for the know-how and references! I would like to eventually write an article (or at least heavily commented source) to hopefully explain all this nonsense for other people like me.

mort963• 2 years ago

TFA mentions that you can use sync after cp to do the same thing as fdatasync. You can't "unmount the file system" because you're writing directly to the block device, the thumb drive isn't mounted.

formerly_proven2• 2 years ago

Syncing as you go (ideally asynchronously) when you have to sync anyway (like when you're writing an image to a thumb drive, or write to NFS) has the big advantage that you don't end up saying "I'm done writing all data" and then blocking for five minutes waiting for the kernel to flush a couple gigs to disk.

codetrotter4• 2 years ago

Speaking of thumb drives.

In the case of copying files to a mounted file system, I’ve sometimes found it faster to use a tar pipeline than cp when copying data to an USB stick or SD/microSD card.

Instead of:

    cp -r ~/wherever/somedir/ /media/SOMETHING/

I would do

    cd ~/wherever/
    tar cf - somedir/ | ( cd /media/SOMETHING/ && tar xvf - )

And it would be noticably faster.

Not the same use case as linked article, but wanted to bring this up since it’s somewhat related.

xorcist• 2 years ago

There's also rsync which works perfectly with local paths and can resume from interruptions (by default), with or without crc checking the material ("-c"). That can be useful for removable storage which can sometimes be a bit unreliable.

Just take care that cp, tar and rsync each have slightly different handling of extended attributes and sparse files.

(By the way, I believe "tar -C /path" is the canonical way of doing "cd /path ; tar" without resorting to subshells.)

codetrotter4• 2 years ago

rsync is a strange beast to me

First thing that makes it so weird is how it assigns different meaning between paths including vs not including trailing slash. Completely different from how most command line tools I am used to behave in Linux and FreeBSD.

That alone is enough to remind me every time I try to use rsync why I don’t like and generally don’t use rsync.

legends2k2• 2 years ago

Interesting.

My guess: speed up is due to buffering involved in the latter's case, not sure though.

tyingq• 2 years ago

There's probably more than one reason it's faster. For example, tar ignores extended file attributes by default. Cp -r would have to check them for every file.

mortehu• 2 years ago

It's because reading and writing happen in separate processes, i.e. simultaneously instead of interleaved.

In general writing to disk is handled asynchronously by the kernel (`write` just copies to a buffer and returns), but metadata operations like creating files are not, so this should help the most for many small files.

legends2k2• 2 years ago

I've found when there are many files but overall size being small i.e. many many small files, archiving, copy and unpacking works faster. Perhaps due to enumerating, analysing overall size and then copying vs archiving directly.

Would be happy to learn the real reason.

formerly_proven2• 2 years ago

cp is single-threaded and only blocks on one IOP at a time.

With a tarpipe, you can block on two IOPs at a time, and they're decoupled by the pipe buffer.

This primarily makes a difference because the kernel cannot issue the I/O for small files ahead of time, like it does when you sequentially read a large file, so you do actually end up blocking and waiting.

naitgacem• 2 years ago

This is the strategy I always use with that use-case. Especially with "assets" folders where thousands of little icons and drawables and the like.

inglor_cz• 2 years ago

On a tangential note, I always found unzipping/untarring in Midnight Commander excruciatingly slow, and copying of many files just barely tolerable.

BenjiWiebe• 2 years ago

I've switched to the following:

  <infile pv >outfile

And it seems to work very well plus gives a good progress indicator.

Comparing dd and cp when writing a file to a thumb drive

Related Stories