File handling in Unix: tips, traps and outright badness

https://rachelbythebay.com/w/2020/08/11/files/

by janvdberg • 6 years ago

134 72 6 years ago

rachelbythebay.com

janvdberg• 6 years ago

Also reminds me of this talk by Dan Luu: https://www.deconstructconf.com/2019/dan-luu-files

cryptonector3• 6 years ago

He doesn't mention ZFS at all.

jwilk• 6 years ago

Let's say it returns a value of 8192 instead, setting errno to EINTR.

errno is meaningful only after the function failed. Short write doesn't count as failure.

jeffbee2• 6 years ago

Indeed, errno is not cleared by successful calls, so you can't check it for success.

After a short write your only legitimate course of action is to advance the pointer and write again.

nitrogen2• 6 years ago

You can set errno to 0 and see if it changes.

rachelbythebay• 6 years ago

That's a derp on my part. Fixed. Thanks. Clearly, this stuff is incredibly complex.

dependenttypes• 6 years ago

note, while this might be true for write it is not true for fwrite

The fwrite() function shall return the number of elements successfully written, which may be less than nitems if a write error is encountered
if a write error occurs, the error indicator for the stream shall be set, and errno shall be set to indicate the error

userbinator2• 6 years ago

You need a signal handler to eat it and do something about it, or you have to explicitly say that you don't care and set it to "ignored", but you can't just pretend it won't happen... because it sure will.

The default action of SIGPIPE is to kill the process, which is exactly what you need for pipelines to work correctly. Related article: https://news.ycombinator.com/item?id=22647539

sfoley• 6 years ago

Not all programs are meant to be used in pipelines.

It’s not even entirely accurate to say that that’s EXACTLY how pipelined programs should work; maybe they want to do some cleanup before they die. SIGPIPE was essentially created as a hack for naive programs that didn’t properly check the return value of write(). EPIPE should have been enough, it’s a perfectly fine solution.

inetknght• 6 years ago

Not all programs are meant to be used in pipelines.

I would argue that programs that aren't meant to be used in pipelines aren't well designed programs.

It’s not even entirely accurate to say that that’s EXACTLY how pipelined programs should work; maybe they want to do some cleanup before they die.

If you need to do some cleanup before you die, then that's exactly what a signal handler is there for you to do. Nothing stops you from exiting immediately after a signal handler.

zaarn4• 6 years ago

If you'd check your errno when write returns -1 on standard in or standard out, SIGPIPE would not be necessary.

userbinator2• 6 years ago

That would mean duplicated logic across all applications and all places in them.

SIGPIPE is like an exception. If you really need to, you can catch it (to do cleanup or similar, as a sibling comment notes), but otherwise the default behaviour is fine and likely what you want.

I've made use of a similar technique when reading input: in code where an EOF is a rare case, but reads occur in multiple places, you can simplify the main logic considerably if you create a function that contains the read and a longjmp() upon EOF to a specific handler which then uses state variables to decide what else needs to be done; it gets rid of all the "if EOF" duplication and puts all the special-case EOF-handling logic in one place.

greggman3• 6 years ago

This sounds to me like seriously bad API design. I'm not saying the low-level functions that allow handling fancy situations aren't important but shouldn't the default be "it just works" and not "you must write 1000s of lines code exactly correctly or else your program will fail and you'll be powned"

This reminds me of Jonathon Blow's rant on similar issues on XBox and PS2/3 where the API, instead of doing the right thing by default required the developer to spend days/weeks using it correctly and having their app rejected over and over and in this particular case there was zero good reason for it not to handle the issues itself rather than pass it all on to the developer.

asveikau6• 6 years ago

I think interrupting I/O can be tricky for an application to handle. If write(2) didn't have this behavior, it's likely that a common outcome would be frozen processes that you can't quit with ^C.

There either needs to be a notion of partial writes (what the article describes) or a write that gets rolled back if interrupted (probably really hellish to implement ... Maybe impossible for something like a socket where the remote machine may have already seen some packets).

EINTR or SIGPIPE are famously criticized as poor mechanisms for this. However even without that, interrupting a blocked write would remain a tricky problem.

nine_k2• 6 years ago

Writing to a file should be transactional. It either succeeds, or gets rolled back. Journaling filesystems implement it as a crash protection mechanism, but an API where you can commit changes without closing, and can roll back explicitly, would be nice.

Writing to a socket is unlike writing to a local file, and cannot offer such guarantees.

Thorrez• 6 years ago

But it sounds from the article that write(2) is more complicated than necessary. Why do some errors cause an error return while other errors cause a signal? I would think consistency in how errors are propagated would be better.

Joker_vD5• 6 years ago

interrupting a blocked write would remain a tricky problem.

Would it? I believe kill(pid, SIGTERM) interrupts it all right. Or do you want to do this without terminating the program? But then we are back to square one: the programs have to handle failed writes somehow. How? "Abort, Retry, Ignore"? And then again, most of this stuff happens deep inside libraries, and every library makes different choices in handling such errors, and when you combine them in a single program, you end up with a mess. As always.

lmm2• 6 years ago

Oh, absolutely. The Unix system API is what was easy to implement in 1970, not what's sane for developers nowadays.

Don't invoke system calls directly. Use a library that handles all this stuff for you properly. Better yet, use a language that gives you a better interface than C does.

thayne2• 6 years ago

Advice (which I agree with) that conflicts with the author's previous post referenced in the first paragraph.

GoblinSlayer2• 6 years ago

I think the post is dramatized a little. Mentioned SA_RESTART should fix the EINTR problem for most synchronous functions and is used for default signals.

iamtew• 6 years ago

"Doing the right thing by default" is a weird concept.

Who's default are we talking about here? The developers desired state of default, or the users?

KSteffensen• 6 years ago

Wasn't that the point of the whole O_PONIES thing?

fsckboy• 6 years ago

This sounds to me like seriously bad API design

this illustrates the sad irony of unix's genius. All the people who think like Dave Cutler (or whoever) are eager to bring Cutlerian ideas to unix. But unix is where they are bringing these ideas because unix is what won, and unix won because it did not make Cutlerian design choices.

I'm not saying that an API call that did the right thing wouldn't be great for a programmer; just that there are a lot of (simple) things going on in the kernel and tradeoffs for a simple userspace security model and some of this comes with that territory. And neither am I saying that a Cutlerian model is theoretically unsound. But things play out practically and this is how things have played out, and observers along the way discussed it the whole time.

More to say for sure, but please forgive me. My comment is for people who understand what I'm talking about. Explaining to people who don't understand is a useful and noble thing, but this ground has been covered so many times over ("worse is better"), and it's very detailed and historical and somewhat subtle, and I have limited time atm.

asveikau6• 6 years ago

Maybe I have skimmed too much, but I didn't see anything about the problems with using O_EXCL to resolve races over NFS. I believe (correct me if I am wrong) it doesn't work.

cryptonector3• 6 years ago

No, O_EXCL works on NFSv3 and NFSv4 just fine. NFSv4 is stateful and solves a number of problems that NFSv3 had.

asveikau6• 6 years ago

Googling around has this for Linux: http://nfs.sourceforge.net/#faq_d10

Tldr: it used to not work. It was fixed in Linux in 2004.

No idea about other unix-like OSs.

ezekiel68• 6 years ago

I enjoyed this post and the one before it (linked in the article). +1 for a resurrection of the old PC loser-ing situation made semi-famous by the seminal "worse is better" article (also linked therein). Really this all comes down to, "Where should the complexity live?" People who enjoy languages, tools, and OSes which cater to systems programming (myself included) eat this stuff up. I can definitely appreciate the dismay this causes in folks who simply want to glue things together to produce higher-level results more quickly.

thayne2• 6 years ago

I really don't see how this post supports the one before it. Much of the complexity here can and in many, perhaps most, cases should be abstracted into a library, whether that is the standard library or a well-established high-quality third-party library. Given how easy it is to make mistakes, common operations, like atomically writing a buffer to a file, or creating a new file and ensuring it didn't already exist should not have to be re-implemented in every application.

cryptonector3• 6 years ago

Sometimes you want to use a temp file named ${target_name}.new, then rename it after writing to it -- the point of this is to avoid leaving garbage if you have unclean exits, but it requires using flock(2). Even better is to create an unlinked file using O_TMPFILE (Linux-only) and then use linkat(2) and then rename(2) (renameat2(2) doesn't quite support what's needed, and it's a darned shame). If you're doing this in a /tmp/ directory, well, you do have to worry about attacks by other users if you try to do the .new thing.

asveikau6• 6 years ago

Wanting a few more details I googled that and found: https://lwn.net/Articles/559147/

Second time I googled something you mentioned in this thread, eager to know more. This guy filesystems, folks.

self_awareness• 6 years ago

I wonder if EINTR can also happen on macOS? macOS is certified UNIX, and the post talks about Unix in general, that's why I wonder.

I've inspected the family of fopen() functions (fread, fwrite), and they don't handle EINTR. Do they delegate this task to the caller? Or that EINTR doesn't exist there? I mean, the error code itself is used, and by quickly skimming XNU sources I can see it's used in some NFS code, but can it happen when reading from or writing to local storage (hard disk)?

Edit: some minutes later, I can see that e.g. libstdcxx supports EINTR, so maybe it can happen on macOS as well (the xwrite() function):

https://opensource.apple.com/source/libstdcxx/libstdcxx-104....

comex2• 6 years ago

I just tested it on macOS and Linux with the two main sources of EINTR: signal handlers and attaching a debugger. I tested by interrupting a read from stdin, but the behavior for disk reads and writes should be similar.

Signal handlers: POSIX specifies that, after a process runs a signal handler which was triggered when it was in the middle of a system call, either the system call will automatically be resumed or it will fail with EINTR, depending on a per-signal-handler setting. If the signal handler was set with sigaction(), this setting is the SA_RESTART flag; otherwise it can be set by siginterrupt(), but the default seems to be implementation-defined(?). Anyway, it seems that both macOS and Linux enable restarting by default. (But well-written library code should take into account the possibility that something else in the process could have decided to disable it.)

Attaching a debugger (using ptrace): This interrupts system calls because the debugger expects to see a userland state. On Linux it seems to automatically restart the system call, but on macOS it produces EINTR. This behavior doesn't seem to be configurable on either OS.

On both OSes, when EINTR does occur in read(), fread() does not cover up for it by automatically retrying, regardless of whether it successfully read any bytes beforehand.

saagarjha4• 6 years ago

Which libc were you using on Linux, or were you making raw system calls? Because I think glibc’s behavior is to EINTR unless you define _GNU_SOURCE, which seems to me to be incompatible with your description…

deathanatos• 6 years ago

Put it another way: if your idea of avoiding races in your temp file creation is to just have a wider namespace, what keeps an evildoer (me) from just grabbing every possibility first? Disk space is cheap, right? You can only have so many pids, and the time of day is obviously predictable, and your random number space is also similarly limited. I could just set up every one of them first and when you hit it, I win!

A UUID, generated by the system's CSPRNG. Ideally through a syscall if the OS has one, so I don't need to burn a FD. You can't pre-fill all UUIDs without running out of space, and the CSPRNG prevents prediction, short of using gdb to attach to the process mid-name-generation, at which point you're on the other side of the airtight hatch.

I do wish linkat could do an atomic replace of an O_TMPFILE file descriptor. It'd avoid the whole random-name business, which is an ugly wart.

for_xyz• 6 years ago

Does write situation apply to linux as well?

Since 99% of the disk I/O is cached, write either success or fails completely. It's beyond scope of the userspace application to know what and when data arrives on the physical disk. Same applies to Windows as well.

Using direct I/O is different story where this is the correct behaviour.

iforgotpassword• 6 years ago

NFS. Also the gdb/strace case still holds. It's really rare so many programs get away with not doing it, and then when it happened once in a honeymoon your shrug it off and restart the program.

saagarjha4• 6 years ago

I hope you’re not fixing short writes during your honeymoon…

lenkite• 6 years ago

So...basically use SQLite for this then as recommended. https://www.sqlite.org/aff_short.html

GoblinSlayer2• 6 years ago

It has its own share of legacy and poor tradeoffs. Transaction log is an overkill for most applications, but the only alternative is full cache in RAM, there's no middle ground like you have with file system.

praptak• 6 years ago

The scenario described in the article suffers from one more problem, not mentioned in the article. Quote:

"Create a file adjacent to the target path using mktemp or similar.

Write your data to it

rename() it from the temporary name to the final name."

I believe this is prone to data loss when the system goes down and the cause is that POSIX guarantees on write reordering are loose. The rename can make it to disk before the data gets written.

I think it caused actual data loss with one of the fancier filesystems which actually pushed the limits of write reordering and exposed this (pretty common) scenario as buggy.

Edit: I found an article about this problem. "Ext4 and data loss": https://lwn.net/Articles/322823/

sgerenser• 6 years ago

Of course, between finishing the write and doing the rename you have to call fsync() on the fd (as that article pointed out).

clktmr• 6 years ago

The first half of the article basically complains about the write()'s syscall possibility to fail. Errors can occur and you need to handle them, there is no way around it. Given that "Everything is a File", the OS shouldn't try to make any guarantees around that. There are cases where you want to handle an early return by write().

This is a syscall after all. First priority is to provide a very general interface and remain compatible, not developer ergonomics.

Hitton• 6 years ago

And this is still only scratching the surface. For more file related horror:

Files are hard - https://danluu.com/file-consistency/

icedchai• 6 years ago

In my experience, most developers ignore the "partial write" scenario with files. Then they get experience working with sockets... where this happens all the time.

jlokier2• 6 years ago

Then they get experience working with sockets...

Ha. I have seen TCP sockets code which breaks in a "partial write" scenario, and even a "partial read" scenario.

Code worked fine on their own LAN. Broke for someone else, they couldn't see why...

They were very surprised when I explained that you don't always get one message per read(), because they had for years. Or maybe they hadn't, but assumed something else caused their occasional application glitches.

perlgeek• 6 years ago

If you're doing async IO, I strongly recommend using a library that abstracts much of the pain away, like libuv. Deal with just the libc functions (or system calls) is a way too low level API for many applications.

ur-whale• 6 years ago

I remember being bitten by write doing only partial work early in my career.

This definitely felt like an API shortcoming, ended up writing a wrapper for write that did what I expected.

File handling in Unix: tips, traps and outright badness

Related Stories