Everything you never wanted to know about file locking

http://apenwarr.ca/log/?m=201012#13

by duck • 16 years ago

122 14 16 years ago

apenwarr.ca

tptacek• 16 years ago

Worth reading all the way to the bottom; the Python stuff made me giggle.

yread• 16 years ago

Cool writeup!

This is apparently because some versions of Windows don't understand shared locks

Which versions? It seems even the Windows 95 had FILE_SHARE_READ and FILE_SHARE_WRITE which are effectively shared locks (it just didn't have the FILE_SHARE_DELETE)

TimJYoung• 16 years ago

The author is referring to the LockFile API call, which only supports exclusive byte-range locks, vs. the LockFileEx API call, which does support shared byte-range locks, but is only available on the Windows NT variants (NT 4 and above), not the Windows 9x variants (95, 98, and ME).

danwolff• 16 years ago

Ah, the cross-disciplinary have vs. understand.

glhaynes2• 16 years ago

Good article. I don't quite understand this part, though:

Still not convinced [that you shouldn't use mandatory locking]? Man, you really must like punishment. Look, imagine someone is holding a mandatory lock on a file, so you try to read() from it and get blocked. Then he releases his lock, and your read() finishes, but some other guy reacquires the lock. You fiddle with your block, modify it, and try to write() it back, but you get held up for a bit, because the guy holding the lock isn't done yet. He does his own write() to that section of the file, and releases his lock, so your write() promptly resumes and overwrites what he just did.

If you're going to read some data and then potentially write modified data back over what you just read, shouldn't the first step before you even read it be to get a lock on the file or that range? Or, if your calculation might take a long time, at least get a lock right before you write it back, verifying first that you're writing over data that hasn't changed while you were off calculating?

stingraycharles• 16 years ago

That was my first intuition too. It sounds like a classic race condition, in which case you would simply acquire the highest level of locking (exclusive) before reading the data, to make sure all your accesses to the shared data are synchronized.

But I'm pretty sure I'm misunderstanding what the author was trying to say.

IgorPartola• 16 years ago

He is talking about mandatory locks, where if I lock the file and do things properly, you cannot do operations on it, even if you disregard the locks. In other words, it's a "convenience": I can structure my program in such a way as to "guarantee" that I will have the exclusive lock on the file and even your "buggy" code won't be able to overwrite it despite the fact that your code does not acquire any locks. Seems like the perfect solution, until the madness described in the OP ensues.

Also, mandatory locks are used in Windows. Ever try to delete that virus.exe while it's still running? Yeah.

js2• 16 years ago

This excellent writeup inspired me to publish some old dotlocking code I had lying about - https://github.com/jaysoffian/dotlock

known• 16 years ago

http://www.beej.us/guide/bgipc/output/html/singlepage/bgipc....

tedunangst2• 16 years ago

Every BSD system I'm aware of has something called lockf, which may or may not be layered on top of flock (or the other way around).

Note that all of these locking schemes fall down hard on NFS, depending on client and server, regardless of what any man pages say about support. Homework question: How does a stateless protocol like NFS remember when one client locks a file and a different client tries locking it?

The failure mode on NFS can vary from "lock always succeeds, regardless of actual status" to "lock always fails, hanging forever", or my favorite, "locks work, unless your process crashes, and then you can never lock that file again until you reboot the NFS server".

DanielRibeiro• 16 years ago

I had discovered this the hard way. Replaced with Apache Zookeeper, which now is much more stable than at the time. It is meant to be a port of Google's distributed lock server Chubby.

kelnos• 16 years ago

So it turns out only one sentence near the end really matters:

"I guess lockfiles are the answer after all."

Yup. Don't use the locking APIs. Just use lockfiles be done with it.

I'd actually go a step further and suggest only using lock directories. Using lockfiles assumes O_CREAT|O_EXCL works properly everywhere (which is probably a safe assumption, but...). mkdir() will return EEXIST if you fail to acquire the 'lock'.

Everything you never wanted to know about file locking

Related Stories