How to change symlinks atomically (2005)

http://blog.moertel.com/posts/2005-08-22-how-to-change-symlinks-atomically.html

by ffggvv • 10 years ago

70 63 10 years ago

blog.moertel.com

josefbacik4• 10 years ago

This is just an accident of how some file systems are implemented and isn't actually garunteed. If you did this on xfs you could still end up with the a 0 length symlink if you crashed at just the right time.

jjnoakes• 10 years ago

xfs doesn't guarantee atomic renames in the same directory?

I thought that requirement was from POSIX.

Does xfs not conform to POSIX?

josefbacik4• 10 years ago

The rename is atomic, the data being in the file is the problem, there is no garuntee unless you fsync.

adrianratnapala4• 10 years ago

I think what the josefbacik means is that there is no guarantee that the original symlink under its temp name has actually been written to disk. After the `rename()`, there is still no guarantee.

I think this is true in many filesytems, not just `xfs`.

If your atomicity requirement you never want the file to disappear from the POV of an external process, then the OP's method is sufficient. If you want crash-proofing as well, then you will need an fsync() -- preferrably on the tempfile BEFORE the rename().

caf4• 10 years ago

The article is talking about atomic with respect to processes running at the same time as the operation, not with respect to a system crash.

userbinator• 10 years ago

As much as the Win32 API is criticised for many of its functions containing often-unused parameters, symlink() having a flag parameter to control whether to overwrite the original link's contents if it exists would've made this much easier.

adrianratnapala4• 10 years ago

You would need the extra parameter + the guarantee that the symlink syscall did the overwrite atomically. And I don't think proliferating such guarantees througought the API is going to end well.

In Unix we have this convention that `rename()` is our atomicity swiss-army knife. I guess it makes it easier for implementers to only have to make their strong guarantees for one system call (+ a few related friends like `renameat`).

Now you could argue that `rename()` is a bit too under-powered for this job, and maybe we want transactions or something. But NTFS tried that too and deprecated them.

What I would like to see in Unix is better support for anonymous files and directories. You could use this with things like `renameat()` so that the temporary never touches the filesystem and is automatically cleaned up if the rename fails.

JoshTriplett5• 10 years ago

What I would like to see in Unix is better support for anonymous files and directories. You could use this with things like `renameat()` so that the temporary never touches the filesystem and is automatically cleaned up if the rename fails.

Exactly. This approach makes the various calls orthogonal. Various syscalls create anonymous reference-counted inodes on a filesystem (which disappear when all references to them do), and syscalls like linkat or renameat or installs such an inode (referenced by file descriptor) into the filesystem with a name.

O_TMPFILE allows creating an anonymous file inode. Ideally, a syscall would exist to do the same for an anonymous symlink inode.

Animats• 10 years ago

Ah, yes, atomic file system operations in UNIX-like systems.

May not be available for your operating system variant. May not be available for your file system. Not applicable to some remote file systems. May not function properly in some virtual machines. Consult your storage area network vendor for additional information.

wfunction• 10 years ago

Yeah, it's another one of the many things that the Windows kernel gets right but *nix programmers never care to give it credit for. `FILE_LINK_INFORMATION` in Windows has a `ReplaceIfExists` flag just for this purpose.

rincebrain• 10 years ago

I'm not sure which is wrong, the person writing or the documentation, but rename() claims to be atomic with few exceptions even when overwriting an existing item.[1]

The only reason I suspect things do unlink-then-rename is that (evidently) rename() doesn't promise anything about what happens to the "src" item if rename() fails, just that something will still be in "dest".[1]

But I don't know why the commands do something different, and don't want to try reading the source of either GNU or BSD mv to find out why.

[1] http://pubs.opengroup.org/onlinepubs/009695399/functions/ren...

pif• 10 years ago

It's funny how you name Windows kernel in a comment about symbolic links, considering that it _misses_ symbolic links.

CyberShadow• 10 years ago

It goes much further than that:

https://en.wikipedia.org/wiki/Transactional_NTFS

marcosdumay• 10 years ago

You know, that fixes the "May not be available for your operating system variant." disclaimer.

All the other still apply, and the GP forgot to tell you to check your HD model's documentation.

AnonymousPlanet• 10 years ago

It is one thing to get it right from a disgn perspective, but another to get it right from an implementation and common usage perspective (see the sibling threads to this comment).

If it doesn't exist in practice or does something different in a thousand edge cases, it's rightfully dismissed as nonexistant.

johnwheeler2• 10 years ago

As a Python/web developer whose never written a line of production C/C++ code in his 16 year career, it's always humbling to stumble into these threads on HN.

Rapzid4• 10 years ago

I highly, highly recommend getting in the habit of consulting OS syscall documentation for stuff. The behaviour of the syscall determines most of the behaviour for this stuff in every language(few exceptions like stdlib buffering, etc). The linux syscall docs are pretty easy to consume even if you don't write C. "The Linux Program Interface" is an awesome book based on the man pages(by the main man page maintainer). Not sure what exists for Windows in the same vein book wise, but technet or something surely has what you NEED.

johnwheeler2• 10 years ago

Thank you. I will check it out. I've been looking into the `select` call a little because I'm working with websockets, but I'm not sure if that's related to syscall. It's amazing what you can get away with not knowing with computers nowadays. It doesn't necessarily mean you should strive for ignorance, of course.

ashitlerferad• 10 years ago

Some related articles:

https://yakking.branchable.com/posts/moving-files-1-copying/ https://yakking.branchable.com/posts/moving-files-2-sparsene... https://yakking.branchable.com/posts/moving-files-3-faster/

How to change symlinks atomically (2005)

Related Stories