A way to do atomic writes

https://lwn.net/SubscriberLink/789600/101b40d06e0dfb80/

by Tomte • 7 years ago

140 53 7 years ago

kabdib4• 7 years ago

Note that the SQL Server team at Microsoft had the Windows folks add write-through and cache control to NTFS so that the database could have stability guarantees.

It's mind boggling that this is still an issue, given the number of decades we've had applications (databases, and other things) that need to efficiently guarantee bits are on stable storage.

zzzcpan3• 7 years ago

It shouldn't be mind boggling, given that this guarantee is impossible to achieve. A little bit better or worse is not that big of a deal, still very far from 100%.

kabdib4• 7 years ago

Er, no. I'm using a storage system right now that makes this guarantee; when the store responds to a block write, it's going to be stable. The implementation in question uses battery-backed RAM for performance, but that's just an implementation detail.

This is definitely not impossible. It's not even that difficult (and yes, I've written a number of transactional storage systems over the past 30 years).

jchrisa• 7 years ago

One way modern distributed databases handle this is by assuming they can't trust the disks. When you are writing to multiple datacenters it makes recovery from disk errors just another kind of chaos resilience. The Calvin protocol takes this to its logical conclusion by moving the write-ahead-log to a Raft-like distributed consensus algorithm, where commits to the log span the cluster: https://fauna.com/blog/consistency-without-clocks-faunadb-tr...

I will add that the different flavors of fsync, and what they do (and don't do), is always a source of entertainment in database engineering chat rooms.

13415• 7 years ago

When I got fed up with the way the OS handled writes 15 years ago in one of my (ordinary) applications, I wrote to a temp file in the same directory, closed the file, did the atomic filename swap, and then verified the integrity of the new file under new name by opening it again and comparing it byte for byte to the content to be stored. Only after that did the operation succeed. On failure, the file names were swapped again and an error was signaled to the user. The temp file remained on disk to avoid further data loss.

I suppose that's not fast enough for a DB.

jjnoakes• 7 years ago

Are you guaranteed that the data you read back didn't come from kernel caches of the file system blocks? How do you know the data actually came back from physical storage?

alain94040• 7 years ago

Yes, but why can't we add an API, all the way down to the lowest levels of the hardware, that says "once I say this write is done, it will be readable again even after power-cycle".

Achieving that guarantee is not impossible, but it needs to be explicit. It can't be inferred from other API calls.

wyldfire• 7 years ago

This feature would work for mmap() regions as well, not just traditional write() calls.

I always wondered about writable disk-backed pages -- back when I last looked it up munmap() manpage made no suggestions about whether it would flush the data or whether you need an explicit msync() to make it happen. Since pages could get written back without an explicit msync(), it's always tricky to speculate how the behavior should be based on observation.

namibj• 7 years ago

Read the code. Only way to know for sure. Then use something maybe based on clang-query to check if the path that leads to flushing base on msync() and/or munmap() changed. Like, if any code between your syscall and the kernel-internal write-page-to-disk-now-i'm-waiting call changed. You should be able to convince clang-query to spit out a normalized AST of the code, and still be able to easily map it back to the source file location so you can check the source-file diff for the specifics once you demmed the AST diff ass non-trivial (some things should be easier to ignore in an AST diff).

In case you end up creating such a tool, please share it with us.

ww5205• 7 years ago

Appending log based file system has a lot of appeals. A lot of hard problems, like atomic write, become trivial. A log based FS shares similar properties as the transaction log in RDBMS, making consistency and recovery easy. Write performance is fantastic. With enough cache memory, read performance should be good, too. The only down-size is the performance hit when doing garbage collection. More research in this area would be helpful.

futureastronaut• 7 years ago

https://en.wikipedia.org/wiki/Journaling_file_system

https://en.wikipedia.org/wiki/Page_cache

ww5205• 7 years ago

Journaling FS only puts metadata and FS commands in log, not structuring the FS itself as an appending log.

Scriptor• 7 years ago

Haven't log-based file systems been a thing for a while now? https://en.wikipedia.org/wiki/Log-structured_file_system

notacoward• 7 years ago

They have, but they had fallen way out of favor for a couple of reasons - write amplification, cost/complexity of segment cleaning, poor performance for some workloads, etc. They've experienced a bit of a resurgence thanks to flash and SMR, though. Personally, I've always thought log-structured filesystems were elegant and wish they'd been more actively developed during the long "winter" during which their COW/write-anywhere cousins stole all the limelight.

ww5205• 7 years ago

Was hoping there are more research attempt at the Log Structured FS. For example,

- Borrow some ideas from generational garbage collection. Young generation in SSD (or mirrored in RAM) with copying GC to get rid of old versions of fast changing data blocks.

- Utilize some deduplication techniques with content based signature.

burmecia• 7 years ago

Totally agree. Log based file system makes things much easier, such as atomic write, data consistency and failure recovery. Along with COW semantics, transaction isolation also becomes easier. By its append-only writing nature, file versioning is trivial as well. Garbage collection is performance downside indeed, but it can be mitigated by COW switching and delayed bulk collection IMHO. Overall, the cost is worthwhile and that is what I am currently doing in ZboxFS: https://github.com/zboxfs/zbox.

ww5205• 7 years ago

That looks promising. Worth taking a look.

ape4• 7 years ago

Dumb question: is close() the same as fsync() for getting data written?

nwallin• 7 years ago

No. (and it's only a dumb question if you don't ask)

close() flushes the application buffer into the vfs buffer, but it does not guarantee the vfs buffer is flushed to the disk buffer, nor does it guarantee the disk buffer flushes to disk. A successful fsync() call guarantees these things.\*

\* except if the disk drive controller lies about flushing its buffer to disk, but there's nothing you can do about that besides buying a better disk.

zAy0LfpBZLC8mAC2• 7 years ago

close() flushes the application buffer into the vfs buffer

No, close() only closes the file descriptor. fclose() potentially also flushes a userspace buffer.

tarruda• 7 years ago

When in full control of the file format, it is hard to find a cross platform user-space solution that beats SQLite.

cafxx• 7 years ago

I agree with some of the comments on the article: there are good reasons to keep durability and atomicity separated, and to allow apps to specify what they need.

Also, a while back there was another paper that had a much better API as it allowed atomicity across files: https://github.com/ut-osa/txfs (better, but obviously not perfect: https://twitter.com/CAFxX/status/1017204557003173888)

caf2• 7 years ago

Hellwig said that originally he was using an open() flag, but got reminded again that unused flags are not checked by open() so using a flag for data integrity is not really a good idea.

This problem will be solved if, as looks likely, the openat2() syscall is merged for the path resolution restriction patchset. As a new syscall, it can do what open() should have always done and return EINVAL on flags it doesn't understand.

chris_wot• 7 years ago

In that system, users can write as much data as they want to a file, but nothing will be visible until they do an explicit commit operation. Once that commit is done, all of the changes become active.

Isn't this Multiversion concurrency control[1]?

1. https://en.wikipedia.org/wiki/Multiversion_concurrency_contr...

A way to do atomic writes

Related Stories