Disk write buffering and its interactions with write flushes

https://utcc.utoronto.ca/~cks/space/blog/tech/WriteBufferingAndSyncs

by ingve • 2 years ago

76 58 2 years ago

utcc.utoronto.ca

mgerdts2• 2 years ago

Rather than allowing multiple gigabytes of outstanding buffered writes and deferring writeback until a gigabyte or more has accumulated, you'd set things to trigger writebacks almost immediately and then force processes doing write IO to wait for disk writes to complete once you have more than a relatively small volume of outstanding writes.

This is especially true when the thing doing a bunch of buffered writes is in a VM. If the VMM is buffering writes to the host fs, you get the described effects in the host OS and the guest OS.

Borg32• 2 years ago

And thats what direct I/O is for. Leave that to the guest OS.

mgerdts2• 2 years ago

Edit: maybe you were suggesting the host does direct IO. That is exactly what I recommend. I initially read this differently.

Original: If you are running the host OS you often have little or no say about what happens in the guest OS. Even if you control both it is likely not trivial to get the apps to use direct IO.

It could even be harmful to use direct IO because that would mean that writes would not stay in the guest buffer cache, forcing what would have been a cached read or minor fault in the guest into what it sees as a physical read.

The written blocks are not going to be shared, except maybe due to KSM. But KSM would do the same if that data was in the guest’s buffer cache if huge pages are not used.

M95D4• 2 years ago

I noticed this problem when I switched to Linux.

On Windows, when I copied a file, disk writes started immediately. On older systems, like Win98, I had to tweak Total Commander's disk buffer to improve copy speed on the same drive. Total Commander even had separate settings for same disk vs. different disk copy buffer sizes.

When I switched to Linux I was immediately surprised that disk writes did not start until the memory was full, and then it would stop reading while flushing dirty data. This happens even if the copy is between different drives: reads stop, writes only, then reads again with no writes to the other disk, repeat. It basically halves the copy speed.

It even happens when I copy to network mounts: reads 20 GB of data into memory, then reading stops and tries to flush the data over the nfs. Nfs times out, transfer fails. I had to use nfs timouts of 1h just to be able to do a backup.

It drives me crazy. Is there any way to make it write immediately, or at least to put a memory limit on dirty data?

nolist_policy2• 2 years ago

https://docs.kernel.org/admin-guide/sysctl/vm.html#dirty-byt...

https://docs.kernel.org/admin-guide/sysctl/vm.html#dirty-bac...

The values are crazy high by default (on modern hardware anyway): 10% of memory for dirty_background_bytes and 20% for dirty_bytes. I wonder why no distro touches these.

suprjami3• 2 years ago

Because people complain their system is "slow" if it blocks on disk I/O.

Another set of people also complain Linux takes too long to safely unplug USB drives.

qwerty4561274• 2 years ago

To me this seems great and even humble. As RAM is dirt-cheap this can significantly improve performance (especially when external or a remotes drives are involved and not only NVMe SSDs), also prolong SSDs life (which means saving not just money but also the hassle of replacing them, money as well though - when it's a Mac and you can't just replace the SSD).

I wish I could configure Windows the same way: whenever it can use RAM to avoid an extra disk write/read - it should.

bobmcnamara• 2 years ago

Is there a max yet or per disk limits? IIRC those would start writeback. I always thought wiping a slow USB disk shouldn't consume all available RAM. But it used to. Maybe it still does.

pradn2• 2 years ago

fsync() guarantees that writes have hit the disk. But is there a guarantee about what's written before an fsync()? Can it be anywhere between "nothing" and "everything"? I suppose this must be a loose guarantee if the "write-back" parameter can be tweaked at will.

nolist_policy2• 2 years ago

Can it be anywhere between "nothing" and "everything"?

Yes, and that won't change because the hardware with it's own buffers behaves the same way.

toast02• 2 years ago

Can it be anywhere between "nothing" and "everything"?

Yes, and there's not (generally) any ordering constraint, either. The last thing you wrote may be persisted, and not the first, etc.

loeg3• 2 years ago

There's some user input to this via posix_fadvise POSIX_FADV_DONTNEED but it doesn't guarantee anything.

vlovich1235• 2 years ago

If you're generating an immutable file, a common technique is to play fsync + rename tricks although I think a more modern technique would be:

    1. open directory (dir_fd)
    2. create an unnamed O_TMPFILE file (file_fd)
    3. write to file_fd
    4. fdatasync(file_fd)
    5. linkat(dir_fd, "", dir_fd, "file name", AT_EMPTY_PATH)
    5. fsync(dir_fd)

This should guarantee that either "file name" will have the old contents or the new contents and no transient version is observed. The "old" mechanism is similar in that you write out to a temporary sibling and rename (these days you'd use RENAME_EXCHANGE w/ renameat2 to guarantee the atomicity or get an error) with the differentiating difference being that the temporary data could be observed on the filesystem / left around if you have a machine reboot.

vlovich1235• 2 years ago

Correction to typo:

    5. linkat(file_fd, "", dir_fd, "file name", AT_EMPTY_PATH)

pradn2• 2 years ago

Thank you - it's baffling to me that something as basic as saving some data to a file has become so convoluted!

fao_• 2 years ago

SSDs and HDD on-board firmware will actually cache-locally and then lie about if it's been written to the disk or not. Pretty much every level of the stack caches and lies about it to the point that if you do a deep dive into "has information been written to disk or not" the answer comes up as "this is impossible to verify".

Rygian4• 2 years ago

My systems routinely report transferring files at speeds much faster than what the physical medium asked, only to have them get stuck and fail on a time out a while after.

Having such behavior be the default is, to my limited understanding, a bug in the Linux kernel.

adr1an• 2 years ago

There's "$ sync" to ask the Kernel to start the actual write. And "dd" command has the option to do so, too. It's just not the default, as many things on Linux, unfortunately.

JackSlateur2• 2 years ago

You can see this the other way

When I write lots of data to a rusty device and don't sync, the kernel gives back more quickly. From there, I can move on with my work : everything is "as-if" the data were actually copied (except power-cutoff protection, indeed)

As such, I see this behavior as a feature, it allows me to wait less and do my work quicker.

Unless windows, on which I must wait for the whole data to be written on the disk.

Rygian4• 2 years ago

As such, I see this behavior as a feature, it allows me to wait less and do my work quicker.

So, when (if?) you notice that the transfer has failed with a timeout, do you still consider your work was done quicker?

toast02• 2 years ago

Rather than allowing multiple gigabytes of outstanding buffered writes and deferring writeback until a gigabyte or more has accumulated, you'd set things to trigger writebacks almost immediately and then force processes doing write IO to wait for disk writes to complete once you have more than a relatively small volume of outstanding writes.

I think having the trigger be size based rather than timebased is the real problem. Or bounds on both...

I probably don't want to buffer writes for more than X seconds, or let the buffer grow beyond Y% of ram. At least for the time based limit, you'd really want to be able to say start writing to disk when the buffer has data over 10 seconds old, but still accept writes into a new buffer, only blocking writes when there's a buffer being written out and the current buffer is too old or too big.

loeg3• 2 years ago

Ideally the system starts flushing buffered writes basically immediately, but with low device-level queue depth so that subsequently issued higher priority IOs do not suffer from a ton of additional latency.

kvemkon• 2 years ago

Ideally is only when the size of the complete file is known in advance. To minimize avoidable fragmentation.

magicalhippo• 2 years ago

ZFS has both a time-based and size-based limit. IIRC the default is 15 seconds and lower of 2GB and some % of system memory.

Though in ZFS' case it's not really a regular write cache as such, as it's used to minimize updates to its on-disk copy-on-write structure.

vlovich1235• 2 years ago

You're mostly right in that it is similar to the problem of buffer bloat in networking. However, it's not quite the same thing because for example you can do things like write a file to disk & unlink it or overwrite some portion of contents, meaning that by buffering for longer you can avoid the writeback in the first place. By buffering for longer, the kernel is trying to balance things landing on disk and avoiding touching the disk if the dirtied data will be dirtied again. Granted not a common use-case these days, but consider the case where you have object files being created during a build over & over again. It's easy to construct scenarios where you indeed wouldn't want to write to disk so quickly.

It's not out of hand a terrible idea to avoid flushing data to disk and there's no free lunch here as any workload you optimize for will have a different workload that suffers. People try to come up with general heuristics that work in most situations on consumer machines, but there's no one size fits all for all HW + use-case combos. That's why hyperscalars tune the kernel beyond that / have kernel developers writing code to optimize for their use-case. It's telling that the performance analysis in the article is pretty hand-wavy without any clear demonstration of a concrete problem.

As for sync, I believe the author is mistaken. You can do fsync instead which is more efficient as it only creates a barrier for writeback of the file descriptor rather than a system-wide sync. And invoking fsync I believe is more common than sync. You should be able to have multiple parallel fsync happening concurrently for unrelated files that don't block on each other so much (ideally the kernel would prioritize those writebacks and interleave for fairness, but I doubt it does).

anamax2• 2 years ago

Not doing a write when the disk is idle doesn't save anything even if that write turns out to be unnecessary.

charleshn• 2 years ago

It is the case, see the vm.dirty_expire_centisecs sysctl [0] at the VM layer, and the commit interval for e.g. ext4 [1] at the filesystem layer, and [2] for their interaction.

[0] https://docs.kernel.org/admin-guide/sysctl/vm.html#dirty-exp...

[1] https://www.kernel.org/doc/html/v4.19/filesystems/ext4/ext4....

[2] https://serverfault.com/a/995887

randkyp• 2 years ago

I found this recent thread interesting, specifically about really considering whether you're going to read the data you just wrote in the near future or not (in which case, use direct IO) and a set of (abandoned?) patches for write-behind caching for sequential writes in Linux (https://lore.kernel.org/lkml/156896493723.4334.1334048120714...).

https://lore.kernel.org/linux-mm/45odvhgymm7fxsgwpewoiiggaok...

rwmj2• 2 years ago

Direct IO is very inflexible. A better way is this recommended by Linus in an old post:

https://lkml.iu.edu/hypermail/linux/kernel/1005.2/01845.html

I implemented something along the same lines but a bit less spicy here:

https://gitlab.com/nbdkit/nbdkit/-/commit/aa5a2183a6d16afd91...

rwmj2• 2 years ago

That second link is wrong, should be: https://gitlab.com/nbdkit/nbdkit/-/commit/a956e2e75d6c88eeef...

hsaliak• 2 years ago

The theory is compelling but this is the exact sort of problem that needs a representative benchmark. Based on the benchmark, the optimal strategy can then be picked.

I don’t see any stats/graphs or benchmarks in the linked article.

cafxx• 2 years ago

I think that the idea of not initiating writeback immediately derives mostly from the days of spinning rust, where read latencies would be noticeably impacted if you initiated writeback too aggressively: reads, contrary to writes, are synchronous by default, and spinning rust rarely allowed high (by modern standards) IOPS, so it made a lot of sense to buffer writes as much as possible to minimize the number of I/O operations spent on writes, as this would leave as many of those IOPS as possible available for reads.

This is probably much less of a concern today, as NVMe drives - beside having many orders of magnitude higher IOPS capacity - also have (at least on paper) much better hardware support for high I/O concurrency. It may still make sense, even today, if your hardware (or stack) limits IOPS.

vlovich1235• 2 years ago

As I mention elsewhere in the thread, reading the kernel documentation for the various flags suggests that the kernel devs are also concerned about multiple writes to the same piece of data & thus buffering lets you potential elide unnecessary disk I/O.

pengaru• 2 years ago

One of the things I improved in my recordMyDesktop fork [0] was an awful tendency for the frame cache writer to accumulate heaps of dirty pages until background writeback would flush them out.

I had 16GiB of RAM which meant quite large swaths of dirty pages would become buffered while the SSD sat idle until writeback began. This would cause high-FPS full-screen recordings in particular to become backlogged and start dropping frames / audio dropouts. Just generally broken behavior for a desktop recorder, especially for a deferred-encode mode that's supposed to be optimized for minimizing system-wide effects/overheads during the recording.

The simple solution I found was to proactively initiate writeback regularly via fdatasync() on the cache fd. [1] I haven't decided yet if more should be done to constrain its buffer cache effects though. The cache files will be read back during encoding in post, so if there's enough RAM it can be desirable to enable reading them back entirely from memory instead of having to hit the disk again... but it would also be nice to let the rest of the system's processes keep their stuff in the page cache. memcg can probably be used to find a balanced solution, but I haven't done any experiments yet. Have any of you handled similar scenarios? What did you do?

[0] https://github.com/recordmydesktop/recordmydesktop

[1] https://github.com/recordmydesktop/recordmydesktop/commit/42...

Disk write buffering and its interactions with write flushes

Related Stories