Your Disk Just Lied to You – and Your Checksums Said Everything Was Fine

https://medium.com/@jingyuzhou/your-disk-just-lied-to-you-and-your-checksums-said-everything-was-fine-40e471f40129

by tanelpoder • 5 months ago

9 5 5 months ago

_wire_• 5 months ago

Article encourages a fix to a problem with "lost disk writes" with no investigation nor explanation as to why writes are lost.

defrost• 5 months ago

Generally these are writes that are lost after the kernel confirms data accepted by application write(). The causes can be hardware glitching, the difficulty lies with alerting the application that has already moved on assuming all is good.

This in the realm of Dan Luu's fsyncgate: https://danluu.com/fsyncgate/ (discussed here: https://news.ycombinator.com/item?id=19126824 ) and "Crash Consistency" https://danluu.com/file-consistency/

The article is about "circling back" and re-checking data blocks against checksums seperately stored rather than relying on hardware checksums and data blocks that match but are "old" (not updated when expected).

The "why" of how such things happen is long and turgid exposition into sometimes shit happens .. and that's a book or three by itself.

wmf• 5 months ago

I think only disk manufacturers can answer that but they're not willing to. Lost or "high-flying" writes have been documented for 20 years or so; we don't need to know why they occur to admit that they are real.

tanelpoder• 5 months ago

The issue in the enterprise (SAN) storage space is that there are so many layers where things can go wrong. Lost writes due to the OS kernel (like the fsyncgate), in-kernel storage interaction drivers, the storage array software itself, then disk firmware etc. Theoretically you could read back the just-written block and check if it’s what you wrote, but maybe it’s returned from some cache, before the bug happened.

Another scenario (OS, driver bugs) is that a correct block is written to a wrong location. Yes the write is persisted, but overwriting a wrong location. So now you have two incorrect blocks.

rootsudo• 5 months ago

The text feels AI generated and is unreadable to me.

Your Disk Just Lied to You – and Your Checksums Said Everything Was Fine

Related Stories