Causing ZFS corruption for fun, profit, and quality assurance

https://datto.engineering/post/causing-zfs-corruption

by binwiederhier • 7 years ago

113 6 7 years ago

the8472• 7 years ago

Some device-mapper virtual block device types also provide ways to simulate error conditions. For bad block errors there is dm-dust.

https://www.kernel.org/doc/Documentation/device-mapper/

derekmhewitt• 7 years ago

This was an interesting read! I hope they do a follow up article on how they automatically manage the rest of their recovery process.

pizza2342• 7 years ago

If with "recovery", you intend recovering that file, it's not possible, since all the copies available in the mirror are corrupted.

In this case, the recovery (repair) of the system (that is, to bring it to a clean state) simply consists in deleting the file (see https://docs.oracle.com/cd/E26505_01/html/E37384/gbbbc.html#...).

If there was an intact copy of the file available in the mirror, then one doesn't need to do anything, as ZFS is self healing: https://blogs.oracle.com/timc/demonstrating-zfs-self-healing.

kchamplewski• 7 years ago

Presumably they're referring to Datto's recovery process (i.e. recovering automatically from secondary or tertiary backups), as opposed to just the ZFS mirror.

pizza2342• 7 years ago

ZFS volumes can also be backed by files. In this case, the volumes can be corrupted by opening the backing file in a text editor, and perform a search/replace. This is simpler but of course less interesting :-)

Another interesting article I've mentioned in another post, is a demonstration of ZFS' self-healing capabilities: https://blogs.oracle.com/timc/demonstrating-zfs-self-healing.

hnlmorg• 7 years ago

Being able to back volumes by files is an amazing feature for testing. I've lost count of the number of times I've done this to prove some documentation works before applying it to the actual storage array in question.

Causing ZFS corruption for fun, profit, and quality assurance

Related Stories