HN user
The first section in the README is called "Design goals", with 13 items. None of them is "data integrity", and none of them even talks about validating the data or handling any failures aside from power loss.
Fair enough.
With modern hard disks, no. They work in sectors.
Author here.
I don't know if the authors are here, but if they are - would you comment on fragmentation and the dangers of growing a filesystem past 95-98% full ?
Fragmentation isn't an issue in TFS, at all. Because it is a cluster-based file system. Essentially that means that files aren't stored contagiously, but instead in small chunks. The allocation is done entirely on the basis of unrolled freelists.
This does cause a slight space overhead (only slight, coming from the fact that metadata of the file is stored in the full form), but it completely eliminates any fragmentation.
The data is summed too.
A good design would look at the state of the art and use the best techniques available. If the aim was research, then try one new thing, not a thousand.
That's what it does: It takes from many sources (although mainly ZFS).
I meant ChaCha20 ofc.
Well, my points still remains. You need to store IVs/keys/etc. which makes it pretty unsuitable for a file system.
TFS was created to speed up the development. The issue is that following the design specs makes it much slower to implement, and prevents a "natural" development (like, you cannot implement it like a tower, you need every component before completion). It was started[1] and got far enough to reading images, but implementing it took ages, so we decided to put it off for now.
It is very similar to ZFS.
I understand this filesystem is still nascent, but shouldn't data integrity at least be one of the design goals?
What makes you think it isn't? It definitely is. In fact, it borrows several ideas from ZFS wrt/ integrity.
For example, it uses parent block checksums like ZFS.
ChaCha2 isn't a block cipher. It's a stream cipher, and would thus require storage of IVs ... which, well, is going to expensive space-wise.
I agree. SipHash is certainly strong if you don't know the key.
If you know the key, it is as weak as it gets (as the paper notes too, you can construct collisions easily if the key is known), so I disagree.
Oh, well.
What I think of as "cryptographic hash function" is a function resistent to pre-image attack, second pre-image attack, and collision generation.
Neither of those are satisfied by SipHash, and can thus not classify as a cryptographic hash function by the normal definition.
Please don't. DJB2 is a poor hash function. It's similar to FNV: Entropy only moves upwards, so flipping higher bits doesn't affect lower bits.
In other words, you risk mapping `n` and `-n` to the same value under some modulus.
No, it's not a cryptographic hash function. It's a MAC function.
The paper clearly states that it is not collision resistant.
A lot of stuff.
SHA256 is very slow, and that's no surprise. It's cryptographic after all.
Here's a small list of usecases for non-cryptographic hash functions:
- Checksums and error correction codes, as long as there is no way to maliciously use this.
- Hash tables. These always use non-cryptographic hash functions.
- Bloom filters.
- Heuristic fingerprinting. They're not strong enough to be used for normal data fingerprints, but they can be used as a way to decide if two buffers are "probably equal" or "certainly not equal".
Hash tables are the main one. Cryptographic hash functions are almost never used in them. SipHash is a popular choice, but it is not cryptographic. That is a misunderstanding: It's a MAC function.
In hash tables, you never use cryptographic hash functions. Why? Because they're slower.
Take SHA3, which is around 50x slower than SeaHash. That is really really bad for hash tables.
When hash collisions happen in hash tables, they're resolved through collision-resolution strategy, such a linear proping.
Fingerprints are one very narrow usecase for hash functions, and there are tousands of other uses.
There's a huge difference between cryptographic and non-cryptographic. Note that blake2 has various length, whereas SeaHash is fixed to 64-bit (although I suppose it's not to hard to make a version with bigger length), and thus naturally collisions will happen.
If you need fingerprints, don't use SeaHash, but if you are looking to insert into e.g. a hash table, you shouldn't use BLAKE2. It's awfully slow for that.
MetroHash's main transformation actually loses entropy, so that's, well, pretty bad. It still passes Smhasher, though.
SeaHash is obviously not cryptographic (nor is SipHash), but I hope it is a secure PRF (i.e. the keys cannot be extracted), and this was the best attack I was able to construct. Still, it isn't a practical attack, but I suppose it is possible to improve.
Note that I am not a cryptographer, and my only piece of advice is: For the sake of god, don't use hash functions not designed for cryptographic security, if you need cryptographic security. It's that simple.
BLAKE(2) is a cryptographic hash function, SeaHash is not. Even the fastest implementations of BLAKE only gets around 7.8 cycles/byte (hardware might do it twice as fast). SeaHash gets 0.24 cycles/byte.
That's a wide difference, around 32x faster.