Filesystem and Database are not cutting the problem space right

This seems like an unreasonable criticism. Seeing a problem without a good solution and creating a solution to it seems to me the very essence of hacking.

lookdangerous• 6 years ago

Maybe it’s because it’s late, but I don’t really follow what this is. It seems cool, but the explanation is too abstract

zupa-hu4• 6 years ago

Hey, author here, thanks for the feedback. I've been working on this problem space for a while and decided to write it all out one problem at a time. Yeah this one was quite heavy on the theory department but I plan to follow it up with a demo on how to actually use it. That should be more hands on. The problem was, I felt like I would be immediately asked on the theory behind it, plus there were people already interested so I thought it's better to have it explained. I agree with your point though.

kazinator• 6 years ago

This is wrong; you can implement atomic transactions in filesystems. As in prepare a change, and then publish it, such that the consumers of that event only see the before and after view, and not something in between. There are many ways to do this, on various scales, involving parts of a file, a whole file or multiple files. Filesystems really have no such disadvantage over databases.

Think about a remote git repo. Multiple users can push. It's based on files.

This HN site is based on files.

skissane• 6 years ago

Common filesystems provide some atomic operations; for example, on POSIX systems, rename() is atomic for local filesystems (but not necessarily remote ones)

There are also some filesystems which expose special transactional features beyond what common filesystems offer. For example, NTFS has filesystem transactions, although it was rarely used and Microsoft has deprecated it – part of the reason for its rare use, is that with Win32 it isn't seamless to adopt, all existing filesystem calls have to be replaced with *Transacted equivalents. (Apparently the undocumented NT level API is more seamless, but well, it's undocumented.)

zupa-hu4• 6 years ago

Hi, author here. Sure, multiple users can do a git push but don't neglect that git is also offering a higher level abstraction, basically a storage system. That's no different from running any other storage system, like a filesystem, except you name it differently.

There are also other hacks to go around it but none of them is a simple solution abstracting away the complexity. Not a clean filesystem that a novice developer could start using without messing it up. They all greatly increase mental complexity.

Brian_K_White• 6 years ago

If this problem is as real and fundamental as presented, and the solution really is this equally fundamental new paradigm, then it should be a kernel module and not a product.

zupa-hu4• 6 years ago

The problem is, it's not enough to have a transactional filesystem as it will inevitably have a different API, one that is not POSIX compliant. Also, you can't jut make the filesystem transactional, in the end you also need an entire transactional OS built around it. Some years ago I set out to eliminate the accidental complexity and kept removing parts of the system that caused any specific problem. In the end, the filesystem + OS turned out to be the biggest culprit.

Filesystem and Database are not cutting the problem space right

Related Stories