In principle, ArangoDB behaves similarly to MongoDB here. Both are essentially "mostly-in-memory" databases in the sense that they hold the data in memory and persist it at the same time to disk via memory mapped files. This approach is good for performance and if you run out
of RAM you ought to shard your data.
However, MongoDB often uses a lot of memory for the actual data, since its BSON binary format stores the names of the attributes with every single document. ArangoDB detects similar shapes of documents (see https://www.arangodb.com/faq#how-do-shapes-work-in-arangodb) and thus
avoids this particular problem.
I agree. It is extremely hard to select the correct technology. So, sometimes it is even more valuable to learn, what has not work (even if it looked nice from the description), than what has worked.
Durability is indeed hard to archive - as you pointed out disk controller sometimes simply tell you a lie.
With respect to corruption ArangoDB behaves similar: It uses an append-only log file with CRC checksums. So, if the last bit of storage contains corrupt data, it is discarded.
IMHO nowadays quite a lot of logic is moved into the front-end (for example "single web application"). In these cases you only need to provide some API in the back-end. But normally you need a bit more then "just" the data:
- some access control (e. g. sessions)
- compute some fields (e. g. "age" derived from "birthday")
- combine and filter data before transfering it the client (e. g. for graph traversals)