lxpz

Garage does actually support CORS (via PutBucketCORS). If there is anything specific missing, would you be wiling to open an issue on our issue tracker so that we can track it as a feature request?

Very interesting, thank you. It would probably make sense for most tables but not all of them because some are holding large CRDT values.

I had a very rapid look at these two papers, it looks like none of them allow the implementation of compare-and-swap, which is required for if-match / if-none-match support. They have a weaker definition of a "transaction". Which is to be expected as they only implement causal consistency at best and not consensus, whereas consensus is required for compare-and-swap.

Fair enough, we will work on making the documentation clearer.

So the thing is, different KV stores have different trade-offs, and for now we haven't yet found one that has the best of all worlds.

We do recommend SQLite in our quick-start guide to setup a single-node deployment for small/moderate workloads, and it works fine. The "real world deployment" guide recommends LMDB because it gives much better performance (with the current status of Garage, not to say that this couldn't be improved), and the risk of critical data loss is mitigated by the fact that such a deployment would use multi-node replication, meaning that the data can always be recovered from another replica if one node is corrupted and no snapshot is available. Maybe this should be worded better, I can see that the alarmist wording of the deployment guide is creating quite a debate so we probably need to make these facts clearer.

We are also experimenting Fjall as an alternate KV engine based on LSM, as it theoretically has good speed and crash resilience, which would make it the best option. We are just not recommending it by default yet, as we don't have much data to confirm that it works up to these expectations.

Garage already shards data by itself if you add more nodes, and it is indeed a viable path to increasing throughput.

If I make certain assumptions and you respect them, I will give you certain guarantees. If you don't respect them, I won't guarantee anything. I won't guarantee that your data will be toast either.

They are not solving the same problem.

Syncthing will synchronize a full folder between an arbitrary number of machines, but you still have to access this folder one way or another.

Garage provides an HTTP API for your data, and handles internally the placement of this data among a set of possible replica nodes. But the data is not in the form of files on disk like the ones you upload to the API.

Syncthing is good for, e.g., synchronizing your documents or music collection between computers. Garage is good as a storage service for back-ups with e.g. Restic, for media files stored by a web application, for serving personal (static) web sites to the Internet. Of course, you can always run something like Nextcloud in front of Garage and get folder synchronization between computers somewhat like what you would get with Syncthing.

But to answer your question, yes, Garage only provides a S3-compatible API specifically.

We were not able to get good enough performance compared to LMDB. We will work on this more though, there are probably many ways performance can be increased by reducing load on the KV store.

If you have replication, you can lose one of the replica, that's the point. This is what Garage was designed for, and it works.

Erasure coding is another debate, for now we have chosen not to implement it, but I would personally be open to have it supported by Garage if someone codes it up.

It does not, at least not for a small local dev server. I believe RAM usage should be around 50-100MB, increasing if you have many requests with large objects.

If you know of an embedded key-value store that supports transactions, is fast, has good Rust bindings, and does checksumming/integrity verification by default such that it almost never corrupts upon power loss (or at least, is always able to recover to a valid state), please tell me, and we will integrate it into Garage immediately.

Thank you for your feedback, we will take it into account.

I talked about the meaning of the Jepsen test and the results we obtained in the FOSDEM'24 talk:

https://archive.fosdem.org/2024/schedule/event/fosdem-2024-3...

Slides are available here:

https://git.deuxfleurs.fr/Deuxfleurs/garage/src/commit/4efc8...

Read-after-write consistency : yes (after PutObject has finished, the object will be immediately visible in all subsequent requests, including GetObject and ListObjects)

Conditionnal writes : no, we can't do it with CRDTs, which are the core of Garage's design.

Losing a node is a regular occurrence, and a scenario for which Garage has been designed.

The assumption Garage makes, which is well-documented, is that of 3 replica nodes, only 1 will be in a crash-like situation at any time. With 1 crashed node, the cluster is still fully functional. With 2 crashed nodes, the cluster is unavailable until at least one additional node is recovered, but no data is lost.

In other words, Garage makes a very precise promise to its users, which is fully respected. Database corruption upon power loss enters in the definition of a "crash state", similarly to a node just being offline due to an internet connection loss. We recommend making metadata snapshots so that recovery of a crashed node is faster and simpler, but it's not required per se: Garage can always start over from an empty database and recover data from the remaining copies in the cluster.

To talk more about concrete scenarios: if you have 3 replicas in 3 different physical locations, the assumption of at-most one crashed node is pretty reasonable, it's quite unlikely that 2 of the 3 locations will be offline at the same time. Concerning data corruption on a power loss, the probability to lose power at 3 distant sites at the exact same time with the same data in the write buffers is extremely low, so I'd say in practice it's not a problem.

Of course, this all implies a Garage cluster running with 3-way replication, which everyone should do.

If I can reassure you about Garage, it's not at all abandoned. We have active work going on to make a GUI for cluster administration, and we have applied for a new round of funding for more low-level work on performance, which should keep us going for the next year or so. Expect some more activity in the near future.

I manage several Garage clusters and will keep maintaining the software to keep these clusters running. But concerning the "low level of activity in the git repo": we originally built Garage for some specific needs, and it fits these needs quite well in its current form. So I'd argue that "low activity" doesn't mean it's not reliable, in fact it's the contrary: low activity means that it works well for us and there isn't a need to change anything.

Of course implementing new features is another deal, I personally have only limited time to spend on implementing features that I don't need myself. But we would always welcome outside contributions of new features from people with specific needs.

Doesn't Haskell do this to some degree with the IO monad? Functions that are not supposed to do IO directly simply have a more specific type signature, like taking in a stream and returning a buffer for example.

https://openwrt.org/toh/tp-link/er605_v2?s[]=er605v2

This one doesn't have WiFi, just 5 Ethernet ports. It can be flashed openwrt with a very reasonable amount of tweaking. It's actually quite powerful and has 256M of RAM and 128M of flash memory. I have one, it's very cool.

This is due to SQLite calling fsync a lot by default, to be on the safe side. You can use pragma journal_mode = "wal" and pragma synchronous = "off" and it should be much faster, without risking corruption if your server powers down unexpectedly (at least in theory, you should still make regular backups, which can be done from within a running sqlite instance using the backup command)

Yes! My desktop PC is almost 10 years old at this point and it works perfectly, I've upgraded the CPU to an i7-4790K (the best of that generation) and added some RAM (32G total) which makes it very good for most of my work and even for occasional gaming. Thankfully I don't use windows that much, but I know I can't upgrade to windows 11, so according to Microsoft I would just have to throw it away in a year or two, which is completely ridiculous.

I think as long as your construction is deterministic, both options are possible: you can either add the intermediate nodes that have only one children, or you can skip adding them and allow links between non-consecutive levels. In that second scenario you would add intermediary-level nodes only when necessary, i.e. only when they actually need to have several children. The second approach is better for performance but might have a bit more implementation complexity.

As the main author of the Merkle Search Tree paper, I'm really glad to see the design become more known and start to be used. I did not go on and build a practical system that made use of the structure, although I originally came up with the data structure while thinking about how to build decentralized social networks and data stores for collaborative applications. Thankfully, the design itself is generic and can be adpted to many use cases, and it's great to see that people have gone as far as to build independant implementations in Go and Rust. I'd be curious to see how it performs in practice, in systems that have a real-world use, and not in the simplistic synthetic benchmark scenario of the paper. Hopefully it holds up to the promise :)

Very interesting project and impressive work, thanks for sharing!

Can you talk a bit about how data is replicated between nodes when Stalwart is run in clustered mode, and what kind of data integrity/resilience properties we have when one, two, several nodes go down?

Also, have you considered implementing server-side encryption of e-mail messages so that a "honest but curious" system administrators could not read user's messages? (e.g. using the user's password to derive an encryption key). More generally, what are your thoughts on the "privacy" aspect?

For back-end systems and distributed systems, I'd love to work on a new OS based on a microkernel and capabilities. Ditch Linux+Docker+Kubernetes, replace that with a microkernel that has a minimal set of modules to multiplex storage and networking, build an orchestrator directly on that that schedules processes as micro-VMs. In those VMs, we could run a stripped-down Linux kernel to handle legacy applications, but probaly we would want to design a few standardized abstractions for accessing other components of the system (storage, databases, queues, RPCs, the scheduler itself) as network services which we would use to implement more modern services. In many cases we could just leverage existing protocols that are built over HTTP (e.g. S3 for object storage). The scheduler would be aware of the dependencies of each services (i.e. the other services that it needs to call to function) and could have the task of directly connecting those things together. To contact one of its dependent services, a service or application would no longer use a hostname+port (resolving the hostname using DNS), it would just directly have a file descriptor to a socket or pipe or whatever kind of RPC interface that the OS directly maps to the corresponding service, either locally or on another node of the distributed system.

If you are aware of anyone working on something like this, please let me know!

Can you do conditional writes?

If by that you mean something like compare-and-swap or test-and-set, then no, that would require consensus number > 1 [0]

Can old versions of a register be compacted to reduce metadata space?

Yes, when using last-write-wins registers, old versions are deleted as soon as they are overseded by a newer version. Almost all metadata stored in Garage uses this mechanism.

How does garbage collecting dead blobs work?

Nodes have a shared table of block references, that identifies for each data block the set of objects (or rather, versions of objects) that use this block. Block references have a "deleted" flag that is set to true when the version in question is deleted from the store. Once set to true, it cannot go back, but a new version can always be created that references the same block. Nodes keep a local count of the number of non-deleted block references for each block, so that when it reaches zero, they queue up the block for deletion (after a small delay). There is also a metadata GC system so that block references with the deleted flag are dropped from the table after a certain time.

[0] https://en.wikipedia.org/wiki/Consensus_(computer_science)#C...

I mean, sure, but we're operating under the premise that we don't want to give out our data to a cloud provider and instead want to store data ourselves. Half of the post is dedicated to explaining that, so pretending that you can't infer that from context seems a bit unfair.

Then you didn't look close enough. Have a look at the .drone.yml to see the entry points, we have both unit tests and integration tests that run at each build.

You probably want at least some redundancy otherwise you're at risk of losing data as soon as a single one of your hard drive fails. However if you're interested in maximizing capacity, you probably need erasure coding, which Garage does not provide. Gluster, Ceph and Minio might all be good candidates for your use case. Or if you have the possibility of putting all your drives in a single box, just do that and make a ZFS pool.

It's not a hard requirement, you might just have a harder time handling voluminous files as Garage nodes will in all case have to transfer data internally in the cluster (remember that files have to be sent to three nodes when they are stored in Garage, meaning 3x more bandwidth needs to be used). 10Mbps is already pretty good if it is stable and your ping isn't off the charts, and it might be totally workable depending on your use case.