pvh

It does indeed turn out to be a difficult and subtle problem. We've tried to balance minimizing novelty (always risky in cryptographic systems) with achieving the various security and scaling properties we're looking for. We're very lucky at Ink & Switch to be working with Brooke Zelenka on this one.

Ideally, you would use existing commodity infrastructure but we have found none of it is really super fit for our purposes. Failing that, we have been developing an approach to low-maintenance reusable infrastructure. For now, I would advise running your own but positioning yourself to take advantage of commodity systems as they emerge.

Mechanical merge algorithms can perform better or worse on different kinds of conflicts (the specific example of editing deleted text is just one of many edge cases) but in the end no CRDT can decide if your merged text is what you mean to say.

We go into a bunch more detail in the Upwelling paper about the differences between (what we call) semantic and syntactic conflicts in writing: https://inkandswitch.com/upwelling/

Ultimately, my feeling is that serious collaboration is a document review problem as much as anything else. That said, this is particularly true in journalism and scientific publishing and can be mostly ignored for your meeting notes...

Anyway, if you see this comment, thanks for a nice piece of writing, Alex. Love to see folks wrestling with these problems.

Hey, thanks! I don't know what's regressed here but I've emailed the support people for buttondown.

Of course not. Kurt is up in Heaven now.

Automerge does indeed work with live collaboration, though apparently not currently as efficiently as some other solutions. Everyone working in this space is exploring and looking for solutions that will work for users woth slightly differing priorities. In addition to automerge consider checking out yjs, electricsql, diamond types, replicache, vulcn, or any of the other folks. Hopefully one of them will be just right for you.

Robust undo/redo remains an ongoing research project. Leo Stewen's work was presented at PLF 2023 a few days ago. It turns out to be a subtle problem to really get completely right, but in my experience you can usually get passable results by letting editors default undo behaviour reverse text input.

For applications with more document-structured data, you can now produce inverse patches using Automerge.diff to go between any two points. To implement a reasonable undo in this environment you can record whatever document heads you consider useful undo points and then patch between them.

To perhaps expand on why the problem remains unsolved slightly further, there was a robust discussion about what the expected behaviour of "undo" out to be in even simple cases at the conference.

The sync protocol does indeed calculate the delta between peers and efficiently catches both sides up.

The backends you see are the ones I use, but the API is a binary blob key value store with range queries: supporting other stores should be straightforward.

Authentication isn’t exactly left as an exercise to the reader but is an area of active work. I would say securing access to a URL via whatever mechanism you’re used to should be fine for client server applications and peer to peer folk seem to mostly have their own ideas.

Please accept my unreserved apologies, Heather! No offense is intended. I can speak for everyone working on Automerge when I say that we've very much appreciated Matthew's work and have indeed spent quite a lot of time studying and responding to it. We spoke about it in person last week, in fact.

As for the use-cases, I do not mean to exclude live collaboration from consideration, just to note that it hasn't been our focus or come up often in the use-cases we study. Live meeting notes are definitely a real use-case and I don't dispute the performance results you show.

As for Y-js, it's a wonderful piece of software with excellent performance and a vibrant community made by exceptional people like Kevin Jahns. We simply have slightly different goals in our work, which undoubtedly reflect where our engineering investments lie.

Indeed, your paper did not measure the same things we look at, and that's why it found new results. Hopefully in time we will join the other systems in performing well on your benchmarks as well.

Ink & Switch is behind it; or more expansively mostly Orion Henry, Alex Good, Martin Kleppmann, and myself. As an organization, we have been working on Automerge for about six years now. We also have a wonderful community of other contributors both in industry and research.

Automerge is not VC-backed software. Indeed, for a number of years Automerge was primarily a research project used within the lab. Over the last year, it has matured to production software under the supervision of Alex Good. The improved stability and performance has been a great benefit to both our community and internal users. Our intention is to run the project as sponsored open source for the foreseeable future and thus far we have done so thanks to the support of our sponsors and through some development grants.

Ink & Switch's research interests drive a lot of Automerge development but funding from sponsors allows us to work on features that are not research-oriented or to accelerate work that we'd like to do but that doesn't have current research applications. If you adopt Automerge for a commercial project, I'd encourage you to join the sponsors of Automerge to ensure its long-term viability.

The way I think about it is that if the data should always travel together it should be in one document. For example -- if your TODO list always goes as a unit, then make it an array of objects in a single Automerge document. On the other hand, if you want to build an issue tracker and to be able to link to individual issues or share them individually then a document each is the way to go. Does that help?

As for network transports you can indeed have multiple at once. I usually have a mix of in-browser transports (MessageChannels) and WebSocket connections. I suspect we'll need to do a little adjusting to account for prioritization once people really start to push on this with things like mDNS vs relay server connections but the design should accommodate that just fine.

As for the docs, my apologies. The "tutorial" was merged into the quickstart as part of extensive documentation upgrades over the last few months. We should update the link in the old blog post accordingly.

Here's a link to save you the effort: https://automerge.org/docs/quickstart/

Automerge is a library that anyone can adopt, and we are a research organization, not a product company.

We have built a variety of projects with Automerge, both publicly and for use in private, including recently the markdown-with-comments editor we call Tiny Essay Editor (https://tiny-essay-editor.netlify.app/) by Geoffrey Litt.

That said, sponsoring the Automerge team helps us build faster and is always welcome. (Thanks to our current and past sponsors for their support!)

The benchmarks Matt Weidner has been working on are great and outside scrutiny is always welcome, but I should note that I find there's an element of artificiality to them. In particular, testing the performance of the sync system while simulating many users typing into the same document doesn't really measure behaviour we have observed "in the wild". In our research, we've found that editing is usually serial or asynchronous. (See https://inkandswitch.com/upwelling for further discussion of our collaboration research.)

The benchmark that concerns me (and that I'm pleased with our progress on!) is that you can edit an entire Ink & Switch long-form essay with Automerge and that the end-to-end keypress-to-paint latency using Codemirror is under 10ms (next frame at 100hz).

While these kinds of benchmarks are incredibly appreciated and absolutely drive us to work on optimizing the problems they uncover, we try to work backwards from experienced problems in real usage as our first priority.

Thanks, Scott. This API should make it much, much easier for folks to build with Automerge and kind of just encapsulates everything we've been doing in-house over the last few years.

What fun, jake! I am really enjoying your series here. When we built Pixelpusher (https://inkandswitch.com/pixelpusher) that was the beginning of a long and ongoing journey into the user experience implications of CRDTs.

For instance:

  * how and when should we merge changes?
  * what if we change our minds later?
  * how do we know if our data is “in sync” with another user
  * what does it mean to depend on another change?

Just a note to suggest that giving away your hard work to those who will profit from it in the hope that they will remember you later seems like a pretty dubious exchange.

Have a look at how that worked out for the folks who built node and its libraries versus the ones who maintained control of their work (like npm).

To the former, curious people with an interest in how and why the world came to be as it is.

To the latter, obviously "it" has improved, but ecosystem effects make certain changes very difficult and expensive to coordinate and what we see here is the scars from that process.

Everything you see in the world grew out of things that came before, and was made by fallible people working with limited time, energy, and perspective.

Honestly, I'm a bit surprised someone with a three letter handle wouldn't already recognize this. Surely you have been around here for a while.

Hey there! It's great to see more folks building more CRDTs. On behalf of the Peritext authors, Automerge, and Ink & Switch, welcome to the community.

We tried Web Bluetooth in a project. It was extremely flaky. Notably it also only works as a server, so you cant peer two browsers over it either.

Just so!

Thanks. We'll fix that before the next one.

The article has quite a few performance numbers in it. The short answer is that it's much, much faster but that we will continue to pursue improvements pretty much forever.

In fact, convergence is a very easy property to preserve in all distributed systems. The trivial but technically valid version of convergence is to throw away all the writes and always return an empty document. A "last writer wins" version at the document level is what you get from a blob store like S3, but while it does converge, it's not that great either.

What we probably want from a distributed system is useful convergence properties that preserve the intent of the participants. A CRDT might not be a good fit for a bank account: if we can both withdraw the last $20 from my account, the bank will be upset. On the other hand, it's a pretty great way of combining independent observations into a list: it doesn't matter what order the observations arrive. Easy!

Most CRDTs aim to preserve causality: if I see your change, and then make my change, my new value will win. If we both make changes without knowing about each other, that's a conflict.

Of course if we both edit unrelated fields -- maybe it's not a conflict! At least, that's how we handle it in Automerge.

In the most conservative case, we should never merge data automatically. Most systems have unmodeled constraints. For example, sometimes a `git` merge will produce no conflicts but fail to compile anyway. Git's model (another CRDT) doesn't model program behaviour, nor do we expect it to. In this case, we rely on a combination of our experience, programming tools, and git's version history tooling to figure out what went wrong.

The conclusion I have is that a CRDT should give us robust tools for minimizing conflict, but also needs to be able to explain how things got to be the way they are and what you can do to make them how you want.

We've made a decent amount of progress on this in Automerge and have a paper coming up about this problem soon but I agree there's still more distance to go. If there are particular questions you have about merge semantics, I'm all ears! We'll continue to explore this space for the foreseeable future and I love to hear about new questions.

The last thing I want to add is that when you say "CRDTs are only really useful for a narrow subset of data", you're really drawing a lot of conclusions all at once about other people's needs and interests. From my perspective, CRDTs are useful for a lot of kinds of data. Not everything certainly, but from where I sit, perhaps more kinds of data than a limited single-node relational database and more kinds than a POSIX file which doesn't retain any history at all.

I prefer to think of Automerge as a form of version control: because the full history is retained, if you don't like the merge you can decide what you want to do instead.

Wonderful postmortem, Paul. Appreciate you taking the time to reflect and share what you learned.

These experiments are not the products we think users would adopt if they held them in their hands. They are more like the studies a painter or a sculptor might do as part of planning a more ambitious piece.

It's hard to predict where this work might eventually lead (that's the nature of research) but I will just say that we continue to explore the space and we have another piece we'll be sharing soon.

For one exploration of that direction, please see our paper Crosscut: https://www.inkandswitch.com/crosscut/

We may in the future.

The RSS feed is a good long-term bet, and in the past we have posted on Twitter at @inkandswitch.

It's a good name, thanks for sharing it.

Databases should add support for lenses and also support type conversations, column relocations, new default values, and everything else a user might need.