How We Synchronized Editing for Rec Room's Multiplayer Scripting System

https://www.tyleo.com/blog/how-we-synchronized-editing-for-rec-rooms-multiplayer-scripting-system

by tyleo • 4 months ago

35 15 4 months ago

I find it fascinating when different people independently arrive at the same architecture when working on a hard problem like this. In my company we built our offline-first apps with PowerSync, which has the same idea of optimistic local changes while waiting for the central server to acknowledge the definitive changes. In PowerSync's case, the sync engine reads Postgres replication logs directly.

tyleo5• 4 months ago

Yeah, I'm surprised this pattern doesn't have a more general name.

I've also used it to synchronize content for local multi-window editing applications: each window is its own process but routes through one "coordination" process to ensure consistent ordering.

Easily solves some cross-process concurrency issues.

Rohansi3• 4 months ago

In this case it's just a (transaction/commit) log. Very common in distributed systems, databases, and filesystems.

munificent• 4 months ago

It does have a name:

https://en.wikipedia.org/wiki/Client-side_prediction

wpietri• 4 months ago

For those interested in a keep-everything-hot approach like this, it's worth checking out the 25-year-old library Prevayler. Full ACID guarantees, and radically faster than a database. I happily used it for a project forever ago and was disappointed to see it so thoroughly ignored.

tyleo5• 4 months ago

Oh, this is interesting. What sort of project did you use it for?

h4ch1• 4 months ago

I did a bit of digging after reading this comment and came across its' currently maintained implementation in Clojure by author Klaus Wuestefeld[0] :)

[0] https://github.com/klauswuestefeld/prevayler-clj

davidanekstein• 4 months ago

I’m surprised there was no mention of operational CRDT’s, or CRDT’s generally.

Rohansi3• 4 months ago

There's no need for CRDTs here because there is always a central authority.

tyleo5• 4 months ago

Yeah, I glossed over a bunch of the things we tried near the bottom of the "In-Memory Database" section. One of the things was CRDTs.

We did a few days of CRDT investigation along with the other techniques and decided the juice wasn't worth the squeeze when we found a simpler solution to our problem.

That being said, we've actually retained a CRDT-like implementation in our codebase and have been considering moving shapes (which have a slightly different set of tradeoffs) to that model.

forrestthewoods2• 4 months ago

Cool post!

I don’t quite understand the “funnel” section. Users see some change local immediately (S1->S2). And all users send all commands to the host. The host then effectively chooses the sort order and broadcasts back.

So does the initial user effectively rewind and replay state? If action applied to S1 was actually from another player they rewind to S1 and apply the sequence?

How many state snapshots do they need to persist? There was no mention of invertible steps.

I feel like I’m missing a step.

tyleo5• 4 months ago

Whether users see the action locally or not is decided on a per-action basis. For more visual actions like movement, we use a local imposter that gets overwritten on completion of the action (success or failure). These revert to the most recent “committed” state which already passed through the action funnel.

We have some actions which happen when saving config UI which don’t really need realtime sync when “save” is pressed and we treat them more like a normal POST request (e.g. we wait for full commit before rerendering). The form fields are kinds a built-in imposter.

For state snapshots we only do big ones. The “full save” that happens every 25 actions or so. These aren’t generally used for reverting people.

So it kinda works like:

1. Do I have an outstanding imposter? If so read from it.

2. Otherwise read from the committed state.

forrestthewoods2• 4 months ago

Ahh that’s interesting.

So instead of “rollback and replay” it’s more like “have a committed state and transient preview on top that may or may not commit”.

Implementation could vary between “commit the preview” vs “delete preview and create commit”. But that’s just an implementation detail.

Anyhow. That’s neat and clever. Thanks!

How We Synchronized Editing for Rec Room's Multiplayer Scripting System

Related Stories