Flux over the Wire using Websockets and Immutable JS

http://blog.rotenberg.io/flux-over-the-wire-3/

by elierotenberg • 11 years ago

81 40 11 years ago

blog.rotenberg.io

stucorbishley• 11 years ago

The single source of truth diagram is so great! :) Was getting tired of turning my head sideways..

sehr• 11 years ago

cache: http://webcache.googleusercontent.com/search?q=cache:http://...

elierotenberg9• 11 years ago

Wow, a crashed site presenting a web architecture surely isn't classy! Sorry about that, I'm working on it.

ac2u• 11 years ago

Y'know... I quite liked jumping in and reading right away! Didn't notice it was lacking certain assets at first.

wereHamster5• 11 years ago

I see that Remutable uses 'patches'. How does it compare with ShareJS? Does it allow collaborative editing (ie multiple people changing the same data structure concurrently, is patch application commutative)?

The idea behind ShareJS (or OT for that matter) is awesome. You never have to think about what to do if two people edit the same document concurrently. It just works. You get undo for free. And you can write the code that it saves automatically in the background. Never again do users have to press a 'save' button.

A while ago I wrote a library which builds on top of the same ideas as ShareJS (ie. OT) but with slightly different focus. You define object types and their properties. This allows the library to automatically parse JSON into JavaScript class instances. It also tracks changes to these objects in a very similar way as ShareJS: each change has a 'path' (where something was changed) and then the actual chang record (currently set and splice).

I use Avers in an SPA which is an editor of fairly complex data structures (90 different object types, nested ~5 levels deep). For rendering I use React. The integration is really simple: Attach a change listener to the root and re-render when anything changes.

https://github.com/wereHamster/avers https://caurea.org/2015/02/08/the-future-of-avers.html

elierotenberg9• 11 years ago

I didn't now about ShareJS, but I'll definitely check it out. At its core, the patch abstraction is commutative (and more importantly, reversible), so I should be able to implement the 'rebase'-like operation. My feeling is that it would be more efficient to perform this at a higher level though, by replaying Actions in the right order in every client to get eventual consistency.

wereHamster5• 11 years ago

ShareJS is just one implementation of OT. If you want to check something out, then start with OT (http://en.wikipedia.org/wiki/Operational_transformation).

dugmartin• 11 years ago

You should also checkout ot.js:

https://github.com/Operational-Transformation/ot.js/

They have a nice visualization of OT here:

http://operational-transformation.github.io/visualization.ht...

aidos5• 11 years ago

Interesting. I'm surprised at how little work seems to have been done around OT (operational transformation, for those who haven't heard of it before).

I have an application where I hold a lot of synced data in the client and the server but I've struggled to find good libraries to help manage that. Generally the work seemed to be around text, rather than arbitrary objects. Sharejs was the best I could find, but my BE system is python (and the object support in ShareJS was a bit sketchy when I looked at it).

Unrelated but looking at your project I saw you were using Sets and Maps. I didn't even know JS had those yet (and they're something I really miss coming from python). Will be switching immediately.

elierotenberg9• 11 years ago

Native (mutable) Maps and Weakmaps are coming up with ES6, and are already (mostly) polyfilled by things like core.js.

Immutable provides (immutable) implementations of Map, OrderedMap, Set, etc. See http://facebook.github.io/immutable-js/ :)

wereHamster5• 11 years ago

Avers supports only two operations, set and splice. It is very simple to implement in any language. My backend is written in Haskell. The code dealing with applying the patches is about 200 lines.

With these two operations you can edit arbitrary data structures. The user experience may not be the best, for example if two people edit the same text, because if you only have set then latest change wins. But maybe your data structures are not text heavy. Or they have many small text fields where the 'latest change wins' semantic is actually desireable.

At my previous company we also used and OT based patch system. And we didn't have text-editing support, we only had an equivalent to set in Avers. And it worked alright. Which operations you need depends on the application, ShareJS is good for one particular type, but too complex (or maybe even lacking) for others.

As for maps and sets, see http://kangax.github.io/compat-table/es6/ how good the support is. There are polyfills though if you need to support older browsers.

riobard• 11 years ago

OT is notoriously hard to get right. That's probably why there's little work done around it.

lootsauce• 11 years ago

Google wave used OT, but Operational Transform can fail sometimes and seems to have been superseded by CRDTs (Commutative/Convergent Replicated Data Type) http://stackoverflow.com/questions/26694359/differences-betw...

Take a look at SwarmJs it uses CRDTs for multi user sync. http://swarmjs.github.io/articles/todomvc/

unconed• 11 years ago

Indeed, Share.js is awesome. I've worked extensively with it, and recently hooked up Share.js to immutable React using a modified React-Cursor. It takes local mutations and converts them into JSON-OT. Remote OTs are merged back in, the server is always ground truth.

You get local reactive UI with immutable state. On the server-side, you can validate the ops as they come in using e.g. the entire infrastructure of passport in node. OAuth2-validated Share.js with strict per-property write access, no problem.

I also extended the couchDB driver to merge in server-side changes, thus allowing daemons to make live changes to the DB without having to care about any of the OT stuff. If share.js changes were made in the meantime, it will save a merged snapshot.

Basically, Share.js can be your insurance policy against pesky humans messing up your data structures, in a world of atomic data-driven robots and asynchronous feeds.

amelius4• 11 years ago

Hmm, I don't really get it yet, but "single source of truth" sounds to me like it needs extra round-trips to this "source of truth" in order to update the display (even if the updates come from the client itself). Is that correct?

elierotenberg9• 11 years ago

You're right, although its fairly easy to implement optimistic updates on top of this. Since patch objects are reversible, you can use a simple logical clock and an undo/redo stack to perform optimistic updates locally, and undo them/redo them when the remote SSoT sends updates.

amelius4• 11 years ago

Well, it could be me, but that doesn't sound very simple. Isn't that something the framework should take care of?

Also, what happens when simultaneously transmitted actions/patches are in conflict?

zzzaim• 11 years ago

Alas, I've been working on something like this for the last few weeks on and off, especially the "patching" of immutable data, my solution was to use jiff[1], which generates JSON Patch operations[2], which is then applied to the Immutable object using immpatch[3] (a lib I wrote).

A cursory look at Remutable seems it to be the better and faster option :)

[1] https://github.com/cujojs/jiff

[2] https://tools.ietf.org/html/rfc6902

[3] https://github.com/zaim/immpatch

(edited formatting)

elierotenberg9• 11 years ago

That was nearly exactly my initial approach! :)

However I realized actual diffing was really slow, especially for large collections, as it involves scanning the entire structure for changes, when in fact all changes could simply be flagged upon mutation. Conceptually, it performs diffing (ie. it gives you a diff object), its just much, much faster!

rattray3• 11 years ago

slightly offtopic question; what are all those `should.be.exactly(whatever)` calls in the Remutable readme?

elierotenberg9• 11 years ago

Its shouldJS. An assertion lib with a fun chainable DSL. Basically whatever.should.predicate desugars to assert(predicate(whatever)). See https://github.com/shouldjs/should.js :)

verroq• 11 years ago

Synchronising state over the wire? Surely this is a solved problem in games development already - serialise your state into a consistent binary form, send over the binary diff of the previous state (AKA delta compression), deserialise on client side.

rattray3• 11 years ago

Remutable looks really impressive. I'm a little leery of the rest of it, though, as it means my backend is no longer a simple REST API. I'm also rather nonplussed that the full-stack Flux doesn't include optimistic display with loading icon, error state, and rollback handling out of the box. It doesn't even seem like an easy thing to do manually with this setup, though I haven't played with it enough to know.

Few minor questions: it looks like a base Remutable must be an Immutable.Map(). Can child items be any of the Immutable datatypes? Do we get updateIn() and similar?

elierotenberg9• 11 years ago

Thanks for your feedback. Several other comments point in the same direction: automatic (or assisted) handling of optimistic updates. I think this is actually relatively easy, since there are clear hooks in the Nexus Flux API to implement dispatch-time and update-time behaviour, but I will definitely provide an example, or even better, an implementation of this that just works out of the box.

rattray3• 11 years ago

Awesome! Excited to see how this goes, especially with a bigger example!

drabinowitz• 11 years ago

This is a really cool implementation! I'm wondering if it wouldn't be possible to handle optimistic updates by submitting a request to the server and then submitting an optimistic response from the client to itself. Then, when the server comes back, you hook into the already present optimistic response and just confirm or deny it based on the response from the server. That way you keep as little state on the client as possible.

Rapzid• 11 years ago

The best example of single source of truth may be chat, but I believe chat is the best example of a replicated log. This throw me off a bit while reading the article :| Left it, came back to the comments later to see people talking about OT and realized I missed something. Just an obesevation.

mandeepj• 11 years ago

We will rather record the mutations (or more accurately, the transitions between immutable states), and just replay them on the client.

Few questions -

How can we replay all the mutations?

How about instead of replaying, if we just show the last mutation? Will it not be the same thing?

I think this flux architecture will work best only in chatting applications. This approach is also making the app itself very chatter so performance concerns in case you do not have strong machines.

I think on form pages this flux approach may not be required.

mikaelemtinger• 11 years ago

Take a look at swarm.js for a fantastic way to keep stores in sync - even works in offline. Matrix.io could also be an interesting option for this, but focuses more on communication than shared data.

Flux over the Wire using Websockets and Immutable JS

Related Stories