The Basis story: an accidental sync service in Clojure for iOS devices

http://sapient-pair.com/blog/2014/04/21/basis-story/

by mphillips • 12 years ago

51 9 12 years ago

notjosh• 12 years ago

And in fact Couch DB would have been close to perfect … if I could have run a Couch DB server on the client, which would is an insane thing to contemplate.

Couchbase Lite (http://developer.couchbase.com/mobile/develop/guides/couchba...) does exactly this - except it's not an actual Couch instance, but rather an SQLite database, and a client that speaks the necessary Couch protocols for sync/replication. Not so insane, and (as far as I've seen so far..besides git!) the only tool that really handles sync well across platforms.

edit: Ah, CBL (and Datomic) are mentioned in part 3 of the article. All is well :)

orandolabs• 12 years ago

EnduroSync (https://orandolabs.com) is a new service for syncing object stores. It has clients for iOS and Android now, with several others right around the corner.

Data is modeled with objects. The object store works offline and online. If you sign up for the service, then you get syncing.

EnduroSync also has a very nice permission model, enabling sharing of object stores in a variety of ways (per user, per app, ...).

bsaul2• 12 years ago

Seems to me like the merge part of the algorithm is a bit overkill :

Generate a uuid for each item on your list upon creation ( on the client side), then sync the actions on the item ( create, update, delete). There's never the need to "merge" on the server. All you need is to create a serial "history" of all the actions on the server side, and replay them in that order on clients. You do that by locking on the DB when a client wants to send new actions.

PS : i'm currently coding exactly that. So it's more a way for me to share my ideas rather than criticize.

mphillips3• 12 years ago

I'd be really interested to hear more about your approach. Maybe you’d write it up?

I did consider simply storing and replaying actions, but realised I'd still want a way to merge them: consider a series of actions like: "add item", "update item", "add item", ...<lots more changes>..., "delete items 1 through 20". If there’s no merge, a lot of redundant actions will end up building up in the offline case.

There are a lot of ways to approach this (I’m reading Brent Simmon’s adventures (http://inessential.com/2014/05/24/vesper_sync_diary_follow-u...) with interest at the moment).

bsaul2• 12 years ago

Yes i realized that it may be a bit long if one needs to sync everything from the start. You may have to build a "compacted" history where all the items that will later be removed are excluded. The thing is it's more of a compression of serial actions than a merge between forked branches ( which makes things easier).

habosa• 12 years ago

This is a great blog series, I'm really enjoying the read. Did you ever consider using an implementation of the Operational Transform (OT) algorithm? Google famously used this for Google Docs, which asynchronously merges a lot of small changes from many clients into a single, real-time document. Sounds a lot like what you built.

mphillips3• 12 years ago

OT is not so much an algorithm as an approach: a way of thinking about sensibly merging parallel sequences of change operations against a data model of some kind.

It’s a really interesting field, going back a long way: the canonical paper that the Google Wave (RIP) people referenced is nearly 20 years old (http://dl.acm.org/citation.cfm?doid=215585.215706). This is another, more recent, good paper: http://hal.archives-ouvertes.fr/docs/00/13/97/04/PDF/OsterIC....

In fact, Basis does use an OT approach in the delta merge algorithm: it breaks the deltas down into a canonical series of “micro operations”, merges them piecewise, then generates a new delta.

Thanks for the kudos!

jeffmax• 12 years ago

Just curious, is the basis client/server available going to be made available? Thanks for the post.

mphillips3• 12 years ago

In the short term, no. I’d like to think that the (somewhat unplanned) overhead of developing Basis will be paid off in future as a commercial advantage for my apps — I guess we’ll see whether that pans out ;)

In the longer term, yes absolutely I’d like to make it open source. In the meantime, I’m happy to share some of the lessons I’ve learnt along the way.

The Basis story: an accidental sync service in Clojure for iOS devices

Related Stories