Derby v0.5.0 – The first OT-powered realtime web app framework

http://blog.derbyjs.com/2013/06/04/derby-v0-dot-5-0/

by nateps • 13 years ago

154 55 13 years ago

blog.derbyjs.com

I'm simultaneously disappointed and relieved that this is not a web framework powered by Operating Thetans.

However, I will have to inform Mr. Cruise that his e-meter will not, in the near future, be able to predict his website's uptime.

Ahh, the tribulations of modern development.

josephg2• 13 years ago

OT stands for Operational Transformation, which is the technology underpinning Google Docs, Google Wave, Etherpad, ShareJS etc. http://en.wikipedia.org/wiki/Operational_transformation

obviouslygreen2• 13 years ago

I appreciate the information, but my comment was entirely an off-topic jab at Scientology. Where applicable, I apologize for the lack of sarcasm indicators. :)

gnok• 13 years ago

For anyone else wondering what OT is: http://en.wikipedia.org/wiki/Operational_transformation

The gist of it seems to be that its a set of data models that make handling concurrency control and conflict resolution easier in the context of a single blob of data being edited by multiple users simultaneously (similar to Google/Apache Wave)

u041578• 13 years ago

Is anyone aware if Racer uses OT to sync the model as a whole (OT on a tree structure), or if it is used on a per-leaf basis to merge operations on values? Firebase seems to have solved this issue (with their Firepad project) by using their tree/json sync engine to transmit the OT operation primitives.

nateps4• 13 years ago

OT is done at the document level. Documents are JSON objects of arbitrary complexity. Any mutation of the JSON object can be composed against any other mutation of the JSON object.

Firepad doesn't actually do OT of JSON. In Firepad, a string OT algorithm is used to merge text edits, and the metadata describing rich text is separately managed as span objects, which get modified in accordance with text changes.

Firebase provides conflict resolution of JSON structures using their Transaction primitive, which allows them to create a Software Transactional Memory used to update the journal that powers the text OT in Firepad. ShareJS inside of Derby uses a similar approach to allow multiple servers to access the journal, but it is done in Redis with Lua scripting.

nateps4• 13 years ago

Definitely try playing the collaborative story game demo explained in the post: http://stories.derbyjs.com/

switz2• 13 years ago

I currently work on one of the most trafficked production Derby applications[0] and I must say that this is a very welcome change. Derby's creators have time and time again proven that they care about the nodejs ecosystem. By funding ShareJS's development, they're ensuring that an unbelievable tool (simplified OT) will be sticking around for quite some time. Writing code with Derby has been far and away the most pleasant development experience I have ever had.

If there's been one issue with Derby so far, it's been Racer. The data sync-ing engine leaks memory and is largely unstable. We've suffered far too much downtime due to these issues. Moving to a fresh backend with ShareJS enables Nate, Brian, Joseph and the rest of the team to rework the scope of the framework with some of the larger goals in mind – things like horizontal scaling and powerful conflict resolution. We have yet to upgrade to 0.5 (the upgrade guide was literally just released), but it appears to be a much more stable foundation on which to build a "real-time" web application. I'm sure we'll have to put up with bugs for quite a few weeks, but this is a powerful step towards reaching 1.0. Thanks guys!

[0] https://habitrpg.com (source: http://github.com/lefnire/habitrpg)

aioprisan• 13 years ago

how many page views/month and uniques? is there a list of most trafficked Derby apps out there?

adamdicarlo2• 13 years ago

That is a really cool site, and a really great idea! Nice work.

I'm curious why a "real-time" framework is needed for this type of app, though -- why not Backbone with REST? Is it so that you can view others' pages to see their habits/dailies/todos get updated in real-time...?

wizonesolutions• 13 years ago

As far as I know, this was a project that wasn't expected to get this big — more about learning Derby for the developer. And then it got big :)

tl;dr: No big reason. He likely agrees with you.

gizzlon• 13 years ago

Wow, that looks really cool :)

yefim• 13 years ago

How does Derby stand up to Meteor? I'm really glad Derby has updated but Meteor (at least to me) seems miles ahead of the curve right now.

josephg2• 13 years ago

Meteor is more mature, but it doesn't do server side rendering or operational transform for document updates. As far as I know, meteor is also limited to a single server. Derby now supports scaling over multiple processes on multiple machines. On http://sink.derbyjs.com/ I get DOMContentLoaded (including all the page content) in 70ms.

steveklabnik3• 13 years ago

I thought Meteor added server-side rendering a while ago? http://www.meteor.com/blog/2012/08/09/search-engine-optimiza...

Or do you mean something else? I'll admit to only really watching all of this from afar, I could have my terminology backwards.

monkmartinez• 13 years ago

Congrats and keep pushing forward! Although I have really focused on Python in the recent months, I keep an eye on the Node landscape... Derby always looked promising, along with half a dozen or so frameworks as well.

dgesang2• 13 years ago

Am I the only one who would look up a possible product name before using it? There already is a (well known) software from ASF named 'Derby': http://db.apache.org/derby

Why would you start such a confusion? Why would you want to fight with that software for page ranking? Why does this mostly happen to JS technologies?

wheaties• 13 years ago

No you aren't. However this happens to many projects/libraries/languages. There's Elixir the language but also the python library for example.

marshray4• 13 years ago

> Why would you start such a confusion?

There are far more software project starts than good names out there. Finding a good name that is also not used by any other project can take a bunch of time and energy away from your code.

> Why would you want to fight with that software for page ranking?

The vast majority of software projects never make it far enough for the name to matter. You don't know which category your project will end up in when you start it.

> Why does this mostly happen to JS technologies?

Because JS is popular and makes it easy to start new ad-hoc code libraries. There are also a lot of half-baked libraries out there that will waste your time. So many developers end up making their own.

examancer• 13 years ago

> You don't know which category your project will end up in when you start it.

So true. I don't know how many times I've started building a native audio streaming app and all-of-a-sudden I realize I've really built a web-based real time strategy game.

Your points are valid... just thought that one statement was a bit absurd and quite humorous.

antoko• 13 years ago

the actual quote was

>The vast majority of software projects never make it far enough for the name to matter. You don't know which category your project will end up in when you start it.

The "categories" are:

1) Projects where the name will matter because the project is successful

and

2) Projects where the name won't matter because the project dies before reaching critical mass.

EDIT: apologies, this was supposed to be a reply to examancer

dgesang2• 13 years ago

  There are far more software project starts than good names out there.

I doubt that. Define 'good name'. Also, for open source software it does not matter that much what it is named. Who would care if Python was named e.g. Snake or Juice or Foo? As long as it is somewhat unique and catchy it should be fine, shouldn't it?

  You don't know which category your project will end up in when you start it.

Exactly, you do not know it. If your product is successful you want it to be unique. Just imagine the number of wrong tags on SE. And if you product is not successful it is even worse to add unnecessary confusion.

  Because JS is popular

Surprisingly and unfortunately, yes. But other technologies have had their time as well, e.g. I can't remember any ambiguous naming in Java. Anyone?

  There are also a lot of half-baked libraries out there that will waste your time. So many developers end up making their own.

Ok, why not just name it jQuery then? Hardly an excuse for poor naming choice.

macspoofing• 13 years ago

My thoughts exactly. I thought the Apache Derby project was dabbling in web-frameworks. Poor poor choice of name.

PommeDeTerre• 13 years ago

It's too bad that they didn't notice, or didn't bother to find out, that the name "Derby" has already been used for years by an open source relational database project: http://db.apache.org/derby/

d0c5• 13 years ago

O.k. I live in Derby and that is also slang for a blowjob in America and a horse race. Plus much more I don't care to mention because the others already did. Do you have to do this?

adamdicarlo2• 13 years ago

Derby is a slang for oral sex in the US? I've lived in the US all my life and have never heard that. Can't find that definition on urbandictionary either (at least not the first page of results)....

dylanhassinger3• 13 years ago

This is cool but I'm on the @SailsJS bandwagon -

http://sailsjs.org

However Sails wants to add OT's, and add-ons that do Meteor-style syncing. I know this is wishful thinking, but maybe Sails and Derby could find some way to merge, or otherwise work together?

Here's the Sails group - https://groups.google.com/forum/#!forum/sailsjs

mr_luc• 13 years ago

Wait - I just looked at the SailsJS docs, and it looks like Sails is like Tower -- that is to say, a Rails-style framework with better WebSocket support (really, more like Merb+DataMapper, because of the easy json api + agnostic ORM).

But if that'd be the case, it's a very different thing from Derby/Meteor/SocketStream, where so much code is shared from client+server that the apps can often run offline and then sync data.

dylanhassinger3• 13 years ago

Yes, they have different goals - by design, Sails is frontend-agnostic. But we want to add a layer that supports the "template syncing" concept, however it will be an add-on on top of Sails core.

Aside from that, Sails brings a lot to the table (awesome ORM, asset pipeline, built-in API, etc). All we need are OTs and things'll be rockin :D

hardwaresofton2• 13 years ago

Just looked at sails, and I'm not going to lie, it looks amazing. I had no idea the nodeJS ecosystem was like that...

It essentially does the work of a Django + Tastypie with... no work.

dylanhassinger3• 13 years ago

yeah its pretty cool huh! :) plus its a cool community. We just need to get OT support

victorhooi• 13 years ago

Hi,

This is very awesome stuff! =). The real-time syncing is definitely awesome, can't wait to try this in some of my apps.

Good props on the video tutorial as well.

One thing sorely lacking form DerbyJS is tutorials, or any kind of help for beginners - would be awesome to see them release more of these.

Cheers, Victor

mambodog• 13 years ago

I've been following both ShareJS and Derby because, while I believe that the future of web application development is in realtime 'holy grail' frameworks, I very much dislike many aspects of Meteor, such as disregarding npm, the lack of proper server-side rendering (without the overhead of DOM) and the lack of importance placed on conflict resolution between client writes.

I really want Derby, or something like it (eg AirBnB's rendr combined with realtime, OT models and collections) to succeed, however it really seems like Derby needs more community support and evangelism behind it, because I can't help but get the impression that most of the community focus in this space currently is on Meteor. As far as I am concerned, Meteor is the PHP of realtime 'holy grail' frameworks, as enjoys from just the same kind of popularity through 'worse is better' (where 'worse' is for the reasons I mentioned above). However, without a palpable, visible sense of community enthusiasm, a lot of fence-sitters such as myself are less likely to jump on board.

Also guys please do something about racerjs.com, it doesn't inspire confidence in the project when it is continuously down.

trungonnews2• 13 years ago

Very cool. But Derby still needs to support IE in order to be taken seriously.

Otherwise look into AirBnB Rendr and Yahoo Mojito if you want to support both client side and server side rendering.

trungonnews2• 13 years ago

Also, Derby needs to decouple from MongoDB.

nateps4• 13 years ago

While we've only written a MongoDB plugin so far, we have designed specifically to be work with any similar datastore. Contributions of other database plugins are welcome. The Mongo implementation is here for reference: https://github.com/share/livedb-mongo

madmax108• 13 years ago

Noob question here, but how is OT different from using, say, websockets?

I get that it's cool, but the question is why? What are the advantages??

Void_• 13 years ago

WebSockets is technology used for real-time updates.

OT is an algorithm that allows sending only changed parts of a document and resolving conflicts made if multiple users are editing the same thing.

flog• 13 years ago

How long till we can have an OT PaaS?

I love OT RT editing, but want to use it for small parts of my products and don't want the hassle of setting up servers.

If you're looking for your next idea; please, take that and run with it :)

marshray4• 13 years ago

The hard part is the conflict resolution. CRaaS FTW.

spicavigo• 13 years ago

I am using Firebase for RT editing on http://codebunk.com.

boothead• 13 years ago

Can any one who's looked at event sourcing comment on the relationship between ES (which is basically a fold over a sequence of events onto a state object) and OT. Superficially it would seem that they are very similar, with OT perhaps being a way to combine the events before applying them to the state.

The reason I ask is that I'm thinking of the feasibility of modelling a disconnected client that continues working from some state. Could you think of this as collecting its list of edits and then applying them on top of the state of the world when that client re-connects (sort of like a git rebase)?

dnr• 13 years ago

I've been working on a real-time collaborative app that does just that: the client collects a list of edits and tries to push them to the server. It always displays the state as the last known accepted state with the local edits replayed on top. The server accepts edits from clients, gives them a total order, and rebroadcasts them. It's intended to be used while connected, but the protocol has no problem dealing with arbitrary gaps in connectedness, so adding an official disconnected mode would be trivial.

Importantly, "edits" here are discrete and independent. They can conflict (in which case it just does last one wins), but they can't affect the meaning of other edits. This scheme couldn't be used for collaborative editing of a linear text document, since e.g. "insert 'a' at position 15" means something different depending on whether it comes before or after "delete positions 5-10".

That problem is what OT solves (by the transform function that lets you commute edits while preserving their meaning). So, yes, that kind of scheme is much simpler than OT, but less powerful. Whether you can use it depends on the semantics of your edits.

bribriinlondon2• 13 years ago

Nice work!

xyproto• 13 years ago

This isn't a first, a lot of discussions are "off topic"-powered.

Derby v0.5.0 – The first OT-powered realtime web app framework

Related Stories