Announcing Firepad — Our Open Source Collaborative Text Editor

https://www.firebase.com/blog/2013-04-09-firepad-open-source-realtime-collaborative-editor.html

by avolcano • 13 years ago

148 62 13 years ago

karl_gluck• 13 years ago

Very cool technology! I really appreciate your decision to allow us to just give it a try without having to set up an account or log in.

gnuvince• 13 years ago

And it's web based, and thus I will never use it.

macspoofing8• 13 years ago

That's a strange restriction you put on yourself? Today you have to have a good reason for your product to NOT be web-based.

yuchi5• 13 years ago

I'm interested too. Is the browser environment which bugs you or the web "platform"? (you know, HTML and brothers)

lutusp9• 13 years ago

> Today you have to have a good reason for your product to NOT be web-based.

For a document that the visitor cannot afford to lose, or for anything in which privacy, security or personal control are issues, a Web-based document is obviously a mistake. Why do you think the big players in cloud-based and Web-based storage and applications are having such a hard time getting people to adopt them?

The answer is obvious -- the drawbacks greatly outweigh the advantages. The risk of losing sensitive content or having it be compromised or stolen is too great.

Even games, a less serious endeavor, suffer when they adopt a Web-based approach. Look at the brouhaha that followed from making the most recent SimCity version work online-only:

http://www.gamesindustry.biz/articles/2013-03-15-ea-defends-...

A quote:

------------------------------------

"So, could we have built a subset offline mode? Yes," Bradshaw admitted. "But we rejected that idea because it didn't fit with our vision...The SimCity we delivered captures the magic of its heritage but catches up with ever-improving technology."

A number of upset fans in the comments section to Bradshaw's update were not assuaged by the explanation. As one user going by the handle klymen wrote, "With all due respect, to write an article about why the new SimCity has to be always online and not to mention DRM or the anti-piracy measure even once, is dishonest and downright disrespectful. It seems like that you don't think very highly of your audience. Yes, the DRM issue is a sensitive topic, but to avoid it, not to mention it as one of the reason why SimCity is always online, shows absolute disconnect and contempt for your fans. We're not stupid."

------------------------------------

The above is just about a game, not a business transaction or potentially sensitive communication.

As to data loss, consider incidents like this:

Title: Amazon's Cloud Crash Disaster Permanently Destroyed Many Customers' Data

Link: http://articles.businessinsider.com/2011-04-28/tech/29958976...

Quote: "In addition to taking down the sites of dozens of high-profile companies for hours (and, in some cases, days), Amazon's huge EC2 cloud services crash permanently destroyed some data.

The data loss was apparently small relative to the total data stored, but anyone who runs a web site can immediately understand how terrifying a prospect any data loss is."

In conclusion, and to second the SimCity gamer's quote above, we're not stupid.

toddmorey2• 13 years ago

One interesting and important bit about how Firebase works is that all write operations to the data happen locally, and then they are synced back to Firebase (and to other connected clients) on a best-effort basis.

That approach is designed to prevent network lag / connectivity issues from impacting the responsiveness of the application. Advancements like that are starting to make newer web apps feel more and more like their desktop counterparts.

yesimahuman• 13 years ago

While your preferences are perfectly valid, I think it's safe to say web based editors and tools are very popular to a large amount of people. For example, everyone I know loves Google Docs and few of them use MS Office. That's real and shouldn't be discounted so quickly.

yuchi5• 13 years ago

I'm happy to see [Substance](http://substance.io) work (Tim's OT) to spread to other related project!

mikelehen7• 13 years ago

Yes! His work is great. Made for a great starting point for building Firepad. :-)

macspoofing8• 13 years ago

Is it based on etherpad?

yuchi5• 13 years ago

No it's based on [Substance](http://substance.io) OT library, but uses CodeMirror instead of the [Surface](http://interior.substance.io/modules/surface.html)

(Substance Team member here)

unwind• 13 years ago

By the way, unfortunately HN doesn't do Markdown. Links should just be links, no separate display text.

yuvipanda• 13 years ago

Sweet! This will be fun to integrate into Wikipedia for editing...

/me plots

nahimn• 13 years ago

The homepage turned into a giant chat room

mbrock2• 13 years ago

Does this depend on proprietary server tech hosted by Firebase?

toddmorey2• 13 years ago

Yes, it does.

"Firepad has no server dependencies and instead relies on Firebase [hosted service] for real-time data synchronization."

So the front-end editor portion of it is open source, but the data sync between connected clients happens via Firebase.

You can find more info at the bottom of this page: http://www.firepad.io/

mbrock2• 13 years ago

Thanks. Seems cool, I just wasn't sure, and phrasing like "Firepad has no server dependencies" had me picturing some advanced WebRTC-based peer syncing or something, which is pretty far-fetched but you never know these days...

prg318• 13 years ago

This is a really neat idea! It would be nice to see the editor widget be re-sizable (like an HTML text area) so that you don't have to deal with scroll bars for larger portions of text.

Pretty impressive though!

mikelehen7• 13 years ago

This should be a pretty easy thing to add, and it's all open source... :-)

joeblau• 13 years ago

You guys are awesome! I love your hot sauce!

Johnyma22• 13 years ago

Check out the Etherpad plugins, stuff like showing cursor position is available http://beta.etherpad.org has it enabled for example.. Etherpad is extendable and does support code editing with syntax highlighting, you just have to jump into /admin/plugins and click "install" on whatever plugin you want :)

glavata• 13 years ago

Loved etherpad before, and still do :)

civilian• 13 years ago

I think there's a bug with people's text cursors writing over eachother.

http://imgur.com/1IvMccd

mikelehen7• 13 years ago

Can you provide more detail? Here or firepad@firebase.com or as a github issue? From the screenshot it's not immediately obvious to me what went wrong.

People are certainly allowed to write over each other (e.g click in the middle of your sentence and write something). If that's not what happened, let me know

saurik4• 13 years ago

So, as with most things built with Firebase, I have to ask how the security works. Last I talked to the Firebase team, they were building expression-only rules for managing server-side validation. This allows you to express some reasonable subset of permissions, but not all possible ones.

In this case, the OT history for the document (required to synchronize clients) is stored in Firebase (with each op being a separate object with a massive ID, I imagine this will become awkward with large numbers of old documents, but I digress). Additionally, snapshots are occasionally stored.

Rather better than previous offerings I've seen using Firebase, this demonstration has been put together to solve the first few obvious problems: I am not allowed to dump the set of documents[1], nor am I allowed to arbitrarily corrupt the database by deleting random objects[2]. So far, so good.

[1] https://news.ycombinator.com/item?id=4780495

[2] https://news.ycombinator.com/item?id=3824775

A set of example security rules for Firepad is actually provided as part of the GitHub project, so we can do some analysis of the kinds of checks we will need to bypass in order to break this particular demo ;P. (Of course, this just makes us faster, it isn't what makes this possible.)

https://github.com/firebase/firepad/blob/master/examples/sec...

Reading these, it turns out that the only verification that is being done on the snapshots is that 1) they look reasonably valid (have the correct set of fields) and 2) they have the correct author field associated with them (as in, the same one that is used on the history revision item).

However, it doesn't do any consistency checks on the data itself. It doesn't even verify that the snapshot we are uploading is different than the one currently on the server, so the problem of corrupting the state is really easy: we just need to pull the current snapshot and modify its data.

The only pain we could run into is that it could also verify that the author of the snapshot is the current user; but that doesn't help: all we need to do is to make a quick edit to the document and then use our new revision (which we legitimately own) to build our new corrupted snapshot.

That said, while that check is present in the example rules on GitHub, that check isn't actually used in the deployed copy of Firepad on this server as this server is entirely anonymous and thereby none of the users have any auth information at all... we can just pretend to be other users.

For users who wish to follow along at home, you just need to have node.js installed, and then do "npm install firebase". You can then use the following script to destroy any document you want: you just need to set the "room" variable to the ID # of the document you want to modify.

    #!/usr/bin/nodejs
    var Firebase = require('firebase');

    // parseInt(window.location.hash.substring(1))
    var room = 44;

    var shard = room % 15;
    var db = new Firebase('https://firebase-firepad' +
        shard + '.firebaseio.com/' + room);

    var check = db.child('checkpoint');
    check.once('value', function (value) {
        value = value.val();
        check.set({
            a: value.a,
            id: value.id,
            o: [''], // random data would be better
            // but I'm both lazy and busy today ;P
        });
    });

When the client then restores this snapshot and attempts to play back the resulting history to "catch up" it will instead end up outputting tons of errors to the console, as the operations stored in the history will be referring to document positions that no longer exist or are different.

    Firepad: Invalid operation. https://firebase-firepad14.firebaseio.com/44 C2GK

The client then has two options: it can either skip the history entry or it can decide the entire document is corrupt. In this case, it seems that Firebase believed the better of the two options was to simply skip these operations: new clients then manage to resync their state.

But, existing clients now have desynchronized state, so all operations that are being synchronized live between the various clients on either side of this split (ones that started from this snapshot, and ones that pre-existed it) are going to result in this error; that didn't really help.

To be very explicit for a second: this is a different scenario than just "well, its an anonymous system, so anyone can delete the data": we didn't just go in and delete the data in the document, we actually corrupted the state of the document, rendering further attempts to edit it useless.

It is currently my belief that Firebase's security rules system is simply not powerful enough to secure an OT-based text editor, whether or not it uses snapshots (at least assuming it supports offline; there might be tricks you can play if all users are required to be online at all times).

(edit: I am finding it interesting that my previous security analyses of Firebase projects, combined with example code, had been voted up quite high, and this one has now been downvoted to 0. I wonder if people just don't care as much about security anymore? Is it because it is open source? Is the Firebase team themselves going around voting down? ;P)

mikelehen7• 13 years ago

Hey Saurik,

Thanks for the thorough and correct analysis as usual. :-)

The key things I would point out are that:

1) The checkpointing is an optimization. You could either remove it (which will hurt initial load time) or delegate it to trusted server code (which will be very lightweight; you could run hundreds of rooms off of a tiny EC2 instance or whatever).

2) In general, the whole point of collaborative editing is that you trust your collaborators. If they're malicious, they can already cause mayhem on your editing experience with constant edits, obscene content, etc.

saurik4• 13 years ago

1) I do not need to modify the snapshots: I can simply inject corrupt history state. The problem is that the server in these kinds of systems is normally supposed to be running the OT algorithm in order to verify that the data being uploaded and stored as part of the permanent document record is valid.

(edit: That said, you would be hard-pressed to do this kind of OT-based text editor without the snapshots, especially with the very large number of separate objects being used to store the history state. While looking into how you were storing the data for this in Firebase, I had tried resetting the snapshot for a document to A0=[''], and attempting to open the document then bogged down so far that I wasn't certain if it would even recover; this problem will just get worse as the document ages... that only had a few hours of history behind it: a real document would just be screwed.)

2) There is a difference between trusting your collaborators with your data, and trusting your collaborators with your program state. Yes: if I am collaborating with people using Google Docs, the other people can ctrl-a+del all of the "data". However, they shouldn't be able to break the editor itself :(.

(edit:) As an example of this, if you remove the snapshots from the mechanism, then you can make the argument that "well, if I validate and ignore all history state that is invalid, this isn't a problem: I just need to keep the clients in sync and skipping things that are broken is valid" (so I'm happily willing to cede that my having added "whether or not it uses snapshots" was going too far). I personally think that this is still a problem, as the document record is still corrupt.

However, with the snapshots certainly, it isn't that I'm able to delete the data: it is that I'm able to break the synchronization system itself. I can setup situations where one party thinks they are editing the document, but their edits are being discarded. I can make it so that one person sees a document different than other people. In addition to doing all of this, I can make it nearly impossible to figure out who's doing it and to fix the situation. This is simply not the same problem as "well, you can always just ctrl-a+del the data from the document".

flywheel• 13 years ago

Security? Client-side encryption.

batgaijin• 13 years ago

http://accumulo.apache.org/notable_features.html

cell labels

Announcing Firepad — Our Open Source Collaborative Text Editor

Related Stories