Ex-Googlers Building Cloud Software That’s Almost Impossible to Take Down

http://www.wired.com/2014/07/cockroachdb/?r

by eridal • 12 years ago

92 42 12 years ago

na853• 12 years ago

Personally I don't think takedowns are the biggest threat facing cloud users. By far the larger threat is having your cloud data harvested and used against you by adversaries such as advertising firms and three-letter agencies.

xionon• 12 years ago

That's not what this article is about at all. The technology described in this article is a distributed database.

Github project: https://github.com/cockroachdb/cockroach/

lern_too_spel• 12 years ago

Neither of those are realistic problems. (How and why would an advertising firm use your data against you, anyway?)

Servers go down every day, and data centers go down every month. This project solves a real problem.

na853• 12 years ago

Targeting me with ads is absolutely an adversarial situation. I and most of the population would rather not see ads at all. I do not want advertising agencies building a profile of me so they can sell me stuff. Do you?

Maybe you'll come back with a completely-bogus, typical hackernews-ish response, deluding yourself, saying that yes you like it when advertisers target you based on the data they collect about you because of reasons X Y and Z, something something "better for me". Ads targeted at you do not benefit you in any way unless you work for an advertising firm.

And are you seriously arguing that NSA et al do not peek into cloud storage? Have you been living under a rock?

jflowers45• 12 years ago

I like the name: "CockroachDB" - and find it interesting that a bunch of the guys are working on it while they also work at Square - and also that it's supposedly based off a Google research paper for "Spanner" which I hadn't previously read about. Lot of good nuggets here.

mathattack2• 12 years ago

I see a headline like that thinking someone will take it as a dare. Of course the source is prone to overhype, so it's worth a grain (or twenty) of salt.

jahewson2• 12 years ago

You got your idiom backwards: more salt = more meat and substance in the story, less salt = less substance.

jcbrand• 12 years ago

That idiom doesn't work the way you think it works :) https://en.wikipedia.org/wiki/Grain_of_salt

curiousDog• 12 years ago

Spanner truly is remarkable. I've always wondered why google hasn't opened it up!

toast0• 12 years ago

The article mentions that Spanner has dependencies on several other Google projects. Assuming Google wants to open it up (more than publishing a paper), they would need to stub out all the dependencies first, which is a major effort.

dekhn• 12 years ago

It would make more sense to host it as a cloud API, or perhaps use it as a backend for Datastore: https://developers.google.com/datastore/ however according to this page: http://googledevelopers.blogspot.com/2013/05/get-started-wit...

it seems like it uses Megastore http://www.cidrdb.org/cidr2011/Papers/CIDR11_Paper32.pdf

outside1234• 12 years ago

Google does open source when it benefits them. Giving everyone else Spanner doesn't benefit them.

cle• 12 years ago

It wouldn't benefit anyone. Spanner is most likely tightly coupled with their internal services and infrastructure.

If you want to know how spanner works, read the paper. If you want to use it yourself, you'll need to build it on top of your own infrastructure, just like Google did.

nostrademons• 12 years ago

Or more when it doesn't harm them. Google's open-sourced several projects (Protocol Buffers, Closure, Gumbo) that didn't directly benefit Google, but also don't harm them or give away highly-valuable IP either.

eloff• 12 years ago

So what's special about this database? What properties does it have that are superior to the currently available alternatives? This article is all hype and no substance.

EDIT: GitHub has more information. It's a scalable ordered key-value store (think a distributed version of Berkeley db.) Storage is based on RocksDB (a variant of LevelDB) and consensus is achieved using Raft. The database is written in Go. It's meant to "tolerate disk, machine, rack, and even datacenter failures with minimal latency disruption and no manual intervention."

What's not clear at all is where it sits CAP wise. It says it's available and strongly consistent. Which would be CA, which is not an option (especially for something claiming to be failure tolerant.) It must either sacrifice write availability or consistency in the event of a partition. No idea which way it leans.

Goranek2• 12 years ago

I think they're trying to create open source Spanner. Here is Google scientific paper about Spanner. http://static.googleusercontent.com/media/research.google.co...

boomzilla• 12 years ago

In my opinion, one of the most profound ideas in Spanner is the introduction of TrueTime API and the guarantee provided in the implementation. I wonder if this project is going to have something similar?

peterwwillis3• 12 years ago

Cockroach provides snapshot isolation (SI) and serializable snapshot isolation (SSI) semantics, allowing externally consistent, lock-free reads and writes--both from an historical snapshot timestamp and from the current wall clock time. SI provides lock-free reads and writes but still allows write skew. SSI eliminates write skew, but introduces a performance hit in the case of a contentious system. SSI is the default isolation; clients must consciously decide to trade correctness for performance. Cockroach implements a limited form of linearalizability, providing ordering for any observer or chain of observers.

Either you use a strongly consistent mode that can have poor performance under contention, or a strongly consistent mode that will have good performance under contention but lots of failed transactions. So you get to decide on performance vs availability.

To answer your question, it sacrifices availability, not consistency. It's an MVCC, after all.

jbooth3• 12 years ago

Why are they using RocksDB rather than LMDB?

If it's based on Raft, then it sacrifices availability if there aren't a quorum of nodes online.

peterwwillis3• 12 years ago

Probably because 1. LMDB is limited to logical address space, 2. it has one big global lock, 3. It's a B-Tree, and both of those contribute to the fact that 4. LMDB is a read-oriented database [performance wise]. I would conjecture that Rocks could also be 'more easily embeddable', but i'm talking out my ass there :)

And yeah, you kind of have to sacrifice availability if you want to stay consistent in the face of write skew...

kjksf• 12 years ago

I benchmarked lmdb vs. leveldb once and on a write-heavy workload leveldb destroys lmdb (think 10x better perf).

The author of LMDB makes pretty bold performance claims and people are too eager to believe them.

You shouldn't propagate those claims unless you've done benchmarking to verify them.

Goranek2• 12 years ago

Just noticed that it's written in Go. Lately more and more databases are written in Go.

grogenaut• 12 years ago

Why, when I see a wired article title, do I immediately assume that it is going to be completely untrue?

Ex-Googlers Building Cloud Software That’s Almost Impossible to Take Down

Related Stories