Dynomite – Making Non-Distributed Databases Distributed

http://techblog.netflix.com/2014/11/introducing-dynomite.html

by stefanwild • 12 years ago

185 60 12 years ago

techblog.netflix.com

jedberg3• 12 years ago

When I saw this internally a few months ago, I said that I haven't been more excited about a new project in a long time. I personally think this will solve a lot of scalability problems people have while letting them keep their favorite database.

stefanwild• 12 years ago

It looks very promising. I like the fact that it doesn't only work for AWS environments.

evantahler• 12 years ago

Are there any deployment notes/examples?

raxen• 12 years ago

not really, the best you are going to find so far: https://github.com/Netflix/dynomite

nrbafna• 12 years ago

Setup and Installation Wiki - https://github.com/Netflix/dynomite/wiki/Getting+Started#set...

void_star2• 12 years ago

It's a little unfortunate that consistency is only mentioned as a footnote. If this is being used internally at Netflix, how are they dealing with consistency issues in the face of failed or conflicting writes?

antirez3• 12 years ago

This currently looks like the Dynamo-alike sharding without the read-repair part making values eventually convergent in all the replicas. This latter feature is much needed, but they are already planning it apparently.

void_star2• 12 years ago

Except no mention of vector clocks, so it's unclear how one would detect or repair a conflict. Maybe that is an implementation detail that is intended to be ironed out later...

halayli2• 12 years ago

If it's based on Dynamo, then it's using vector clocks.

fizx2• 12 years ago

For projects with similar goals, see also Twitter's Gizzard (abandoned) and LinkedIn's Voldemort.

wowmsi• 12 years ago

Also Riak. However, one interesting aspect of Dynomite seems to be offer default support for clustering across multiple data centers with pluggable stores.

darkr• 12 years ago

Riak (enterprise version) does that, and has pluggable stores (albeit limited to in-memory, bitcask + eleveldb)

anonfunction• 12 years ago

Add Mashape's Astronode to the list: https://github.com/Mashape/Astronode-Broadcaster

jlafon2• 12 years ago

It's interesting that Netflix decided to write this rather than using Amazon's own DynamoDB. I wonder specifically if DynamoDB was too expensive (as I have found), or was there some other reason?

jetblackio• 12 years ago

This is completely different than DynamoDB. Essentially this is a framework for sharding or horizontally scaling multiple datastores (MySQL, Memcache, Redis, etc).

jlafon2• 12 years ago

That makes sense. I had only seen the Memcache & Redis datastores being used (and both being comparable to DynamoDB).

amelius• 12 years ago

Why would you use multiple datastores? Isn't it much more difficult to atomically store data this way? Assume a transaction succeeds in store 1, and fails in store 2, you'd need to roll back the transaction in store 1 after it has committed (!)

fizx2• 12 years ago

This is the sort of project you do to get cross-region replication on your existing data stores, many of which look more like PostgreSQL than KV stores.

grogers2• 12 years ago

Keep in mind that they define

DC := AWS region

Rack := AWS AZ

That should tell you that they need multi-region availability, something that DynamoDB will likely never provide.

ppearcy• 12 years ago

Any ideas if this supports Postgres?

Also, I noticed comments about an architecture document here: https://github.com/Netflix/dynomite

Anyone know where the doc they are referencing is located?

jedberg3• 12 years ago

It doesn't support Postgres yet but that is on the (longer term) roadmap.

However, since it is open source, contributions are welcome! :)

halayli2• 12 years ago

nitpick, the code check for NULLs at a low level but misses checking NULLs at higher levels like here:

https://github.com/Netflix/dynomite/blob/master/src/dyn_dnod...

personZ7• 12 years ago

Netflix always makes for an interesting case, but what always strikes me is how incredibly "trivial" their application really is: It's a simple application at a very large scale. This doesn't dismiss their accomplishments, of course, but often it does seem like Netflix spends an enormous engineering effort on fairly low value aspects of the service.

jrallison• 12 years ago

You're discounting the "very large scale" portion of your statement.

What may seem fairly low value from the outside or consumer perspective is likely of huge value internally to ensure you can always watch Netflix without even thinking about everything that has to happen to deliver as much video as they do to their customer's browsers.

themartorana• 12 years ago

Well, they make it seem trivial, while dealing with every watch and every pause, every rating updating complex recommendation engine algorithms. They deal with licensing and region awareness and proxying by ISP and running their own CDNs and...

And that's the magic.

To boot, they do it at enormous scale on a multitude of platforms all while their own destructive code (Chaos Monkey) runs around destroying servers.

It's true, some things like "stream me this" are maybe not rocket science, but at Netflix scale, everything becomes a bit more difficult.

jcampbell1• 12 years ago

Netflix is good at providing the service on any device, and having 100% uptime.

Their requirements are 100% availability, and an API's that never breaks, but consistency rarely matters.

The engineering effort is cool, but it is solving a problem that I don't have. The only thing that matters to netflix is they list the catalog somewhat correctly, and always deliver a stream.

I have a different problem, thus I am uninterested in their magic. If I had a similar problem or they signed my paycheck, I'd care.

brown9-2• 12 years ago

this seems like a pretty common fallacy for people to think about an organization from the outside

personZ7• 12 years ago

How is it a 'fallacy'? To be clear, this has nothing to do with the actual streaming (which does not happen from EC2) but instead are the basics like viewing history and categories. It absolutely is the simple part of the application, albeit, as mentioned, at a very large scale. Given that someone else mentioned it, does Netflix even bother with the whole recommendation algorithm thing anymore? My personal experience has been that it gives the most superficial of recommendations.

And if we're being honest, the Netflix app layer is kind of terrible. I could enumerate the problems, discoverability being king, but it is not the compelling part of the service.

jeangenie• 12 years ago

An idea's utility is often orthogonal to its complexity.

morgante• 12 years ago

Comments like this really annoy me.

What exactly is trivial about building a complicated multisource streaming video service with sophisticated recommendations (plus all the other "little" stuff like payments)?

Google is "trivial." It's just a box for doing text search on a graph database. Apple is "trivial." They just make pretty skins for commodity hardware & software. Etc.

When you consider 99% of everything in your field trivial, maybe you should consider refining your definition of trivial.

personZ7• 12 years ago

Comments that contrive absurd strawmen annoy me, even if they provide the author that righteous sense of striking down an imagined position.

chaz• 12 years ago

Netflix currently has 94 open engineering open reqs, plus another 22 data/analytics open reqs. Perusing through the list should give you a sense of how complicated their products are: http://jobs.netflix.com/jobs.php

nteon• 12 years ago

Netflix is a Java-heavy shop (the client talked up in the blog post is the Java client). I was pleasantly surprised to see that dynomite is implemented in C with an eye towards both BSD + Linux. I'm curious if this was because a GC'ed language like Java or Go was tried and proven not up to the task, or what.

viksit• 12 years ago

My guess would be that the ecosystem in their (seed) target set of databases and protocols is written in C. In one case, they're using Twitters twemproxy to provide connection counts for redis/memcached.

zerobfd• 12 years ago

Not just connection counts; when I first looked at the github I was surprised at how close the code organization was to twemproxy, and when I looked further a huge amount of the code is built on top of existing twemproxy code.

Netflix probably saw an opportunity to built the Dynamo layer on top of what Twitter had already written, and decided to just keep building on in C instead of rewriting a fairly large codebase.

mumrah• 12 years ago

My guess would be that they went with C to make it easier to hook into existing data systems like MySQL or Redis.

padho• 12 years ago

My guess is that this has a performance reason

lobster_johnson• 12 years ago

Funnily enough Dynomite was also the name of an Erlang implementation (https://github.com/moonpolysoft/dynomite) of Dynamo that seemed to get off to a very promising start before it was abandoned a few years ago.

diltonm• 12 years ago

I don't know if it was intentional but the name reminds me of the character JJ in the TV series Good Times. Awesome show that was. It sounds solid overall but one thing caught my eye, "A client can connect to any node to read the data. Similar to writes, a node serves the read request if it owns the data, otherwise it forwards the read request to the data owning node in the same rack.". I wonder if that could lead to an owning node arbitration storm when more than one node in a rack owns the data?

NKCSS• 12 years ago

Cool, it's something that Microsoft SQL Server (Enterprise edition) offers out of the box (Peer-2-Peer replication). The only problem with P2P replication is that, at large scale, requires massive amounts of bandwidth to keep everything synced, and it gets worse the more nodes you add.

diptanu• 12 years ago

Depends how things are replicated. In Dynamo style replication everything isn't replicated everywhere. Based on consistent hashing, and how vnodes etc are configured, a key lands itself in more than one nodes so that things are generally available. So scale-out in a dynamo-like system happens more gracefully.

RealCasually• 12 years ago

Perhaps I missed it, but is there any discussion on how their sharding mechanism works? How do they define a shard and where is ownership of various shards managed so a "non-owning node can redirect to an owning node"?

marktangotango• 12 years ago

I questioned tbat as well, could be a property of the underlying store (memcache hash). How would mysql sharding work without being application managed?

Dynomite – Making Non-Distributed Databases Distributed

Related Stories