Netflix: Run Consistency Checkers All The Time To Fixup Transactions

http://highscalability.com/blog/2011/4/6/netflix-run-consistency-checkers-all-the-time-to-fixup-trans.html

by yarapavan • 15 years ago

66 8 15 years ago

cookiecaper2• 15 years ago

The described approach just feels wrong to me. I understand that the problems Netflix faces are challenging, but usually when I run into messy systems like that, where you do one thing to address one problem, and then have to do four or five other things to fix the problems that one workaround caused, there's a fundamental misengineering in the mix somewhere. Systems should be simple and consistent as much as possible -- doing something like this across the entire service is crazy.

I understand if they do it a little bit to test things, or whatever, but I think if they sat back and were willing to carefully consider this problem, they could come up with a much simpler workflow. The ability to do that is a key part of quality engineering. A big cascading infrastructure of workarounds that eventually only kind of gets things in a correct state is usually a sign of a problem.

gte910h• 15 years ago

I've long felt that NoSQL does not jive with a certain type of mindset many computer professionals have.

There is this sterile perfect world mindset many computer professionals idealize.

Transactions and traditional RDBMSes make this sort of person really happy. They love to see everything always working out. That when it doesn't, everything is taken care of for them, etc. I do not know if you're this type of person, but your comment sounds like the type of sentiment that type often expresses when they see NoSQL in real world apps

This is not a work around of a messy system. This is adding more consistency on a system which is designed to be fast a scalable more than consistant.

Fast and scalable is more important with something like netflix. It's going to bother someone a hell of a lot more if they get fits and starts with their streaming, if it takes forever to get a pageview, etc, than if something occasionally get's misordered in a queue.

If they went with a system that demanded consistency over speed/scalability, then what they MIGHT get is a backend database that never went out of sync, but the UI of the website would show that queue items had moved, but the write conflicts would cause the transaction caused by the UI to unroll, and thus the user would still experience inconsistency. Or the website could wait for a confirmation for every step and offer a horrendously slow user experience during peak times.

I think this sort of NoSql data garden with automated gardeners running through all the time will be a fairly typical path from now on for many types of systems. I think financial apps may even adopt it eventually!

cookiecaper2• 15 years ago

I've actually used NoSQL multiple times in the past. I don't have a problem with that. Maybe it was the way the article was written, but it just sounded messy and disorganized, and they really should be able to do better on the end result (not jump around so much on queue/position in movie).

tybris• 15 years ago

> Systems should be simple and consistent as much as possible

CAP theorem says otherwise.

Distributed systems are complicated, almost like reality. You can feel the constraints of physics and math everywhere. You simply cannot get the best of all worlds (e.g., consistency, availability, and partition tolerance). If you have the money and need the scale, which often go hand in hand, you're simply better off avoiding hard consistency models, because the other two are more important to your business.

I love their approach. It's exactly what I think people should do to scale up their system. Avoid two-phase commits and other elegant availability mine fields and go for pragmatism. Note that this is only relevant for a few dozen companies.

lsd5you2• 15 years ago

It is not 100% clear from the examples cited that scaling should necessitate the consistency issues they have. Surely it is quite straight forwards to scale user private data (i.e. 'sharding' on user data) such as position in play and the queue.

For the data which is cross user, such as that which drives their recommendation system (are there other parts, I don't know, not a user) then consistency would be compromised - but given the soft/heuristic nature, presumably also more or less irrelevant. Here eventual consistency would be good enough (and ideally should not require extra processes)?

Like the halting theorem, the implications of the CAP theorem are only unavoidable in pathological cases. Until it is identified as a pathological case then I believe it is premature to invoke such theorems.

edoloughlin• 15 years ago

Very good and very pragmatic overview of NoSQL here: http://www.infoq.com/presentations/Enterprise-NoSQL

Netflix: Run Consistency Checkers All The Time To Fixup Transactions

Related Stories