Netflix Billing Migration to AWS

http://techblog.netflix.com/2016/06/netflix-billing-migration-to-aws.html

by hepha1979 • 10 years ago

219 119 10 years ago

techblog.netflix.com

merb3• 10 years ago

They used a DRBD replicated MySQL. Wonder why they used MySQL over PostgresSQL then.. Would be great to know if they looked that up aswell.

setheron• 10 years ago

What does DRBD have to do with MySQL vs PostgresSQL ? DRBD is just block device replication

merb3• 10 years ago

Actually you could do DRBD with PostgreSQL, too. There is a Master-Master Replication with MySQL that PostgreSQL don't has, but since they didn't used that it would be great to hear we they've choosen MySQL over PostgreSQL. Would be great if they would share that, too.

gdulli2• 10 years ago

I think it's just a reflex for (ex-MySQL) Postgres users to ask that anytime they see someone using MySQL. If you have significant experience with both you know the quality of life is different between the two.

yeukhon3• 10 years ago

The interesting thing is cross-region read replication has existed for MySQL since in 2013, whereas cross-region read replication for PG has been out a few days out officially.

https://aws.amazon.com/blogs/aws/cross-region-read-replicas-...

https://aws.amazon.com/about-aws/whats-new/2016/06/amazon-rd...

I am interested in non-read replica replication...

frik2• 10 years ago

In the last 15 years every news (on /. and HN) that mentions MySQL is interrupted with such questions from Postgres fanboys. How about: MySQL is the very best solution for such tasks and Postgres isn't the universal holy grail? Wouldn't it be better to try to convert people over from (Postgres direct competitors) MSSQL, OracleDB, DB2 to Postgres?

deathanatos• 10 years ago

Frankly, no; I've used all of Postgres, MySQL, MSSQL, and SQLite, and MySQL is the worst of the lot. No, PostgreSQL isn't a holy grain, but any user of MySQL would overall be better served by PostgreSQL; especially on AWS where the replication is so simple to set up, which IMO is the only good argument I've heard in favor of MySQL (but I hear even that is fraught with peril).

MySQL is so completely riddled with bugs and utterly baffling behavior — it is the PHP of relational DBs. What I remember off the top of my head includes attempting to subtract datetimes causing the punctuation to be removed from the datetimes and the resulting "integers" to be subtracted, FK integrity being violated in a number of easy-to-hit corner cases, the "utf8" encoding not being able to encode UTF-8 (and the default being latin1…), GROUP BY allowing obviously (i.e., catchable to the parser) broken queries, `SELECT * FROM table` on <10k row tables taking minutes in some cases, the SQL dialect swapping the words "key" and "index" inappropriately; Read [1] if you want more.

Thus far the only other tool I've seen come close to this level of insanity is MongoDB. The thing about a tool so willfully discards any sort of reasoned approach to its topic area is that the people who use it — who inevitably are not well versed in how a relational database works "in theory" — cannot derive from its behavior its rules, because its behavior is irrational, bordering on psychotic, and the people using it tweak query after query while having no understanding of why one query might work better until some abomination that someone usually works is crafted; `-- Don't touch`. A good tool — I believe, somewhere deep inside me — will teach the novice user. MySQL will not; it will drive you insane.

[1] http://grimoire.ca/mysql/choose-something-else

crisopolis• 10 years ago

At least they ditched Oracle (licensed)...

emcrazyone2• 10 years ago

I worked at a Fortune-5 that was heavily invested in Oracle.

Oracle has a nasty licensing model where they charge you per core regardless of if that core is a physical one or not (hyper-threading). While I was there, it suites told all the engineering managers that Oracle was out and the going forward solution was Microsoft SQL which, as I understand, has more relaxed licensing model.

Another thing I'm wondering about is I would figure Netflix to be big enough to have SAN storage. Just about every large company I worked at always used SAN replication technologies instead of open source stuff. And it's not a debate about open source solution vs. commercial. It's more about support. Large companies want a throat to grab when things break bad.

matt_wulfeck• 10 years ago

Netflix doesn't have SAN because it doesn't run anything outside of AWS. AWS provides the throat to hold in your analogy.

c17r• 10 years ago

MS SQL used to be per socket pricing. With 2012 they switched to per core.

That was a sad day.

Annatar7• 10 years ago

Oracle has a nasty licensing model where they charge you per core regardless of if that core is a physical one or not (hyper-threading).

That's why you then switch to their T5 or M-series with up to 3,072 processors, and then Oracle comparatively charges you peanuts, since those systems have relatively few physical sockets. And if you know SPARC, and you know Solaris, you can squeeze some serious savings. The problem, it seems to me, is that most system administrators and managers today are neither familiar with SPARC nor with Solaris, so they end up paying more in the long term in licensing and maintenance for running something else like MySQL on GNU/Linux.

But that's their problem, not the guy's who knows SPARC and Solaris and how to save money, isn't it?

Annatar7• 10 years ago

And why is that good? Oracle is worth every penny of those $47,000 per core for an enterprise license, and then some. Provided my company makes a commercial product and I can pay, I would gladly pay Oracle that kind of money (and more) for their database.

Large companies have an enterprise wide license of the Oracle database, so they have thousands of instances (worked at a place like that, that's how I know), and at that point, with that kind of scale, Oracle becomes dirt cheap, considering it delivers the only database I have seen that can withstand the kind of abuse a large institution can throw at a relational database management system.

hetfeld• 10 years ago

Dropped Oracle, using MySQL. Why not use PostgreSQL instead?

eterm• 10 years ago

Is there a version of poe's law for hacker news / slashdot?

A large company describes their very real efforts and shares their experience and knowledge, and the first comment is "Why not use <preferred thing> instead?".

Rodeoclash• 10 years ago

Because knowing how they worked through the decision to come to a conclusion on using a particular product is valuable.

Postgres is usually the first choice for applications like this so knowing why they chose something else over that could influence others that need to make a similar decision.

justinhj• 10 years ago

Assuming MySQL meets their needs there's no hugely compelling reason to choose PostgreSQL over MySQL.

cpollard• 10 years ago

I know regarding HIPAA, Aurora isn't supported but mySQL is. Maybe there's a reason why it's not HIPAA compliant that made it unusable for Netflix? And cost?

singlow• 10 years ago

What aspects of HIPAA does Aurora cause problems for which wouldn't be a problem for other storage engines? The Aurora engine RDS product supports all of the normal MySQL access controls, SSL in transit and at-rest encryption for the data, snapshots and backups.

ams6110• 10 years ago

Where is the intersection between HIPPA and a back-office customer billing system?

lapitopi• 10 years ago

I work on the Netflix Billing Team.

PostgreSQL was indeed a very attractive option, but we wanted to keep a path to Aurora open. When we were working on the migration, Aurora was still in beta, so instead of going to Aurora directly, we decided run our own MySQL instances on EC2.

lilbobbytables• 10 years ago

Ahhh, now it makes sense. Thanks for answering.

philliphaydon2• 10 years ago

Why would you go to RDS with such large amounts of data when AWS do not provide ability to get data out easily. If you moved away from AWS in the future for what ever reason your data is more or less stuck in AWS.

Annatar7• 10 years ago

Does "Aurora" fix the inherently broken SQL and integrity constraint mishandling associated with MySQL?

Which security models does "Aurora" offer, for instance, are you able to use OS authentication like in Oracle? What about PKI on SmartCards?

ForHackernews• 10 years ago

Does this mean Amazon can mine Netflix's data to improve their Prime Video services?

justicezyx• 10 years ago

There is explicit policy inside AWS to not access any user data.

mgrennan• 10 years ago

And Sodden is wrong. The government is not collecting your data.

alttab• 10 years ago

No, but they can probably look at how Netflix is utilizing AWS and make that better.

ascendantlogic• 10 years ago

AWS is a gigantic cash cow for Amazon now. If they were caught with their hands in a competitor's cookie jar the fallout would cost them in the billions. They probably wouldn't risk it, but this is just me hypothesizing.

thramp• 10 years ago

Customer data is sacrosanct within Amazon. Cannot touch it without the customer’s consent.

source: I work for Amazon Web Services.

kevin_b_er• 10 years ago

Amazon already showed its hand at maliciousness when it blackballed all chromecasts from stores, including 3rd party sellers. I don't put suddenly blackballing netflix beyond amazon's consideration. Amazon is cutthroat with public customers, corporate customers, and with its own employees. I think netflix is stupid to put more eggs in the vulture's nest.

In fact, I can pretty much guarantee that at the first opportunity where the lawyers agree it is a usable hole, they'll try to kill netflix through denying it service. Taking out netflix for a week or two while the engineers rebuild the backends with a different provider would be excellent for amazon's video division.

the_arun• 10 years ago

I would safely assume this data will be encrypted and "anonymized" to minimize the risk of Amazon using this data.

mikikian• 10 years ago

Dropped Oracle, using MySQL. Why not use AWS Aurora instead?

nemothekid• 10 years ago

FTA: While our subscription processing was using data in our Cassandra datastore, our payment processor needed ACID capabilities of an RDBMS to process charge transactions. We still had a multi-terabyte database that would not fit in AWS RDS with TB limitations.

jon-wood2• 10 years ago

Somewhere else in the comments here one off the engineers on this project said Aurora was still in beta at the time but they went with MySQL so they had a reasonably simple migration path later.

ww5203• 10 years ago

DRBD works very well for high availability, especially good to provide failover for master database, since that usually requires fast failover like under 10 seconds. Five or six years ago, did couple HA setup with DRBD along with Linux HA and virtual IP. The failover work great.

kbenson4• 10 years ago

I was under the impression the accepted way to fail-over with mysql and DRBD was to fail out the old server, and then start repairing tables, because you couldn't be sure they were in a valid state. That's about a decade old info though.

I always just set up master/master replication with mysql. You can get free distributed reads that way if you architect your application right.

Rapzid• 10 years ago

DRBD supports flushes and barriers. You just switch the primary and startup the database. InnoDB will recover as necessary and you are good to go. This is assuming you are using all InnoDB tables(you likely should be).

ww5203• 10 years ago

DRBD produces two mirror copies of data. When the primary failed, the standby has whatever data the primary has just before the crash. When the standby starts up, the RDBMS goes through the normal recovery and bootup. It's same as the primary crashed and being started up again.

MySQL's master/master has a number of complication and problems: 1. data loss due to async nature of replication, 2. update conflict on same data on multiple masters, 3. two masters mean two IP so all the clients need to know how to fail over to different IP, 4. complication in adding or removing master.

With DRBD, the disks are mirrored so that's no chance of data loss once a transaction is committed. There's only one master so no complicate conflict resolution. Linux HA's virtual IP means the standby will take over the primary IP so the clients don't need to know there's a server failover. Adding or removing standby is easy. DRBD will sync the disks automatically, no downtime on the primary.

Annatar7• 10 years ago

DRBD is failover - there is an interruption while the switch is made.

High availability is when you can afford to lose one or more nodes, and there is no service interruption. Moreover, the processing and I/O are symmetric and performance scales linearly by adding more nodes.

tzakrajs• 10 years ago

How many times is Netflix going to finalize its transition to AWS?

yeukhon3• 10 years ago

They won't 100% be on AWS, but probably close to 95%. Netflix owns its own media CDN.

dexterdog• 10 years ago

So almost all of their bandwidth is handled outside of AWS. I would wager a guess that bandwidth is quite a bit more than 5% of their ops bill.

adrianggg• 10 years ago

I got really excited because I read the heading as Netflix migration costs paid for by AWS. I thought they worked out deal to get a free tier during migration. Wow...oh...nevermind... :-)

ryanmerket• 10 years ago

I work for AWS. I believe we do offer some migration assistance for bigger startups. Hit me up if you want to learn more: rmerket@amazon.com

partiallypro2• 10 years ago

I find it to be a peculiar business decision to completely (as much as you can) migrate to one of your largest competitors' cloud service. It seems like Microsoft is the only, of the larger cloud providers, that doesn't -really- compete with Netflix. (Google has YouTube, and I guess even Microsoft has a much smaller Windows Store presence.) Even if Amazon can't access the raw data, they could see how you're utilizing it to improve their own video service, and they get the benefit of billing you and potentially using their pricing leverage to squeeze your margins.

I feel like Apple's approach of utilizing multiple providers makes more sense (though they do this for uptime and redundancy.) Maybe I'm being a pessimist.

truncate• 10 years ago

As per this article, Netflix only migrated their billing infrastructure not their video streaming service.

luhn• 10 years ago

They only migrated their billing infrastructure because the rest of Netflix is already in AWS. As far as I'm aware, the only non-AWS Netflix component remaining is their CDN. (And it will likely remain that way.)

niftich2• 10 years ago

That's because they've already migrated their videostreaming infrastructure to AWS (and some in Google Cloud Storage as a failsafe).

This Ars Technica article from February summarizes the multi-year effort: http://arstechnica.com/information-technology/2016/02/netfli...

hkmurakami• 10 years ago

If you think about it, Apple does something similar by having Samsung fab their processors.

kiwijamo• 10 years ago

Amazon could, if they wanted to, obtain a lot more useful data from the data Netflix hosts in AWS than Samsung could from just making chips for Apple. So I'm not sure if it's a good comparison.

hiuhsdf• 10 years ago

Yeah, this decision makes no sense, AWS is not even cost-effective past a certain, relatively small, amount of resource usage. Netflix could have easily built their own infrastructure for this, plenty of smaller companies do it and save tons of money. The only thing that would make sense here is if Amazon offered them their services at a huge discount, which could be a move looking to either acquire Netflix in the near future or cripple it.

drewrv• 10 years ago

It's an easier task for a smaller company to pull off, and I'll bet Netflix is getting a better deal than most of us could negotiate.

GrinningFool• 10 years ago

Netflix started with their own infrastructure[1] and migrated it all to AWS. Presumably this was more cost-effective for them.

[1] https://media.netflix.com/en/company-blog/completing-the-net...

kchoudhu• 10 years ago

At this point, I'd imagine Netflix's main expenses are not engineering/bandwidth/compute -- it's content. My guess is that there's better ROI in paying Amazon a premium to not have to worry about the CDN engineering while concentrating investment on expanding their library.

nitwit005• 10 years ago

Microsoft also sells through the Xbox platform, which is in direct competition for the TV space.

stoic• 10 years ago

I don't think the differentiation in that space is in technical implementation, but rather in content quality (produced or "acquired")

curun1r• 10 years ago

I don't see Netflix and Amazon being serious competitors...yet. For both of them, the main competitor is cable. The more that either/both of them are able to get people to "cut the cord," the better it will be for both of them.

I can't speak for others, but for me, the decision wasn't, "should I choose Netflix or Prime Video?" Instead, it was, "Can I find enough programming to satisfy me for less than I'm paying for cable?" The only way my answer was yes was with multiple streaming subscriptions.

tomphoolery2• 10 years ago

Even if Amazon can't access the raw data, they could see how you're utilizing it to improve their own video service, and they get the benefit of billing you and potentially using their pricing leverage to squeeze your margins.

So you're saying Amazon is going to risk millions of dollars so they can make a few more bucks on video streaming, which is like 3 levels down from their primary business?

Prove it. I had an argument with my last company about this very issue. If Amazon's primary business was video delivery, then that would make a lot of sense. But where does Amazon's primary revenue stream actually come from? That's right, it's AWS. Amazon may be an excellent retailer, but they spend just as much money as they make on the shipping and fulfillment side to get shit on your doorstep faster and cheaper than anyone else out there. Each person that spends money on Amazon can only really spend a few hundred dollars per year. But even a small company that's entirely hosted on AWS, like 70% of the companies I've worked for, pays Amazon thousands of dollars per month for hosting. There's definitely more people than companies, but shipping stuff to people costs a lot more money than Amazon needs to pay in order to have your stuff hosted by them. Plus they do a lot of R&D into making their own systems faster and more efficient, eventually passing savings down when they re-work their pricing tiers.

Basically, my argument is/was that the whole idea of Amazon stealing your IP to make a few extra bucks on whatever they happen to be doing is totally bunk. Amazon is really in the infrastructure business, and if you're a video startup...you're NOT. And take it from someone who watched a company try and fail to build a competing private cloud with no budget and a skeleton crew...it's stressful and not fun at all.

The main reason to use multiple providers is, as you said, uptime and redundancy...and "not putting all your eggs in one basket", so to speak. It's an engineering, not political, decision. My last company could have probably saved themselves by moving everything to AWS and shutting down their Level3 internet-backbone connectivity and direct fiber from the office to the datacenter (which is the same technology AWS is using anyway, except they have an actual cloud API and not just a pile of servers), but they were too busy conflating this engineering/performance decision with one that must be made for political reasons.

chipperyman573• 10 years ago

Amazon may be an excellent retailer, but they spend just as much money as they make on the shipping and fulfillment side to get shit on your doorstep faster and cheaper than anyone else out there. Each person that spends money on Amazon can only really spend a few hundred dollars per year. But even a small company that's entirely hosted on AWS, like 70% of the companies I've worked for, pays Amazon thousands of dollars per month for hosting

But there are a lot more people than businesses, and an order of magnitude more people than businesses who both need custom hosting infrastructure and choose AWS.

e12e2• 10 years ago

Good points. I'd just like to point out that, there are "political redundancy" reasons to diversify to multiple (cloud) providers: you could easily get in a billing dispute with a single vendor - if that means your entire business grinds to a halt, then that puts you in a very weak and vulnerable position.

I don't know if the extra effort of using eg. Google and AWS is worth in order to "stay up when AWS goes down" - but it might be worth it to stay up if/when AWS cuts you off for some reasons (quite possibly due to human error, billing, a take-down notice or other legal dispute etc).

None of that helps if you might accidentally find yourself on the wrong side of the "war on terror" by publishing news - in such a case all your US funds and assets might be frozen, and you would need a non-US presence in order to stay up while sorting out the potential error. But I suppose it's no worse than being subject to other kinds of arbitrary censorship...

tmptmp• 10 years ago

>Prove it. I had an argument with my last company about this very issue. If Amazon's primary business was video delivery, then that would make a lot of sense. But where does Amazon's primary revenue stream actually come from? That's right, it's AWS.

Let me try. Earlier Amazon's primary business was selling books, then it "became" selling almost every object that can be legally sold, and now you are saying that it "is" AWS. What about tomorrow? Tomorrow, it easily may "become" selling videos too. With Amazon it's very much possible.

So, it's not just technical issue, it's a political/business issue too. Of course, as you have said, they must take into consideration the trade-off. If the trade-off is more like "killing yourself under the technical burden of setting up a good network" vs "potentially allowing/helping Amazon to take advantage of your hosted service on their AWS and thus to become a future competitor" then they may go to AWS and/or other cloud provider(s).

edit: allowing/helping

partiallypro2• 10 years ago

For starters, your company is not the largest video provider in the world, like Netflix. As someone else mentioned here, diversification can be for "political" reasons as well. Especially if you're as large as Netflix; diversifying could keep your billing in check. Let's say you split between AWS, Azure and GCS. That gives you the ability to scale based on what is cheapest. You could also do public and private clouds in tandum. You can scale your private cloud/on-premise to the public cloud if need be with most large cloud providers (though I never suggested they use on-premise.)

As far as this not being Amazon's primary business, well, they are a lot bigger and diverse. But let's use a Wal-Mart analogy. Wal-Mart is so powerful (maybe not anymore, because of Amazon) that if it doesn't like your wholesale pricing to them, they can move your shelf space and practically destroy your business. They have the leverage in those relationships. Now, with AWS being the #1 cloud provider you have a similar lock-in, at the very least your switching cost would be quite high.

So, let's just for example sake say Amazon make a decision to give their own video streaming priority on their own cloud. That's not breaking net neutrality laws, because...it's their servers, this has nothing to do with telecom. So now Amazon Prime Video, streams 4K at a much better rate than Netflix (and Netflix would likely never know.) Then, perhaps another assumption, AWS decides to increase their pricing tier for their media servers for streaming. So suddenly they have squeezed you in two ways. Quality of Service and pricing.

Amazon could easily do a calculation to compare themselves to Azure and GCS to tell what is the proper amount they could get away with and still make it more expensive just to migrate to a competitor. You're locked-in, and just to make up for your switching, it would cost, let's say 1 years worth of AWS service. Hard to explain your sudden blip in earnings to short minded investors.

Anyhow, all I am trying to say is...it would make more sense, to me, from a business stand point to diversify across multiple providers instead of being all in on AWS. I never said you had to have an on-premise set-up. It would be much different if they were a small start-up, but Netflix is not. It can at times take up more internet traffic than torrenting.

I will point out though that services like Docker are making cloud lock-ins harder to do, but they don't solve reliance on APIs or just proprietary offerings. There's a reason why Google, Microsoft and Amazon all underline, push and constantly improve those offerings as a form of lock-in.

It is possible Netflix has some sort of pricing agreement with Amazon that locks in a rate for x amount of years. But either way, when you're as large as Netflix, I think diversification is the better long term strategy.

thwarted• 10 years ago

But where does Amazon's primary revenue stream actually come from? That's right, it's AWS.

Looking at Amazon's Q1 2016 Financial Results [0], page 8, we see that net sales of non-AWS amounts to ~$26B, and AWS is ~$2.5B. From the Segment Highlights section (same page), AWS sales is 9% of total sales.

AWS beats non-AWS income only because there were losses in international; ignoring international, it's a $16m difference.

Looking at page 14, Media sales in North America and International sum to $5.6B. Media sales alone is double AWS's sales of $2.5B (page 13). Profit margins are way higher for AWS, so there's still a lot of room for a larger income difference between the two segments.

Based on this, someone doing media streaming as their primary business needs to be aware of who they are in bed with infrastructure-wise, but I agree that it doesn't mean that AWS isn't worth using just because Amazon is in the same market/area. Media sales are a big portion of Amazon's non-AWS sales, and being digital are most likely fewer headaches than physical goods; but not so big of a portion that one needs to worry about Amazon being the 800lb gorilla that would need to be contended with. It's more likely that Amazon is a threat to Netflix on licensing deals and content offerings (IMO, Netflix is slightly stronger, but neither are great, in content offering).

Is Amazon going to pull the rug out from under Netflix? Not if they want anti-trust attention. Is Amazon going to disallow the Netflix app to run on their devices? I suspect no, since they seem to want to create a platform rather than a walled garden when it comes to devices and media. Is Amazon going to undermine the trust in AWS by using AWS customer's data, or metadata about their customers, for their own gain? Probably not. If I'm going to put money on who's going to do "the right thing", for businesses and tech as a whole, it's going to be on the likes of Amazon rather than, say, Oracle.

Looking closer at (just) this (document), the growth numbers are such that it could go either way as to which revenue stream will eventually be the major contributor to Amazon revenue streams. That's a significant reduction in losses (-60%) over Q1 2015, so International turning around over the next year or so could keep AWS and non-AWS income neck and neck.

[0] http://phx.corporate-ir.net/External.File?item=UGFyZW50SUQ9N...

JohnTHaller• 10 years ago

Amazon has spent years screwing with Amazon Prime video subscribers on Android to the detriment of the Prime brand and their brand as a whole in the eyes of many of those affected users. Amazon could very well screw with AWS to the benefit of Android Prime video for reasons only upper management can discern.

travmatt• 10 years ago

That was my first thought. Sabotaging one of the most conspicuous brands (and arguably their flagship customer) in the United States today sounds like an excellent way to commit business suicide - how many other tech companies would they cede to their competitors? Could a company ever defend using AWS again if they even tangentially competed with Amazon? Moreover, I bet that AWS is much more central to their long-term than Amazon video is. That just seems like a ton of risk for very little upshot.

justinmk• 10 years ago

I find it to be a peculiar business decision to completely (as much as you can) migrate to one of your largest competitors' cloud service.

This was directly addressed in a 2011 talk[1] by a director at Netflix:

Competition drives cost down
[...]
We would prefer to be an insignificant customer in a giant cloud.

[1] https://justinmk.github.io/2016/03/17/selfish-cooperation.ht...

antaviana• 10 years ago

How do you know Microsoft will not compete with you sometime next quarter? Did you anticipate the LinkedIn purchase with which Microsoft enters the job search market?

limeyx• 10 years ago

I think for smaller companies it could make sense but Netflix is big enough and visible enough that if AWS cut them off, I think you'd see a huge exodus off of AWS of people fearing they would be next ... simply not worth it to Amazon to shoot itself in the foot like that IMO

cia48621793• 10 years ago

What if the Netflix side on AWS was hacked say like their security credentials was leaked?

innocenat• 10 years ago

Am I the only one who initially thought that Netflix is passing the bill for migration to AWS?

Annatar7• 10 years ago

Considering how much code and data was interacting with Oracle, one of our objectives was to disintegrate our giant Oracle based solution into a services based architecture. Some of our APIs needed to be multi-region and highly available. So we decided to split our data into multiple data stores. Subscriber data was migrated to Cassandra data store. Our payment processing integration needed ACID transaction. Hence all relevant data was migrated to MYSQL.

Considering that Cassandra is not ACID compliant with her "eventual consistency", and that MySQL is notorious for corrupting data and not functioning correctly, I am compelled to wonder just what kind of people work at Netflix. And who gets the idea to go to AWS and pay the full virtualization on Linux performance penalty?

Now, I've done Oracle engineering at some very large databases (hundreds of millions of rows, OLTP and DWH), and I know that Oracle is a smoking fast database when the right people develop on it. Also makes me wonder what kind of code they had running, and what kind of people selected it, when they managed to gum up what is essentially the Bugatti Veyron of databases.

Given this information from "Netflix" I won't be considering them as a potential employer any time soon. It has to be a mess over there.

tedivm• 10 years ago

The majority of the corruption issues with MySQL comes from using the innodb engine. AWS built their own MySQL engine called Aurora that I would be shocked if Netflix wasn't using. It's designed for distributed workloads and should be harder to corrupt.

I'll admit I'm confused about picking Cassandra as well, but not for the same reasons you are. They're only storing subscriber data (billing address, subscription type, etc). That data is going to remain static for months at a time. When changed the only potential problem that could occur is the billing process using old data, but I'm guessing their system is smart enough to try again in thirty minutes.

Oracle may be fast, but it's also expensive. This is billing, which means batched jobs running in the background- they don't care how long each individual transaction takes, and I'm positive it's going to be cheaper to roll up more servers to compensate than it is to pay Oracle's licensing fees.

ec109695• 10 years ago

Amazon doesn't offer instances large enough on RDS, so they are managing their own.

Do you have any sources for the corruption comparison between Aurora vs InnoDB?

Annatar7• 10 years ago

Oracle may be fast, but it's also expensive.

So? That's capitalism: one gets what one pays for, and Oracle is not just fast, it can do a lot, and it can be configured to be paranoid about protecting data, and it has clustering technology meant for scaling, RAC.

Truth be told they could have picked PostreSQL and it would have still been a better solution than Cassandra and MySQL.

tokensimian• 10 years ago

I'm not sure if you saw this other thread, but I thought it answered one of your questions so was worth sharing. Sounds like Aurora wasn't stable when this migration began, so they did the migration with an eye towards the next migration (to Aurora).

Sorry about the c/p but I don't see a permalink on lapitopi's comment.

lapitopi 1 day ago I work on the Netflix Billing Team. PostgreSQL was indeed a very attractive option, but we wanted to keep a path to Aurora open. When we were working on the migration, Aurora was still in beta, so instead of going to Aurora directly, we decided run our own MySQL instances on EC2.

edit: begun/began

buro9• 10 years ago

Thanks for writing this up, I'm just at the tail end of having re-architected the CloudFlare billing system, also a subscription system written in Java with a MySQL back-end, but fronted by a Go API that insulates the rest of the business from the internals of the billing system.

The blog post covers a lot of the high level stuff really well, but I'm interested to learn whether you experienced any issues along the way, what they were, how you dealt with them.

In CloudFlare's case our migration was made more complex by also adding PayPal, and changing our processing gateway. Both of which created risk that we've had to work hard to understand and mitigate, i.e. how different gateways may, with the same card, return different results, etc.

pamonrails• 10 years ago

We do a lot of migrations from custom systems or SaaS integrations to Kill Bill (open-source subscription billing and payments platform) [0] and we've summarized our strategy and usual pain points in our migration guide [1]. You might find it useful.

Happy to chat offline too if you want to go into specifics.

[0] http://killbill.io/ [1] http://docs.killbill.io/0.16/migration_guide.html

back_beyond• 10 years ago

This thread is missing Google's PR team and links to its SRE book

jason46• 10 years ago

Is this why my daughter can't watch Young Justice? I've noticed quite a few titles show unavailable. Curious if Netflix is positioning to sell to amazon.

Netflix Billing Migration to AWS

Related Stories