Rebuilding Netflix's video processing pipeline with microservices

https://netflixtechblog.com/rebuilding-netflix-video-processing-pipeline-with-microservices-4e5e6310e359?gi=d48b3333df75

by samaysharma • 2 years ago

197 168 2 years ago

netflixtechblog.com

Zetobal• 2 years ago

You know I read articles like that all the time but the user experience of all these app gets worse. The time to first video frame on Netflix is not great to say the least. The rich metadata seems also only to be used internally...

bodantogat2• 2 years ago

My pet peeve with Netflix is that it occasionally forgets where I last paused my video. I don't often watch Netflix, so when I do, it's annoying that I can't resume from where I left off

0x6c6f6c• 2 years ago

This is Disney+ but nearly every time. Moved from the living room to bedroom and had to seek to get back to where we were just last night

_puk• 2 years ago

I've found actively pausing a video before exiting seems to improve the accuracy of the resume point.

I'm presuming this is because it fires a beacon on pause, but only every X (10?, 30?) seconds on playback.

Not had many issues with it completely forgetting where I am though (it does sometimes get confused as to which episode I watched last though).

Grazester• 2 years ago

I watch Netflix regularly and never had this issue. I wonder if it is your client? I use an Nvidia Shield(Android TV).

SebFender2• 2 years ago

There you go - 100% right you are.

vergessenmir• 2 years ago

It might not be, but who are we comparing them to? Secondly who out there is doing this scale as "cheaply"

Disney is already finding out that a streaming platform isn't cheap to run and hard to do efficiently.

eterpstra3• 2 years ago

Wait, who am I supposed to believe here?!? Prime Video tore down their micro services in favor of a monolith just last year! Which trillion dollar globocorp is my tiny, insignificant company supposed to emulate?

https://thenewstack.io/return-of-the-monolith-amazon-dumps-m...

Hnrobert42• 2 years ago

My problem with these two articles is neither provide sufficient detail to learn anything meaningful.

Are there any good resources from trillion dollar globocorps that get down into the weeds?

eterpstra3• 2 years ago

AirBnB comes to mind. https://airbnb.io

sgift• 2 years ago

Well .. which one is better for your chances to get more money in the performance review?

Traubenfuchs• 2 years ago

I was highly commended for bringing up 6 new microservices a year ago during my performance review. Late during development I noticed that at least 2 of them were useless (they are essentially message routing gateways, I planned to "enrich" messages from inside, but ultimately never did). It was already done and I did not want to waste time to integrate those two services into the others so I left it as is and, well, my boss loved it.

traspler• 2 years ago

They moved one of their use-cases back to a monolith, not their whole suite of applications/services.

nonameiguess• 2 years ago

They're pretty clear here on what the benefits are. Microservices allow you to scale up and down individual system components without having to carry along the rest of a vertically scaled monolith for the ride. This makes for more efficient utilization of compute resources. For a company renting compute resources from the cloud, like Netflix, this can save a lot of money.

What Amazon did, according to this needlessly snarky article that is not Amazon's tech blog, does not conflict with this. It's all theory. In reality, you should not be dogmatic and religious about your architecture choices, but empirical wherever possible. They measured utilization and cost and found they could do better in some cases with monolithic sub-systems. This doesn't mean all of Prime Video abandoned SOA.

Solvency• 2 years ago

Honestly, I get that it somehow became cool to ignore the grammatically correct "computation" and "computational resources" in favor of just grunting "compute" — why not go all the way?

"efficient utility of compute resources", etc. Just shorthand everything.

Does it take a famous developer to do it first for everyone to feel comfortable doing it?

ndriscoll2• 2 years ago

This makes for more efficient utilization of compute resources.

No it doesn't. The rest of the monolith is just a chunk in your compiled binary sitting on disk, which is trivial in terms of resource cost. If that code is not running, it is not using any runtime resources.

Microservices will, however, greatly increase resource requirements if they lead to additional serialization/deserialization, which is relatively expensive. If you're doing video encoding, this isn't such a big deal. For web services, it is likely to be the bulk of the resource cost. This is only exacerbated in modern infrastructures where services are more and more expected to use TLS to talk to each other.

pookha• 2 years ago

Swapping out silicon calls for network calls is the definition of complicated...And starting complicated because you want to achieve some premature optimization is insane. Monoliths are preferable for 90% of the use cases that you see in public or private industries.

redberryi• 2 years ago

Be thankful. That madness is what feeding a lot of families. Yours maybe one of those.

eterpstra3• 2 years ago

Mine is definitely one of those.

sbt567• 2 years ago

IIRC, the rebuild to monolith doesn't tell the whole story. They aren't ditching all of their microservices in favor of monolith. There is a person responsible from Prime giving a clear picture of this in twitter (sadly I'm not saving the tweet source)

tdb7893• 2 years ago

If you read the Prime Video blog post the takeaway is definitely not "always use a monolith". I haven't used Step Functions but they specifically mention step functions with a lot of state transitions (and the pricing model is per state transitions) and storing things in S3 and having to access things there all the time (which I've used S3 before and I was a bit shocked by since it seemed obvious to me that was gonna be really expensive). The takeaway for me was it's important to actually understand the tools that you're using.

As an aside the Prime video article as a bit funny, at one point they have the line (which I hope is sarcastic but I fear that it isn't) "We experimented and took a bold decision: we decided to rearchitect our infrastructure" when their original design just obviously chose tools that didn't fit their workflow.

pas2• 2 years ago

The takeaway for me was it's important to actually understand the tools that you're using.

no no no, that takes time, the hype train doesn't wait for anyone. the sacred monolith it is. all hail the Monolith! crush the microgerms, destroy the filthy tiny services.

mk89• 2 years ago

And just to add: that decision (to start with serverless for your POCs) came from above, CTO level.

Needless to say, microservices are not serverless/step functions.

iamflimflam1• 2 years ago

IIRC The Prime Video guys were processing videos using AWS step functions - which probably makes sense if you are processing a few videos every so often. If you are processing videos continuously then it’s much more cost effective to just have some big boxes running 24x7 crunching through a queue of jobs.

dboreham2• 2 years ago

Which trillion dollar globocorp is my tiny, insignificant company supposed to emulate?

None. They have a different set of problems than you.

mhitza• 2 years ago

I watch ThePrimagen (coding content creator on YouTube/Twich) from time to time, and he works as an engineer at Netflix.

My impression seems to be that he doesn't like the container infrastructure, reflecting my own opinion, though he never calls out explicitly the infrastructure at Netflix as something bad. But every time he talks about work at Netflix, sounds about as complex as I'd image if I'd give the job to a CV driven engineer.

jitl2• 2 years ago

Netflix is huge, has wide breadth of activities, needs to move terabytes of video around, and so has a lot of essential complexity.

matthewmacleod2• 2 years ago

I mean you could just read a variety of different sources, look at the different trade-offs, and make informed decisions based on the compromises of both different architectural designs and your own products’ needs?

Jesus the quality of conversation here is not good today.

elwell3• 2 years ago

The meta takeaway is that you shouldn't be afraid to resist trends (in either direction: monolith or microservices). If you're 'wrong' today, you may be 'right' in 5 years.

hcarvalhoalves• 2 years ago

"Micro service" and "Monolith" don't have precise definitions anyway. Ideally, there's only one right architecture: the one that's sufficient for the problem at hand, all things considered (latency, availability, cost, provider, maintenance, conceptual integrity, ...).

wesselbindt• 2 years ago

I feel like the term "microservice" is open to misinterpretation (people tend to focus on the micro part a bit much; I remember someone waxing poetic about something like a csv parsing service they had). But surely monolith is quite unambiguous: a system which is deployed as a whole. That is, you cannot deploy some part of the system on its own. You want to get changes in part X to prod? Better be prepared to deploy A through W and Y and Z as well.

That feels fairly precise. But maybe some folks would disagree with this definition of a monolith.

SebFender2• 2 years ago

These are conversations just like talking about code editors and so on. No one is right or wrong. But if I had to choose - micro services hard harder to control and secure. And vi? not for me ;)

highspeedbus• 2 years ago

Besides the debate about microservices being good or bad, it's clear that netflix developers are very passionate about what they do. To me this seems to play a big role in the success of a software product.

imjonse• 2 years ago

They are good for Netflix, bad for 99% of the other companies/projects, at least initially until they clearly know what they will build.

zer00eyz3• 2 years ago

Did any of this:

Make my nextflix better? How about cheaper? Did it deliver better content? Is this the work product of 2000 engineers focused on delivering me the worst content in the best way possible? What exactly am I getting for my 12, 20 ... wait what the hell is netflix charging now for their garbage content...

1000? 2000?d engineers at netflix and this is the article we get, this is their flex?

I am underwhelmed.

throwaway894345• 2 years ago

I don’t like Netflix’s content either but I’m pretty sure that’s not up to the engineering department.

matt_heimer2• 2 years ago

This is a bit like complaining about AWS because you don't like the product selection on Amazon.

barbazoo2• 2 years ago

Is Netflix offering their services as a platform now?

ActionHank• 2 years ago

Except for any aspect of the post to actually justify the engineering effort it would either have to improve efficiency or save money. All while Netflix has increased cost, stopped account sharing, and introduced ads.

Net negative improvement for users whilst there was presumably some net savings or gains for Netflix.

They've from offering a competitive offering to offering a compelling investment, shedding any guise of caring about their users along the way.

munro2• 2 years ago

And then I always think, Netflix only has ~3,600 movies... My friend's Plex server has 4x that (in just movies). I'm also often underwhelmed by Netflix's engineering posts

zer00eyz3• 2 years ago

He should write a mastrubatory blog post about how effective his set up is... Hell he could do a cost comparison for running his service vs Netflix per month.

I think this is the year where I go back to stealing content, none of these services are worth it.

alt2275• 2 years ago

But your mate doesnt have to stream them on demand to millions of people round the world.

They not only have to store the movies, but access and simultaneously stream them thousands of times from anywhere on the globe.

Imagine how many people are watching things like Stranger things series premiere at the same time.

pitaj• 2 years ago

How much does that cost to put together, just in storage?

scarface_744• 2 years ago

I am assuming you got the statistic from here

https://www.comparitech.com/blog/vpn-privacy/netflix-statist...

It also has 1800 TV series and how many episodes does the average TV series have?

johnchristopher• 2 years ago

Netflix has rotations though, does it paint a better picture ?

It also has TV shows, I wonder how shows and movies compare regarding bandwidth, storage and offers.

pas2• 2 years ago

so we can maintain our rapid pace of innovation

so it will make things better, pinky-promise!

I'm happy to pay them for the occasional good content, that I'll then torrent (because fuck smart TVs), but ... their app/client/website and their system seems to just work. I'm sure there are many things to optimize, etc, etc.. but probably they could reduce their development (and ops) budget by 70-80% if they would stop fucking with the system.

though, of course, that'd require a drastically different mindset, different people, etc.

phailhaus2• 2 years ago

Yes.

While it is still early days, we have already seen the benefits of the new platform, specifically the ease of feature delivery.

barbazoo2• 2 years ago

What features though? Oh maybe a sleep timer for the kids section so it turns off after 1 or 2 episodes? No, that would be actually useful but wouldn’t warrant a blog post.

kredd• 2 years ago

I mean, it depends? Obviously Netflix has extremely different priorities than 99.99% of the software in the world. Scale of operations is much different as well.

It’s available in most of the countries in the world, in a lot of varying devices, requiring a ton of different video processing pipelines, content delivery networks, infrastructure and etc. Even very “straightforward things” like downloading for offline viewing can be a significant effort to implement. Now think of audio sync, post processing, sub delivery, localization, partnerships and etc., you can see how you would need a ton of engineering effort to achieve it. Just the scale makes it much nuanced perspective during implementation.

You and me can dislike whatever content they’re delivering, but it’s very obvious how there are millions and millions of people who still enjoy it.

zer00eyz3• 2 years ago

>> It’s available in most of the countries in the world, in a...

You get that your post right here is a better pitch for Netflix engineering than their own engineering. Blog about some top those problems, the things that your doing that make your domain hard and interesting...

trunnell3• 2 years ago

Don't be underwhelmed. It definitely makes your Netflix better: whatever you watch can be encoded better, which enhances quality and lowers the chance of a rebuffer interrupting your experience. And the improved encoding efficiency frees up money that can be spent on content production.

But you can also just enjoy the story of developer achievement!

geodel3• 2 years ago

The way I see is Netflix is mostly sprawling implementation of mediocre Java stack which is typical of large IT department in F500 companies. But their relentless tech marketing and extremely high pay for work which is a wrapper around enterprise IT has created an aura of sophistication and cutting edge in many people's mind.

matthewmacleod2• 2 years ago

Yes probably all of those things.

I’m not being wide.

smegger001• 2 years ago

I doubt the backed engineers have any say over the content side of the company so blaming them for that isn't really reasonable. And while it may not make it cheaper for you or improve the user facing interface again another team, it probably made it easier for them to maintain and debug and administrate, which is something all sysadmins and engineers should respect.

Tainnor• 2 years ago

If you hate Netflix, why don't you cancel your subscription?

cletus• 2 years ago

Story time: I worked at Facebook and had to work with someone who came from Netflix. He was one of those people who, when he went to a new company, simply tried to reinvent everything he came from with no care or consideration given to what's already there.

FB very much does not use microservices. The closest is in infra but the www layer is very much a massive monolith, probably too massive but that's another story. They've done some excellent engineering to make the developer experience pretty good, like you can commit to www and have it just push to prod within a few hours automatically (unless someone breaks trunk, which happens).

Anyway, this person tried to reinvent everything as microservices and it pretty much just confirmed every preconceived notion (and hatred) or microservices that I already had.

You create a whole bunch of issues with orchestration, versioning and deployment that you otherwise don't have. That's fine if you gain a huge benefit but often you just don't get any benefit at all. You simply get way more headaches in trying to debug why things aren't working.

One of the key assumptions built into FB code that was broken is RYW (read your write). FB uses an in-memory write-through grraph database. On any given www request any writes you make will be consistent when you read them within that request. Every part of FB assumes this is true.

This isn't true as soon as you cross an RPC boundary... much like you will with any microservices. So this caused no end of problems and the person just wouldn't hear this when it was identified as an issue before anything was done. So th enet effect was 2 years spent on a migration that ultimately was cancelled.

Don't be that guy. When you go into a code base, realize that things are the way they are for a reason. It might not be a good reason. But there'll be a reason. Breaking things for the sake of reinventing the world how you think it should've been done were you starting from zero is just going to be a giant waste of everybody's time.

As for Netflix video processing, they're basically encoding several thousand videos and deploying those segments to a VPN. This is nothing compared to, say, the video encoding needed for FB (let alone Youtube). Also, Netflix video processing is offline. This... isn't a hard problem. Netflix does do some cool stuff like AI scene detection to optimize encoding. But microservices feels like complete overkill.

belval3• 2 years ago

He was one of those people who, when he went to a new company, simply tried to reinvent everything he came from with no care or consideration given to what's already there.

You have to have a very bad case of god complex if you look at a codebase that serves >3B users and experiences very little downtime thinking "oh yeah I could completely rearchitect that thing to be better"...

elwell3• 2 years ago

Maybe, but that's also the mindset of serious innovation. The question is whether or not the idea is a good one or not.

willio582• 2 years ago

It’s either god complex or pure lack of foresight and an inability to learn from or plain lack of experience.

emptysea• 2 years ago

Or looking to play the promo game, build a bunch of unnecessary stuff and get a pat on the back and better comp

dboreham2• 2 years ago

The fact that something a) works and b) serves very high volume traffic should always be a powerful counterargument against any suggestion to reinvent it.

fragmede2• 2 years ago

Chesterton's fence. A simple rule of thumb that suggests that you should never destroy a fence, change a rule, or do away with a tradition until you understand why it's there in the first place.

willio582• 2 years ago

Very cool! I think the “until you understand..” part is most important there. Like you still should be free to, but you absolutely need to understand why it’s like that in the first place.

VyseofArcadia3• 2 years ago

Does anyone have any reading material on the reliability of systems that use microservices? I've had a bit of basic probability rattling around in the back of my head that makes me suspect microservices are in general less reliable. I'd be interested in seeing a real-world analysis.

My thinking goes like this, with some simplifying assumptions. Let's say you have a monolith with 99% uptime that you rearchitect into 5 microservices, each with 99% uptime, and if any one of those services goes down your whole system is down. Let's also assume for the sake of simplicity that these microservices are completely independent, although they are almost assuredly not.

From basic probability, 99% uptime means there is some chunk of time t for which P(monolith goes down) = 1%. But

P(microservice system goes down) = P(service A down or service B down or...) = P(service A down) + P(service B down) + ... = 5%

In reality P(microservice system goes down) < 5% because they aren't independent and the chunk of time in which service A can go down will overlap that of service B. But still, that means the upper bound of the whole system going down is higher than for a monolith.

But microservices are pretty popular, and I'm sure someone has thought along these lines before. One potential rebuttal is that each microservice is in fact more reliable than the monolith, although from what I've seen in my career I am skeptical that's truly the case.

Where's the hole in my reasoning? (Or maybe I'm right. That would be fine too.)

dmattia• 2 years ago

In general, one of the goals of microservices should be that if one of the five services goes down, the other four should be able to operate in some capacity still.

In practice, this can make the math quite a bit messier, but I don't think it necessarily has been worse overall from my perspective.

So instead of having your system be up or down 99% of the time in a monolith, you'll have it fully up 95% of the time (using your numbers), but of that 5% of downtime, 20% of the time one of your products will be running slowly, or 10% of the time some new feature you launched won't work for specific customers in some specific region, etc.

At my company it makes things like SLA/SLO guarantees for "our services" pretty complicated in that it's hard to define what uptime truly means, but overall I think the five microservice approach, when done well, should have less than 1% of complete downtime, at the cost of more partial downtime

VyseofArcadia3• 2 years ago

In general, one of the goals of microservices should be that if one of the five services goes down, the other four should be able to operate in some capacity still.

This is an excellent point, but what brought this to my mind was that the microservices in the Netflix article I don't think have this property. It looks to me if any of the VIS, CAS, LGS, or VES go down, then the whole service is effectively down.

Indeed, in my own career what I've seen is that if one microservice goes down the user won't be seeing 500 errors or friends, but the service will be completely useless to the user. You've just gone from a hard error to a spinning load icon, which might in fact be an even worse user experience.

It could be argued that this is just "you're doing microservices wrong", but then we start getting into no true Scotsman territory.

jolux• 2 years ago

I don’t know if our architecture truly qualifies as microservices but in my experience one of the advantages is that the system is able to limp along in a degraded state much more effectively when one service goes down whereas it’s a lot easier to bring the whole system to its knees with a single change in a monolith.

This suggests an addition to your model which is that not all outages are equally costly.

bick_nyers• 2 years ago

I'm not an expert in this discussion by any means, but my two cents is that collaboration on larger teams is easier on the microservice path than the monolith path.

I would expect that over the course of years, the uptime and feature velocity of the monolith will decline at a rate faster than that of microservices, if a monolith has 99% reliability, then that inflection point might take a while to occur due to the calculation that you provided.

However, as a company you might be willing to go from 99% to 95% reliability (or 3 9's to 2 9's) to double the speed of feature development.

To my understanding, microservices are primarily implemented as an architecture for collaboration due to the inherent inefficiencies and communication difficulties of large teams. This is why they are often not recommended unless you could be categorized as "Big Tech".

VyseofArcadia3• 2 years ago

Anecdata incoming. This was when I was working in "Big Tech".

When I was in web dev, my experience was that there was actually no good separation, and collaboration with microservices was in fact more difficult. Every single feature I worked on required changes across several microservices, and having Team A run service A and Team B run service B etc. just meant that I had to get buy in from every team. My team was easy enough to work with, but then for the work needed on service B I would have to learn and start using their processes, attend their standups and meetings in addition to mine, and so on.

Frankly it was a nightmare. But in a monolith, the same work is usually just a few quick arguments during code reviews. Maybe inviting a few extra people to design reviews.

In my experience, microservices just make teams more territorial.

zmgsabst• 2 years ago

Your assumptions elide the benefits:

- micro services don’t always block the pipeline; often the failing one can catch up later

- scaling can happen for each micro service

- removing faulty components from the main path means the key services are less likely to crash

- you haven’t explained why feature X is more likely to crash in a micro service than a monolith, eg, you’re assuming components A and B have 0.5% crash in a monolith but 1% when run independently

Your model ignores that most of your crashes come from the same code paths between the two models; only a small contribution is to the crash is from hosting.

geodel3• 2 years ago

I did about something same. If there are 5 services each can fail independently (unlikely but lets assume) and uptime is 99%. Then P(All up at same time) = 0.99^5 which gives any one is down at any moment is 1- 0.99^5 ~5%. So this increase the failure 5 times of original 1% downtime. And with 100s of micro services in overall architecture with some indirect connections between many of them I think this number could possibly go much higher.

Further at least where I work it is clearly that failure rate is higher than 5%. But with cottage industry of observability tools, cloud native solutions ..blah..blah.. telling basic maths to people in responsible positions is sure shot way to get fired. I am already being marked as someone opposed to progress so I can basically take my statistics and shove up mine. There is million times more data about reliability of micro services and they all can't be wrong.

com2kid2• 2 years ago

My thinking goes like this, with some simplifying assumptions. Let's say you have a monolith with 99% uptime that you rearchitect into 5 microservices, each with 99% uptime, and if any one of those services goes down your whole system is down. Let's also assume for the sake of simplicity that these microservices are completely independent, although they are almost assuredly not.

I've worked on Microservices at HBO. IIRC over 2 years my team's multiple services only had 1 complete outage, and 2 or 3 impactful incidents.

Also a nice benefit of Microservices is that you can shove queues and retry logic between services, and replay messages later on when a service is back up. Obviously not appropriate for anything that needs to give real time results, but there are a surprising number of features that don't mind a 30 second or even 5 minute delay so long as success is guaranteed eventually.

trevor-e2• 2 years ago

The flaw in your logic/math is assuming that the resulting 5 microservices each have the same 99% uptime as the original monolith. In practice some microservices are much simpler and therefore more reliable than others, especially when broken out.

andy_ppp• 2 years ago

Well, even that depends, the overhead to do microservices well is a lot - versioning of what services are deployed and guarantees around compatibility between said deployments can be a huge amount of work. I’ve seen systems where 2 month old containers are just floating around still being sent incompatible messages. Then there’s teaching developers about CAP theory properly, at Netflix great everyone is sharp enough to get it but at AverageCompanyX, well my experience is 30% of developers actually can’t reason about eventual consistency.

Therefore I would question the assumption that things are simpler, code certainly, infrastructure, debugging and deployment certainly not.

I would say breaking things down into services, slowly, as it makes sense is enough. They don’t need to be micro.

tester756• 2 years ago

In practice some microservices are much simpler and therefore more reliable than others, especially when broken out.

Much simpler in what way?

You mean less close = less crash/segfault potential?

If yes, then oh c'mon, modern stacks are incredibly reliable that they almost never crash.

More microservices = more infra level stuff needed = WAY more potential problems

jameshart• 2 years ago

Several reasons this doesn’t pan out in practice.

1) Retries. When one replica of a microservice is down, the calling service can retry, get service from an up replica, and the outage is routed around

2) Queues. Microservices lend themselves to queue and worker patterns where downtime on individual services has less effect on overall service availability

4) Outages have narrower impact. One microservice losing access to its database breaks the functionality that relies on that microservice; other functionality runs fine.

5) Changes have smaller blast radius. Most outages are caused by changes; changes in monoliths that cause outages are more likely to take the whole system offline (eg stack overflows and infinite loops crash processes). Changes that cause outages in microservices can’t knock other services offline.

belval3• 2 years ago

Processing Ad creatives posed some new challenges: media formats of Ads are quite different from movie and TV mezzanines that the team was familiar with, and there was a new set of media processing requirements related to the business needs of Ads.

Nice to see them rearchitecture their service around enshittification.

bjornlouser• 2 years ago

'Long release cycles: The joint deployment meant that there was increased fear of unintended production outages as debugging and rollback can be difficult for a deployment of this size. This drove the approach of the “release train”. Every two weeks, a “snapshot” of all modules was taken, and promoted to be a “release candidate”. This release candidate then went through exhaustive testing which attempted to cover as large a surface area as possible. This testing stage took about two weeks. Thus, depending on when the code change was merged, it could take anywhere between two and four weeks to reach production.'

I guess I'm just old, but I prefer the delay with a couple of weeks of testing versus pushing to prod and having the customer test the code.

atomicfiredoll• 2 years ago

What's interesting to me about this is that some companies seem to struggle to get even one reliable deployment process in place. In this case they were able to actively select the right process for the right job, even if it isn't the one they're normally geared toward using.

It's not necessarily anything Earth shattering, but it may be an issue at some smaller places with fewer resources.

mannyv2• 2 years ago

Netflix is a company that works with media people. You either understand what that implies or you don't.

And remember, this is the backend for video encoding. Issues in this aren't necessarily user visible.

The big benefit to their velocity is responsiveness. Apparently the backend team understands their customer and the timelines that the customer wants, and adjusted appropriately.

Just dealing with ads would have been problematic, because those tend to be straight 1080 or 4k with stereo. Nothing fancy, but I'll bet they didn't fit inside the chunk size they were expecting since ads are usually 30 seconds or less. And they don't need the dynamic encoding etc that normal titles do.

I wonder how much benefit dynamic encoding brings in space reduction?

neilv• 2 years ago

Netflix is a company that works with media people. You either understand what that implies or you don't.

It implies everyone is hopped up on drugs?

b33j0r• 2 years ago

It’s not full-circle, as much as industry maturity.

More stories lately are about “why we went pack to monoliths and building with borland C++”

Not long ago it was more likely “how microservices solved everything at our company, and why only morons disagree.”

So are we moving towards or away from microservices? Both. We’re maturing to use the right tools for the system.

sirsinsalot3• 2 years ago

Using the ... right ... tool for the job? In software development?

Surely not! I want my dogmatic clickbait and LinkedIn-style grandstanding thank you very much!

How else will I stay on the hedonic treadmill of staying up-to-date with a new framework or architecture every 3 months?!

chrishare• 2 years ago

True, and yet people are inclined to think the other grass is greener.

trunnell3• 2 years ago

Amazing to read about the continued re-invention of this part of Netflix's internal systems.

I worked with this team (and its predecessors) during my time at Netflix. They achieved several "holy grails" of video encoding: a perceptual quality metric (VMAF), optimal bitrate selection per 2 second video chunk, and then optimal video chunking to be scene based rather than a fixed 2 seconds. Doing any of that in a research lab would be a challenge, but pulling it off at Netflix's scale is epic.

You might need some background on how adaptive video streaming works to fully grok this article.

But this is also just a story about a massive refactoring of a large, critical system. How many companies have you worked for that aggressively pursued refactoring/re-engineering their central systems? At most other places, I've seen risk aversion, fear, and mismanagement conspire to kill innovation. Not so at Netflix.

entropicdrifter• 2 years ago

That sounds like an epic series of projects to have gotten to work on.

devindotcom• 2 years ago

I'm curious about grain encoding - did you work on that at all? My friend heard they were doing a grain extraction layer that was re-added client side. Feel free to contact devin@techcrunch if you have any interesting insight.

mochomocha• 2 years ago

Yes, that's part of the AV1 specs actually. See https://norkin.org/pdf/DCC_2018_AV1_film_grain.pdf

Andrey (who works for Netflix) drove the effort. Chatted with him about it.

aidenn0• 2 years ago

I used to do a poor-mans version of that with low-bitrate mpeg4 (ASF or divX) movie downloads back in the day; the resolution would be low, and the MS MP4 codec tended to blur as well (plus noise killed the quality so much that there was often a denoiser in the encode pipeline), so adding any noise to the playback significantly improved the look of the movie.

ksec5• 2 years ago

Removing Grain and then re-adding it after encoding isn't specific to Netflix or even AV1 as some have suggested. I believe a recent part of H.265 HEVC, or H.266 VVC or MPEG-5 has something similar.

ilrwbwrkhv• 2 years ago

Pornhub serves more diverse data across the world with a much simpler pipeline. They are the ones who should be emulated. Netflix needs a bit of housecleaning to get rid of this faction in the company.

BiteCode_dev2• 2 years ago

Adult website stacks are underrated. They manage to be scrappy and yet deliver a lot.

I've seen streaming services serving half a million HD video users a day running on 7 servers, 2 of them being cheap vps.

martindbp• 2 years ago

As a customer I can't really say that I can tell any difference between all the streaming services, so presumably this type of optimization helps Netflix's bottom line somehow?

sirsinsalot3• 2 years ago

Honestly, many business could learn how to beat-off the competition by emulating the adult industry.

They're often first to come to market, aren't intimidated to try new things, resist being bound by incumbents or gagged by censors and are generally wide open to grasping the full thro ... that's enough now.

solarpunk• 2 years ago

where can one read about pornhub's infra? does mindgeek have an engineering blog?

overstay8930• 2 years ago

You are so right, I have no idea who is running other streaming services but its pretty obvious they have absolutely no idea what they're doing when they're doing bullshit like pushing blocky h.264 streams to iPhone 15 Pros with AV1 and other crap that costs a shit ton of money for a worse outcome.

Paramount looks like Youtube and uses twice as much bandwidth, that is almost impressively bad.

rldjbpin• 2 years ago

my 0.02 on the matter - i can commend that they got a reliable system but the deliverables are not quite comparable imho.

not all "HD" is made equal. bitrate is a major component of the perceived video quality. some experienced it first hand when youtube premium started pushing higher bitrate version for 1080p (https://news.ycombinator.com/item?id=34918698).

(disney+) hotstar is an indian streaming service which recently handled more than 50M+ concurrent viewers (https://news.ycombinator.com/item?id=38344265) for a live sports event. but if you actually watch the stream, it is of a very poor quality.

long story short: hub pushes for bitrate/quality much lower than netflix needs to in order to satisfy its paying customers, especially when they discriminate based on how much one pays.

but i wonder how much it would help with non-cpu scenarios when we need to do transcoding for upcoming codecs which might benefit from using hardware accelerators.

manmal2• 2 years ago

I‘m not impressed by the quality Netflix gives me when on my phone, all while paying for a 4K plan. Streaming a locally hosted 4K file looks just way better. Apple TV+ is approaching that level of quality, so it’s definitely doable.

ksec5• 2 years ago

That has more to do with Bitrate. Giving enough bitrate your video will be great even with older Codec such as baseline H.264. AppleTV+ uses anywhere from double to quadruple the bitrate. So it is much more about cost of business rather than technology barrier.

ksec5• 2 years ago

They achieved several "holy grails" of video encoding: a perceptual quality metric (VMAF), optimal bitrate selection per 2 second video chunk, and then optimal video chunking to be scene based rather than a fixed 2 seconds.

Both being extremely hard problem but I wouldn't say they are "holy grails" level. The latter part is currently being offered by other encoding services. While the first one, VMAF, is much better than all other metric we were previously using, still has many short comings. I was actually hoping all the AI / LLM model we could have something better than VMAF by now.

n4r9• 2 years ago

Are there other examples of LLMs discovering novel high-performing algorithms, models or metrics? I didn't think they were close to that yet.

issafram• 2 years ago

It's pretty easy, just stream the results of ffmpeg. I could create a Netflix platform in one day.

/s if anyone took this seriously

dmillar• 2 years ago

Would be interested to also know how they handle per-title audio. With stereo -> 5.1 -> 7.1 and the sides and wide layout variants, how do they think about this during the inspection and encoding process? Being completely naive to Netflix's source media, and assuming it comes in a variety for formats and media, it seems like there are decision to make there. Though audio obviously has a much lower bandwidth burden, one would think there could still be QoE gains (and bandwidth savings) by doing things that AV1 can do with different scenes in something like OPUS.

devit• 2 years ago

I can't understand the article.

If you want to encode video, use ffmpeg. Netflix serves static movies, so encoding is going to be relatively rare and can probably be done on whatever computing resources are already available. Quality-wise, the ffmpeg/x264/x265 people probably are doing a good job already.

If you want to serve video, serve it with HLS or similar from static files stored on a CDN with a bunch of bitrate profiles. Here the problem is more creating or finding the CDN that anything to do with video.

Can't quite figure out what the purpose of all the stuff in the article could be (maybe to justify the jobs of the people doing the work?)

phantomathkg• 2 years ago

Disclaimer, not working for Netflix, but working in this area.

Ultimately it will still likely to be ffmpeg or their internal fork of it? But they are talking about per-title/per-shot optimisation here and not apply a blanket quality profile for every single video.

Netflix also produce their own series and allow them to optimise their encoding further.

They also have a Video Validation Services and Video Quality Services that do automatic quality check of the encoded video as the post indicated.

Is it complex, yes it is. But is it overly complex, maybe not?

ddorian43• 2 years ago

Thats what they do, but better. Example they have their own CDN and put the nodes in your ISP facility.

Using per-title-encoding, allows you to have less renditions for a given video, having better cache hits.

Another example is pre-fetching data to the CDN nodes(when a new episode or season of a popular show comes out).

Its just extra optimizations that come up and are worth it at scale.

ksec5• 2 years ago

Can't quite figure out what the purpose of all the stuff in the article could be

Netflix encoding is actually much more complex. Both per Scene and per TiTle encoding, optimal bitrate selection etc are a lot more than what ffmpeg offers if you do it manually.

Arguably you are right though Netflix could ignore all of that and just brute force the problem with much higher cost in terms of encoding, storage and bandwidth. But i guess at their scale it make sense to do all these optimisation.

ksec5• 2 years ago

I am much more looking forward to Serving Netflix Video Traffic at 1600Gb/s and Beyond. Hoping drewg123 will share something soon :)

jiggawatts• 2 years ago

Okay, now that that's done, can someone at Netflix please figure out how to use their multiple data centres worth of distributed clusters to serve more than 4 or 5 subtitle languages please?

If anyone at Netflix would like some assistance, I've previously consulted in the areas of large-scale compression optimisation, and I'm sure we can get those 100KB text files down to under 20KB!

I'll help build distributed Kubernetes buzzword-compliant architectures, if that helps anyone get internal promotions as a part of this pan-cultural effort of inclusivity.

Rebuilding Netflix's video processing pipeline with microservices

Related Stories