Netflix buffering issues: Boxing fans complain about Jake Paul vs. Mike Tyson

https://www.sportingnews.com/us/boxing/news/netflix-buffering-livestream-issues-boxing-jake-paul-mike-tyson/327ee972d4b14d90cc370461

by storf45 • 2 years ago

492 921 2 years ago

sportingnews.com

suzzer992• 2 years ago

On a few forum sites I'm on, people are just giving up. Looking forward to the post-mortem on how they weren't ready for this (with just a tiny bit of schadenfreude because they've interviewed and rejected me twice).

itsthecourier• 2 years ago

They sabotaging OP just for a reverse schadenfreude play

Amfy• 2 years ago

Can you share which forums

suzzer992• 2 years ago

Chiefsplanet.com, unstuckpolitics.com, my buddies on group text :)

boppo12• 2 years ago

/r/netflix and /sp/

dyauspitr4• 2 years ago

It’s not everyone. Works fine for me though I did have to reload the page when I skipped past the woman match to the Barrios Ramos fight and it was stuck buffering at 99%.

icameron• 2 years ago

You skipped the best part.

moralestapia7• 2 years ago

The post-mortem will be interesting indeed.

scruple• 2 years ago

AB84 streamed it live from a box at the arena to ~5M viewers on Twitter. I was watching it on Netflix, I didn't have any problems, but I also put his live stream up for the hell of it. He didn't have any issues that I saw.

almost_usual• 2 years ago

He didn't have any issues that I saw.

He’s definitely got issues..

HDThoreaun2• 2 years ago

were calling antonio brown ab84 now? What happened to Mr. BC?

tiffanyh2• 2 years ago

Dupe: https://news.ycombinator.com/item?id=42153906

josh2600• 2 years ago

FWIW, works fine for me.

iancmceachern• 2 years ago

Us too

roamerz• 2 years ago

Been working great for me as well. Starlink in Oregon.

whoknowsidont5• 2 years ago

Please don't make these types of comments, they mean nothing and they serve no purpose.

pentagrama• 2 years ago

It means, it is different if the service goes down to 100, 50, 10 percent of users. I watched the show with no issues.

rascul2• 2 years ago

Why is it acceptable to share that it doesn't work but not acceptable to share that it does?

gbil• 2 years ago

The stream never buffered on my side but quality was for the whole duration of the stream pretty basic I doubt it was even 720p

Justin_K• 2 years ago

I love how I can come to HN to instantly find out if it’s Netflix or my WiFi.

jfdi• 2 years ago

Right?!

slicktux2• 2 years ago

This! I was checking my WiFi and then I instinctively checked HN and what do you know!

Jiahang• 2 years ago

metoo！

dghughes• 2 years ago

Wifi wifi or wifi as in your ISP Internet connection? Sp many people now call an Internet connection "wifi".

Anyway, network cable is the only way to go!

anakaine6• 2 years ago

"Many people", those who call their ISP connection WiFi, are technology potato's.

rascul2• 2 years ago

I have the wifis with the geebees.

bluSCALE4• 2 years ago

Looks like shit for me. Buffered a bit as well.

freditup• 2 years ago

I wonder if there will be any long term reputational repercussions for Netflix because of this. Amongst SWEs, Netflix is known for hiring the best people and their streaming service normally seems very solid. Other streaming services have definitely caught up a bit and are much more reliable then in the early days, but my impression still has always been that Netflix is a step above the rest technically.

This sure doesn't help with that impression, and it hasn't just been a momentary glitch but hours of instability. And the Netflix status page saying "Netflix is up! We are not currently experiencing an interruption to our streaming service." doesn't help either...

ocdtrekkie• 2 years ago

So the issue is that Netflix gets its performance from colocating caches of movies in ISP datacenters, and a live broadcast doesn't work with that. It's not just about the sheer numbers of viewers, it's that a live model totally undermines their entire infrastructure advantage.

See: https://openconnect.netflix.com/en/

pinkmuffinere• 2 years ago

Damn that sucks. I wonder if they could have intentionally streamed it 5 min late? I don’t have all the context around the fight though — maybe a competing service would win if Netflix intentionally induced delay?

stingraycharles• 2 years ago

Correct, this is not Netflix’ regular cup of tea, and it’s a very different problem to solve. They can probably use their edge caches, but it’s challenging.

sgarland7• 2 years ago

I wonder how effective it would be to cache live events with a delay. Write to the tail, read from the head.

bushbaba• 2 years ago

Not sure I fully buy that. The “live” stream is rarely “live”. It’s often a highly cached buffer that’s a few mins from latest. Those in isp caches can still help here.

h4l• 2 years ago

That model still works for streaming. You have a central source stream only to the distributed edge locations, then have clients only stream from their local edge location. Even if one region is overwhelmed, the rest can still work. Load on the central source is bounded.

_dark_matter_2• 2 years ago

I don't think that live doesn't work with caches. No one watching live would care about a O(s) delay, which is highly amenable to caching at ISPs and streaming changes from there to downstream clients. Offhand I'd say that would support O(ms) delay but no less.

ethbr12• 2 years ago

I'm curious if the root cause is more variable than usual latency.

Sample size 1, but...

I saw a ton of buffering and failure on an embedded Netflix app on a TV, including some infinite freezes.

Moved over to laptop, zero buffering.

I assume the web app runs with a lot bigger buffer than whatever is squeezed into the underpowered TV.

tnvmadhav2• 2 years ago

they will learn :)

jimbob45• 2 years ago

Yeah, the funny part is that Hulu, Amazon Prime, and Peacock have all demonstrated the ability to handle an event of this caliber with no issue. Netflix now may never get another opportunity like this again.

dyauspitr4• 2 years ago

Sure they will. They’ll just set up the next event and outside of some tech folks no one will remember this.

silisili4• 2 years ago

AFAIK those three farm out to CDNs with tons of edges who know what they're doing.

I have a feeling Netflix said 'how hard could this be?' and is finding out right now.

anshumankmr3• 2 years ago

Chances are Jake might fight Connor McGregor. Sure, Connor is not as famous as Tyson but that will also invite a large number of people to stream it.

bsimpson3• 2 years ago

I don't spend much time streaming, but I got a glimpse of the Amazon Prime catalog yesterday, and was surprised at how many titles on the front page were movies I'd actually watch. Reminded me of Netflix a dozen years ago.

nicce3• 2 years ago

Amazon Prime front page includes currently a lot of ads for movies that you can rent or ”buy”. Are you sure these movies weren’t them?

LollipopYakuza• 2 years ago

Prime Video has to be the worst of all major streaming services. The video quality is horrible, its crippled with ads (3 not skippable ads for an episode of 45 minutes, lastly), and a lot of interesting titles are behind a "partner paywall".

stackghost• 2 years ago

Reminded me of Netflix a dozen years ago.

It's been pretty rough the last few years. So many great films and series, not to mention kids programming, removed to make way for mediocre NeTfLiX oRiGiNaLs and Bollywood trash.

Spooky232• 2 years ago

Netflix pivoted to be a platform to waste as much of your time as possible vs entertain.

The Amazon originals are way better imo. They do the dark pattern crap with paid content, as one would expect from Amazon.

spike0216• 2 years ago

Amazon Prime isn't so great. Lots of for rent/purchase content or content with ads these days. And they end up repeating slots of content in all the rows in their UI, so I end up seeing the same suggestions everywhere rather than much that's new (other than first party productions).

To me they're basically padding their front page.

But honestly that's most of the major streaming platforms these days. I recently cancelled Disney Plus for similar reasons. The only reasons I don't cancel prime or Netflix are because I have family members I split the memberships with to share.

ilaksh4• 2 years ago

I think they have to refund the fees for a month to anyone who streamed this fight. That's the only thing that seems fair.

It has been pretty useless. At the moment seems to be working only when running in non-live mode several minutes behind.

So if there are 1 million trying to stream it, that means they would lose $15 million. So.. they might only give a partial refund.

But people should push for an automatic refund instead of a class action.

SoftTalker5• 2 years ago

Is it really that big a deal if you are watching a few minutes behind?

I've watched ball games on streaming networks where I can also hear a local radio broadcast, and the stream is always delayed compared to the radio, sometimes by quite a lot. But you'd never know it if you were just watching the stream.

jeromegv• 2 years ago

Not the same demographic but their last large attempt at live was through a Love is blind reunion. It was the same thing, millions of people logging in, epic failure, nothing worked.

They never tried to do a live reunion again. I suppose they should have to get the experience. Because they are hitting the same problems with a much bigger stake event.

barkingcat2• 2 years ago

yup wanted to say that live stream stuttering has happened before on Netflix - I don't think the reputation is deserved.

From a livestreaming standpoint, netflix is 0/x - for many large events such as love is blind, etc.

From a livestreaming standpoint, look to broadcast news, sports / Olympics broadcasters, etc and you'll see technology, equipment, bandwidth, planning, and professionalism at 1000x of netflix.

Heck, for publicly traded quarterly earnings livestream meetings, they book direct satellite time in addition to fiber to make sure they don't rely only on terrestrial networks which can fail. From a business standpoint, failure during a quarterly meeting stream can mean the destruction of a company (by making shareholders mad that they can't see and vote during the meeting making them push for internal change) - so the stakes are much higher than live entertainment streaming.

Netflix is good at many things, livestreaming is not one of those things.

KennyBlanken• 2 years ago

Is anyone surprised? I don't see how their infrastructure can handle this when it was designed for non-realtime precaching of prerecorded content.

tylerchilds2• 2 years ago

this is false, the tom brady roast was live streamed

yes, love is blind failed, but was definitely not the most recent attempt. they did some other golf thing too, iirc

Spooky232• 2 years ago

It may vary by ISP. It’s been fine for me.

graton• 2 years ago

I think you are correct. Ziply Fiber said they were seeing 2.1 times their normal peak [1].

But also people were saying they weren't having any issues streaming on Ziply.

[1] https://www.reddit.com/r/ZiplyFiber/comments/1gsenik/netflix...

rvz3• 2 years ago

I don't think Netflix is even designed to handle very extreme multi-region live-streaming at scale as evidenced in this event with hundreds of millions simultaneously watching.

YouTube, Twitch, Amazon Prime, Hulu, etc have all demonstrated to stream simultaneously to hundreds of millions live without any issues. This was Netflix's chance to do this and they have largely failed at this.

There are no excuses or juniors to blame this time. Quite the inexperience from the 'senior' engineers at Netflix not being able to handle the scale of live-streaming which they may lose contracts for this given the downtime across the world over this high impact event.

Very embarrassing for a multi-billion dollar publicly traded company.

gonzo412• 2 years ago

They must have try to do this on the cheap, thinking they could dynamically scale on the fly for this. Big mistake.

2OEH8eoCRo0• 2 years ago

Is cable and broadcast better for live TV? No scaling issues. Doesn't matter how many people tune in.

Valord• 2 years ago

Seems like it

betaby2• 2 years ago

Interestingly that TV nowadays is delivered through IP.

vitus6• 2 years ago

It's fundamentally different, for sure.

Most third-party internet-based streaming solutions are overlaid on top of a point-to-point network, while broadcast is one-to-many, and even cable tends to use multicast within the cable provider's network.

You have potentially different problems, e.g. limited bandwidth / spectrum. If, say, there are multiple games going on at the same time, you can only watch whichever feed the broadcaster decides to air. And, of course, regardless of the technology in use, there are matters of acquiring rights for various events. One benefit of internet-based streaming is that one service can acquire the rights and be able to reach everyone, whereas an individual cable provider might only reach its direct subscribers.

oaththrowaway2• 2 years ago

I used to work for a live streaming platform once. We always joked that VOD (Netflix) was "easy" compared to live.

taeric• 2 years ago

Not really a joke, though? VOD has obvious methods to cheat a bit. Redundancy abounds and you can even network shape for costs. Could probably get even better compression for clear reasons.

Live, not so much. One source that you have to fanout from and absolutely no way to get cheap redundancy. Right?

noncoml• 2 years ago

their streaming service normally seems very solid

Not trying to downplay their complexity, but last I heard Netflix is splitting the shows in small data chunks and just serves them as static files.

Live streaming is a different beast

treflop3• 2 years ago

Static files have been pretty much the standard streaming protocols for both VOD and live for the last 15 years ago. Before, it was Adobe Flash (RTMP).

With the way that they are designed, you can even use a regular CDN.

astrange• 2 years ago

That's how streaming (usually) works. The main URL is a playlist of transport stream files and it just downloads them in the background as you go.

a_random_name2• 2 years ago

If Netflix still interviews on hacker rank puzzles I think this should be a wake up call. Interviewing on irrelevant logic puzzles is no match for systems engineering.

rvz3• 2 years ago

For systems design and engineering, absolutely this. I expected the very highest standards and upmost uptime from Netflix, similar to Google and Amazon.

Tells you the uselessness of their engineering blogs.

VirusNewbie2• 2 years ago

I did a round of netflix interviews, didn't get an offer (but passed the technical coding rounds) they absolutely had the best interview process of any company I've interviewed at my entire career.

They do make you code but the questions were 1. Not on hacker rank or leetcode 2. Pratical coding questions that didn't require anything more than basic hashmaps/lists/loops/recursion if you want. Some string parsing, etc.

They were still hard, you had to code a fast, but no tricky algorithms required. It also felt very collaborative, it felt like you were driving pair programming. Highly recommended even though didn't get an offer!

that_guy_iain5• 2 years ago

I don't think it'll be long-term. Most people will forget about this really quickly. It's not like there will be many people saying "Oh, you don't want to sign up for Netflix, the Tyson fight wasn't well streamed" in even 6 months nevermind 10 years.

lawgimenez• 2 years ago

There's an upcoming NFL game on Netflix next month. They need to get their shit together.

caseyohara2• 2 years ago

Two games actually, both on Christmas Day. A day when most people are at home or the home of family or friends, and they are both pretty good late-season matchups (Chiefs-Steelers and Ravens-Texans) so I imagine viewership will be high.

If they botch the NFL games, it will surely hurt their reputation.

jpalawaga2• 2 years ago

From what I've heard, Netflix has really diluted the culture that people know of from the Patty McCord days.

In particular, they have been revising their compensation structure to issue RSUs, add in a bunch of annoying review process, add in a bunch of leveling and titles, begin hiring down market (e.g. non-sr employees), etc.

In addition to doing this, shuffling headcount, budgets, and title quotas around has in general made the company a lot more bureaucratic.

I think, as streaming matured as a solution space, this (what is equivalent to cost-cutting) was inevitable.

If Netflix was running the same team/culture as it was 10 years ago, I'd like to say that they would have been able to pull of streaming.

neilv2• 2 years ago

Were they not able to hire enough top-skilled people? If not, why not?

Or did they have a lot of needs that they decided didn't require top-skilled people?

Or was this a beancounter thing, of someone deciding that the company was paying more money on staffing than they needed to, without understanding it?

squeaky-clean3• 2 years ago

Has Netflix ever live streamed something before? People on reddit are reporting that if you back up the play marker by about 3 minutes the lag goes away. They've got a handle on streaming things when they have a day in advance to encode it into different formats and push it to regional CDNs. But I can't recall them ever live streaming something. Definitely nothing this hyped.

ipaddr• 2 years ago

Love is blind reunion which had major problems. Chris Rock comedy which only had a few issues. The Netflix Cup which had issues.

StevePerkins• 2 years ago

Chris Rock comedy special, and the Tom Brady roast. Nothing on this scale, though.

Fire-Dragon-DoL• 2 years ago

For me, netflix constantly forget the last episode/spot I was in a TV show. Beyond frustrating

geodel3• 2 years ago

I think Netflix will have even more sw engineers looking to work there once they notice even for average quality of work they can get paid 3 times more than their current pay.

test6554• 2 years ago

Yea, it’s a bad look. But I switched to watching some other Netflix video and it seemed fine. Just this event had some early issues. Looks fine now though.

rapind2• 2 years ago

ut my impression still has always been that Netflix is a step above the rest technically.

I always assumed youtube was top dog for performance and stability. I can’t remember the last time I had issues with them and don’t they handle basically more traffic than any other video service?

normie30003• 2 years ago

Maybe a client issue, but i've got a low-end smart tv which handles netflix fine, but youtube is unwatchable due to buffering and failed cuts to adverts

kenjackson• 2 years ago

Streamed glitch free for me both on my phone and Xbox. The fight wasn’t so great though, but still a fun event. Jake Paul is a money machine right now.

tnvmadhav2• 2 years ago

same. could be a bandwidth issue at CDN/ISP level at a certain region(s) ?

ikt• 2 years ago

Based on this I'm wondering whether it was straight up they did not expect it to be this popular?

Some Cricket graphs of our #Netflix cache for the #PaulVsTyson fight. It has a 40 Gbps connection and it held steady almost 100% saturated the entire time.

https://fosstodon.org/@atoponce/113491103342509883

dh20222• 2 years ago

I think why I will remember about this fight is not the (small) streaming issue I encountered as much as the poor quality of the fight itself. For me that was the reputational loss. Netflix was touting “NFL is coming to Netflix”. This fight did not really make me want to watch that.

BLKNSLVR• 2 years ago

I don't care about boxing or UFC or the grade-A douchebags that are the Paul brothers, but I tuned in just because I had the time and a Netflix subscription.

It was actually great that the fight itself was so boring because it justifies never having to spend time / money on that kind of bullshit. It was a farce. A very bright, loud, sparkly, and expensive (for some people) farce.

The value I got from it was the knowledge that missing out on that kind of thing isn't really missing out on anything at all.

tass2• 2 years ago

Was live streaming much of a use case for them before this?

They stream plenty of pre recorded video, often collocated. Live streaming seems like something they aren’t yet good at.

someothherguyy• 2 years ago

https://www.cnn.com/2023/03/04/entertainment/chris-rock-netf... (2023-03-06)

fred_is_fred2• 2 years ago

If places like Paramount+ can figure it out, Netflix, given their 10+ year head start on streaming and strong engineering culture, should also have been able to. And if you don't like my example, literally every other streaming service has streamed live sports without issue. YT TV, Hulu, Paramount+, Amazon Prime, Peacock, even Apple TV streams live sports.

It may be "new" to them, but they should have been ready.

apexalpha• 2 years ago

but my impression still has always been that Netflix is a step above the rest technically.

Maybe if we're not counting Youtube as 'streaming', but in my mind no one holds a candle to YT quality in (live)streaming.

Yeul3• 2 years ago

There's a difference between live broadcasts and serving up content that's sitting on a server I guess?

In my country every time there's a big football match the people who try to watch it on the internet face issues.

MyFedora• 2 years ago

Netflix won't take a hit here.

Most people pay Netflix to watch movies and tv shows, not sports. If I hadn't checked Hacker News today, I wouldn't even know they streamed sports, let alone that they had issues with it. Even now that I do, it doesn't affect how I see their core offering, which is their library of on-demand content.

Netflix's infrastructure is clearly built for static content, not live events, so it's no shock they aren't as polished in this area. Streaming anything live over the internet is a tough technical challenge compared to traditional cable.

oplav• 2 years ago

Netflix is trying to expand into live sports. This event wasn’t a one off thing. There is an NFL game they are streaming at the end of the year.

Willingham• 2 years ago

I can feel the pressure on the network engineers from here XD

nightowl_games2• 2 years ago

From my experience, it works if your not watching it 'live'. But the moment I put my devices to 'live' it perma-breaks. 504 gateway timed out in web developer tools hitting my local CDN. probably works on some CDNs, doesnt on others. Probably works if your not 'live'

edit: literally a nginx gateway timed out screen if you view the response from the cdn... wow

jeffbee2• 2 years ago

Why do they want to get into the live business? It doesn't seem to synergize with their infrastructure. Sending the same stream in real time to numerous people just isn't the same task as letting people stream optimized artifacts that are prepositioned at the edge of the network.

dyauspitr4• 2 years ago

They want to break into sports because it’s such a big business and if you do sports you need to be able to stream live.

smegger0012• 2 years ago

Because its the one area traditional networks have the advantage in

VirusNewbie2• 2 years ago

Most PPV is what, $50-$70? So subscribing to Netflix for $20 or whatever per month sounds like a bargain for anyone who is interested and not already a customer. Then assume some large percentage doesn’t cancel either because they forgot, or because they started watching a show and then decided to keep paying.

dilyevsky5• 2 years ago

Not sure why this is being downvoted. I can see your point - it’s much harder to this live but a lot of their cdn infra can be reused.

SkyPuncher• 2 years ago

Live is the only thing that won’t be commodified entirely. “Anyone” can pump out stream-when-you-want TV shows. Live events are generally exclusive, unpredictable, and cultural moments .

jairuhme• 2 years ago

Every time it buffers for me, Netflix does an internet test only for it to come back and say its fast...

cranberryturkey• 2 years ago

yeah i'm using iptv which is just a rip of NF and its stuck buffering.

rvz3• 2 years ago

All these engineering blog posts, distributed systems and these complex micro-services clearly didn't help with this issue.

Netflix is clearly not designed nor prepared for scalable multi-region live-streaming, no matter the amount of 'senior' engineers they throw at the problem.

ilrwbwrkhv6• 2 years ago

This. Overly complex nonsense that they have built up is crazy and the fact that we on a tech forum agree to all of this is crazier.

It's almost like this platform has been taken over by JavaScripters completely.

sgarland7• 2 years ago

taken over by JavaScripters

That is an incredible way to phrase the sentiment, thank you.

robertlagrant6• 2 years ago

Netflix is clearly not designed nor prepared for scalable multi-region live-streaming

Well, yes. Who would think Netflix was designed for that? They do VOD. They're only trying to move into this now.

yoshamano• 2 years ago

I did some VPN hopping and connecting to an endpoint in Dallas has allowed me to start watching again. Not live though, that throws me back into buffering hell.

runjake• 2 years ago

It’s like watching a Minecraft cosplay of the event.

yapyap• 2 years ago

Hell, I’d complaing about Jake Paul vs Mike Tyson as well if I was a boxing fan. Even without buffering issues

koolba2• 2 years ago

They better get have some better judges and refs too. The co-headline title fight was a joke.

smolder• 2 years ago

Yes, it was utterly boring, but they made their money. I don't like either Paul brother, so I only watched in hopes shorter, much-older Tyson would make Jake look as foolish as he is.

ackbar03• 2 years ago

This reminds me of that scene in Silicon Valley

neilv2• 2 years ago

https://www.youtube.com/watch?v=9IGvzb-KCpY&t=50s

abixb• 2 years ago

I clicked on this thread to type that exact thing, holy smokes.

You're referring to Hooli's streaming of UFC fight that goes awry and Gavin Belson totally loses it, lol. Great scene and totally relevant to what's happening with Netflix rn.

a_random_name2• 2 years ago

Ota broadcasts are clearer

causality0• 2 years ago

Weird that an organization like Netflix is having problems with this considering their depth of both experience and pockets. I wonder if they didn't expect the number of people who were interested in finding out what the pay-per-view experience is like without spending any extra money. Still, I suppose we can all be thankful Netflix is getting to cut their live event teeth on "alleged rapist vs convicted rapist" instead of something more important.

marxisttemp• 2 years ago

alleged rapist vs convicted rapist

And you’ll never guess which Presidential candidate they both support!

benreesman• 2 years ago

This Serrano fight is just an insane display of excellence.

If anyone was waiting for the main card to tune in, I recommend tuning in now.

Waterluvian• 2 years ago

I know nothing about boxing and this fight was just ridiculously impressive. I kept tuning out of the earlier fights. They felt like some sort of filler. I didn’t get the allure. But Taylor v Serrano was just obvious talent that even I could appreciate it.

vFunct• 2 years ago

Serrano was robbed!

slicktux2• 2 years ago

That was a savage fight!

anakaine6• 2 years ago

Absolutely excellent fight. 10 full rounds with full effort until the end. Fantastic.

Also, no buffering issues on my end. Have to wonder if it's a regional issue.

29athrowaway2• 2 years ago

Serrano should have won.

agrippanux• 2 years ago

What was an amazing fight - that Serrano won. I have no idea how Taylor was scored the winner.

cco• 2 years ago

naw, taylor head butting the whole fight was dirty and really took the wind out of it

sgarland7• 2 years ago

A friend and I, in separate states, found that it wouldn’t stream from TVs, Roku, etc. but would stream from mobile. And for me, using a mobile hotspot to a laptop; though that implies checking IP address range instead of just user-agent, so that seems unlikely.

Anyway, I wouldn’t be surprised if they were prioritizing mobile traffic because it’s more forgiving of shitty bitrate.

arcbyte• 2 years ago

I just left a bar streaming it on a smart TV and back in my home it's streaming on the Roku just fine.

sgarland7• 2 years ago

Guess I was looking for explanations too hard.

anakaine6• 2 years ago

I wonder if this points to network peering and edge nodes. Mobile network vs cabled network likely being routed to different places.

wonderwonder• 2 years ago

Is this potentially an aws issue?

spencerchubb2• 2 years ago

I would assume not because twitch runs on aws. I think netflix engineers haven't optimized as much for livestreaming like twitch has

anshumankmr3• 2 years ago

I do not think twitch has the amount of concurrent users netflix might have had today morning for the fight.

chevman• 2 years ago

Guess they should have livestreamed it on X to be safe!

kevinventullo• 2 years ago

Or Facebook. Or YouTube. Or Vimeo. Or LiveLeak.

spyda56• 2 years ago

Working okay for me

voyagerfoil• 2 years ago

Over promised and under delivered. That’s a bad look

boppo12• 2 years ago

I'm watching on a 'pirate' stream because my netflix stream is absolutely frozen.

walrushunter• 2 years ago

I'm an engineering manager at a Fortune 500 company. The dumbest engineer on our team left for Netflix. He got a pay raise too.

Our engineers are fucking morons. And this guy was the dumbest of the bunch. If you think Netflix hires top tier talent, you don't know Netflix.

moralestapia7• 2 years ago

I can +1 with a similar anecdote.

They obviously have some really good engineers, but many low-tier ones as well. No idea how long they stay there, though.

I'm watching the fight now and have experienced the buffering issues. Bit embarrassing for a company that fundamentally only does a single thing, which is this. Also, yeah, 900k TC and whatnot but irl you get this. Mediocre.

spencerchubb2• 2 years ago

livestream is quite different from streaming pre-processed video, so I'm not surprised by the scaling issues

anakaine6• 2 years ago

I have to wonder if its a regional thing. I'm watching from the southern pacific in HD, and its been excellent.

reagan83• 2 years ago

Curb your enthusiasm had a good segment of episodes that were a parody on Netflix and how they shifted hiring from merit to other criteria.

indigodaddy• 2 years ago

Curb did? What season/episodes?

ldjkfkdsjnv2• 2 years ago

Its all ego when these companies think they hire the best.

0xpgm2• 2 years ago

An engineering manager who thinks his engineers are morons and dumb?

I have questions..

wordofx• 2 years ago

None of the Fortune 500 companies hire top talent. They have a few good people but 98% of the engineers are average at best. Over paid.

qingcharles• 2 years ago

This is every dev house I've worked at. For most people (mostly not the ones on HN), coding is a 9-5 job. No ambition. Just lines of code. Go home. I don't know there is anything particularly wrong with that.

You just have to accept most staff at any corporation are simply just average. There has to be an average for there to be better and worse.

AlotOfReading• 2 years ago

I've never seen a team that has somehow managed to hire exclusively morons. Even the shittiest of call center jobs and construction crews have a few people with enough brain cells to tie their shoelaces.

Have you considered that maybe you're being overly harsh about your co-workers? Maybe take the fact that one of them was hired by a top paying employer as a sign that you should improve your own ability to judge skill?

theendisney2• 2 years ago

I've seen tons of them! The formula is to create conditions that will make even slightly competent people leave. They hire their morron nephew, he is always 30 min late then they moan when you are 5 min late because the parking lot was blocked by their car. He always leaves 2 hours early while you do overtime that they regularly forget to pay for. Your day is filled with your own work PLUS that of your retarded coworkers who only drink coffee while joking about you doing their work. You are not as fast as the last guy! haha! If something goes wrong the morrons collectively blame you, just like the last time. You get a formal warning. etc etc The other normal person they hire is let go after 2 days because they complaint which means they didnt fit the team.

And so on

If he still works there the morron who left was less of a.

djbusby3• 2 years ago

Why is your team morons? Kind of disparaging maybe? Fish rots from the head situation?

that_guy_iain5• 2 years ago

Having worked with a bunch of guys who have gone on to "top teams", I no longer believe they have top teams. My fav was the guy who said the system could scale indefinitely after it literally fell on its ass from too much traffic. He couldn't understand that just because Lambdas my themselves can scale, they are limited by the resources they use, so just ignored it and insisted that it could. The same guy also kept on saying we should change the TPEG standard because he didn't like how it worked. And these companies are seriously pretending they've got the best and brightest. If that's really true, I really need to find another profession.

YZF6• 2 years ago

I've worked for many companies that said they hired the best. And to be honest when I hire I also try to hire good people. I think I could hire better if a) I had an open cheque, b) I was running coolest project in the universe. I did hire for some interesting projects but nothing close to an open cheque. Even under these conditions it's tough to find great people. You can go after people with a proven track record but even that doesn't always guarantee their next project will be as successful.

The reality though is that large companies with thousands of people generally end up having average people. Some company may hire more PhD's. But on average those aren't better software engineers than non-PhD's. Some might hire people who are strong competitive coders, but that also on average isn't really that strong of a signal for strong engineers.

Once you have a mix of average people, on a curve, which is the norm, the question becomes do you have an environment where the better people can be successful. In many corporate environments this doesn't happen. Better engineers may have obstacles put in front of them or they can forced out of the organization. This is natural because for most organizations can be more of a political question than a technical question.

Smaller organizations, that are very successful (so can meet my two criterias) and can be highly selective or are highly desirable, can have better teams. By their nature as smaller organizations those teams can also be effective. As organizations grow the talent will spread out towards average and the politics/processes/debt/legacy will make those teams less effective.

kkielhofner• 2 years ago

I'm an engineering manager

How are you involved in the hiring process?

Our engineers are fucking morons. And this guy was the dumbest of the bunch.

Very indicative of a toxic culture you seem to have been pulled in to and likely have contributed to by this point given your language and broad generalizations.

Describing a wide group of people you're also responsible for as "fucking morons" says more about you than them.

pfannkuchen• 2 years ago

Reputation usually lags reality by 5+ years. See: Google.

renewiltord2• 2 years ago

Absolutely right. Netflix was once all about the sports team mentality. Now they’re Man Utd.

Lammy2• 2 years ago

I hope to never have a manager who is mentally stack ranking me and my coworkers in terms of perceived dumbness instead of in terms of any positive trait.

geodel3• 2 years ago

Btw how do you know your current manager is not doing that.

bryanrasmussen4• 2 years ago

dumbness is ranking intelligence, which is a positive trait, dumbness is just a metric for how often intelligence fails.

Example - the manager who started this sub-thread may be a pretty smart guy and able to accurately rate the intelligence of the engineers at his organization - but he had a minor momentary failing of intelligence to post on HN calling those engineers fucking morons.

You've got to rank how often the intelligence fails in someone to be able to figure out how reliable their intelligence is.

JamesBarney• 2 years ago

Almost everyone I know manager or not is usually ranking everyone they work with on various attributes.

In fact it would be incredibly weird to ask a close friend who at their work kicks ass and who sucks and have them respond back, "I've never really thought about how good any of my coworkers were at their jobs"

strken3• 2 years ago

I'm not a manager and I don't stack rank people, but I am 100% capable of knowing when one of my co-workers or predecessors is a fucking moron.

The trick is to use my massive brain to root cause several significant outages, discover that most of them originate in code written by the same employee, and notice that said employee liked to write things like

    // @ts-nocheck
    // obligatory disabling of typescript: static typing is hard, so why bother with it?
    async function upsertWidget() {
      try {
        // await api.doSomeOtherThings({ ... })
        // 20 line block of commented-out useless code
        // pretend this went on much longer
        let result = await api.createWidget({ a, b, c })
        if (!result.ok) {
           result = await api.createWidget({ a, b }) // retries for days! except with different args, how fun
           if (!result.ok) {
             result = await api.updateWidget({ a, b, c }) // oh wait, this time we're updating
           }
        }
        // notice that api.updateWidget() can fail silently
        // also, the three function calls can each return different data, I sure am glad we disabled typescript
        return result
      } catch (error) {
        return error // I sure do love this pattern of returning errors and then not checking whether the result was an error or the intended object 
      }
    }

    function doSomething() {
      const widget = await upsertWidget()
    }

...except even worse, because instead of createWidget the name was something far less descriptive, the nesting was deeper and involved loops, there were random assignments that made no goddamn sense, and the API calls just went to an unnecessary microservice that was only called from here and which literally just passed the data through to a third party with minor changes. Those minor changes resulted in an internal API that was actually worse than the unmodified third party API.

I am so tired of these people. I am not a 10x rockstar engineer and not without flaws, but they are just so awful and draining, and they never seem to get caught in time to stop them ruining perfectly fine companies. Every try>catch>return is like an icy cat hand from the grave reaching up to knock my coffee off my desk.

spike0216• 2 years ago

Hmm. Engineering managers should be setting the team culture and determining best criteria for extending an offer to a candidate. If theres a problem with the hiring process I'd look for the closest source that could or should be fixing it.

I don't think I'd want to work for you.

geodel3• 2 years ago

It mostly makes sense to me. From their bombastic blogs to github projects full of overwrought Enterprise java design patterns. The only thing great about Netflix is it pays a lot more.

globalnode• 2 years ago

this is why managers get a bad rap. what proportion think like this? hopefully not a large one but i do worry. ultimately if the team sucks its because of the management. theyre the ones with the greatest power to change things for the better.

AdieuToLogic• 2 years ago

I'm an engineering manager at a Fortune 500 company. The dumbest engineer on our team left for Netflix. He got a pay raise too.

Apparently he was smart enough to get away from the Fortune 500 company he worked at, reporting to yourself, and "got a pay raise too."

Our engineers are fucking morons. And this guy was the dumbest of the bunch.

See above.

If you think Netflix hires top tier talent, you don't know Netflix.

Maybe you don't know the talent within your own organization. Which is entirely understandable given your proclamation:

  Our engineers are fucking morons.

Then again, maybe this person who left your organization is accurately described as such, which really says more about the Fortune 500 company employing him and presumably continues to employ yourself.

IOW, either the guy left to get out from under an EM who says he is a "fucking moron" or he actually is a "fucking moron" and you failed as a manager to elevate his skills/performance to a satisfactory level.

JCharante6• 2 years ago

or he actually is a "fucking moron" and you failed as a manager to elevate his skills/performance to a satisfactory level.

sometimes managers don't have the authority to fire somebody and are forced to keep their subordinates. Yes good managers can polish gold, but polishing poop still results in poop.

briansm2• 2 years ago

White-Knighting for 'fucking morons' is not a good look though. You'll end up in a world where packets of peanuts have a label on saying 'may contain nuts'.

ripped_britches• 2 years ago

This is the funniest thing I’ve read today

IshKebab• 2 years ago

failed as a manager to elevate...

Managers aren't teachers. They can spend some time mentoring and teaching but there's a limit to that. I've worked with someone who could not write good code and no manager could change that.

Most people I've worked with aren't like that of course (there's really only one that stands out), so maybe you've just been lucky enough to avoid them.

I do find it unlikely that all of his engineers are morons, but on the other hand I haven't worked for a typical fortune 500 company - maybe that's where all the mediocre programmers end up.

midtake• 2 years ago

Top troll bro

29athrowaway2• 2 years ago

What are the chances that your entire engineering team is entirely composed of low performers or people with bad attitude or whatever you designate as "fucking morons"?

It's more likely that you are bad at managing, growing and motivating your team.

Even if it was true, to refer to your team in this way makes you look like you are not ready for management.

Your duty is to get the most out of the team, and that mindset won't help you.

notadoomer236• 2 years ago

Don’t agree. Sometimes you can observe the world around you, and it’s not pretty. Are they not allowed to observe the truth as they see it? What if they are right?

ohyes2• 2 years ago

Sometimes if everyone else is the problem you are the problem.

gamblor9562• 2 years ago

Our engineers are fucking morons

If your "dumbest engineer" got a job and a hefty raise going to Netflix, it means he was very capable engineer who was playing the part of moron at this Fortune 500 company because he was reporting to a manager who was calling him and the entire team morons and he didn't feel the need to go above and beyond for that manager.

Also, highly likely that it was the manager that was the moron and not everyone around him.

throwawaythekey2• 2 years ago

If your "dumbest engineer" got a job and a hefty raise going to Netflix, it means he was very capable engineer who was playing the part of moron

It's also possible that there's very little correlation between capability, reputation and salary.

Don't we all know someone who is overpaid? There are more than a few well known cases of particular employers who select for overpaid employees...

reactordev• 2 years ago

Sounds like he got a better deal. If this is how you describe your team, I suspect they are all submitting their resumes hoping to get away from you.

cultofmetatron• 2 years ago

Our engineers are fucking morons.

I interviewed at Netflix a few years ago; with several of their engineers. One thing I cannot say is that they are morons.

their interview process is top notch too and while I was ultimately rejected, I used their format as the base template for how I started hiring at my company.

silisili4• 2 years ago

I don't have a dog in this fight, but you typically use your A players for hiring/interviews.

It can be both true that Netflix has God tier talent and a bunch of idiots. In fact, that's probably true of most places. I guess the ratio matters more.

tayo42• 2 years ago

what seemed good about it that makes different then any other hiring process that seems detached from the job?

blinded3• 2 years ago

You ever thought they were doing the bare minimum and studying at night to leave?

justmarc• 2 years ago

Your job must be truly awful.

throw10920• 2 years ago

I'm going to avoid leaving a zero-effort response like "actually you're the problem" like half of the replies and contribute:

Why do you call your engineers morons? Is it a lack of intelligence, a lack of wisdom, a lack of experience, inability to meet deadlines, reading comprehension, or something else?

I wonder if Netflix is just hiring for different criteria (e.g. you want people who will make thoughtful decisions while they want people who have memorized all the leetcode problems).

_giorgio_2• 2 years ago

I don't understand why nobody here believes you.

There's no reason to doubt what you say, probably people identify with the mistreated one. Why?

horns4lyfe3• 2 years ago

Because the idea that all the engineers that work at his large company are morons is absurd. Anyone in that situation that believes that and even more, states it, is just making their own character flaws apparent.

horns4lyfe3• 2 years ago

Sounds like you’re a good match for their team then

dang• 2 years ago

(We detached this subthread from https://news.ycombinator.com/item?id=42154036.)

harimau777• 2 years ago

Currently trying to watch it and it's not loading at all for me. Re-subscribed specifically for the fight.

Jabbs• 2 years ago

Looks like I’m playing Tysons Punchout right now

selimthegrim• 2 years ago

Glass Jake?

ralph84• 2 years ago

Main event hasn’t even started yet. Traffic will probably 10x for that. They’re screwed. Should have picked something lower profile to get started with live streaming.

pythonguython• 2 years ago

I don’t work in tech. Is this something that engineers could respond to and reallocate resources to fix mid stream?

GauntletWizard• 2 years ago

Not a chance. This level of infrastructure was set up days in advance - I would be unsurprised if they'd had a code freeze all week for this fight.

jdub• 2 years ago

They've done quite a bit of lower profile live streams... various events, and the Everybody's in LA chat show series.

dools2• 2 years ago

Can Mike Judge please stop predicting everything?

RaftPeople2• 2 years ago

I've been re-watching Silicon Valley the last few weeks and just watched the Nucleus live stream episode 2 days ago, pretty funny seeing it in real life.

dools2• 2 years ago

"Puts data compression in a rear naked chokehold"

Jamie9912• 2 years ago

Works in Australia. Maybe their CDN is under a lot of stress?

gonzo412• 2 years ago

Don't jinx it.

geoffmunn• 2 years ago

In NZ, it was had maybe 2 low-quality moments, but never froze and was in high-definition for the rest of the time.

jameson• 2 years ago

Dumb question

Isn't live streaming at scale already solved problem by cable companies? I never seen ESPN going down during a critical event

ilrwbwrkhv6• 2 years ago

Yes, as I have said again and again on hacker news in different comments Netflix went overboard with their microservices and tried to position itself as a technological company when it's not. It has made everything more complex and that's why any Netflix tech blog is useless because it is not the way to build things correctly.

To understand how to do things correctly look at something like pornhub who handle more scale than Netflix without crying about it.

The other day I was having this discussion with somebody who was saying distributed counter logic is hard and I was telling them that you don't even need it if Netflix didn't go completely mental on the microservices and complexity.

dilyevsky5• 2 years ago

This is not the same streaming - netflix is doing that over HTTP. Totally different tech and scaling issues

hmcq6• 2 years ago

Yes and no. It's not the "same" but this is a solved problem. Fastly regularly delivers the Super Bowl to 10x as many viewers.

Netflix dropped the ball hard

bastard_op• 2 years ago

You would think, but technology always finds a way to screw things up. Cox Communications has had ongoing issues with their video for weeks because of Juniper router upgrades and even the vendor can't fix it. They found this out AFTER they put it in production. Shit happens.

kalesh• 2 years ago

It's down permanently for me in India. We have Hotstar, which has a record of 58 million viewers during the cricket World Cup final. Way ahead.

anakaine6• 2 years ago

Probably less about the level of advancement and more about their ability to stream vs play VOD. Two different kinds of infrastructure optimisation.

ta12439• 2 years ago

59 million at the world cup was concurrent live streams

https://www.icc-cricket.com/news/biggest-cricket-world-cup-e...

An impressive achievement and the scale netflix failed to do a year later.

margaretdouglas6• 2 years ago

Wasn't that the biggest concurrent stream ever?

scrapcode• 2 years ago

They have absolutely shit the bed here, and of course their socials are completely ignoring it.

quickslowdown• 2 years ago

Bet they wish they'd gone with middle out compression

robertlagrant6• 2 years ago

When they come up with that idea it's the most 18-rated and accurate way an engineer would think about it.

ksec2• 2 years ago

Netflix is good only on streaming ready made content, not live streaming, but;

1. Netflix is a 300B company, this isn't a resources issue.

2. This isn't the first time they have done live streaming at this scale either. They already have prior failure experience, you expect the 2nd time to be better, if not perfect.

3. There were plenty of time between first massive live streaming to second. Meaning plenty of time to learn and iterate.

crowcroft• 2 years ago

Yea, the issue here isn't just that they're having issues, it's that they're having the same issues they've had before.

ilrwbwrkhv6• 2 years ago

You can't solve your way out of a complex problem that you have created and which wasn't needed in the first place. The entire microservices thing was overly complex with zero benefits

I spoke to multiple Netflix senior technicians about this.

They said that's the whole shtick.

iLoveOncall2• 2 years ago

That's a ridiculous statement. PrimeVideo is the leader in terms of sports events streaming over internet and it is composed of hundreds of microservices.

Live streaming is just much harder than streaming, and it takes a years of work and a huge headcount to get something good.

that_guy_iain5• 2 years ago

They have the NFL next month on Christmas day. So that'll be a big streaming session but I think it'll be nothing compared to this. Even Twitter was having problems handling the live pirate streams there.

jonlucc• 2 years ago

Even Twitter was having problems

Is that a surprise? They're not who I would think of first as a gold standard for high viewership live streams.

hughesjj3• 2 years ago

Yeah didn't they crash on love is blind or one of their reality shows recently-ish?

freefaler4• 2 years ago

The problem is that provisioning vast capacity for peak viewership is expensive and requires long-term commitment. Some providers won't give you more connectivity to their network unless you sign a 12 month deal where you prepay that.

Peak traffic is very expensive to run, because you're building capacity that will be empty/unsused when the event ends. Who'd pay for that? That's why it's tricky and that's why Akamai charges these insane prices for live streaming.

A "public" secret in that network layer is usually not redundant in your datacenter even if it's promised. To have redundant network you'd need to double your investment and it'll seat idle of at 50% max capacity. For 2hr downtime per year when you restart the high-capacity routers it's not cost efficient for most clients.

treflop3• 2 years ago

Then sign a contract with Akamai, who has been in business for 25 years? You outsource if you aren’t planning to do something very often.

There is no middle ground where you commit a mediocre amount of resources, end up with downtime and a mediocre experience, and then go “but we saved money.”

patrick4512• 2 years ago

What's your point? If they couldn't manage to secure the resources necessary, they shouldn't have agreed to livestream it. As a customer, I don't care AT ALL if it's difficult.

rajnathani2• 2 years ago

Apple was clearly larger than Google when they came out with Apple Maps, and it was issue-laden for a long time. It is not a resource-issue, but a tech development maturity issue.

brokenmachine6• 2 years ago

They already have prior failure experience

What was the previous fail?

curiousDog• 2 years ago

Utter incompetence from senior leadership at Netflix. They had so much time to prepare for this.

PittleyDunkin• 2 years ago

I want to index everyone sneering at this situation and never work with any of them.

seizethecheese• 2 years ago

Eh, punching up, while still punching, doesn’t seem that distasteful to me.

kpw94• 2 years ago

yep, especially knowing this isn't their first rodeo... 18 months since https://time.com/6272470/love-is-blind-live-reunion-netflix/

But the real indicator of how much Sunday’s screw-up ends up hurting Netflix will be the success or failure of its next live program—and the next one, and the one after that, and so on. There’s no longer any room for error. Because, like the newly minted spouses of Love Is Blind, a streaming service can never stop working to justify its subscribers’ love. Now, Netflix has a lot of broken trust to rebuild.

purpleidea• 2 years ago

I think they must be noticing the issues, because I've noticed they've been dropping the stream quality quite substantially... It's a clever trick, but kind of cheap to do so, because who wants to watch pixelated things?

nomilk2• 2 years ago

To be brutally honest if it’s a choice between pixelated and constantly buffering, pixelated is way less bad. Constantly buffering is incredibly annoying during live sports. (but this doesn’t negate your main point which is that if people paid to watch they expect decent resolution)

nomilk2• 2 years ago

It will never not annoy and amuse me that illegal options (presumably run by randoms in their spare time) are so much better than the offerings of big companies and their tech ‘talent’.

Bilal_io3• 2 years ago

I have Netflix purchased legally with hard earned money. But because I had issues I looked for illegal streams, and they were bad, crashes, buffering.. you name it. So I went back to Netflix and watched it at 140p quality.

samschooler• 2 years ago

This twitter stream was the most reliable for me. Completely took Netflix out of the equation; just some dude at the event with his phone: https://x.com/i/broadcasts/1mrxmMRmXpQxy

rk062• 2 years ago

Illegal options would have lot less active users. So it is not a fair comparison

hughesjj3• 2 years ago

Yup. Also a bit more latency since it's effectively restreaming unless it's someone at the actual event.

boramalper2• 2 years ago

Illegal options also have lot less resources (revenue, service providers who are willing host/facilitate illegal activities, and so on) so it’s a fair comparison in my opinion.

johnny_canuck• 2 years ago

It's far from perfect here in Canada, I keep having to pause it or go back and then load it again.

Oddly having watched PPV events via the high seas for years, it feels normal...

owenpalmer• 2 years ago

I'm watching the event as I'm writing this. I've been needing to exit the player and resume it constantly. Pretty surprising that Netflix hasn't weeded out these bugs.

miek• 2 years ago

I switched to watching on the android app and it's been flawless. Sad, but workable

callc• 2 years ago

I couldn’t watch a show a couple days ago. Long time customer, and first time I’ve considered cancelling. Broke the basic contract of I give $ and Netflix give show.

bluedino• 2 years ago

It's been fine since 11:00 EST, I wonder if they started using the CDN more effectively and pushed everyone back a few minutes?

unsnap_biceps3• 2 years ago

Mine just crashed and reloaded to "Envoy Overloaded"

JoyfulTurkey• 2 years ago

Netflix has some NFL games on Christmas Day. Wonder how those will go for them.

I remember when ESPN started streaming years back, it was awful. Now I almost never have problems with their live events, primarily their NHL streams.

zoklet-enjoyer• 2 years ago

Wow I feel scammed. I paid for a Netflix subscription specifically for this but it's not loading so I'm watching on an illegal streaming website

djbusby3• 2 years ago

Mine is glitchy, but if I refresh i get a good steam for a bit, then it gets low res, then freeze. If I wait for auto-reconnect it takes forever. Hard refresh and I'm good. Like, new streams to new server, then overloaded, then does as if their cluster is crashing and healing is rapid cycles. Sawtooth patterns on their charts.

And then all these sessions lag, or orphan taking up space, so many reconnections at various points in the stream.

System getting hammered. Can't wait for this writeup.

magic_man• 2 years ago

Illegal streams are working but netflix is not. That is crazy.

gurjeet• 2 years ago

envoy overloaded

That's the plain-text message I see when I tried to refresh the stream.

Follow-up:

My location: East SF Bay.

Now even the Netflix frontpage (post login, https://www.netflix.com/browse ) shows the same message.

The same message even in a private window when trying to visit https://www.netflix.com/browse

The first round of the fight just finished, and the issues seem to be resolved, hopefully for good. All this to say what others have noted already, this experience does not evoke a lot of confidence in Netflix's live-streaming infrastructure.

smitelli• 2 years ago

Ah, envoy. Now that is a name I have not missed.

tinyhouse• 2 years ago

It's not lagging for me. It crashed and not coming back.

Update: Switched to the app on my phone and so far so good.

KingOfCoders• 2 years ago

Not enough chaos monkey engineering.

margaretdouglas6• 2 years ago

There was some blog post on HN the other day where someone said they don't do chaos monkey anymore... Even then, how do you chaos test a novel event ahead of time?

betaby2• 2 years ago

Sounds like a scene from: Silicon Valley - Nucleus fails

https://www.youtube.com/watch?v=9IGvzb-KCpY

Sateeshm• 2 years ago

My kid woke me up complaining internet is not working. Turns out he is trying to watch the fight and it's not working at all here in India.

blinded3• 2 years ago

It’s a learning experience! I remember Conor and Floyd broke hbo and the ufc. It’s a hard problem for sure!

Some buffering issues for us, but I bet views are off the charts. Huge for Netflix, bad for espn, paramount, etc etc

Bilal_io3• 2 years ago

You're lucky you only had some buffering issues. This is got the case for many people, I don't know the percentage, but many people on reddit were complaining.

This is bad for Netflix imo.

blinded3• 2 years ago

I guess it kinda depends on viewer counts?

everly• 2 years ago

Cable TV (or even OTA antenna in the right service area) is simply a superior live product compared to anything streaming.

The Masters app is the only thing that comes close imo.

Cable TV + DVR + high speed internet for torrenting is still an unmatched entertainment setup. Streaming landscape is a mess.

It's too bad the cable companies abused their position and lost any market goodwill. Copper connection direct to every home in America is a huge advantage to have fumbled.

ericcholis• 2 years ago

The Masters app is truly incredible, I don't know if it gets enough praise.

mmooss3• 2 years ago

What's so great about it?

crote• 2 years ago

The interesting thing is that a lot of TV infrastructure is now running over IP networks. If I were to order a TV connection for my home I'd get an IPTV box to connect to my broadband router via Ethernet, and it'd simply tell the upstream router to send a copy of a multicast stream my way.

Reliable and redundant multicast streaming is pretty much a solved problem, but it does require everyone along the way to participate. Not a problem if you're an ISP offering TV, definitely a problem if you're Netflix trying to convince every single provider to set it up for some one-off boxing match.

nitwit005• 2 years ago

I have wondered if better multicast support will happen just for cost savings, as the amount of live content increases.

So far, no one seems particularly motivated.

heraldgeezer3• 2 years ago

This. Im honestly going to cancel my streaming shit. They remove and mess with it so much. Like right now HBO max or whatever removes my recent watches after 90 days. why?

damontal• 2 years ago

Apple TV MLB games look incredible compared to live cable tv.

muddi900• 2 years ago

I can't see the fight right now.

gamblor9562• 2 years ago

After a few buffering timeouts during the first match, the rest of the event had no technical difficulties (in SoCal, so close to one of Netflix's HQs).

Unfortunately, except for the women's match, the fights were pretty lame...4 of the 6 male boxers were out of shape. Paul and Tyson were struggling to stay awake and if you were to tell me that Paul was just as old as Tyson I would have believed it.

shrubble2• 2 years ago

On X.com someone had a stream that was stable to at least 5 million simultaneous viewers, but then (as I expected) someone at Netflix got them to pull the plug on it. So I would expect this fight to have say, 50 million + watching? Maybe as many as 150-250 million worldwide, given this is Tyson's last fight.

wayoverthecloud• 2 years ago

I watched the whole fight with a 2 minute delay. That was frustrating and it didn't help that Tyson lost.

smegger0012• 2 years ago

Well he is almost 60 and the average retirement age for pro boxers is in the mid 30s. He is well past his prime and in physically demanding sport that is very hard on participants.

Dem_Boys• 2 years ago

What do you think were the dynamics of the engineering team working on this?

I'd think this isn't too crazy to stress test. If you have 300 million users signed up then you're stress test should be 300 million simultaneous streams in HD for 4 hours. I just don't see how Netflix screws this up.

Maybe it was a management incompetence thing? Manager says something like "We only need to support 20 million simultaneous streams" and engineers implement to that spec even if the 20 million number is wildly incorrect.

margaretdouglas6• 2 years ago

Has there ever been a 300m concurrent live stream? I thought Disney+ had the record at something like 60m.

JCharante6• 2 years ago

probably an esports match hosted on bilibili

markus92• 2 years ago

World Cup final, if you add up all streams worldwide?

nonameiguess• 2 years ago

There's no way 300 million people watched this, especially if that number is representing every Netflix subscriber. The largest claimed live broadcast across all platforms is last year's Super Bowl with 202 million unique viewers for at least part of it, but that includes CBS, Nickelodeon, and Univision, not just streaming audiences. Its average viewers for the whole game was 123 million, which is second all-time to the Apollo 11 moon landing.

iamjackg• 2 years ago

Just adding a data point, here in Canada on my nVidia Shield it went down to 360p a dozen times or so, but never paused at all. I guess I got lucky.

jaarse2• 2 years ago

Everyone pointing out that their illegal streams, X streams, etc. work fine are kind of missing the point.

These secondary streams might be serving a couple thousand users at best.

Initial estimates are in the hundreds of millions for Netflix. Kind of a couple of orders of magnitude difference there.

samatman2• 2 years ago

Piracy is distributed, yes.

I think that is the point, in fact.

robertlagrant6• 2 years ago

Everything's easy when it's someone else's content.

robertlagrant6• 2 years ago

They're not used to live. I imagine that's it. All their caching infrastructure is there assuming the content isn't currently being generated.

impulser_• 2 years ago

Hopefully they fix it because they are hosting two Christmas NFL games this year and if you want to really piss people off you have buffering issues during NFL games lol.

0xpgm2• 2 years ago

Maybe this was a stress test for the NFL games?

I'd expect the NFL games to have a largely American audience, but today's boxing event attracted a global audience.

MuffinFlavored2• 2 years ago

I wonder what % of the "60-120m" viewers were from Europe vs America vs Asia

uptownfunk3• 2 years ago

It was so bad. So so bad. Like don’t use your customers as guinea pigs for live streaming. So lame. They need a new head of content delivery. You can’t charge customers like that and market a massive event and your tech is worse than what we had from live broadcast tv.

subless• 2 years ago

I ended up turning my TV off and watching from my phone because of the buffering/freezing. The audio would continue to play and the screen would be frozen with a loading percentage that never changed.

I have Spectrum (600 Mbps) for ISP and Verizon for mobile.

iamzycon• 2 years ago

From my limited understanding, the NFL heavily depends on the Netflix Open Connect platform to stream media to edge locations, which is different from live streaming. Probably, they over-pushed the HD contents.

odinthedog• 2 years ago

Does anyone have any thoughts besides "bad engineering" on what could've gone wrong? It seems like taking on a new endeavor like streaming an event that would possibly draw many hundreds of millions of viewers doesn't make sense. Is there any obvious way that this would just work, or is there obviously a huge mistake deeply rooted in the whole thing. Also, are there any educated guesses on some fine details in the codebase and patterns that could result in this?

normie30003• 2 years ago

Did anyone else see different behaviour with different clients? My TV failed on 25% loaded, my laptop loaded but played for a minute or two before getting stuck buffering, and my iphone played the whole fight fine. All on the same wifi network.

rdtsc• 2 years ago

Live streaming and streaming prerecorded movies is a whole different ballgame.

In fact, optimizing for later can hurt the former.

Would be interesting to read any postmortems on this failure. Maybe someone will be kind enough to share the technical details for the curious crowd.

pests2• 2 years ago

Amazon had issues last year too when they started broadcasting TNF but its fine these days.

I'm sure they will get it figured out.

thr0waway001• 2 years ago

Shoulda used middle out compression.

new_user_final• 2 years ago

I am curious about their live streaming infrastructure.

I have done live streaming for around 100k concurrent users. I didn't setup infrastructure because it was CloudFront CDN.

Why it is hard for Netflix. They have already figured out CDN part. So it should not be a problem even if it is 1M or 100M. because their CDN infrastructure is already handling the load.

I have only work with HLS live streaming where playlist is constantly changing compared to VOD. Live video chunks work same as VOD. CloudFront also has a feature request collapsing that greatly help live streaming.

So, my question is if Netflix has already figured out CDN, why their live infrastructure failing?

Note: I am not saying my 100k is same scaling as their 100M. I am curious about which part is the bottleneck.

vitus6• 2 years ago

Why it is hard for Netflix. They have already figured out CDN part. So it should not be a problem even if it is 1M or 100M. because their CDN infrastructure is already handling the load ... Note: I am not saying my 100k is same scaling as their 100M. I am curious about which part is the bottleneck.

100k concurrents is a completely different game compared to 10 million or 100 million. 100k concurrents might translate to 200Gbps globally for 1080p, whereas for that same quality, you might be talking 20T for 10 million streams. 100k concurrents is also a size such that you could theoretically handle it on a small single-digit number of servers, if not for latency.

CloudFront also has a feature request collapsing that greatly help live streaming.

I don't know how much request coalescing Netflix does in practice (or how good their implementation is). They haven't needed it historically, since for SVOD, they could rely on cache preplacement off-peak. But for live, you essentially need a pull-through cache for the sake of origin offload. If you're not careful, your origin can be quickly overwhelmed. Or your backbone if you've historically relied too heavily on your caches' effectiveness, or likewise your peering for that same reason.

200Gbps is a small enough volume that you don't really need to provision for that explicitly; 20Tbps or 200Tbps may need months if not years of lead time to land the physical hardware augments, sign additional contracts for space and power, work with partners, etc.

junior44660• 2 years ago

This is probably a naive question but very relevant to what we have here.

In a protocol where a oft-repeated request goes through multiple intermediaries, usually every intermediate will be able to cache the response for common queries (Eg: DNS).

In theory, ISPs would be able to do the same with the HTTP. Although I am not aware of anyone doing such (since it will rightfully raise concerns of privacy and tampering).

Now TLS (or other encryption) will break this abstraction. Every user, even if they request a live stream, receives a differently encrypted response.

But live stream of a popular boxing match has nothing to do with the "confidentiality" of encryption protocol, only integrity.

Do we have a protocol which allows downstream intermediates eg ISPs to cache content of the stream based on demand, while a digital signature / other attestation being still cryptographically verified by the client?

dboreham2• 2 years ago

What you describe is called a CDN and has been widely used for 20 years.

flas9sd• 2 years ago

there's Named Data Networking (went by Content-Centric Networking earlier). You request data, not a url, the pipe/path becomes the CDN. If any of your nearest routers have the bytes, your request will go no further.

I don't see it much mentioned the last few years, but the research groups have ongoing publications. There's an old 2006 Van Jacobson video that is a nice intro.

ilaksh4• 2 years ago

I guarantee this is a management issue. Somebody needed to bear down at some point and put the resources into load testing. The engineers told them it probably won't be sufficient.

I assume this came down to some technical manager saying they didn't have the human and server resources for the project to work smoothly and a VP or something saying "well, just do the best you can.. surely it will be at least a little better than last time we tried something live, right?"

I think there should be a $20 million class action lawsuit, which should be settled as automatic refunds for everyone who streamed the fight. And two executives should get fired.

At least.. that's how it would be if there was any justice in the world. But we now know there isn't -- as evidenced by the fact that Jake Paul's head is still firmly attached to his body.

2-3-7-43-1807• 2 years ago

I thought it's only the best of the best of the best working at Netflix ... or maybe we can just put this myth to sleep that Netflix even knows what it's doing. The suggestions are shit, the UX is shit, apparently even the back end sucks.

skwee357• 2 years ago

I’m very disappointed.

Woke up at 4am (EU here), to tune for the main event. Bought Netflix just for this. The women fight went good, no buffering, 4K.

As it approached the time for Paul vs Tyson, it started to first drop to 140p, and then constantly buffer. Restarted my chromecast a few times, tried from laptop, and finally caught a stream on my mobile phone via mobile network rather than my wifi.

The TV Netflix kept blaming my internet which kept coming back as “fast”.

Ended up watching the utterly disappointing, senior abuse, live stream on my mobile phone with 360p quality.

Gonna cancel Netflix and never pay for it it again, nor watch hyped up boxing matches.

brokenmachine6• 2 years ago

You should ask for a refund if that's the only reason you paid for it.

hiyer• 2 years ago

On a tangential note, the match totally looked fixed to me - Tyson was barely throwing any punches. I understand age is not on his side, but he looked plenty spry when he was ducking, weaving and dodging. It seemed to me he could have done better in terms of attacking as well.

adamtaylor_13• 2 years ago

Yeah the biggest thing to me, and the commentators mentioned this as well, his legs looked REALLY wobbly.

All your attacking power comes from your legs and hips, so if his legs weren’t stable he didn’t have much attacking power.

I think he gave it everything he had in rounds 1 and 2. Unfortunately, I just don’t think it was ever going to be enough against a moderately trained 27 year old.

nubinetwork• 2 years ago

I would argue Tyson has a shorter reach, Jake was whiffing a lot of superman punches, and all that does is waste energy. Jake might be able to throw punches, but he clearly wasn't interested in taking them. If they stood closer and slugged it out, the fight could have gone either way.

nightowl_games2• 2 years ago

How is this story not on the front page anymore? 375 comments. Seems like a big story to me.

npilk• 2 years ago

I believe HN's algorithm tends to relatively downrank stories with a high comment-to-upvote ratio, because they are more often flamewars on divisive topics.

yodon• 2 years ago

Another major algorithmic down-ranker is vote wars on comments.

If lots of people are upvoting and downvoting the same comments, that's treated as a signal the topic is contentious and people are likely to start behaving badly.

HN is very clear they prioritize good behavior as the long term goal, and they are as a result comfortable having contentious topics fall in the ranking even if everyone involved in the discussion feels the topic is important.

ajdude• 2 years ago

It wasn't even just buffering issues, the feed would just stop and never start again until I paused it and then clicked "watch live" with the remote.

It was really bad. My Dad has always been a fan of boxing so I came over to watch the whole thing with him.

He has his giant inflatable screen and a projector that we hooked up in the front lawn to watch it, But everything kept buffering. We figured it was the Wi-Fi so he packed everything up and went inside only to find the same thing happening on ethernet.

He was really looking forward to watching it on the projector and Netflix disappointed him.

mmooss3• 2 years ago

My Dad has always been a fan of boxing

What did your Dad think about the 'boxing'?

manquer2• 2 years ago

Commercial boxing has always been like WWE or MMA with a thin veneer of actual sport to it, i.e. it is just entertainment[1].

To rephrase your question then what does someone think of the entertainment on display?

I don't think it was good entertainment.

None of the hallmarks of a good show was present. i.e. It wasn't close, nor was it bloody or anything unexpected like say a KO everything went pretty much as expected. It wasn't nice watch as all,no skill or talent was on disply, all Paul had to do was use his speed to backpedal from the slow weak punches of a visibly older tyson with a bum knee and land some points occasionally to win.

[1] There is a deeper argument here is any spectator sports just entertainment or is truly about skill talent and competition. Boxing however including the ones promoted by traditional four major associations falls clearly on the entertainment side than say another sport like NFL to me.

mvandermeulen2• 2 years ago

Was this necessary? The comment was on a tech forum about the tech issues, do we really need to reprosecute the argument that it wasn’t real boxing here too? There are plenty of other places for those so painfully inclined to do so

beanjuiceII• 2 years ago

i thought they did DSA interviews at netflix what happened? I had to watch the fight on someone streaming to X from their phone at the event and it was better than watching on netflix..if you could watch at all. extremely embarrassing!

bobobob420• 2 years ago

Its because they use medium-hard leetcode instead of hard. I suggest 8 rounds instead of 4

taneq• 2 years ago

Surely more whiteboard questions are the key? Reversing a binary tree is so last year, they should make candidates reverse an octree.

meesles• 2 years ago

My theory is they've so heavily optimized for static content and distributing content on edge nodes that they were probably poorly setup for live-streaming.

moritonal• 2 years ago

This, I feel bad for their engineers who were told every bonus would be a matter of how low can they get the cost-per-GB of transferred data, leading to the glorious Netflix-in-a-box (https://openconnect.netflix.com/deploymentguide.pdf) and then management casually asks for a live show with a wildly different set of guarantee requirements.

gosub100• 2 years ago

someone's going to have to reverse a linked list of subscriptions.

sylens• 2 years ago

Reminds me of Nucleus stuttering during UFC

xerox13ster• 2 years ago

I could hear Gavin Belson screaming during the broadcast when my stream was freezing as they were each making their entrance. Mike Judge is a prophet.

r_hanz• 2 years ago

That show ages better every single day.

criddell2• 2 years ago

I watched on an AppleTV and the stream was rock solid.

I don’t know if it’s still the case, but in the past some devices worked better than others during peak times because they used different bandwidth providers. This was the battle between Comcast and Cogent and Netflix.

eqvinox3• 2 years ago

Your device type has no influence on your provider and its bandwidth characteristics. If you're on Comcast, Apple can't magically make it not suck.

fidotron• 2 years ago

Native apps have a lot more scope for client side load balancing due to having a different security model than browsers.

immibis3• 2 years ago

If you're on Apple, Comcast can make it magically not suck. Not sure if it's relevant though.

Cogent just seems to love picking fights with everyone (see Hurricane Electric). Why are they still in business?

criddell2• 2 years ago

It isn't Apple, it's Netflix.

Remember back in 2014 or so when Netflix users on Comcast were getting slow connections and buffering pauses? It didn't affect people who watched Netflix via Apple TV because Netflix served Apple TV users with a different network.

In a little known, but public fact, anyone who is on Comcast and using Apple TV to stream Netflix wasn’t having quality problems. The reason for this is that Netflix is using Level 3 and Limelight to stream their content specifically to the Apple TV device. What this shows is that Netflix is the one that decides and controls how they get their content to each device and whether they do it via their own servers or a third party. Netflix decides which third party CDNs to use and when Netflix uses their own CDN, they decide whom to buy transit from, with what capacity, in what locations and how many connections they buy, from the transit provider. Netflix is the one in control of this, not Comcast or any ISP.

https://www.streamingmediablog.com/2014/02/heres-comcast-net...

js22• 2 years ago

I watched on an AppleTV and the stream was rock solid.

For me it was buffering and low resolution, on the current AppleTV model, hardwired, with a 1Gbps connection from AT&T. Some streaming devices may have handled whatever issues Netflix was having better than others, but this was clearly a bigger problem than just the streaming device.

booleandilemma• 2 years ago

Was this their first time doing live content? I figured something would go wrong. I'm sure lots of people were watching.

mmiyer• 2 years ago

Not their first time, but the first time at this scale.

sgarland7• 2 years ago

OK, but the last time they tried a livestream (a reality show reunion), it also fell over. I suppose to their credit, my stream never outright died yesterday, it just went to potato quality.

l33t7332273• 2 years ago

I thought Netflix’s biggest advantage was the quality/salary of its engineers.

I think that every time I wait for Paramount+ to restart after its gone black in picture on picture, and yet, I’n still on Paramount+ and not Netflix, so maybe that advantage isn’t real.

A4ET8a8uTh02• 2 years ago

Sigh, none of the competitors are much better. Disney, who has more than enough cash to throw at streaming, is a near constant hassle for us ( after 3 or more episodes it throws an inscrutable error on Playstation ). I would drop it, but this is the only remaining streaming service and wife is not willing to drop it ( I guess until 1 it is one error per one episode ).

immibis3• 2 years ago

I keep hearing that a lot of people are switching to the illegal competitor for better service.

IncreasePosts• 2 years ago

Do you think YouTube couldn't handle it?

grapesodaaaaa2• 2 years ago

I think this was true at some point, but I’ve been disappointed in the quality of the OSS Netflix tools recently. I think before k8s and a plethora of other tools matured, they were way ahead of the curve.

I specifically found the Netflix suite for Spring very lacking, and found message oriented architectures on something like NATS a lot easier to work with.

dylan6048• 2 years ago

People just do not appreciate how many gotchas can pop up doing anything live. Sure, Netflix might have a great CDN that works great for their canned content and I could see how they might have assumed that's the hardest part.

Live has changed over the years from large satellite dishes beaming to a geosat and back down to the broadcast center($$$$$), to microwave to a more local broadcast center($$$$), to running dedicated fiber long haul back to a broadcast center($$$), to having a kit with multiple cell providers pushing a signal back to a broadcast center($$), to having a direct internet connection to a server accepting a live http stream($).

I'd be curious to know what their live plan was and what their redundant plan was.

diggan2• 2 years ago

People just do not appreciate how many gotchas can pop up doing anything live.

Sure thing, but also, how much resources do you think Netflix threw on this event? If organizations like FOSSDEM and CCC can do live events (although with way smaller viewership) across the globe without major hiccups on (relatively) tiny budgets and smaller infrastructure overall, how could Netflix not?

phyrex• 2 years ago

Scale changes everything, I don't think it's fair to shrug this off

dylan6048• 2 years ago

how much resources do you think Netflix threw on this event?

Based on the results, I hope it was a small team working 20% time on the idea. If you tell me they threw everything they had at it to this result, then that's even more embarrassing for them.

throw0101b3• 2 years ago

If organizations like FOSSDEM and CCC can do live events (although with way smaller viewership) […]

Or, for that matter, Youtube (Live) and Twitch.

bostik• 2 years ago

The CCC video crew has its fair share of geeks from broadcasting corporations and studio houses. Their combined institutional knowledge about live events and streaming distribution is probably in the same ballpark as that of giant global TV networks.

They also have the benefit of having practiced their craft at the CCC events for more than a decade. Twice a year. (Their summer event is smaller but still fairly well known. Links to talks show up on HN every now and then.)

Funky anecdote: the video crew at Assembly have more broadcasting and live AV gear for their annual event than most medium-sized studios.

bena2• 2 years ago

It is weird because this was a solved problem.

Every major network can broadcast the Super Bowl without issue.

And while Netflix claims it streamed to 280 million, that’s if every single subscriber viewed it.

Actual numbers put it in the 120 million range. Which is in line with the Super Bowl.

Maybe Netflix needs to ask CBS or ABC how to broadcast

ironhaven2• 2 years ago

Do you live stream the superbowl? Me and everyone I know watch it over antenna broadcast tv. I think it is easier to have millions of tvs catch airwaves vs millions of point to point https video streams.

tempest_• 2 years ago

When Netflix started it was the first in the space and breaking ground which is how they became a "tech" company that happens to stream media however it has been 15 years and since than the cloud providers have basically build "netflix as a service". I suspect most of the big streamers are using that instead of building their own in house thing and going through all the growing pains netflix is.

AlwaysRock• 2 years ago

Solves differently though, right? Cable broadcasts are not the same as a streaming video over the internet, right?

colesantiago6• 2 years ago

This is the whole point of chaos engineering that was invented at Netflix, which tests the resiliency of these systems.

I guess we now know the limits of what "at scale" is for Netflix's live-streaming solution. They shouldn't be failing at scale on a huge stage like this.

I look forward to reading the post mortem about this.

dylan6048• 2 years ago

Everyone keeps mentioning at scale. I seriously doubt this was an "at scale" problem. I have strong suspicion this was a failure at the origination point being able to push a stable signal. That is not an "at scale" issue, but a hubris of we can do better/cheaper than broadcasting standard practices

xyst5• 2 years ago

You are making excuses for a multibillion dollar company that has been in this game for many years. Maybe the first to market in streaming.

This isn’t NFLX’s first rodeo in live streaming. Have seen a handful of events pop up in their apps.

There is no excuse. All of the resources and talent at their disposal, and they looked absolutely amateurish. Poor optics.

I would be amazed if they are able to secure another exclusive contract like this in the future.

DrillShopper• 2 years ago

A company that readily admits it burns out SWRs and SREs in exchange for the big bucks.

Just what the fuck are these people doing?

If I were a major investor in them I'd be pissed.

Xenoamorphous3• 2 years ago

Sorry for the off topic but what’s this thing that I only come across in Hacker News about referring to a company by their stock exchange name (APPL, MSFT, etc) outside of a stock context? It seems really weird to me.

dylan6048• 2 years ago

Um, aksually…

I was pointing out how dumb a multibillion dollar company is for getting this so wrong. Broadcasting live events is something that is underestimated by everyone that has never it, yet hubris of a major tech company thinking it knows better is biting them in the ass.

As many other people have commented, so many other very large dwarfing this event have been pulled off with no hiccups visible to the viewers. I have amazing stories of major hiccups during MLB World Series that viewers had no idea about happening, but “inside baseball” people knew. To the point that the head of the network caught something during the broadcast calling the director in the truck saying someone is either going to be fired or get a raise yet the audience would never have noticed if the person ended up getting fired. They didn’t, btw.

chgs5• 2 years ago

You’re talking about the contribution from the venue to the boardcast centre, increasingly not a full program but being mixed remotely.

That’s a very different area to transmission of live to end users.

dylan6048• 2 years ago

What are you talking about? The signal coming from a live event is the full package. The output of “the truck” has multiple outs including the full mix of all grafix, some only have the mix minus any branding, etc. While the isos get recorded in the truck, they are not pushed out to the broadcast center.

All of the “mixing” as you call it is done in the truck. If you’ve never seen it, it is quite impressive. In one part of the truck is the director and the technical director. The director is the one calling things like “ready camera 1”, “take 1”, etc. the TD is the one on the switcher pushing the actual buttons on the console to make it happen. Next to them is the graphics team prepping all of the stats made available to the TD to key in. In another area is the team of slomo/replay that are taking the feeds from all of the cameras to recorders that allow the operators to pull out the selects and make available for the director/TD to cut to. Typically in the back of the truck is the audio mixer that mixes all of the mics around the event in real time. All of that creates the signal you see on your screen. It leaves the back of the truck and heads out to wherever the broadcaster has better control

avidphantasm• 2 years ago

Is multicast a thing on the commercial internet? Seems like that could help.

dilyevsky5• 2 years ago

If commercial = public, then no - you can not use multicast for this. It is heavily used within some enterprise networks though like if you go to a gym with lots of TVs they are all likely on multicast

_fat_santa2• 2 years ago

When you step back and look at the situation, it's not hard to see why Netflix dropped the ball here. Here's now I see it (not affiliated with Netflix, pure speculation):

- Months ago, the "higher ups" at Netflix struck a deal to stream the fight on Netflix. The exec that signed the deal was probably over the moon because it would get Netflix into a brand new space and bring in large audience numbers. Along the way the individuals were probably told that Netflix doesn't do livestreaming but they ignored it and assumed their talented Engineers could pull it off.

- Once the deal was signed then it became the Engineer's problem. They now had to figure out how to shift their infrastructure to a whole new set of assumptions around live events that you don't really have to think about when streaming static content.

- Engineering probably did their absolute best to pull this off but they had two main disadvantages, first off they don't have any of the institutional knowledge about live streaming and they don't really know how to predict demand for something like this. In the end they probably beefed up livestreaming as much as they could but still didn't go far enough because again, no one there really knows how something like this will pan out.

- Evening started off fine but crap hit the fan later in the show as more people tuned in for the main card. Engineering probably did their best to mitigate this but again, since they don't have the institutional knowledge of live events, they were shooting in the dark hoping their fixes would stick.

Yes Netflix as a whole screwed this one up but I'm tempted to give them more grace than usual here. First off the deal that they struck was probably one they couldn't ignore and as for Engineering, I think those guys did the freaking best they could given their situation and lack of institutional knowledge. This is just a classic case of biting off more than one can chew, even if you're an SV heavyweight.

Loughla• 2 years ago

First off the deal that they struck was probably one they couldn't ignore

If you can't provide the service you shouldn't sell it?

immibis3• 2 years ago

You've never worked in a startup have you? Or any business for that matter. You have to promise something first, then build it.

_fat_santa2• 2 years ago

My speculation here is this was just classic SV cockiness. The team that closed this deal probably knew that they didn't have the capability but I'm sure the arguments for doing it anyways was something along the lines of: "we have the best engineers in the bay area, we can probably figure this out"

mbesto• 2 years ago

There are endless amounts of stories and situations in which selling something before it really exists has helped businesses. It's totally plausible that a team working on video streaming at the scale of Netflix could figure out live streaming.

Pre-optimization is definitely a thing and it can massively hurt (i.e. startups go under) businesses. Let's stop pretending any businesses would say 'no' to extra revenue even before the engineering team had full assurance there was no latency drop.

RicoElectrico• 2 years ago

Execs never listen or even ask engineers about feasibility of projects they sign up to. Hope the exec in question will be let go.

throw0101b3• 2 years ago

> First off the deal that they struck was probably one they couldn't ignore
If you can't provide the service you shouldn't sell it?

Then how will the folks in Sales get their commission?

Besides, not providing the service hasn't stopped Tesla from selling FSD, and their stock has been going gangbusters.

js22• 2 years ago

They now had to figure out how to shift their infrastructure to a whole new set of assumptions around live events

It wasn't their first live event. A previous live event had similar issues.

TheAceOfHearts2• 2 years ago

This isn't Netflix's first foray into livestreaming. They tried a livestream last year for a reunion episode of one of their reality TV shows which encountered similar issues [0]. Netflix already has a contract to livestream a football event on Christmas, so it'll be interesting to see if their engineers are able to get anything done in a little over a month.

These failures reflect very poorly on Netflix leadership. But we all know that leadership is never held accountable for their failures. Whoever is responsible for this should at least come forward and put out an apology while owning up to their mistakes.

[0] https://time.com/6272470/love-is-blind-live-reunion-netflix/

crazygringo• 2 years ago

But we all know that leadership is never held accountable for their failures.

You've never heard of a CEO or other C-suite or VP getting fired?

It most definitely happens. On the other hand, people at every level make mistakes, and it's preferable that they learn from them rather than be fired, if at all possible.

JumpCrisscross5• 2 years ago

Livestreaming is a solved problem. This sounds like NIH [1]. (At the very least, hire them as a back-up.)

[1] https://en.wikipedia.org/wiki/Not_invented_here

gregorygoc• 2 years ago

Saying live-streaming is a solved problem is like saying search is a solved problem.

oehpr• 2 years ago

Look. I'm a small startup employee. I have a teeny tiny perspective here. But frankly speaking the idea that Netflix could just take some off the shelf widget and stuff it in their network to solve a problem... It's an absurd statement for even me. And if there's anyone it should apply to it would be a little startup company that needs to focus on their core area.

Every off the shelf component on the market needs institutional knowledge to implement, operate, and maintain it. Even Apple's "it just works" mantra is pretty laughable in the cold light of day. Very rarely in my experience do you ever get to just benefit from someone else's hard work in production without having an idea how properly implement, operate, and maintain it.

And that's at my little tiny ant scale. To call the problem of streaming "solved" for Netflix... Given the guess of the context from the GP post?

I just don't think this perspective is realistic at all.

namaria5• 2 years ago

"Solved" merely means you don't need to invent something new to solve it. It doesn't mean trivial nor easy. And it definitely doesn't mean the problem is above trade-offs.

intelVISA• 2 years ago

Not sure why Netflix is held in high regard - this proves they're just as much clowns as the other 'big players' in the circus.

thinkingkong• 2 years ago

They arent clowns at all. Ita a totally different engineering problem and you cant just spin up live streaming capacity on demand. The entire system end to end isnt optimized for live streams yet.

draw_down• 2 years ago

I mean, maybe? You just made all this up.

grapesodaaaaa2• 2 years ago

I hope they do a postmortem

yarrowy• 2 years ago

they should also do a business postmortem, how did the exec greenlight this without livestreaming infrastructure in place?

ldjkfkdsjnv2• 2 years ago

We all know netflix was built for static content, but its still hilarious that they have thousands of engineers making 500-1M in total comp and they couldnt live stream a basic broadcast. You probably could have just run this on AWS with a CDK configuration and quota increase from amazon

hawk_• 2 years ago

I thought Hooli was Google, but may be it was Netflix after all.

Trasmatta• 2 years ago

Silicon Valley predicted this: https://youtu.be/ddTbNKWw7Zs

cryptozeus• 2 years ago

Everyone here talking like this something unique netflix had to deal with. Hotstar live streamed india va Pakistan cricket match with zero issues with all time high live viewership ever in the history of live telecast. Why would viewers paying $20 month want to think about their technical issues, they dropped the ball pure and simple. Tech already exists for this, it’s been done before even by espn, nothing new here.

al_borland2• 2 years ago

The Independent reports 35m viewers of that cricket match [0].

Rolling Stone reported 120m for Tyson and Paul on Netflix [1].

These are very different numbers. 120m is Super Bowl territory. Could Hotstar handle 3-4 of those cricket matches at the same time without issue?

[0] https://www.the-independent.com/sport/cricket/india-pakistan...

[1] https://www.rollingstone.com/culture/culture-news/jake-paul-...

dilyevsky5• 2 years ago

Majority of superbowl viewers watch it on cable. Streaming gets fewer than 10M concurrents

achow3• 2 years ago

India - Australia is the one of interest, scored cricket’s highest concurrent audience ever, 59 Million.

https://www.icc-cricket.com/news/biggest-cricket-world-cup-e...

kelvie• 2 years ago

The Games Awards claims to have 118 million livestreams [1], and went off without a hitch.

I watched it for the game trailers, actually shocked that it's also superbowl viewership territory.

https://variety.com/2023/digital/news/game-awards-2023-break...

al_borland2• 2 years ago

Too late for me to edit, but Netflix is now reporting 60m, half of what Rolling Stone said.

https://x.com/netflix/status/1857906492235723244?s=46

lxgr4• 2 years ago

But that's exactly the point: Netflix didn't do this in a vacuum, they did it within Netflix.

It might just have been easier to start from scratch, maybe using an external partner experienced in live streaming, but the chances of that decision happening in a tech-heavy company such as Netflix that seems to pride itself on being an industry leader are close to zero.

rldjbpin• 2 years ago

with zero issues

depending on whom you ask, the bitrate used by the stream is significantly lower than what is considered acceptable from free livestreaming services, that albeit stream to much, much smaller audience.

without splitting hairs, livestreaming was never their forte, and going live with degradation elsewhere is not a great look for our distributed computing champ.

colesantiago6• 2 years ago

If you're going to be having intense algorithm interviews, paying top dollar for only hiring senior engineers, building high intensity and extreme distributed systems and having SRE engineers, we best see insanely good results and a high ROI out of it.

All of the conditions was perfect for Netflix, and it seems that the platform entirely flopped.

Is this what chaos engineering is all about that Netflix was marketing heavily to engineers? Was the livestream supposed to go down as Netflix removed servers randomly?

nova22033• 2 years ago

Amazon prime streams the Thursday night NFL game and they seem to have no problem.

margaretdouglas6• 2 years ago

Isn't the scale a bit different though? Surely this event was an order of magnitude more concurrent viewers than some NFL game.

jarsin• 2 years ago

I blame RTO and AI

ironhaven2• 2 years ago

This is why we need ipv6. If ipv6 was fully rolled out this livestream could have been an efficient multicast stream like what happens with ipTV.

BrandoElFollito• 2 years ago

One similar crash I remember very well was CNN on 9/11 - I tired to connect from France but is down the whole day.

Since then I am very used to it because our institutional web sites traditionally crash when there is a deadline (typically the taxes or school inscriptions).

As for that one, my son is studying in Europe (I am also in Europe), he called me desperate at 5 am or so to check if he is the only one with the problem (I am the 24/7 family support for anything plugged in). After having liberally insulted Netflix he realized he confirmed with his grandparents that he will be helping them at 10 :)

softwaredoug• 2 years ago

The way to deal with this is to constantly do live events, and actually build organizational muscle. Not these massive one off events in an area the tech team has no experience in.

_dark_matter_2• 2 years ago

Agreed. This is a management failure, full stop. Unbelievable that they'd expect engineering to handle a single Livestream event of this magnitude.

mbrumlow5• 2 years ago

I have this argument a lot in tech.

We should always be doing (the thing we want to do)

Somme examples that always get me in trouble (or at least big heated conversations)

1. Always be building: It does not matter if code was not changed, or there has been no PRs or whatever, build it. Something in your org or infra has likely changed. My argument is "I would rather have a build failure on software that is already released, than software I need to release".

2. Always be releasing: As before it does not matter if nothing changed, push out a release. Stress the system and make it go through the motions. I can't tell you how many times I have seen things fail to deploy simply because they have not attempted to do so in some long period of time.

There are more just don't have time to go into them. The point is if "you did it, and need to do it again ever in the future, then you need to continuously do it"

parasti• 2 years ago

This is golden advice, honestly. "If you don't use it, you lose it" applied to software development.

andai2• 2 years ago

This is great, but what possible counterargument is there? We should prolong indefinitely a spooky ambiguity about whether the system works or not?

skybrian2• 2 years ago

Doing dry runs regularly makes sense, but whether actually shipping it makes sense seems context-dependent. It depends on how much you can minimize the side effects of shipping a release.

Consider publishing a new version of a library: you'd be bumping the version number all the time and invalidating caches, causing downstream rebuilds, for little reason. Or if clients are lazy about updating, any two clients would be unlikely to have the same version.

Or consider the case when shipping results in a software update: millions of customer client boxes wasting bandwidth downloading new releases and restarting for no reason.

Even for a web app, you are probably invalidating caches, resulting in slow page loads.

With enough work, you could probably minimize these side effects, so that releasing a new version that doesn't actually change anything is a non-event. But if you don't invalidate the caches, you're not really doing a full rebuild.

So it seems like there's a tension between doing more end-to-end testing and performance? Implementing a bunch of cache levels and then not using it seems counterproductive.

unoti• 2 years ago

1. Always be building: It does not matter if code was not changed...
2. Always be releasing...

A good argument for this is security. Whatever libraries/dependencies you have, unpin the versions, and have good unit tests. Security vulnerabilities that are getting fixed upstream must be released. You cannot fix and remove those vulnerabilities unless you are doing regular releases. This in turn also implies having good unit tests, so you can do these builds and releases with a lower probability of releasing something broken. It also implies strong monitoring and metrics, so you can be the first to know when something breaks.

01HNNWZ0MV43FF• 2 years ago

There's two other ways I've seen it phrased:

"Test what you fly, and fly what you test" (Supposedly from aviation)

"There should be one joint, and it should be greased regularly" (Referring to cryptosystems I think, but it's the same principle. Things like TLS will ossify if they aren't exercised. QUIC has provisions to prevent this.)

ravenstine2• 2 years ago

There should be a caveat that such this kind of decision should be based on experience and not treated as a rule that juniors might blindly follow. We all know how "fail fast and early" turned out (or whatever the exact phrase was).

MisterBastahrd• 2 years ago

The WWE is moving their programming to Netflix next year. If I were them, I'd be horrified at what I saw.

geor9e2• 2 years ago

They've been doing live events since 2023. But it's hard to be prepared for something that's never been done by anyone before — a superbowl scale event, entirely viewed over the internet. The superbowl gets to offload to cable and over the air. Interestingly, I didn't have any problems with my stream. So it sounds like the bandwidth problems might be localized, perhaps by data center or ISP.

burntalmonds• 2 years ago

Yeah, I think people are incorrectly assuming that everyone had the same experience with the stream. I watched the whole thing and only had a few instances of buffering and quality degradation. Not more than 30 seconds total during the stream.

mastazi• 2 years ago

Maybe they considered this event as a rehearsal for the upcoming NFL streams which I am guessing might have a wider audience

firesteelrain2• 2 years ago

I had issues here and there but there was workarounds. Then, towards the end, the quality either auto negotiated or was forced down to accommodate the massive pull.

elcritch• 2 years ago

I suspect a lot of it could be related to ISP bandwidth. I streamed it on my phone without issue. Another friend put their TV on their phone’s WiFi which also worked. Could be partly that phone hotspots lower video bandwidth by default.

I suspect it’s a bit of both Netflix issues and ISPs over subscribing bandwidth.

positr0n2• 2 years ago

I would guess the majority of the streamed bandwidth was sourced from boxes like these in ISP's points of presences around the globe: https://openconnect.netflix.com/en/

So I agree the problems could have been localized to unique (region, ISP) combinations.

uep• 2 years ago

My suspicion is the same as yours, that this may have been caused by local ISPs being overwhelmed, but it could be a million other things too. I had network issues. I live in a heavily populated suburban area. I have family who live 1000+ miles away in a slightly less populated suburban area, they had no issues at all.

patrick4512• 2 years ago

The ISP hypothesis doesn't make sense to me. I could not stream the live event from Netflix. But I could watch any other show on netflix or youtube or hulu at the same time.

ignoramous• 2 years ago

...the tech team has no experience in

Unless Netflix eng decides to release a public postmorterm, we can only speculate. In my time organizing small-time live streams, we always had up to 3 parallel "backup" streams (Vimeo, Cloudflare, Livestream). At Netflix's scale, I doubt they could simply summon any of these providers in, but I guess Akamai / Cloudflare would have been up for it.

giantg2• 2 years ago

Wow, building talent from within? I thought that went out of fashion. I think companies are too impatient to develop their employees.

eh9• 2 years ago

that’s difficult to reproduce at scale; there are only so many “super bowl” events in a calendar year

don-code• 2 years ago

Sometimes this just isn't feasible for cost reasons.

A company I used to work for ran a few Super Bowl ads. The level of traffic you get during a Super Bowl ad is immense, and it all comes at you in 30 seconds, before going back to a steady-state value just as quickly. The scale pattern is like nothing else I've ever seen.

Super Bowl ads famously seven million dollars. These are things we simply can't repeat year over year, even if we believed it'd generate the same bump in recognition each time.

NoPicklez• 2 years ago

I think Netflix have a fair bit of organisational muscle, perhaps the fight was considered not as large of an event as the NFL streams would be in the future.

Also, "No experience in" really? You have no idea if that's really the case

simple10• 2 years ago

I watched the event last night and didn't get any buffering issues, but I did notice frequent drop in video quality when watching the live feed. If I backed the video up a bit, the video quality suddenly went back up to 4k.

I had some technical experience with live video streaming over 15 years ago. It was a nightmare back then. I guess live video is still difficult in 2024. But congrats to Jake Paul and boxing fans. It was a great event. And breaking the internet just adds more hype for the next one.

lunatuna• 2 years ago

I wonder how localized the issues were. I watched the Taylor/Seranno fight and the Paul/Tyson without issue and the picture quality was the best in every seen for live sports. Was blown away by how good it was. No where near what I’m getting with steaming NFL. This is what I want the future of live sports to look like. Though the commentary was so so.

I’m in the Pacific Northwest. I wonder if we got lucky on this or just some areas got unlucky.

ctvo2• 2 years ago

It’s insane the excuses being made here for Netflix’s apparently unique circumstances.

They failed. Full stop. There is no valid technical reason they couldn’t have had a smooth experience. There are numerous people with experience building these systems they could have hired and listened to. It isn’t a novel problem.

Here are the other companies that are peers that livestream just fine, ignoring traditional broadcasters:

- Google (YouTube live), millions of concurrent viewers

- Amazon (Thursday Night Football, Twitch), millions of concurrent viewers

- Apple (MLS)

NBC live streamed the Olympics in the US for tens of millions.

csallen• 2 years ago

They failed. Full stop.

It's not full stop. There are reasons why they failed, and for many it's useful and entertaining to dissect them. This is not "making excuses" and does not get in the way of you, apparently, prioritizing making a moral judgment.

y-c-o-m-b• 2 years ago

Amazon had their fair share of livestream failures and for notably less viewers. I don't think they deserve a spot on that list. I briefly worked in streaming media for sports and while it's not a novel problem, there are so many moving parts and points of failure that it can easily all go badly.

deanCommie2• 2 years ago

There is no one "Amazon" here, there are at least 3:

* Twitch: Essentially invented live streaming. Fantastic.

* Amazon Interactive Video Service [0]: Essentially "Twitch As A Service", built by Twitch engineers. Fantastic.

* Prime Video. Same exact situation as Netflix: original expertise is all in static content. Lots of growing pains with live video and poor reports. But they've figured it out: now there are regular live streams (NHL and NFL), and other channel providers do live streaming on Prime Video as a distribution platform.

[0] https://aws.amazon.com/ivs/

fredgrott• 2 years ago

it could be that they made use of the same advice X followed :)

mikeryan• 2 years ago

The big difference of all the examples you’ve mentioned is dedicated full-time crews on the ground where the events are produced.

I’m pretty confident that when the post mortem is done the issues are going to be way closer to the broadcast truck than the user.

mfiguiere• 2 years ago

The examples given here are not on the same scale. The numbers known so far:

- 120m viewers [1]

- Entire Netflix CDN Traffic grew 4x when the live stream started [2]

[1] https://www.rollingstone.com/culture/culture-news/jake-paul-...

[2] https://x.com/DougMadory/status/1857634875257294866

prasadjoglekar• 2 years ago

Disney HotStar managed to stream ~60M livestreams for the Cricket world cup a year ago. The problem has been solved. Livestreaming sports just have a different QoS expectations than on demand.

https://www.geeksforgeeks.org/how-did-hotstar-managed-5-9-cr...

silisili4• 2 years ago

I guess one question I have is did Netflix partner with other CDNs?

Despite their already huge presence, Amazon for example has multiple CDNs involved for capacity for live events. Same for Peacock.

Thaxll2• 2 years ago

Netflix has 280m subscriber highly doubt half of them tuned in to watch the match, is that 130m figure official?

toast02• 2 years ago

I don't disagree that Netflix could have / should have done better. But everybody screws these things up. Even broadcast TV screws these things up.

Live events are difficult.

I'll also add on, that the other things you've listed are generally multiple simultaneous events; when 100M people are watching the same thing at the same time, they all need a lot more bitrate at the same time when there's a smoke effect as Tyson is walking into the ring; so it gets mushy for everyone. IMHO, someone on the event production staff should have an eye for what effects won't compress well and try to steer away from those, but that might not be realistic.

I did get an audio dropout at that point that didn't self correct, which is definitely a should have done better.

I also had a couple of frames of block color content here and there in the penultimate bout. I've seen this kind of stuff on lots of hockey broadcasts (streams or ota), and I wish it wouldn't happen... I didn't notice anything like that in the main event though.

Experience would likely be worse if there were significant bandwidth constraints between Netflix and your player, of course. I'd love to see a report from Netflix about what they noticed / what they did to try to avoid those, but there's a lot outside Netflix's control there.

freefaler4• 2 years ago

As a cofounder of a CDN company that pushed a lot of traffic, the problem with live streaming is that you need to propagate peak viewership trough a loooot of different providers. The peering/connectivity deals are usually not structured for peak capacity that is many times over the normal 95th percentile. You can provision more connectivity, but you don't know how many will want to see the event. Also, live events can be trickier than stored files, because you can't offload to the edges beforehand to warm up the caches.

So Netflix had 2 factors outside of their control

- unknown viewership

- unknown peak capacities outside their own networks

Both are solvable, but if you serve "saved" content you optimize for different use case than live streaming.

JSDevOps• 2 years ago

Why didn’t they use Netflix AI to solve the problems?

frankharv• 2 years ago

How dare you insult the AI Gods

aucisson_masque• 2 years ago

Honestly you didn't miss much, every (real) boxing fan thought of this as a disgrace and a shame when announced. putting a 58 year old Tyson against a crackhead filled with steroids (Jake Paul) ? Either case it would have been a shame on Jake Paul for even getting in the ring with such an old boxer.

In boxing you are old by 32 or maybe 35 year old for heavy weight, and everything goes down very very fast.

End of rant.

chx• 2 years ago

People still pay real world money to Netflix after they cancelled and how and why Warrior Nun just to see grandpa being beaten up.

I guess in the year when Trump is being reelected this is hardly a surprise.

bloomingeek• 2 years ago

Serves Netflix right for killing my beloved DVD rentals.

duxup• 2 years ago

I’m a little amused at folks tuning in for meme / low quality personalities doing things … and getting the equivalent production values.

eigenvalue• 2 years ago

They should have partnered with every major CDN and load balanced across all of them. It’s ironic how we used to be better at broadcasting live events way back in the day versus today.

badgersnake• 2 years ago

I’m not sure buffering was the biggest issue with this event. How was as 58 year old Tyson fighting a man in his 20s?

xyst5• 2 years ago

This wasn’t a “real” fight in the ring. It was clearly a hype/money fight only. The late 20 year old boxer has a massive following (or hate following) with younger age groups; and Mike Tyson brings the older age groups out. Mike has earned somewhat of a legendary status.

Leading up to the fight, there were many staged interactions meant to rile up the audience and generate hype and drive subscription revenue, and subsequently make ad spots a premium ($$$).

Unfortunately, American television/entertainment is so fake. I can’t get even be bothered to engage or invest time into it anymore.

phatfish• 2 years ago

It may as well have been a WWE special. As good as scripted. It was a business venture not a competitive fight.

xyst5• 2 years ago

What a massive blow to NFLX. They have been in the streaming game for years (survived COVID-19) and this silly exhibition match is what does them in?

I didn’t watch it live (boxing has lot its allure for me) but vicariously lived through it via social feed on Bluesky/Mastadon.

Billions of dollars at their disposal and they just can’t get it right. Probably laid off the highly paid engineers and teams that made their shit work.

wannacboatmovie• 2 years ago

Probably laid off the highly paid engineers and teams that made their shit work.

More likely overpaying a bunch of posers without the chops, a victim of their own arrogance.

thesausageking• 2 years ago

It's not a "massive blow" at all. Consumers will only vaguely remember this in a month. Netflix got a lot of new signups and got to test out their streaming infrastructure to figure out what needs work.

The fight itself was lame which worked in their favor. No one really cared about not being able to see every second of the "action". It's not like it was an NBA game that came down to the last second.

Thaxll2• 2 years ago

In 2012 Youtube did the Red Bull stratos live stream with 8m concurrent users. We're 12 years later, Netflix fucked up.

MichaelZuo• 2 years ago

The average quality of talent has gone way down compared to 2012 though.

E.g. the median engineer, excluding entry level/interns, at YouTube in 2012 was a literal genius at their niche or quite close to it.

Netflix simply can’t hire literal geniuses with mid six figure compensation packages in 2024 dollars anymore… though that may change with a more severe contraction.

spaceywilly• 2 years ago

To me the difference is that in 2012, you had companies focusing on delivering a quality product, whether it made money or not. Today, the economic environment has shifted a lot and companies are trying to increase profits while cutting costs. The result is inevitably a decline in quality. I'm sure that Netflix could deliver a flawless live stream to millions of viewers, but the question is can they do it while making a profit that Wall Street is happy with. Apparently not.

Mistletoe• 2 years ago

The funny thing is I was just reading something on HN like three days ago about how light years ahead Netflix tech was compared to other streaming providers. This is the first thing I thought of when I saw the reports that the fight was messing up.

lxgr4• 2 years ago

But is there a way that Netflix might have learned from all of Youtube's past mistakes?

The only reasonable way to scale something like this up is probably to... scale it up.

Sure, there are probably some generic lessons, but I bet that the pain points in Netflix's architecture (historically grown over more than a decade and optimized towards highly cacheable content) are very different from Youtube, which has ramped up live content gradually over as many years.

SoftTalker5• 2 years ago

2012 live video was what, 480p?

jeffgreco• 2 years ago

That’s about what Paul Tyson was looking like at times.

heraldgeezer3• 2 years ago

No? In 2012 most already had fiber net here. Are you young or poor?

reaperducer3• 2 years ago

In 2012 Youtube did the Red Bull stratos live stream with 8m concurrent users

8m vs 60m. And not in 4K. Not a great choice for comparison.

MuffinFlavored2• 2 years ago

Source for Netflix pulling 60m streams to watch Tyson/Paul?

throwawayUS9• 2 years ago

Why no one mentioned the term “vaporware”? Isn’t this a classic example of one?

sourcecodeplz• 2 years ago

This livestream broke the internet, no joke. youtube was barely loading and a bunch of other sites too. 130M is a conservative number given all the pirate streams.

SG-• 2 years ago

don't confuse your ISP breaking with every other provider or the rest of the Internet. It was more than fine here.

brokenmachine6• 2 years ago

My internet was fine.

m3kw9• 2 years ago

I would have just made it simple, delay the live stream a few seconds and encode it into the same bucket where users already is playing static movies. Just have the player only allow start at the time everyone is at.

vouwfietsman• 2 years ago

Reading the comments here, I think one thing that's overlooked is that Netflix, which has been on the vanguard of web-tech and has solved many complicated problems in-house, may not have had the culture to internally admit that they needed outside help to tackle this problem.

A combination of hubris and groupthink.

kaptainscarlet• 2 years ago

Not invented here syndrome works at first but as time progresses the internally built tools become a liability

nikolay• 2 years ago

The arrogant Netflix! They always brag about how technologically superior they are, and they can't handle a simple technological challenge! I didn't have a buffering issue, I had an error page - for hours! Yet, they kept advertising the boxing match to me! What a joke! If you can't stream it, don't advertise it to save face with people like me who don't care about boxing!

notimetorelax• 2 years ago

I think you’re oversimplifying it. Live event streaming is very different from movie streaming. All those edge cache servers become kinda useless and you start hitting peering bottlenecks.

YZF6• 2 years ago

Edge caches are not useless for live streaming. They're critical. The upstream from those caches has no way of handling each individual users. The stream needs to hit the edge cache and end users should be served from there.

A typical streaming architecture is multi-tiered caches, source->midtier->edge.

We don't know what happened but it's possible they ran out of capacity on their edge (or anywhere else).

StressedDev• 2 years ago

Every organization makes mistakes and every organization has outages. Netflix is not different. Instead, of bashing them because they are imperfect, you might want to ask what you can learn from this incident. What would you do if your service received more traffic than expected? How would you test your service so you can be confident it will stay up?

Also, I have never seen any Netflix employees who are arrogant or who think they are superior to other people. What I have seen is Netflix's engineering organization frequently describes the technical challenges they face and discusses how they solve them.

gunapologist99• 2 years ago

Nucleix needs to focus on fixing middle-out compression instead of kicking cameras.

RyeCombinator• 2 years ago

Chaos testing, nothing to see here.

pchwalek• 2 years ago

Sounds like a job for Pied Piper

manav• 2 years ago

It seemed to be some capacity issue with the CDNs. When I stopped and restarted the stream it worked again. Perhaps they do not use real time multi-cdn switching.

gardenhedge• 2 years ago

So much for Netflix engineering talent aura

sporkland• 2 years ago

Was this the plot of a silicon valley episode?

abc-1• 2 years ago

I thought Netflix engineers were the best and could even do mythical leetcode hards. What happened? Why are they paid half a million dollars a year?

thrw42A8N• 2 years ago

Isn't this more of a management problem, trying to turn a not-livestream system into a livestreaming one?

throwaway1063822• 2 years ago

It's incomprehensible to me that Netflix, one of the most highly skilled engineering teams in the world - completely sh*t the bed last night and provided a nearly unwatchable experience that was not even in the same league as pre-internet live broadcast from 30 years ago.

ilaksh4• 2 years ago

My bet is that a technical manager told his executive (multiple times) that he needed more resources and engineering time to make live work properly, and they just told him to make do because they didn't want to spend the money.

It could come down to something as stupid as:

Executive: "we handled [on demand show ABCD] on day one, that was XX million"

Engineering: "live is really different"

Executive: (arguing about why it shouldn't be that different and should not need a lot of new infrastructure)

Engineering: (can't really argue with his boss about this anymore after having repeated the same conversation 3 or 4 times) -- tells the team: we are not getting new servers or time for a new project. We have to just make do with what we have. You guys are brilliant, I know you can do it!"

throwaway1063822• 2 years ago

hopefully their stock takes a big hit monday - these types only understand one thing

laweijfmvo• 2 years ago

hopefully that technical manager has a paper trail and that executive has someone to answer to above them. in cases like this, i always throw together a doc and ask for sign offs.

uptownfunk3• 2 years ago

Hahaha this reads very close to what actually happened

mountainriver2• 2 years ago

What’s amazing is that they have had several streaming flips before and are unable to fix it

gist2• 2 years ago

I had buffering issues but then backed off and let a bit of it buffer up (maybe 1 or 2 mintues?) and then it was fine for the entire Tyson Paul match. There was no reason I needed it to be live vs. a 1 or 2 minute delay.

Edman2742• 2 years ago

If you had money riding on it because of legalized sports gambling in your state and sports regulators clearing it to be a thing to bet on, then it was probably pretty psychologically important whether or not there was a 1 or 2 minute delay on you seeing the results of the game.

lizknope• 2 years ago

Does anyone remember IP multicast?

I remember a lot of trade magazines in the late 1990's during the dot com boom talked about how important it would be.

https://en.wikipedia.org/wiki/IP_multicast

I never hear about it anymore. Is that because everyone wants to watch something different at their own time? Or is it actually working just fine now in the background? I see under the "Deployment" section it mentions IPTV in hotel rooms.

grogenaut4• 2 years ago

dumbing it down a bit: Imagine if anyone in your neighborhood could broadcast video and take up %N of the bandwidth to all of the routers in the neighborhood. Imagine this on your campus or at your office. This works for cable tv, as there's only 200 channels. You're just going to slurp up all of the bandwidth instead and maybe no one is watching the tv.

Sure you get these black swan events that everyone wants to watch, but they're just that, really infrequent. So instead you have to provision capacity if on the interent to do big events like this. The upside is that you have a billion people able to point to point to another billion people instead of 30 companies who can send to everyone.

DanAtC• 2 years ago

Multicast isn't broadcast; bandwidth is only used on links with clients who have joined the group.

shermantanktop9• 2 years ago

Every time a big company screws up, there are two highly informed sets of people who are guaranteed to be lurking, but rarely post, in a thread like this:

1) those directly involved with the incident, or employees of the same company. They have too much to lose by circumventing the PR machine.

2) people at similar companies who operate similar systems with similar scale and risks. Those people know how hard this is and aren’t likely to publicly flog someone doing their same job based on uninformed speculation. They know their own systems are Byzantine and don’t look like what random onlookers think it would look like.

So that leaves the rest, who offer insights based on how stuff works at a small scale, or better yet, pronouncements rooted in “first principles.”

grogenaut4• 2 years ago

you are so right about that. tho I'm sure that many of the netflix folks are still doing their after action analysis in prep for Dec 25 NFL.

now take this realization and apply it to any news article or forum post you read and think about how uninformed they actually are.

fragmede6• 2 years ago

If NFL decides to keep Netflix for that, that is. The bandwidth for that fight was rookie numbers, and after that fiasco, why would the NFL not break their contract and choose someone with a proven track record doing bigger live events, like the World Cup?

dpkirchner• 2 years ago

Right? A common complaint by outsiders is that Netflix uses microservices. I'd love to hear exactly how a monolith application is guaranteed to perform better, with details. What is the magic difference that would have ensured the live stream would have been successful?

miked853• 2 years ago

It's not guaranteed, but much fewer points of failure.

leptons4• 2 years ago

I doubt a "microservice" has anything to do with delivering the video frames. There are specific kinds of infrastructure tech that are specifically designed to serve live video to large amounts of clients. If they are in fact using a "microservice" to deliver video frames, then I'd ask them to have their heads examined. Microservices are typically used to do mundane short-lived tasks, not deliver video.

ilrwbwrkhv6• 2 years ago

I am one of the ones who complain about their microservices architecture quite a lot.

This comes from both first-hand experience of talking to several of their directors when consulted upon on how to make certain systems of theirs better.

It's not just a matter of guarantees, it's a matter of complexity.

Like right now Google search is dying and there's nothing that they can do to fix it because they have given up control.

The same thing happened with Netflix where they wanted to push too hard to be a tech company and have their tech blogs filled with interesting things.

On the back end they went too deep on the microservices complexity. And on the front end for a long time they suffered with their whole RxJS problem.

So it's not an objective matter of what's better. It's more cultural problem at Netflix. Plus the fact that they want to be associated with "Faang" and yet their product is not really technology based.

karaterobot• 2 years ago

The only time I worked on a project that had a live television launch, it absolutely tipped over within like 2 minutes, and people on HN and Reddit were making fun of it. And I know how hard everyone worked, and how competent they were, so I sympathize with the people in these cases. While the internet was teeing off with easy jokes, engineers were swarming on a problem that was just not resolving, PMs were pacing up and down the hallway, people were getting yelled at by leadership, etc. It's like taking all the stress and complexity of a product launch and multiplying it by 100. And the thing I'm talking about was just a website, not even a live video stream.

swyx• 2 years ago

what was the ultimate cause/fix of issues in your case? a database thing?

jillyboel• 2 years ago

people were getting yelled at by leadership

this is where you get up and leave

ryoshu• 2 years ago

Those are the times when you identify who is there to help and who is there to be performative.

pdimitar• 2 years ago

You cannot leave us hanging like that. What was the issue?

adamredwoods2• 2 years ago

Some breaks are just too difficult to predict. For example, I work in ecommerce and we had a page break because the content team pushed too many items into an array, that caused a back-end service to throw errors. Because we were the middle-service, taking from the CMS and making the request to back-end, not sure how we could have seen that issue coming in advance (and no one knew there was a limit).

seanp2k2• 2 years ago

Shoulda used Varnish.

survirtual2• 2 years ago

For an event like this, there already exists an architecture that can handle boundless scale: torrents.

If you code it to utilize high-bandwidth users upload, the service becomes more available as more users are watching -- not less available.

It becomes less expensive with scale, more available, more stable.

The be more specific, if you encode the video in blocks with each new block hash being broadcast across the network, just managing the overhead of the block order, it should be pretty easy to stream video with boundless scale using a DHT.

Could even give high-bandwidth users a credit based upon how much bandwidth they share.

With a network like what Netflix already has, the seed-boxes would guarantee stability. There would be very little delay for realtime streams, I'd imagine 5 seconds top. This sort of architecture would handle planet-scale streams for breakfast on top of the already existing mechanism.

But then again, I don't get paid $500k+ at a large corp to serve planet scale content, so what do I know.

nemothekid5• 2 years ago

Torrents are awful for live events.

1. Everyone only cares about the most recent "block". By the time a "user" has fully downloaded a block from Netflix's seedbox, the block is stale, so why would any other user choose to download from a peer rather from netflix directly?

2. If all the users would prefer to download from netflix directly rather than a p2p user, then you already have a somewhat centralized solution, and you gain nothing from torrents.

Edman2742• 2 years ago

The protocol for a torrent is that random parts of a file get seeded to random people requesting a file, and that the clients which act as seeds are able to store arbitrary amounts of data to then forward to other clients in the swarm. Do the properties about scaling still hold when it's a bunch of people all requesting real time data which has to be in-order? Do the distributed Rokus, Apple TVs, Fire TVs and other smart TVs all have the headroom in compute and storage to be able to simultaneously decode video and keep old video data in RAM and manage network connections with upload to other TVs in their swarm - and will uploading data to other TVs in the swarm not negatively impact their own download speeds?

transcriptase2• 2 years ago

I don’t pay my ISP each month to be part of a streaming sites infrastructure. I pay the streaming site each month to use theirs.

miki1232112• 2 years ago

Then, instead of people complaining about buffering issues, you'd get people complaining about how the greedy capitalists at Netflix made poor Joe Shmoe use all of his data cap, because they made him upload lots of data to other users and couldn't be bothered to do it themselves.

kmeisthax• 2 years ago

Yes, and then some idiot with an axe to grind against Logan Paul starts DDoSing people in the Netflix swarm, kicking them out of the livestream. This is always a problem because torrents, by design, are privacy-hostile. That's how the MAFIAA[1] figured out you were torrenting movies in 2004 and how they sent your ISP a takedown notice.

Hell, in the US, this setup might actually be illegal because of the VPPA[0]. The only reason why it's not illegal for the MAFIAA to catch you torrenting is because of a fun legal principle where criminals are not allowed to avail themselves of the law to protect their crimes. (i.e. you can't sue over a drug deal gone wrong)

[0] Video Privacy Protection Act, a privacy law passed which makes it illegal to ask video providers for a list of who watched what, specifically because a reporter went on a fishing expedition with video data.

[1] Music and Film Industry Association of America, a hypothetical merger of the MPAA and RIAA from a 2000s era satire article

ryandv• 2 years ago

I've noticed this amongst the newer "careerist" sort of software developer who is stumbling into the field for money, as opposed to the obsessive computer geek of yesteryear, who practiced it as a hobby. This character archetype is a transplant, say, less than five years ago from another, often non-technical discipline, and was taught or learned from overly simplistic materials that decry systems programming, or networking, or computer science concepts as unnecessary, impractical skills, reducing everything to writing JavaScript glue code between random NPM packages found on google.

Especially in a time where the gates have come crashing down to pronouncements of, "now anybody can learn to code by just using LLMs," there is a shocking tendency to overly simplify and then pontificate upon what are actually bewilderingly complicated systems wrapped up in interfaces, packages, and layers of abstraction that hide away that underlying complexity.

It reminds me of those quantum woo people, or movies like What the Bleep Do We Know!? where a bunch of quacks with no actual background in quantum physics or science reason forth from drastically oversimplified, mathematics-free models of those theories and into utterly absurd conclusions.

lclarkmichalek• 2 years ago

What does this have to do with the topic being discussed?

az09mugen• 2 years ago

Even before LLMs were trendy, at the time of covid 19, a lot of people surprisingly became "experts" on the matter of virology and genetics on social networks.

dartos4• 2 years ago

even for enthusiasts, to go from no programming experience to understanding how Netflix handles video streaming at scale would take more than 5 years.

doctorpangloss• 2 years ago

You don’t belong to either group. What does this make you?

shermantanktop9• 2 years ago

You may have belonged to one of those groups in the past, or maybe you will someday. I certainly have. Many of the more seasoned folks on HN have.

Stuff goes wrong, random internet people jump on the opportunity to speculate and say wildly off-the-mark comments, and the engineers trying to keep the ship from sinking have to sit quietly for fear of making the PR backlash worse.

david-gpu• 2 years ago

Completely agreed. There are also former employees who have very educated opinions about what is likely going on, but between NDAs and whatnot there is only so much they are willing to say. It is frustrating for those in the know, but there are lines they can't or won't cross.

Whenever an HN thread covers subjects where I have direct professional experience I have to bite my tongue while people who have no clue can be as assertive and confidently incorrect as their ego allows them to be.

shermantanktop9• 2 years ago

nods knowingly

fragmede6• 2 years ago

Some people can just let others be wrong and just stay silent, but some people can't help themselves. So if you say something really wrong, like this was caused by Netflix moving to Azure, they should have stayed on AWS! someone will come along to correct you. If you're looking for the right answer, post the wrong one, alongside some provoking statement (Windows is better than Linux because this works there), and you'll get the right answer faster than if you'd asked your question directly.

https://xkcd.com/386/

croes• 2 years ago

You are basically saying, everybody who criticizes Netflix now has no clue.

That’s a bold claim given that people with inside knowledge could post here without disclosing they are insiders.

Is that some kind of No True Scotsman?

shermantanktop9• 2 years ago

I’m just pointing out that there are Netflix engineers reading all these words.

For every thread like this, there are likely people who are readers but cannot be writers, even though they know a lot. That means the active posters exclude that group, by definition.

These threads often have interesting and insightful comments, so that’s cool.

pfraze• 2 years ago

At the scale that Netflix just dealt with? Yeah I honestly think this is a case where less than 5000 people in the world are really qualified to comment.

tomcam3• 2 years ago

You are basically saying, everybody who criticizes Netflix now has no clue.

GP clearly meant some people not everybody. You are the one making bold claims.

anothernewdude• 2 years ago

who offer insights based on how stuff works at a small scale, or better yet, pronouncements rooted in “first principles.”

And looking through the comments, this is just wrong.

eqvinox3• 2 years ago

3) the people supplying 1) and 2) with tools (hard- or software)

We (yep) don't know the exact details, but we do get sent snapshots of full configs and deployments to debug things... we might not see exact load patterns, but it's enough to know. And if course we can't tell due to NDAs.

johnnyanmac4• 2 years ago

I'm sure 2) can post. But it won't be popular, so you'll need to dig to find it.

Most people are consumers and at the end of the day, their ability to consume a (boring) match was disrupted. If this was PPV (I don't think it is) the paid extra to not get the quality of product they expected. I'm not surprised they dominate the conversation.

bobdvb5• 2 years ago

I am 2, I absolutely will get argued with by people who think they know better.

I'm also not going to criticise my peers because they could recognise me and I might want to work with them one day.

arduanika• 2 years ago

And nonetheless, it freezes up.

grogenaut4• 2 years ago

This topic is really just fun for me to read based on where I work and my role.

Live is a lot harder than on demand especially when you can't estimate demand (which I'm sure this was hard to do). People are definitely not understanding that. Then there is that Netflix is well regarded for their engineering not quite to the point of snobbery.

What is actually interesting to me is that they went for an event like this which is very hard to predict as one of their first major forays into live, instead of something that's a lot easier to predict like a baseball game / NFL game.

I have to wonder if part of the NFL allowing Netflix to do the Christmas games was them proving out they could handle live streams at least a month before. The NFL seems to be quite particular (in a good way) about the quality of the delivery of their content so I wouldn't put it past them.

marchie_uk• 2 years ago

The NFL seems to be quite particular (in a good way) about the quality of the delivery of their content

Alas, my experience with the NFL in the UK does not reflect that. DAZN have the rights to stream NFL games here, and there are aspects of their service that are very poor. My major, long-standing issue has been the editing of their full game “ad-free” replays - it is common for chunks of play to be cut out, including touchdowns and field goals. Repeated complaints to DAZN haven’t resulted in any improvements. I can’t help but think that if the NFL was serious about the quality of their offering, they’d be knocking heads together at DAZN to fix this.

grogenaut4• 2 years ago

I don't think they think this is a problem actually. Content edited replays are actually very popular with sports fans who are time shifting. Time shifting is also an afterthought for the NFL / MLB / NHL from what I can tell. I live in Seattle but grew up in the midwest so time shift a ton of sports and it's always been horrific.

I'm more comparing Thursday Night Football and the quality of the encoding than anything. Delivery glitches are a seperate issue that I think they care about less.

NFL: 90+ minutes after the match on NFL Gameday, and it auto plays the most recent video for that team, which is always the post game interview. So you load it up, go to your team and it auto plays the "we won" or "it was a tough loss" like why the f*ck am I paying for a dvr solution when you do that. NFL Sunday Ticket: you can watch the games sometime monday after the fact but not the sunday night games. Good thing I paid well below half price for it with a disciount.

NHL: constantly shifting networks each year with worse solutions and not letting you get to half the previous games after a week. Totally useless for deferred unless you only want to watch the game a day or more after. Fubo, you have to 'record' the game and sometimes it's on a slightly different network and doesn't record. And their blackout system is the worst of all, who cares about your mediocre local team sorry you can't watch Chefs/Bills because they overlapped by some amount.

MLB: always broken at the top of the year, constantly changing the interface. You often get stuck watching the commercial break which is not actually commercials and is just the same "ohtani / judge highlight video from 2 years ago" and a "stat" about the sluggers that is almost an entire season out of date. The app resets when switching from the live CDN to the on demand one once the game ends which often resets the game and jumps you 6 innings forward, or makes the game unavialable for 30 minutes.

And EFF you if you want to watch playoffs on any of them.

mountainriver2• 2 years ago

Netflix’s snobbery of engineering is so exhausting. Then seeing them be unable to fix this problem after several previous streaming failures is a bit rich.

To me it speaks to how most of the top tech companies of the 2010s have degraded as of late. I see it all the time with Google hiring some of the lower performing engineers on my teams because they crushed Leetcode.

devit• 2 years ago

Why is live a lot harder?

Aside from latency (which isn't much of a problem unless you are competing with TV or some other distribution system), it seems easier than on-demand, since you send the same data to everyone and don't need to handle having a potentially huge library in all datacenters (you have to distribute the data, but that's just like having an extra few users per server).

My guess is that the problem was simply that the number of people viewing Netflix at once in the US was much larger than usual and higher than what they could scale too, or alternatively a software bug was triggered.

michaelt• 2 years ago

Latency is somewhat important for huge sporting events; you don't want every tense moment spoiled by the cheers of your neighbours whose feed is 20 seconds ahead.

With on-demand you can push the episodes out through your entire CDN at your leisure. It doesn't matter if some bottleneck means it takes 2 hours to distribute a 1 hour show worldwide, if you're distributing it the day before. And if you want to test, or find something that needs fixing? You've got plenty of time.

And on-demand viewers can trickle in gradually - so if clients have to contact your DRM servers for a new key every 15 minutes, they won't all be doing it at the same moment.

And if you did have a brief hiccup with your DRM servers - could you rely on the code quality of abandonware Smart TV clients to save you?

MattDaEskimo• 2 years ago

I'm not an expert in this, but at least familiar with the trade.

I'd imagine with on-demand services you already have the full content and therefore can use algorithms to compress frames and perform all kinds of neat tricks to.

With live streaming I'd imagine a lot of these algorithms are useless as there isn't enough delay & time to properly use them, so they're required to stream every single pixel and maybe some JIT algorithms

nemothekid5• 2 years ago

On demand is easier precisely because having a huge library in all data centers is relatively cheap. In actuality you just have a cache, collocated ISPs that pulls from your origin servers. Likely you have users all watching different things so you can easily avoid hot spots by sharding on the content type. Once the in demand content is in the cache its' relatively easy to serve.

Live content is harder because it can't really be cached, nor, due to TLS, can you really serve everyone the same stream. I think the hardest problem to solve is provisioning. If you are expecting 1 million users, and 700,000 of them get routed to a single server, that server will begin to struggle. This can happen in a couple different ways - for example an ISP who isn't a large consumer normally, suddenly overloads its edge server. Even though your DC can handle the traffic just fine, the links between your DC and the ISP begin to suffer, and since the even is live, it's not like you can just wait until the cache is filled downstream.

avidiax• 2 years ago

I'm in an adjacent space, so I can imagine some of the difficulties. Basically live streaming is a parallel infrastructure that shares very little with pre-recorded streaming, and there are many failure points.

* Encoding - low latency encoders are quite different than storage encoders. There is a tradeoff to be made in terms of the frequency of key frames vs. overall encoding efficiency. More key frames means that anyone can tune in or recover from a loss more quickly, but it is much less efficient, reducing quality. The encoder and infrastructure should emit transport streams, which are also less efficient but more reliable than container formats like mp4.

* Adaptation - Netflix normally encodes their content as a ladder of various codecs and bitrates. This ensures that people get roughly the maximum quality that their bandwidth will allow without buffering. For a live event, you need the same ladder, and the clients need to switch between rungs invisibly.

* Buffering - for static content, you can easily buffer 30 seconds to a minute of video. This means that small latency or packet loss spikes are handled invisibly at the transport/buffering layer. You can't do this for a live event, since that level of delay would usually be unacceptable for a sporting event. You may only be able to buffer 5-10 seconds. If the stream starts to falter, the client has only a few seconds to detect and shift to a lower rung.

* Transport - Prerecorded media can use a reliable transport like TCP (usually HLS). In contrast, live video would ideally use an unreliable transport like UDP, but with FEC (forward error correction). TCP's reaction to packet loss halves the receive window, which halves bandwidth, which would have to trash the connection to shift to a lower bandwidth rung.

* Serving - pre-recorded media can be synchronized to global DCs. Live events have to be streamed reliably and redundantly to a tree of servers. Those servers need to be load balanced, and the clients must implement exponential backoff or you can have cascading failures.

* Timing - Unlike pre-recorded media, any client that has a slightly fast clock will run out of frames and either need to repeat frames and stretch audio, or suffer glitches. If you resolve this on the server side by stretching the media, you will add complication and your stream will slowly get behind the live event.

* DVR - If you allow the users to pause, rewind, catch up, etc., you now have a parallel pre-recorded infrastructure and the client needs to transition between the two.

* DRM - I have no idea how/if this works on a live stream. It would not be ideal that all clients use the same decryption keys and have the same streams with the same metadata. That would make tracing the source of a pirate stream very difficult. Differentiation/watermarking adds substantial complexity, however.

db48x• 2 years ago

It probably depends more on the ISP than on Netflix. Engineers over in my ISP’s subreddit are talking about how flows from Netflix jumped by over 450Gb/s and it was no big deal because it wasn’t enough to cause any congestion on any of their backbone links.

_HMCB_• 2 years ago

They didn’t miss anything.

heraldgeezer3• 2 years ago

Even people on hacker news do not understand.

Internet live streaming is harder than cable tv sattelite live streaming over "dumb" TV boxes cable. They should not have used internet for this honestly. A TV signal can go to millions live.

tiffanyh2• 2 years ago

IPTV

A see in the comments multiple people talking about how "cable" companies who have migrated to IPTV has solved this problem.

I'd disagree.

I'm on IPTV and any major sporting event (World Series, Super Bowl, etc) is horrible buffering when I try to watch on my 4K IPTV (streaming) channel. I always have to downgrade to the HD channel and I still occasionally experience buffering.

So Netflix isn't alone in this matter.

more_corn• 2 years ago

Streaming is hard.

j45• 2 years ago

Streaming live can be a very different thing than on-demand.

Hopefully Netflix can share more about what they learned, I love learning about this stuff.

I had joked I would probably cancel Netflix after the fight.. since I realized other platforms seemed to have more content both old and new.

Then the video started stuttering.

ravenstine2• 2 years ago

How is this not a solved problem by now?

I think this is a result of most software "engineering" having become a self-licking ice cream cone. Besides mere scaling, the techniques and infrastructure should be mostly squared away.

Yes, it's all complicated, but I don't think we should excuse ourselves when we objectively fail at what we do. I'm not saying that Netflix developers are bad people, but that it doesn't matter how hard of a job it is; it was their job and what they did was inadequate to say the least.

Jonathan Blow is right.

dilap• 2 years ago

Technical issues happen, but I wish they would've put up a YouTube stream or something (or at least asked YouTube to stop taking down the indie streams that were popping up). It seems like basically their duty to the boxers and the fans to do everything in their power to let the match be seen live, even if it means eating crow and using another platform.

chupchap• 2 years ago

Live streaming is hard. Most companies that do live streaming at 2024 scale did it by learning from their mistakes. This is true for Hotstar, Amazon and even Youtube. Netflix stack is made to stream optimised, compressed , cached videos with a manageable concurrent viewers for the same video. Here we had ~65m concurrent viewers in their first live event. The compression they use, distribution etc have not scaled up well. I'll judge them based on how they handle their next live event

latortuga• 2 years ago

I don't think this is their first live event. They have hosted a pro golf promotional match and they had a live pro tennis match between Nadal and Alcaraz off the top of my head.

jaarse2• 2 years ago

Seems like the magic number was 60 million concurrent streams

https://www.theverge.com/2024/11/16/24298338/netflix-mike-tt...

ta12439• 2 years ago

That's a suspicious number, given the previous world record is 59 million.

yusyusyus2• 2 years ago

I wrote an analysis on doing this kind of unicast streaming in cable networks a decade ago. For edge networks with reasonable 100gig distribution as their standard, these would see some of the minor buffering issues.

There is a reason that cable doesn’t stream unicast and uses multicast and QAM on a wire. We’ve just about hit the point where this kind of scale unicast streaming is feasible for a live event without introducing a lot of latency. Some edge networks (especially without local cache nodes) just simply would not have enough capacity, whether in the core or peering edge, to do the trick.

ta12439• 2 years ago

Saw an Arista presentation about the increase in SFP capacity, it's Moore law style stuff. Arm based kit has a shockingly efficient amount of streams-per-watt too.

I can't see traditional DVB/ATSC surviging much beyond 2040 even accounting for the long tail.

You're right that large scale parallel live streams has only become feasible in the last few years. The BBC has some insights in how the BBC had to change their approach to scale to getting 10 million in 2021, having had technical issues in the 3 million range in 2018

https://www.bbc.co.uk/webarchive/https%3A%2F%2Fwww.bbc.co.uk...

Personally I don't think the latency is solved yet -- TV is slow enough (about 10 seconds from camera to TV), but IP streaming tends to add another 20-40 seconds on top of that.

That's no good when you're watching the penalties. Not only will your neighbours be cheering before you as they watch on normal TV, but even if you're both on the same IPTV you may well 5 seconds of difference.

The total end-to-end time is important too, with 30 seconds the news push notifications, tweets, etc on your phone will come in before you see the result.

yusyusyus2• 2 years ago

Saw an Arista presentation about the increase in SFP capacity, it's Moore law style stuff.

SFP itself isnt much the issue. Serdes is, and then secondarily the operating power envelope for the things (especially for the kinds of optics that run hot). Many tradeoffs available.

I can't see traditional DVB/ATSC surviging much beyond 2040 even accounting for the long tail.

Tend to agree in well-developed infra, but rural and poorly-developed are well served with more traditional broadcast. Just saying “starlink!” 5 times in a dark bathroom won’t fix that part.

Personally I don't think the latency is solved yet -- TV is slow enough (about 10 seconds from camera to TV), but IP streaming tends to add another 20-40 seconds on top of that.

I dont think it will get better. Probably worse, but with net better service quality. HLS/DASH are designed for doing the bursty networking thing. Among good reasons for this, mobile works much better in bursts than strict linear streams, segment caching is highly effective, etc.

But I think this makes sense: its a server-side buffering thing that has to happen. So assuming transmuxing (no transcoding lol) and wire latency are 0, we’re hitting the 1-5 seconds for the segment, probably waiting for a fill of 10 seconds to produce the manifest, then buffering client-side another 10 or so. Throw in more cache boxes and it’ll tick up more. It is quite high, but aside from bookies, i dont know how much people will actually care vs complain.

onetokeoverthe• 2 years ago

off topic but.

i thought tyson was in eldercare.

cryptica• 2 years ago

I don't understand why the media is pushing this a Jake Paul vs Mike Tyson stuff so hard and why people care about it. Boxing is crude entertainment for low intelligence people.

I'm tired of all this junk entertainment which only serves to give people second-hand emotions that they can't feel for themselves in real life. It's like, some people can't get sex so they watch porn. People can't fight so they watch boxing. People can't win in real life so they play video games or watch superhero movies.

Many people these days have to live vicariously through random people/entities; watch others live the life they wished they had and then they idolize these people who get to have everything... As if these people were an intimate projection of themselves... When, in fact, they couldn't be more different. It's like rooting for your opponent and thinking you're on the same team; when, in fact, they don't even know that you exist and they couldn't be more different from you.

You're no Marvel superhero no matter how many comic books you own. The heroes you follow have nothing to do with you. Choose different heroes who are more like you. Or better; do something about your life and give yourself a reason to idolize yourself.

achow3• 2 years ago

This is Netflix's statement on the fiasco

https://x.com/netflix/status/1857906492235723244

rajnathani2• 2 years ago

Maybe if jedberg and Brendan Gregg were still a part of Netflix, that this wouldn’t have happened.

Publius_Enigma• 2 years ago

I'm sure the architecture and scale of NetFlix's operations is truly impressive, but stories like this make me further appreciate the elegant simplicity of scalability of analogue terrestrial TV, and to a similar extent, digital terrestrial TV and satellite.

PeterStuer3• 2 years ago

Pure speculation as I have 0 knowledge.

Assuming Netflix used its extensive edge cache network to distribute the streams to the ISPs. The software on the caching servers would have been updated to be capable of dealing with receiving and distributing live streamed content, even if maybe the hardware was not optimal for that (throughput vs latency is a classic networking tradeoff).

Now inside the ISPs network again everyting would probably be optimized for the 99.99% usecase of the Netflix infra: delivering large bulk data that is not time sensitive. This means very large buffers to shift big gobs of packets in bulk.

As everything along the path is trying to fill up those buffers before shipping to the next router on the path, some endpoints aware this is a live stream start cancelling and asking for more recent frames ...

Hilarity ensues

vdm• 2 years ago

TreeDN: Tree-Based CDNs for Mass Audience Live Streaming https://www.youtube.com/watch?v=wRUwsvept-8

uptownfunk3• 2 years ago

All your corporate culture, comp Structure, Interview process etc etc is all so much meta if you can’t deliver. They showed they can’t deliver. Huge let down.

Netflix buffering issues: Boxing fans complain about Jake Paul vs. Mike Tyson

Related Stories