Netflix Performance on Top ISP Networks

http://techblog.netflix.com/2011/01/netflix-performance-on-top-isp-networks.html

by carson • 15 years ago

178 64 15 years ago

techblog.netflix.com

This is not a very useful graph to assign blame. :) I've always wondered why I don't always get movies in streamed in HD even for movies available in HD.

Almost anytime of the day, I can saturate my DSL line's download bandwidth from fast sites such as Amazon S3. But Netflix doesn't always stream in HD even though my link is faster than what's required (4800 Kbps).

I suspect the bottleneck is at Netflix's end.

illumin8• 15 years ago

The bottleneck is most likely not at Netflix's end. They use a number of CDNs, such as Akamai and Level 3. These CDNs usually co-locate a box right at your ISP. So, it is entirely up to your ISP how much of that bandwidth makes it through to you. I can assure you, Akamai content is fully capable of saturating your tiny broadband connection. I regularly download large files hosted on Akamai (commercial software) and achieve download rates of 2MB/sec on my 20 megabit Cablevision line.

However, ISPs do "manage" Netflix traffic. How they choose to manage it varies by ISP, but it might be something as simple as giving streaming traffic a different priority over web traffic, which tends to be more latency sensitive. Joe sixpack tends to complain when his browser takes longer than a few seconds to load a page, which ties up a customer service line. If your Netflix stream automatically switches from HD to non-HD, you might not even notice because they've made their software pretty good at adapting to changes in available bandwidth.

SoftwareMaven• 15 years ago

Are you using a dns provider like opendns or google? I am and it drastically affects streaming rates because you get sent to the wrong place in the cdn. I changed my streaming box's dns server to my isp's and got much better service from Netflix.

anonymous2462• 15 years ago

Thanks a lot. Yes, I use Google DNS. I will be sure to try this out.

bbatsell• 15 years ago

This is a fantastic idea.

There are a few elements to it that one could use to dispute its value — for example, Netflix has no way of knowing what level of service a customer has paid for. They also don't appear to differentiate between different types of service from a single ISP (e.g., they lump together AT&T's old ADSL customers in with ADSL2/U-Verse, and, presumably Verizon's DSL with FiOS). I would hope that they are also filtering for sessions where a higher bitrate is requested, which Netflix's Silverlight app doesn't do unless the window/screen is large enough to benefit from the extra resolution.

Of course, one could argue that those differentiators don't ultimately matter, and that if its customers are opting for older or slower service levels, that can tell you something about the ISP's pricing competitiveness.

Nrsolis• 15 years ago

With respect to the service tiers of the various networks, a much more useful graph would include the mean, median, and standard deviation of the observed throughput. It should then be really easy to determine if DSL vs. FiOS is responsible for the skew towards cable companies presented in these graphs.

techsupporter• 15 years ago

"they lump together AT&T's old ADSL customers in with ADSL2/U-Verse, and, presumably Verizon's DSL with FiOS"

I was wondering why both AT&T and Verizon scored lower on the graph than I expected, though it's a pleasant surprise to see my local cable operator (Charter) parked at the top should I ever need to switch.

Perhaps Netflix can't easily separate the two? I know the hostname attached to the IP I get from Verizon reveals that I'm using FiOS, but the IP address itself seems to be assigned out of the same blocks that Verizon DSL customers use. I suspect Netflix isn't--at least for an external graph like this--putting in the computing resources needed to reverse look-up all the connecting IPs and parsing them for the individual provider offerings.

TomOfTTB2• 15 years ago

I'm sorry but I don't see how this is a fantastic idea at all.

First, this is next to useless from a consumer perspective. Because anyone whose actually tested a single provider can tell you speeds vary by geography. Time Warner might be middle of the pack overall but the fastest in your particular area.

Second, they're really p#ss#ng off the ISPs here. If you want faster bandwidth out of the ISPs its better to privately talk to them than publicly embarrass them.

Finally this reinforces the reputation Netflix is already getting among content producers which is the company doesn't play well with others. There's a reason why the head of HBO and Time Warner are taking hard lines against Netflix.

Bottom line: Netflix is part of a content delivering ecosystem. If they offend every other part of that ecosystem it will be much harder for them to survive in the future

elq• 15 years ago

> There's a reason why the head of HBO and Time Warner are taking hard lines against Netflix

A gift from my employer - http://lh4.ggpht.com/_kt8r19d1Kpw/TUCXRpFxEjI/AAAAAAAAA_Y/t_...

(context - http://www.nytimes.com/2010/12/13/business/media/13bewkes.ht...)

And Bewkes is probably a bit more pissed about http://www.google.com/finance?chdnp=1&chdd=1&chds=1&...

thetrumanshow2• 15 years ago

I respectfully disagree. This is a fantastic move by NetFlix.

NetFlix knows that the ISPs are already pissed at them and are moving to throttle them, so they fire back by showing consumers which of the ISPs are a better fit for their NetFlix viewing habits. NetFlix is flexing its muscle a bit here to show it can redirect customers to choose different ISPs and affect their bottom line.

alexqgb2• 15 years ago

@TomOfTTB - How do you know Netflix hasn't already tried the private route? And if you think your "go along to get along" outlook had any merit in this arena, why do you think major entertainment companies spent the last decade trying to undo the internet?

Remember that "the others" with whom "Netflix does not play well" include members of the RIAA, the MPAA, and primary backers of ACTA. I'm sure you're also aware that cable operators have a similarly anti-customer track record in Washington DC.

These not-minor details matter. Why? Because the audience is also part of "the entertainment ecosystem". Suing them, DRMing them, and bribing their elected representatives in Congress all count as fundamentally offensive - and all that was going on well before Netflix could get anyone to return its calls.

I'm not saying that your advice isn't good. I'm just saying that Netflix isn't the first company to which it should be directed.

jauer• 15 years ago

"Netflix has no way of knowing what level of service a customer has paid for"

I think that partially explains the clump of RBOCs in the 1,800 - 2,000 band.

I manage a network that has several hundred subscribers wholesaled off of AT&T's legacy DSL network in the midwest.

Many of our users can't get speeds beyond that range because of line errors. The copper plant is degrading. Those RBOCs primarily use ADSL. Their speeds will probably degrade until someone overbuilds with FTTH.

ShabbyDoo• 15 years ago

Yes. The only thing I found useful about this graph was the variability over time by provider. I'm a (surprisingly) happy WindStream customer, and I routinely get the 6Mbit down that I pay for. If I wanted to pay more, I could get 12Mbit, but I really don't need it. Conversely, if I wanted to save money, I could buy only 1.5Mbit down. If many Windstream customers only pay for 1.5Mbit but still stream Netflix, then the Windstream average will look crappy even though Windstream is delivering as promised. I wonder how Netflix could reliably collect up/down SLAs and report metrics compared to them.

jerf2• 15 years ago

Ah, interesting thing I just barely noticed: Compare the vertical scales on the US and Canadian graphs.

InnocentB• 15 years ago

You have to bear in mind that the Canadian graph is measuring kilbots per second, which is a far scarier metric :)

jerf2• 15 years ago

Wow, I was just talking about the numbers on the scale, I didn't even notice that.

sachinag• 15 years ago

They really should have labeled the lines themselves on the right side instead of just using the default legend in Excel. I can't really tell who the hell is on the top since there are like three different greens (let alone the colorblind).

thamer• 15 years ago

In comparison, YouTube has a page with your connection and your ISP compared to others: http://www.youtube.com/my_speed

Cyndre• 15 years ago

Thanks for that. A nice little tool for isp admins to check how they are doing in general.

roadnottaken• 15 years ago

great link, thanks.

danilocampos• 15 years ago

For all the shit I give Comcast, I must take my hat off to them for being at the top of the heap. And I'm not surprised – my Comcast service here in the Bay Area has been awesome. I get every last bit of my 20 Mbit pipe, Watch Instantly starts almost instantly and I always get the HD version of my programming when available.

I hasten to add that part of this must be a function of competition here. I can choose from Comcast, DSL or AT&T's U-verse thing, so they have some incentive to provide a compelling bandwidth offering. Still – at least where data service is concerned, Comcast delivers.

noodle2• 15 years ago

comcast's end product is not bad at all, compared to what is available in america.

their problem is customer service and their corporate policies. i avoid calling them at all costs and only do so once i know for sure that my problem is unfixable on my side.

Cymen• 15 years ago

Did you know they have a direct support forum at BroadBandReports.com? I used it a couple days ago and it is useful for smaller issues that you don't want to call in. Must be a subscriber and only you and Comcast can read your posts in the direct forum:

https://secure.dslreports.com/forum/comcastdirect

dugmartin• 15 years ago

Netflix over Comcast is also great out here in Western Mass - but it's probably because we are the only people streaming on our segment.

ktsmith• 15 years ago

I'm with Charter, 30Mbit down 5Mbit up at a not too outrageous price. I typically see 32-33Mbit down in actual use (sustained). 60Mbit down is available but is still too expensive for my use. They don't cap your bandwidth at this time either. Their customer service really isn't that bad in my experience. They haven't hesitated to send out techs to look into potential line problems. They've swapped out modems (leased) on request. They proactively send out hardware when upgrading (DOCSIS 2.0 to 3.0 for example) so that consumers are ready ahead of time etc. They certainly aren't perfect but my other option is AT&T which is atrocious in this area and has trouble providing their top speed U-Verse. My only complaints with Charter are with their television packages which stink and are overpriced. I'm pretty sure that's only partially their fault.

rdoherty• 15 years ago

I'm glad someone is showing average bandwidth rates for this many ISPs; I'm not sure I've ever seen any tech site test that many.

I gotta nitpick the graph though, it's incredibly hard to figure out which line is which ISP (too many lines and similar colors). A regular bar chart with the ISP names under each one would have worked fine (since the avg. bandwidth didn't change much over the course of the study).

mthoms• 15 years ago

This community could make some very interesting visualizations if Netflix would simply provide the raw data used to make the chart. This would be even better if the data were broken down by zip code. C'mon Netflix, let us have at it!

mmaunder2• 15 years ago

This definitely fits with my experience using Qwest DSL for my business in Seattle and Comcast cable at home. Comcast has consistently high throughput, low latency to my Seattle data center, fewer hops to the local DC, and for some reason Comcast routing seems to be smarter about routing via the higher quality of our 4 X 1 gigabit providers when I'm connecting to our local DC.

btucker• 15 years ago

I'm a very happy Charter customer in Massachusetts. I pay $30/mo for a 10Mbit connection

mmaunder2• 15 years ago

They should also filter for geographies that providers serve. e.g. If one provider is serving predominantly rural areas, most of their trunks will be using directional wireless, leaving them at a severe disadvantage which is no fault of theirs.

If we were then to rank ISP's on throughput, even though a rural ISP is serving a critical and under-served part of our community, they would be rewarded with negative perception.

chc• 15 years ago

A lot of people I've heard from in rural areas have much better Internet connections than I can affordably get living in San Diego.

chadgeidel• 15 years ago

I'll second this. When I lived in Mitchell, SD (15k people - look it up) I had 3 ISPs to choose from (in city limits). I had pretty good throughput from 2 of them - one was wireless.

wildmXranat• 15 years ago

I am on their fastest tested network in Canada, Rogers. Just few months ago, Rogers upped our prices and dropped our monthly quota to 60 gb/month. That means I can use up my monthly quota by watching Netflix for about 3-4 days.

Yeah, we are getting raped either way. It's pretty much impossible to win this argument as a consumer, even if Netflix wins their with the ISPs.

I used to be with Teksavvy where 200 gb/ month was the norm. I just might have to convince my g/f and get her to switch.

Just to compare, Rogers is $59 for 60 gb/ month, Teksavvy is $29.99 for 200 gb.

kmfrk4• 15 years ago

Please don't use "raped" like that.

lincolnq2• 15 years ago

Offtopic, but do you find this especially offensive? (The GP's quote was "Yeah, we are getting raped either way").

If "fucked" was substituted for "raped", would you still be offended? What about "violated"? "shat on"? "murdered"?

I'm trying to determine whether it's the content of the metaphor, the language, or something else which is offensive. (FWIW, my friends use "raped" a lot, and I sense that some are offended by it, so I'm trying to distinguish why.)

wmoxam• 15 years ago

Teksavvy won't be able to offer 200GB for $29 anymore now that UBB is a reality (at least for their DSL offering, not sure about their cable offering).

charlief• 15 years ago

Rogers is the top ISP in the set, but according to some official statements, their systems inadvertently began severe DPI shaping on encrypted and P2P traffic since September 2010 around when Netflix was rolled out to Canada. This issue still remains today.

I have a 50 megabit plan with Rogers and all my SSL traffic throttles the entire line with an exponential dropoff to 25 Kilobytes a second. The packet loss is severe and persists for all packets even HTTP. It takes a long time sometimes an hour for the connection to recover. You can imagine how many applications this would impact (syncs with large files, P2P, Dropbox make the Internet virtually unusable).

I don't speculate on the matter, but here is some more information.

http://www.christopher-parsons.com/blog/isps/rogers-network-...

http://torrentfreak.com/rogers-bittorrent-throttling-experim...

jarek• 15 years ago

A graph of throughput vs bandwidth usage limit per ISP, pretty please?

aidenn0• 15 years ago

My ISP (cox) is near the top, and dealing with their customer service has been a breath of fresh air compared with any other ISP I've dealt with.

On an absolute scale, it's so-so customer service, but on a relative scale it's amaizing how many things they don't do wrong compared to other ISPs I've worked with (Comcast in particular)

tlrobinson• 15 years ago

This is great. I'd also love to see some sort of "ISP score" based on how "net-neutral" each one is.

losvedir• 15 years ago

Wow, interesting, but a textbook example of how not to make a chart. I had to break out the DigitalColor Meter utility in OS X to tell which was which. Here's the list, fastest to slowest based on the leftmost point of each line: 1)Charter 2)Comcast 3)Cox 4)Time Warner 5)Suddenlink 6)Cablevision 7)Cable One 8)Verizon 9)ATT 10)Bellsouth 11)Windstream 12)Embarq 13)CenturyTel 14)Frontier 15)Clearwire

edit: Ah well, I guess that will do, can't make them stand up in a column.

ebbynezer• 15 years ago

It's things like this that make me love this Netflix. By opening up and showing their customers the courtesy of their inner workings, they invite them to become more than customers.

By showing that they care ALOT about what their users care about, content delivery performance, they demonstrate credibility, a key component of trust.

Consistent trust building exercises such as these does more than a temporary sale ever could for long term customer loyalty, even if it can’t be perfectly measured.

I would like to see more companies try the same, even if they can’t prove the ROI on a spreadsheet before hand.

Why use a spreadsheet when you have a perfectly good gut you can check to see if it is the right thing to do?

jjguy• 15 years ago

Welcome to the opening salvos of the Age Of Preferred Internet. By publishing the streaming performance metrics of consumer ISPs, Netflix is making sure providers know they can influence the ISP choice of their customers. This as a warning, pure and simple.

p.s., in case you missed it -- 21 Dec FCC ruling: http://www.thedailybeast.com/blogs-and-stories/2010-12-21/ne...

HilbertSpace• 15 years ago

My ISP says I get 15 Mbps download and 2 Mbps upload, and speed tests confirm I do. For sustained flows, I get a solid 15 Mbps when downloading software from Microsoft.

So the 15 Mbps is way above anything in the Netflix graph.

So, the ISP can't always be the bottleneck keeping data rates below 3 Mbps.

wmf• 15 years ago

Netflix (apparently) can't measure anything above 4.8 Mbps, so 15 counts as 4.8 and the average is brought down by people who are on the el cheapo 1.5 Mbps plan.

bnycum• 15 years ago

All the traffic seems to follow the same curve, with the same dips and spikes. The Canada curve is different than the US curve, but there is still a curve there too. I'm curious why that is.

trotsky• 15 years ago

I think people are misunderstanding this report, which is surely their intent. This isn't a chart of how well each ISP performs in general, or even how well they perform in relation to each other.

As a baseline, all of the ISPs on this list that I've used (ftth, coax and dsl providers) have been able to push to 90%-100% of quoted line speed 99% of the time.

What this chart shows is netflix performance, which is quite different than general network performance. What goes into netflix performance? Total netflix service load, CDN pop load, CDN circuit size, SLA/QOS terms, the choices the netflix client makes about adaptive bitrates and shaping the provider applies to the traffic.

Take a look at the US chart and note the correlations in spikes and dips across all providers at certain times - 10/21 peak, 10/23 valley, 11/13 valley etc. These are indications of aggregate stream load netflix wide and are an indication that netflix manages their peak throughput to be pretty close to their max throughput (as well they should). But it's also a strong indication that if 10% of netflix customers switched to a competitor overnight average stream performance for netflix would jump noticeably, perhaps even 10%.

How their adaptive streaming works is another huge factor. First they measure your current performance to the CDN POP and then explicitly chooses to use only 60% of that[1]. So add 40% to all the ISP numbers out of the gate. TCP+HTTP overhead is another ~4%. Downloading something else while you stream? If the streaming sees more than a few headroom faults (ie 1000ms worth of data takes more than 600ms) the system will quickly adapt down in bitrate trying to do its best to never hard fault (buffering). It likely is considerable more conservative in moving the stream speed back up. Customer using 802.11b? Max throughput there is ~5mbps, so netflix will use a max of 3mbps. In an urban area on a congested channel? You may only see 1-2mbps at times. Microwave on? etc.

Lastly, with the news that netflix is 20% of ISP traffic at night the network providers have every incentive to manage that traffic down. Look at the best US performer at ~2700kbits. The fastest HD stream is 4800kbits - Nearly twice the speed. Worst case if an ISP allowed full rate streaming they might increase their peak network load by 10% or more.

Who was it that announced netflix is now 20% of night time traffic? Our old friend sandvine, provider of the DPI gear made famous by comcast using it to throttle bittorrent. This tells us two things - 1) sandvine is on net at at least a few major broadband providers and 2) by announcing that number they are implicitly telling network managers that they can help reduce that traffic.

Netflix makes it very easy to do fine grained control on bitrates. If you are watching the HTTP headers you know exactly which customers are on which stream rate without having to count the throughput. Most users won't notice a QOE change from 4800kbits to 3200kbits. Want to cut your network traffic? Use DPI to identify the higher rates and label them bulk-toofast. Then either explicitly rate limit them, give them a low queue priority or simply route the acks through a big buffer under performing router or indirect route. Once the adaptive bitrate slows down to a rate you like switch the label. The more subtle of these methods are much, much harder to detect than the comcast RST's were, and wouldn't technically violate any net neutrality rules in place.

[1] http://www.breakingpointsystems.com/community/blog/emulating...

adestefan• 15 years ago

There's a major flaw with this chart: companies that offer more than one connection service have all their services lumped together. The reason Comcast, Cox and Charter are at the top is because they're cable-only systems. Most of the other companies on that chart offer a mix of DSL, wireless and FTTH services.

Netflix Performance on Top ISP Networks

Related Stories