Verizon Using Recent Net Neutrality Victory to Wage War Against Netflix?

http://davesblog.com/blog/2014/02/05/verizon-using-recent-net-neutrality-victory-to-wage-war-against-netflix/

by ceterumnet • 12 years ago

560 226 12 years ago

transfire• 12 years ago

Can anyone else verify this or similar?

o0-0o• 12 years ago

I think Time Warner has started doing this now as well. Right after the ruling, Netflix quality was so poor that I bought a new modem. No improvement. I will have to check the other AWS services I use including my own stiff. Thanks for posting.

leobelle3• 12 years ago

There is a lot of complexity involved with network connections. There are many plausible alternative explanations for your and the author's problems. One or two people talking about their experience isn't the kind of data you would need to jump to the conclusion that your ISP is throttling your internet connection.

Justsignedup• 12 years ago

They literally stated that during a support call "we throttle cloud services"

Can anyone set up a "speed test" online on AWS or something similar? This way we can experiment with connections.

orenmazor2• 12 years ago

if only we had a distributed group of people, all of whom with a vested interest in figuring this out, that could help us with doing speed tests in variety of configurations…

MichaelTieso• 12 years ago

Not sure the person in the chat understood the question. Doesn't appear to be a native English speaker.

squigs254• 12 years ago

Totally true, but let's be honest even if the chat representative understood English perfectly and denied up and down that Verizon was throttling, he probably doesn't have the expertise to know one way or the other what is actually happening.

I think the evidence speaks for itself however.

ef4• 12 years ago

I think the ISPs are going to eventually regret this. Once you get a reputation for having a crappy product, it's hard to lose it. And they will get that reputation.

Their own advertising has often touted how well their service can stream Netflix. People are already primed to judge them on it, because the industy's own advertising has taught them to.

Stupid stupid.

squigs254• 12 years ago

IN the perfect world of competition, true.

Unfortunately most people have very few ISP options.

zaraflan• 12 years ago

Comcast/Verizon/AT&T/et al already have terrible reputations. They also have dominant local monopolies and the legislative power to defend them.

nhangen2• 12 years ago

I disagree. Here in Florida, Verizon FIOS has an excellent reputation and the product is far superior to the other choices we have.

It's also the first place I've lived where I have choice (Brighthouse being the other player).

Time Warner and Comcast are definitely among the worst though, and I suspect that if the OP is correct, it won't be long until Verizon joins them.

bhouston• 12 years ago

Someone should write a speed test website that looks to read from a set of common cloud services and then compares the speeds and records this based on ISP and location.

This shouldn't be hard to setup and you'll get a ton of page views probably for ever.

One can then see degradations over time and based on ISP. Would be even nicer to have some controls in there too so you can see if well known cloud services are slower than non-cloud services.

trigger• 12 years ago

Netflix already do this, http://ispspeedindex.netflix.com/

sonnyz• 12 years ago

Things are not looking good in my area... http://imgur.com/pxlpxO9

paulgb2• 12 years ago

Unfortunately their US data only goes to December. Looks like Verizon dipped prior to the net neutrality ruling. http://imgur.com/xOfjRf0

erjjones• 12 years ago

Did you check out the other countries. We get the shaft on speeds for sure.

JangoSteve• 12 years ago

I'm a little confused by the average speed reported in their speed index. Not sure about the others, but I know Comcast and AT&T both offer several speed tiers. Is their average speed reported here affected by the percentage of customers who purchase the lowest tier?

cheald• 12 years ago

...and some rather heady bandwidth bills.

bnchdrff• 12 years ago

M-Lab provides a suite of tools to test a lot of net neutrality related stuff: http://www.measurementlab.net/

tobylane• 12 years ago

This doesn't at all answer your question, but it's close. http://broadband.mpi-sws.org/transparency/bttest.php

forcer• 12 years ago

You inspired me to do exactly that - coded within 6 hours - http://netneutralitytest.com/

colinbartlett• 12 years ago

I would not take some offhanded remark from an outsourced, minimum wage, offshore worker as confirmation of anything.

RyanZAG• 12 years ago

By far the most correct observation. These guys are just following a script - they probably don't even know what half of the words mean. Please don't read anything into it or assume the outsourced support is somehow privy to anything at all that goes on inside Verizon other than what the FAQ system he's using tells him on screen.

squigs254• 12 years ago

I wouldn't read into this even if it wasn't an outsourced support person.

But the proof is in the pudding, the speed tests speak for themselves, no?

waps2• 12 years ago

The netflix data itself points to a gradual lowering of bandwidth to their service. Now here's 2 scenarios :

1) Verizon decided to enter some router configurations that affect the packet forwarding to Netflix (well, most likely FROM netflix, as the internet is not actually symmetrical). What we should see as a result of this is a sudden change in bandwidth.

2) The load on the interfaces in between is changing, either because of more netflix subscribers, more AWS traffic, or more netflix traffic. Since these are things happening slowly over time, you'd see a gradual worsening of traffic conditions.

I think it's pretty fucking likely that what's happening here is 2). Now granted, Verizon could solve this problem by cooperating with Netflix. But for them that's an expense with no upside, so likely they're demanding Netflix make it worth their while to upgrade the interconnects.

This is one of the basic problems with the internet today. It used to be the case that bandwidth between providers was much more plentiful than last-mile bandwidth. So we upgraded last-mile bandwidth. Problem : doing that resulted in an exponential increase in load on the core network and the interconnects between different networks. Needless to say, nobody can keep up with exponentially rising demands (especially without subscriber growth). The basic problem is that a linear increase in bandwidth for the customer results in an exponential increase in costs for the ISP (so -surpise- small ISPs don't feel the effects nearly as badly).

So ISPs are desperate. Their costs are spiralling out of control, with everyone demanding they follow this exponential curve, but of course their customers not willing to pay for it (and all the money in the world would only buy time if thrown at this problem). Demanding the ISPs pay for it is simply not going to work. Right now, yes, they could theoretically pay for it, but that won't last long.

Plus Netflix is being a rather bad netizen themselves. The polite thing to do is to carry the traffic to very close to the user on your own network, and only use other's links, especially transit, as a last resort ("cold-potato" routing, for content providers). Netflix has caches, but no own network.

There's other problems that result from this cost problem. Already transcontinental and other long-haul links are strained to the limit. This has resulted in a massive degradation of non-local internet traffic, and it's getting worse fast.

It is really a very simple problem. You have consumers, and you have producers. And you get more and more of each. To a limited approximation you can assume everyone on the internet is both. To simplify things, let's say everyone uses a tiny amount of everyone else's services. Now given N participants in the network, what is the load on the core network ?

N * (N - 1) =~ N^2 (assuming full duplex links, which would be generally correct for the internet)

So adding more participants in the internet increases resources required to give every existing participant in the network his "old" speed. These need to be paid for. But unless everyone want to start paying an amount for your internet connections that rises with the square of the size of the internet, we're going to see progressively worse and worse throttling. Right now the strategy is fast becoming limiting interconnect capacity to prevent the core network overloading, which would have far worse consequences.

While, yes, historically this has mostly been monopolies cheating the market, that's less and less true. This is going to get worse, fast. And since those resources are obviously not going to be made available, there's only one thing to say :

Get used to it.

rosser2• 12 years ago

I'd be with you if you'd said, "By for the most likely correct observation."

But you're asserting the truth of this position, not its probability, and I don't think anyone participating in this discussion thus far has sufficient information to do that.

waterlesscloud• 12 years ago

You're right. It would take a top executive paid a minimum of 7 figures before we could trust what they tell us.

nashashmi• 12 years ago

And this is the reason why I cannot make a complaint about Verizon that will stick. Verizon purposely hires or makes it seem they hire crappy technical people. But when a person does 'know' either through some confidential/proprietary memo or through technical know-how, people begin to have doubts on the quality of the information.

I had a deep fight with Verizon over a chromebook 3g service that they no longer offered, and I tried to get a copy of the internal memo that said the service was cancelled, but they would not give it to me. And everybody thought that the person I spoke to was lying.

cbaleanu2• 12 years ago

The title of this article is also very misleading. Did not find any reference to the neutrality law in the article.

kevinchen• 12 years ago

Verizon throttling AWS has nothing to do with net neutrality?

lerk• 12 years ago

They don't get a minimum wage. That's the reason they're off shore.

whitef0x• 12 years ago

Also I do not think that Verizon would be targeting Amazon's S3 platform, even if it was a small subsection of it. Too many webservices use it and Amazon is no small player either. There are easier ways to limit Netflix bandwidth through data stream throttling like they do for torrents.

senthil_rajasek• 12 years ago

How did you conclude that this was a "remark from an outsourced, minimum wage, offshore worker" ? The author was talking to a Verizon customer support.

netneutrohno• 12 years ago

Yup, Verizon confirmed that it isn't throttling anything.

http://bgr.com/2014/02/05/verizon-throttling-netflix-amazon-...

darkarmani2• 12 years ago

Too bad there is proof that they are throttling. Whether it is by policy or accident, they speed differences indicate that they are affecting it.

grej• 12 years ago

This is probably only the beginning. On demand video directly threatens the television business model of these companies. They will use every technology available to them to throttle users like crazy until they are compelled not to, either by the market (consumers en masse) or by regulation.

xSwag• 12 years ago

Yes let's arm ourselves with pitchforks based on an anecdotal experience of a stranger on the internet.

eksith4• 12 years ago

"That means a lot of services are going to be impacted by this." Should be easy enough for someone else to verify shortly.

No need for pitchforks, but corroboration would certainly be welcome.

eksith4• 12 years ago

Oh dear. Looks like the work left for Google Fiber isn't done yet. Hoping they'll make a run in New York (or at least upstate where I intend to move), but that's a long shot.

I wonder how much independent ISPs could make if they stick to a small list of customers, but take up the "We Don't Throttle" stance.

lucb1e9• 12 years ago

Google Fiber is the worst idea ever. You know this company makes the vast majority of its income through advertising and collecting data about you, right?

I understand that it's a great service for a great price, but the question is how long that remains sustainable. Right now they're sitting on such a pile of money that it doesn't matter whether they provide internet access in a few cities at a super cheap price. They benefit enough in reputation and in extra usage of other services when people are able to use very high quality internet.

Proof of this is that Google disallowed running servers (the very thing that makes the internet the internet, end to end communication) up until their reputation was almost completely tarnished in the hacker community. Only then did they relent and allow servers to be run. They're not much different from other for-profit companies.

eksith4• 12 years ago

Google disallowed servers initially as a non-insignificant number of people thought of the service as an easy entry into mass virtual hosting as soon as they got it. That started to affect throughput[1]. As mentioned numerous times elsewhere, their ultimate goal is to embarrass ISPs into providing better service. That seems to be working in Texas ATM. The hobbyists that complained got an exemption in their TOS for non-commercial use[2].

Of course, Google is collecting information about me. As is every other ISP (Verizon and Comcast hijack domain misspellings to redirect to their search pages), which is why VPNs and Tor are so popular right now.

[1] http://arstechnica.com/information-technology/2013/07/google...

[2] https://support.google.com/fiber/answer/2659981

Fogest2• 12 years ago

They would be stupid to allow servers. Buy your own business line if you want that. Google doesn't want others losing bandwidth because some people want to run servers.

bestdayever• 12 years ago

Google Fiber is the worst idea ever. You know this company makes the vast majority of its income through advertising and collecting data about you, right?

You never actually made a point here, do you have one? Obviously google makes money through advertising so getting you fast internet allows you to see more ads, is that a bad trade off?

jpswade• 12 years ago

Net Neutrality was dead as soon as Google became an ISP.

We all know where this is going.

alimoeeny2• 12 years ago

So that is what's going on! What should we do?

wheaties• 12 years ago

So there's no issue with a business line? Sounds like they're NOT throttling Netflix per se but rather residential lines. That would make sense since businesses pay more and generally consume less.

Welcome to the future.

Twirrim• 12 years ago

Hanlon's Razor: "Never attribute to malice that which is adequately explained by stupidity."

It could be any number of things, like a problem with one of their peering providers. I wouldn't take one customer support agent's word for it.

veritas20• 12 years ago

File a compliant with the FCC. Yes, I know, but you have to start somewhere to let them know how their action/inaction/decisions affect consumers. The more complaints, the better.

bartkappenburg• 12 years ago

Slightly off topic:

I'm very lucky to live in the Netherlands if I compare the average speeds, even with the mediocre ISPs, with the rest of the countries where Netflix is active.

See: http://ispspeedindex.netflix.com/netherlands

gst• 12 years ago

You can't really compare a small densely populated country such as the Netherlands to a huge country with large rural areas such as the United States. If you live in an urban area here (such as the Bay Area) you get excellent speeds as well.

seanmcq• 12 years ago

What bay area do you live in and who is your ISP?

squigs254• 12 years ago

I think the best way to test this would be to open a VPN connection, and connect to AWS or watch Netflix via VPN

alimoeeny2• 12 years ago

OK, this guy lost his job, I am sure they record all the conversations, and just a grep -i "cloud providers" should be enough to find the conversation, sorry man,

joshuacc• 12 years ago

I have also noticed a similar effect on Verizon FIOS, where some video services work perfectly well during the day, but around 5pm or so start getting incredibly slow to buffer. Whether this is intentional throttling I'm not qualified to say, but it makes me quite hesitant to use Verizon in the future.

riggins• 12 years ago

I have had the same experience with Verizon FIOS in NYC.

brown9-22• 12 years ago

I've noticed this when streaming from the PBS Apple TV app, but not so much the Netflix app or website.

Occum's razor would suggest the problem is likely with the video services themselves, and/or congestion from other people who are now home from work watching videos, than something malicious from Verizon.

polarix• 12 years ago

Back to torrents, anyone?

rmchugh• 12 years ago

might have been a good idea to black out his name. poor fellow might lose his job for being caught saying that.

incongruity2• 12 years ago

Having the raw chat visible would be enough to find it in the logs, if they cared that much… just say'n is all.

lucb1e9• 12 years ago

I'm pretty sure the chats get logged, they can search through that. There are no privacy regulations in the US like there are in Europe about not spying on employees, and even here (Europe) most have an automated message saying "This [chat/call] may be recorded for training purposes."

GunlogAlm• 12 years ago

Net Neutrality is dead [in the United States].

geofflane• 12 years ago

Net Neutrality is fine. The FCC just needs to reclassify internet providers as common carriers rather than information services.

To quote from the Washington Post [1]: "Broadband is currently classified by the FCC as an information service, a category that gives the agency a fairly limited set of regulatory options. If Internet providers were classified instead as common carriers, the FCC's rule would likely stand. In fact, the federal ruling on Tuesday upheld the FCC's net neutrality rules as a matter of principle; the problem is that the agency effectively tried to apply its powers in the wrong context."

[1] http://www.washingtonpost.com/blogs/the-switch/wp/2014/01/14...

wiremine• 12 years ago

So what does the legal roadmap look like going forward for the FCC?

jlgaddis• 12 years ago

The FCC just needs to reclassify internet providers as common carriers rather than information services.

Yeah, it's not quite that easy. With the "common carrier" classification comes a lot of other regulatory issues.

padseeker3• 12 years ago

How do we, IT savvy people capable of monitoring, testing and verifying this stuff, use our collective skills to monitor each ISP and communicate who is doing the best/worst job?

As long as there is at least one ISP provider who is not throttling traffic, we should reward that ISP that does not discriminate on web traffic, and punish the ones that do. You might have to show some differences across different regions and cities, but this is critical. I don't know enough about monitoring web traffic to figure this out but it sounds like an awesome project to spearhead. How do we get started?

It's just that I'm just a little sick of the finger pointing, and would prefer to do whatever I could to fix the problem or use the markets to promote the good companies and punish the bad ones. If there is a market where all the ISP providers are throttling then you have a better case for government regulation. Who's with me?

kordless4• 12 years ago

I'm with you.

The primary problem we'll have is the 'reward' system in locales with only one provider. You'll have to focus on markets with multiple providers. The solution involves building an escrowed Bitcoin fund which does payouts based on provider's aggregate throughput. How that is implemented, I have no idea.

padseeker3• 12 years ago

I was thinking a lot simpler - keep tabs on each ISP in each market. Determine who has the least discriminatory network. Once enough data is collected you can make recommendations. ISP X is the best, ISP Y is the worst, anyone who cares needs to switch from ISP Y to ISP X. No bitcoin needed.

You can have a coordinated event, a cancellation day where all users of ISP Y cancel on the same day and switch to ISP X. Make a big deal about it. Tell your friends about it.

The one concern I have is in certain markets there might not be enough competition. If all the providers in that market are throttling then you have a stronger case for the government to get involved. Heck, there might even be collusion in those markets. But you might be able to bypass any government enforcement of Net Neutrality if there are enough well informed consumers punishing the bad ISPs and patronizing the good ISPs. It aint rocket science.

brianmcdonough2• 12 years ago

I want to express my gratitude for your thoughtful and more effective approach. The aha! nature of this article and the sensationalistic title lead me to believe that it's a knee jerk reaction.

padseeker3• 12 years ago

thanks - the concern is legit, it might be throttled like the article said. OR the guy on the phone is inept and has no idea what he is talking about. Either way the threat is real possibility but whether it is actually happening is hard to verify.

Lack of competition though is a hard barrier to climb. If there are only say 2 ISPs in your area it seems like you are pretty much SOL. And that is where government intervention needs to happen.

mweinbergPK• 12 years ago

As someone who is in DC doing net neutrality advocacy on a daily basis, I love this question. One of our major challenges is that ISPs have data on their networks and we don't. That leaves us in a position to speculate about what is "really" happening and hoping that someone out there happens to have the technical skills to potentially uncover a problem. The HN community's ability to evaluate claims made by ISPs as legitimate or illegitimate based on technical constraints is incredibly valuable.

ajb• 12 years ago

Interestingly though, there is an argument that even ISPs don't have the right data, and that this is a reason - or at least an excuse - for throttling. See http://www.bobbriscoe.net/presents/1005ftw/1005ftw-briscoe.p... There is an ietf working group (ConEx) on exposing the kind of information necessary, with the idea that this would provide the right kind of incentives for ISPs not to throttle. I'm not sure that ISPs wouldn't anyway have the incentive to be a gatekeeper, but it would be a good idea to remove any possible excuse.

wmf5• 12 years ago

Check out these projects: http://projectbismark.net/ http://www.fcc.gov/measuring-broadband-america https://www.samknows.com/

njharman• 12 years ago

It would be easier to just get Net Neutrality into the law / regulations.

Or, just ask the NSA who is already collecting ALL THE TRAFFIC!!!

ninjac0der• 12 years ago

Because more regulation is a good idea when education about the problem is this low...

rosser2• 12 years ago

This is one of the reasons I'm a loyal Sonic customer, despite paying more dollars for less bandwidth, relative to my friends and coworkers who use, for example, Comcast.

ceterumnet2• 12 years ago

I've updated the blog post to also include traceroutes from the 2 sites. I'm not a networking expert, but since the traffic is ultimately going through the same network - peering should be a non issue. Feel free to let me know if this is significant or not.

rnovak• 12 years ago

There are really too many variables to have any idea. For example, in both instances I see that your traceroute identifies the hop to your router, and you're using the 192.168.1.0 private network. Since your business example shows actual reverse dns lookups (from the looks of it), I'm guessing it uses a commercial switch.

Unfortunately, there's no way to determine the actual cause of the slow down.

For instance, you said that the slow down happens after 5pm, which quite literally is when 80+% of the working force gets off work. It could also be an artifact of AWS, and not Verizon FIOS.

Also, 2 instances does not a trend make. There's no way to know if you're one of those people who were sent a letter because they were using terabytes of data a month, and they're simply protecting the network traffic of 99% of their other users. Granted, I assume you aren't, but I can't be 100% sure.

For a truly independent test, you'd need the same modem/access device, without a local switch (i.e. a machine connected directly to the modem), and you would need VPN access to test both identifiable and non-identifiable traffic.

If you could verify that Netflix traffic through the VPN, on the same network, without any interference through the router, was delivering much more bandwidth than the same Netflix traffic, but in the open (not through the VPN), and could do so for a statistically significant portion of the Verizon FIOS user base, then you'd have reason to complain. Until then, I don't see your claim being taken very seriously.

Just my opinion though, ymmv

Touche• 12 years ago

Unfortunately, there's no way to determine the actual cause of the slow down.

Is networking the most frustratingly complex field in technology? As someone who admittedly knows little about it it all seems like a pile of hacks stacked on top of one another. It's the only explanation I can come up for why it's near impossible to figure out why a network isn't behaving correctly.

scurvy• 12 years ago

We'd need more samples to make any serious conclusions, but it does look like you're taking the same route from VZ's gear to AWS. That would rule out any kind of peering spat or congested port theories. If the paths are indeed the same, it means that VZ is CoS'ing down residential traffic over business. You'll need to get more traceroutes during different times of day, too.

jstalin2• 12 years ago

I have experienced exactly this on Comcast. Suddenly, about two months ago, Netflix and Hulu became terrible to watch. I thought maybe it was my cable modem, so I upgraded it and still no luck. So I signed up for a VPN service and tested watching Netflix and Hulu both on the VPN and off.

Sure enough, when on the VPN, I get the highest quality picture. When I disconnected the VPN, Netflix dropped down to about 500kbps. Terrible.

nathan_long• 12 years ago

An excellent test. If a VPN makes something faster, despite the fact that it adds overhead to the process, that seems like proof that it's being throttled.

jstalin2• 12 years ago

Oh, and the even more galling part... when watching Hulu on regular Comcast (without the VPN), the shows average around 500-900kbps. When it flips over to displaying the commercials, it jumps to 8Mbps or more!

twoodfin4• 12 years ago

Not necessarily. It might just be bypassing the congested route that everyone else on your ISP is using. Comcast and Verizon have had disagreements peering with Cogent, one of Netflix's ISPs:

http://gigaom.com/2013/06/20/verizon-that-peering-flap-about...

It's not throttling, but rather the peering points along the "best" route between Cogent and Comcast/Verizon are overloaded, resulting in lousy connections between them. Look at this traceroute hop from Comcast to Cogent I just recorded:

   7  he-0-10-0-1-pe03.111eighthave.ny.ibone.comcast.net (68.86.83.94)  21.377 ms  20.379 ms  19.387 ms
   8  be7922.ccr21.jfk10.atlas.cogentco.com (154.54.13.161)  67.126 ms  67.747 ms  69.083 ms

40 ms to go one short hop in, I believe, the same data center.

rhino369• 12 years ago

It's not being throttled. There is a peering dispute between netflix's CDNs and the ISPs. Netflix's CDN's don't want to pay for delivery to the ISP. Normally CDNs peer with each other for free if the balance in each direction is equal. The problem is that any CDN that delivers netflix is hugely unequal since it's a huge one way stream.

The ISPs could just eat the cost, and let it be delivered for free. Or Netflix's CDNs could just pay the peering fee. Or Netflix could use different CDNs.

The reason the VPN works is because it sends the data over a different CDN that isn't saturated because of the dispute.

At least that is how I understand it, but I'm not an expert (or even a novice).

wil421• 12 years ago

Interesting I have been watching Netflix more than ever but I havent had any problem. I'm also using Comcast in Atlanta I wonder if its based on region?

aroch4• 12 years ago

It's only tangentially related to region. Comcast bring all of Netflix's traffic onboard in NYC and their East Coast routing is handled in New Jersey. Their eastern seaboard routing is about the only area where they're not having massive internal issues. Both their Denver (Midwest) and Oregon (PNW) iBone routing centers are having serious capacity issues that are manifesting as packetloss, giant latency increases and generally shitty route-arounds in the hope that it'll relieve pressure.

650REDHAIR3• 12 years ago

I had this exact same problem with Sonic.net in SF. It was miserable. Each time I called they would tell me everything was fine and that it wasn't on their end- ignoring all proof.

On the verizon topic, soon after they got rid of the unlimited data tier my speeds would drop significantly after I reached 200-250MB of data transfer, but when I cycled the power and updated the roaming the speed would go back to normal until it hit the 200-250MB again. It wouldn't surprise me at all to learn that this was done on purpose or that they're currently limiting streaming traffic.

lucb1e9• 12 years ago

Are you sure you don't mean GB instead of MB? Because 200MB is like one youtube video with some decent quality.

jevinskie• 12 years ago

Does Netflix actually stream video data from AWS or from other CDNs? Can the OP compare the route to AWS to the route to the Netflix server?

kevingadd• 12 years ago

IIRC netflix runs pretty much everything out of AWS.

higherpurpose• 12 years ago

Solution: don't serve Netflix to Verizon customers.

dangrossman• 12 years ago

I just spun up an EC2 instance (US-East) and threw a bunch of 100MB-1GB test files on it.

I'm on Verizon FiOS here at home, residential service, and they all downloaded at ~20Mb/s, which is the line speed I'm paying for.

Anecdotally, I watch Netflix every night and haven't noticed a buffering/quality issue. I'll have to try the download test again later.

legohead• 12 years ago

The author said this was happening specifically after 4pm.

ceterumnet2• 12 years ago

Just FYI - I'm only seeing the slowdowns in the evening.

couradical• 12 years ago

Do you know how many homes in your neighborhood have FiOS? I am wondering if it's a congestion issue - have you checked RTT in the day vs afternoon? What about looking at wireshark to see how many duplicate Acks/Re-transmits you're seeing?

As only the first-hop device appears to be different in the tracert, I doubt it's something upstream - if they were throttling, they'd probably do it at the network core - before it left the Verizon network for transit, which would apply to both circuits.

kordless4• 12 years ago

Does your boss live in close proximity to you?

j_m_b• 12 years ago

To the people saying that you shouldn't trust some tech support guy, I counter with this: He is acting as representative on behalf of Verizon. Verizon trained them and Verizon gave them the scripts to read so that this tech will properly represent Verizon.

elgato999• 12 years ago

Switch to t-mobile. Screw verizon

PanMan• 12 years ago

AFAIK Netflix doesn't stream directly from AWS, but (while using it for their transcoding and so) uses a CDN. So blocking AWS wouldn't help in changing the quality of netflix, right? It certainly could be related though.

vzwthrow2• 12 years ago

They use a form of deep packet inspection to identify and "manage" traffic on their network. Where it sits doesn't matter.

brown9-22• 12 years ago

Let's say Verizon was intentionally throttling all traffic to AWS at certain times of the day (which seems like an awfully blunt weapon, but whatever).

Where is the part of this where they are promoting their video services over Netflix's "inferior" experience?

saskiah2• 12 years ago

They don't have to explicitly promote this.

Honestly, getting people fed up with Netflix (not realizing that Verizon is at fault) and then deciding to go with an actually TV package (with a discount for bundling it with internet, no less) might be the goal here.

That, or the goal is simply to hurt Netflix, which approaches the same end from the side of the provider, not the consumer.

jusben13697• 12 years ago

I'm not sure I believe any of this for reasons others have pointed out. But let's say it's true. Is this an argument for or against Net Neutrality?

By most accounts I read NetFlix is responsible for 30% + of evening internet traffic. Kind of staggering. If an ISP actively throttled Netflix people would leave that ISP for another one that didn't throttle Netflix Netflix is clearly a hugely popular service with enough people that it could exercise that clout. Netflix can sign agreements and tout such providers (heck, bundle in sign up offers etc) So the market can solve this problem.

Secondly, why are they throttling this service? Because they can't charge effectively for it. We all love the simplicity of "all you can eat" pricing. Yet Netflix essentially exploited that model to their own benefit to the point now that model is under risk.

I find it fascinating that a service that has (legally and ethically and followed standard biz practices and sensibilities) grown to gobble 30% of all resources and effectively squeezes out other traffic is seen as a victim here. I'm not trolling so would love to hear what I'm missing.

dankoss• 12 years ago

Most people in the US don't have a choice in their ISP.

jusben13697• 12 years ago

Interesting. That's news to me. The majority of people in the US really only have one choice?

pikachu_is_cool3• 12 years ago

people would leave that ISP for another one that didn't throttle Netflix.

Haha.

jusben13697• 12 years ago

We dumped TWC for ATT because of constant timeouts when trying to access movies on demand at peak times.

edent• 12 years ago

Well, there are two problems with that scenario.

1) There may not be other services in the area - or all services throttle. Your only solution is to move house.

2) The user may not realise it is their ISP which is the problem. "NetFlix sucks - it's so slow. But TimeWarnerCastPlayer is awesome - it's so fast!"

NetFlix isn't "squeezing" anyone. Customers are requesting data through their ISP, NetFlix are sending it as asked.

jusben13697• 12 years ago

Netflix has an ISP tracker on their site that shows performance by ISP. Trust me, if they know your ISP is a "throttler" they have the ways to get this in front of you and highlight "Hey you're having a crappy experience because you're using X and they throttle us. Y wants your business and will give you a $150 coupon to change. Click here now"

ohyes• 12 years ago

How is it squeezing out other traffic? A packet is a packet, under the net neutrality model all packets are routed equally. Capping packet flow from a provider would be like Subaru owning chunks of the national highway and setting a limit on the number of Toyotas that are allowed to be driving at a time. If Verizon doesn't have the capacity to support the evening demand, they should increase their shitty infrastructure so they can deliver the service that they sold to their customers.

And what if I want to compete with netflix myself? They have the cash to make a deal with Verizon, I don't.

jusben13697• 12 years ago

Most networks see a degradation in performance at peak times. That reduction in performance can result in all types of behavior. You may not watch a clip, may click away from a site etc that's loading too slowly (and that would otherwise load fine and capture your view a few hours earlier)

clavalle• 12 years ago

people would leave that ISP for another one that didn't throttle Netflix

That assumes people have a choice for their ISP.

If I want high speed internet I have two choices. Both of them are huge national companies.

If they decide to go the throttle route, where do I go?

Secondly, why are they throttling this service? Because they can't charge effectively for it.

That's an interesting take considering how they offered a product at a certain bandwidth and I agreed to pay for that bandwidth. If I use it to capacity, the services I receive from third parties are somehow 'exploiting' the model? I don't think so.

If the post office offers to ship anything under 10lbs for a flat rate, is a company taking advantage if they offer, say, a 10 lbs cheese of the month package using that USPS service?

jonknee2• 12 years ago

Verizon has special right of way access for its fiber cables and should be regulated as such. I can't change ISPs and the ISP is only able to exist because of exemptions from City, State and Federal governments.

VikingCoder• 12 years ago

leave that ISP

There's your flaw. Most people have 0, 1, or 2 choices for ISP. And those ISPs offer similarly sucky service.

Compare US bandwidth rates to modern countries, and you'll see that the US businesses are not in any way competing with each other to the benefit of the consumer.

masmullin• 12 years ago

There is an important distinction you are failing on.

"NetFlix is responsible for 30% + of evening internet traffic"

This is false. Netflix is not responsible for the traffic usage. PEOPLE WHO WATCH NETFLIX are responsible for this traffic.

I also think you are incorrect on the "all you can eat" pricing. Of course, I am not privy to the actual business deal that Netflix has with their internet provider... but I very much doubt that they have unlimited bandwidth. They, like their customers, are most likely paying for how much they use.

zimbatm• 12 years ago

It's simple. NetFlix is already paying for that traffic. The users are already paying to get access to the Internet. The ISPs are trying to make complicated arguments so that they can get away with charging the same traffic twice.

nevir• 12 years ago

This looks more like an overloaded residential gateway.

jonknee2• 12 years ago

Overloaded just to AWS? Seems strange.

fredgrott• 12 years ago

I like to point out something...

The ISP providers that use fixed wireless to customers have the same upstream infrastructure and costs as Verizon..

Want to guess if they are throttling?

They are not.. for example CSINEt local to me does not

So we should be asking what part of verzion's structure is forcing them to do this..

It cannot be money alone as both entities are entitled to the same broadband federal grants

smutticus• 12 years ago

We need more data. This doesn't prove anything and the author is just speculating.

danielsiders• 12 years ago

We're seing up to 80% packet loss on our routes to Cogent (Verizon FiOS customers).

brianmcdonough2• 12 years ago

The net neutrality issue is a tragedy of the commons. Say you have a common green area and everyone shares it. People come and picnic and everything's great until a family of giants moves into town and sits on the entire green.

Netflix is like that family of giants.

The solution is to build a bigger green, but how is that going to be funded. Customers don't want to pay for it and feel they're already being overcharged. Netflix doesn't want to pay for it just because they are streaming giant files. Such costs are not built into their ability to make profits.

It's a common tragedy.

talmand5• 12 years ago

Your analogy seems off to me. It works as long as you admit that the reason the family of giants is there because everyone else wants them to be there.

Plus, both the customers and Netflix already pay into the system, it's just that the companies in charge of it want more simply because Netflix is a competitor to some of the services they offer.

Zizzle• 12 years ago

Exactly.

It's not like the ISPs and making a profit. They are extremely profitable and just want more.

Makes me wonder how much this discussion is being atrsoturfed.

spinlock3• 12 years ago

This is why I think streaming is a broken model: it spikes usage at the same time every day. BitTorrent (or just off-line downloading) is so much better because you can amortize the download over the whole day and take advantage of off-peak bandwidth. We just need to convince copyright holders that its in their best interest to deliver a superior product.

aroch4• 12 years ago

Except streaming is the only viable option due to how content licensing works. You could do slow (200Kbps) download of content you plan on watching in the evening while you're at work, but because of restrictive licensing the cost becomes prohibitive ($10-40/night to rent and watch the same thing as on Netflix)

shinratdr• 12 years ago

That sounds like a problem for content licensing, not me. Promoting a broken model because the model that model is built on is broken as well hardly bodes well for this burgeoning industry.

wmf5• 12 years ago

If Netflix pre-cached content during off-peak hours it's not clear that their licensing bill would be any higher.

vzwthrow2• 12 years ago

Throwaway here, so take it for what it's worth but I swear my experiences are true.

I deal with Verizon Wireless (as far as I can tell, the article is about the other Verizon entity) reps often, I've heard this mentioned many times in an almost off-handed, positive, revenue-generating manner.

That they're positioning themselves to charge providers like Netflix for "priority" on their network. Except the way they describe it, it sounds like they will be deprioritizing or otherwise "shitifying", for lack of a better word, the provider's traffic. Netflix was specifically mentioned.

It's never mentioned on internal slides, but there's always some vague term on a slide that brings up this topic.

This is going to happen, people, unless legislation interferes with their plan and even then they may be able to loophole it. Net neutrality is the enemy, as far as Verizon (and possibly the other big telcos) are concerned.

talmand5• 12 years ago

I see it more as an opportunity for more VPN services. I can imagine the ads proclaiming something along the lines of getting the bandwidth you're already paying for.

chwahoo• 12 years ago

Those VPN services better pony up to Verizon then. ;)

inkovic• 12 years ago

If companies are serious in battling ISP throttling, they should explain to users that their content will load slowly due to their ISP and they should consider switching.

Make Verizon and whatnot pay with their wallets.

cobookman• 12 years ago

The industry as a whole will eventually just charge a per GB cost. During peak hours its more expensive...etc, just like electricity. The interesting part will be if they still have speed tiers. Its really in both parties best interest.

ChrisLTD2• 12 years ago

I'm guessing there will be speed tiers at first. The telecom industry doesn't leave money on the table without pressure from the government or fierce competition.

wmf5• 12 years ago

Speed tiers at first? Pretty much all Internet connections today are sold by speed already.

specialp• 12 years ago

Verizon may not purposely throttle content but they are certainly uncooperative because they want to be paid from the big bandwidth consumers. This is why Google is trying to make it more transparent to users who is to fault when you can barely watch YouTube in 360P on FIOS during some periods when speed tests to other locations yield 50Mbit. http://www.google.com/get/videoqualityreport/#how_video_gets...

darkarmani2• 12 years ago

This might explain why netflix has been working horribly starting about a month ago.

broknbottle• 12 years ago

The traceroute is useless without a return traceroute..

tim333• 12 years ago

Verizon denies it to the Washington post:

"We treat all traffic equally, and that has not changed," the statement read. "Many factors can affect the speed of a customer’s experience for a specific site, including that site’s servers, the way the traffic is routed over the Internet and other considerations. We are looking into this specific matter, but the company representative was mistaken. We’re going to redouble our representative education efforts on this topic."

Although in the comments there's a guy who differs:

"Chris Walsh wrote: Verizon is absolutely doing this, at least in Silver Spring. I stumbled on the daveblog's post via a facebook friend and was amazed because if finally explains what has happened to my home FIOS connection over the last few weeks, even down to the exact 40 kbps speed. ..."

http://www.washingtonpost.com/blogs/the-switch/wp/2014/02/05...

saskiah2• 12 years ago

Has anyone else tried duplicating some simple tests?

Ping google and netflix while connected to a verizon network. Repeat on a network with a different provider, see if the speeds are proportional, or if there is a larger % decrease for netflix than for google when on the verizon network.

I know this doesn't explicitly prove anything, but if several people all get this result that starts to compile more evidence.

Verizon Using Recent Net Neutrality Victory to Wage War Against Netflix?

Related Stories