Verizon's accidental mea culpa

It's working for me now as well. When I posted the cached version the page on Level3 blog was giving me a 404 although the post was on the "Recent posts" list.

I'm not getting why I'm being down voted for posting the cached version when I was getting the 404.

amalag3• 12 years ago

I have to agree with this assessment. If Verizon is claiming they have green in their network with 34% utilization to Level 3, then the connection between Level 3 and Verizon is lacking.

Is it a simple fix like Level 3 & Netflix claim? If so, what is Verizon's advantage to continue the slow Netflix. Netflix is really slow for Verizon users, it is very obvious for all their subscribers.

shimshim4• 12 years ago

Its not always great on Comcast either. It has gotten better over the last 6 months though, I will admit.

amalag3• 12 years ago

I bought a VPN and it fixes it. Sad that it is necessary.

spuiszis• 12 years ago

I have Comcast as well and it's gotten because of the deal Netflix struck with them... http://knowmore.washingtonpost.com/2014/04/25/this-hilarious...

EDIT: John Oliver also talks quite eloquently about this as well: https://www.youtube.com/watch?v=fpbOEoRrHyU

mikeash2• 12 years ago

Verizon's advantage should be obvious: they also sell TV service which is in direct competition to Netflix. If Netflix works poorly, that's more customers potentially signing up for FiOS TV.

lbotos• 12 years ago

I feel like this is what people often forget in the net neutrality debate. The pipe providers have a horse in the race (FiOS TV On Demand, XFinity On Demand) and we need to make sure they aren't rigging the race so to speak.

Crito2• 12 years ago

Furthermore, if Verizon degrades connections to whichever ISPs Netflix uses, then they have leverage to get Netflix to pay Verizon to peer. If they make it so that the only way that Netflix can offer their service is to pay them for it directly, then that is more money for Verizon. Very clear motivation.

hga• 12 years ago

It's even stated parenthetically in the blog posting when comparing European and US Level 3 and Verizon peering utilization, the former ~18%, and the latter ~100% "(where Verizon sells broadband to its customers and sees Level 3 and online video providers such as Netflix as competitors to its own CDN and pay TV businesses)"

fixermark2• 12 years ago

I'm not a lawyer, but when you put it that way, it almost screams "RICO violation," doesn't it? ;)

induscreep• 12 years ago

Verizon probably needs to pay Level 3 for the bandwidth - so apart from the cost of the hardware (cards + cables) there is a possibly large cost which is not mentioned in the Level 3 article.

bitJericho3• 12 years ago

Netflix has offered to directly connect to Verizon without the need for Level3 to be in the middle.

ktsmith5• 12 years ago

Verizon probably doesn't need to pay Level 3 for the bandwidth. Typically peering points like this are settlement free. In other words it's a voluntary exchange of traffic and each side of the line keeps 100% of the revenue from its customers using the interchange.

higherpurpose2• 12 years ago

They just want the same deal as Comcast, that Netflix, in a moment of stupidity, made:

http://knowmore.washingtonpost.com/2014/04/25/this-hilarious...

xxchan• 12 years ago

It's not stupid at all. By giving into the racket, Netflix demonstrated that it's not a technical issue at all, so that argument no longer flies. It's clear that the ISPs are capable of supporting the service, they just don't want to.

msandford28• 12 years ago

Yes, it absolutely is a simple fix.

This is the increasingly common case of an ISP wanting to have it's cake and eat it too. It's an analogy of course, but it's a good one.

Verizon has either a monopoly or a near monopoly on the last mile from POPs to it's customers. In other words customers have no choice or little choice but Verizon.

And these customers pay Verizon for some amount of bandwidth and expect Verizon to make a good faith effort to deliver that bandwidth. The Terms & Conditions always say "up to X Mbps" because Verizon can't control network congestion once the data is off of Verizon's network. It might be that once the data leaves their network it has to go to Australia which has notoriously poor internet as a result of being kind-of out of the way. Or maybe it heads to Africa and starts making cell tower hops in which case it's going to be abysmal.

But in this case Verizon isn't making a good-faith effort to deliver the bandwidth that their customers are paying for to them. What they want is to extract extra money from someone who also provides a service to their customers, namely Netflix or Level3. Why do they want to do this? It's hard to say for sure. Trying to make extra profits, trying to keep Netflix slow and encourage people to subscribe to Verizon cable TV, spite? We can't know.

But what is clear is that this isn't the fault of Netflix or Level3. Level3 has said they will upgrade their side of the connection for free which would reduce congestion and Netflix has offered to locate servers on the Verizon network for free, which would reduce the Verizon/Level3 network congestion.

But Verizon refuses to accept either of those offers. Why? I'll venture a guess: it's not because they have the best interests of their customers in mind. If they did, they would have upgraded already.

To me it seems like they're abusing monopoly(ish) power.

Zikes5• 12 years ago

Level3 has said they will upgrade their side of the connection for free which would reduce congestion and Netflix has offered to locate servers on the Verizon network for free, which would reduce the Verizon/Level3 network congestion.

They're not necessarily doing this "for free", but as a part of the service their customers have already bought and paid for. It's not doing anything extra, it's normal operations, it's keeping the promise the business made to the customer, which is the opposite of what Verizon is doing in this case.

digikata3• 12 years ago

I wonder if Verizon's customers could form a class action lawsuit accusing it of failing in good faith to deliver bandwidth from Netflix due to intentionally congested handover points.

yxhuvud3• 12 years ago

This is not and has never been a technical problem. It is purely in the business and politics areas.

One way to see the conflict is that it is about what status verizon have - are they considered an end user isp or is it a transit provider? L3 is in the right if they should be considered the former, and Verizon is right if they should be the latter.

bitJericho3• 12 years ago

"If that’s the case, we’ll buy one for them. Maybe they can’t afford the small piece of cable between our two ports. If that’s the case, we’ll provide it. Heck, we’ll even install it."

That's what I would call a mortal wound to Verizon's arguments.

ohashi2• 12 years ago

I love this. All of what I presume was happening behind closed doors is finally getting put into the light. If only there were serious options for consumers to act on based on the information that is coming out publicly.

druidsbane• 12 years ago

L3 as a consumer broadband provider. Cut out all the other middle-men.

sheetjs2• 12 years ago

Clever technique to draw attention away from what is most likely a contractual dispute.

Verizon's argument is clear:

Netflix chose to attempt to deliver that traffic to Verizon through a few third-party transit providers with limited capacity over connections specifically to be used only for balanced traffic flows

Level3 hasn't refuted this point (the imbalance of the traffic flow). If the traffic is not balanced, the contract between Verizon and level3 probably has clauses requiring Level3 to pay Verizon for the imbalance (in this case, since Level3 is pushing more data, they would be the ones paying). That is the real problem, most likely, and the technical argument merely serves to confuse

chimeracoder2• 12 years ago

(in this case, since Level3 is pushing more data, they would be the ones paying).

That's the way the Internet works - since consumers are generally prohibited from running servers (and even if they did, they wouldn't operate at the same scale), traffic from consumers is always going to be orders of magnitude less than traffic to consumers.

All of the data that Verizon is receiving from Level 3 is in response to a request that a Verizon customer has issued. Because the size of a request is much smaller than the size of the payload (response), the traffic will always have this pattern.

Do you have any evidence supporting that Level 3's contracts with Verizon require them to pay? This is the first I've ever heard that implied, and it would be very surprising, since it would be incredibly foolish on Level 3's part (they know that the traffic will always be shaped like this).

msandford28• 12 years ago

in this case, since Level3 is pushing more data, they would be the ones paying

No. That's not how the internet works. Nobody PUSHES data. People PULL data.

Verizon has customers who pay for internet access. Those customers make REQUESTS to Netflix for data and Netflix RESPONDS with data.

They happen to be streaming movies which use a lot of bandwidth, but on average it's about 3Mbps per concurrent stream. That is WELL below the 25/3 or 50/5 or 100/10 that Verizon advertises for purchase.

If someone were pushing data it would be called a DoS or DDoS. An attack is when someone sends unrequested data to try and break your network.

But this data isn't unrequested. Verizon's customers have requested it from Netflix as they are within their rights to do since they have literally paid for it.

This is Verizon wanting to bill their customers once and then their customers' vendors as well. Double billing for a single service is a neat trick if you can pull it off. But it tends not to engender goodwill.

epistasis2• 12 years ago

Netflix chose to attempt to deliver that traffic to Verizon through a few third-party transit providers with limited capacity over connections specifically to be used only for balanced traffic flows

Verizon's argument would be a lot stronger if they didn't attempt to say that Netflix chose balanced flow contracts. They are saying that Netflix went shopping for backbone providers and chose the ones that had balanced traffic flow agreements to deliver video across, which is just silly.

Such rhetorical overreaching is a common debating error, and can ruin an otherwise good argument. Verizon may or may not have balanced flow agreements, I don't know what the contracts are, but now I don't trust Verizon to honestly report on them based on their imputation of Netflix's motivations.

chiph• 12 years ago

This is the exact argument that Verizon is using -- that Level3 is sending "too much" data. Of course, that data is in response to requests sent from Verizon's residential and business customers. So it's not unwanted -- in fact it's the data that their customers are paying Verizon to deliver to them.

smsm424• 12 years ago

I'm not sure why Level3 has to pay anything. Let's imagine there's only 4 entities in the world - User, Verizon, Level3 and Netflix. User buys internet access from Verizon and pays. Netflix hosts servers at Level 3. Level 3 has connection to Verizon. User pays Verizon to be able to access Netflix servers. Netflix pays Level 3 to deliver content up to Verizon. Why when the data crosses between Level3 and Verizon Level3 should pay anything? Verizon has been already paid by the user to deliver the data. That's like a taxi driver demanding payment from the restaurant since he brings them clients. Clients already paid for being brought to the restaurant, that's the whole reason they hired taxi in the first place, and demanding payment from the restaurant sounds like a racket to me. And a fraud towards the users which expect the provider to perform their obligations, not to use them as bargaining chip to extort other businesses.

cortesoft• 12 years ago

Balance-flow contracts are only used when each network is acting as a gateway where the end-point of the traffic is NOT IN either of the parties involved in the contract.

For example, suppose there are 5 separate networks, run by different groups: A, B, C, D, and E. C connects to A,B and D, and D connects to C and E. In this case, the only way networks A and B can connect to D or E is through the connection between C and D.

Now, if in this arrangement, the D or E networks are sending/requesting a lot of traffic to/from networks A and B, while C is rarely requesting data from D or E, then network C might have a fair argument to make that D is abusing their balanced-flow peering agreement. Basically, they are being asked to be a middle-man for all the traffic going between networks that are NOT theirs. They might ask for compensation for doing this job (carrying traffic between two other networks).

However, this is NOT what Verizon does. All of the traffic between level 3 and the Verizon network is bound FOR A VERIZON CUSTOMER. They are not a middle-man, they are the end destination. Now, if Level-3 is sending data over those congested level-3/verizon gateways that is bound for, say, AT&T's customers, then they would have an issue and could fairly demand payment.

That is not the case, however.

malchow3• 12 years ago

@sheetjs: I'm sorry, but I simply cannot follow the logic in the Verizon argument that you quote. You quote:

"Netflix chose to attempt to deliver that traffic to Verizon through a few third-party transit providers with limited capacity over connections specifically to be used only for balanced traffic flows"

But isn't that a misstatement of fact? CDNs like Level3 do nothing but transfer data, usually media files, onto consumer ISP networks when those ISP's customers' request them.

In what dream world are Akamai and Level3 receiving as much data from Verizon's consumer network as they are sending? Why would any ISP expect a link with a CDN to have a balanced traffic flow?

kenrikm• 12 years ago

The Verizon users are REQUESTING the data, they pay Verizon to provide it, it's Verizon's job to make sure the users have access to the "Internet"

Verizon looks at it as if Netflix and Level 3 are pushing the Data, everyone else clearly sees that they are pulling it.

kevin_thibedeau• 12 years ago

It doesn't matter if there is an imbalance in the traffic flow. Verizon is being paid to deliver requested packets over IP. That's all they have to do. Singling out Netflix is pointless. If Netflix were replaced with 1000 video streaming sites with the same aggregate traffic volume who would they blame for all the data their customers are pulling in? Verizon desperately wants to avoid being a humble ISP with no way to pull the strings the way do with their telco service. That's why they're playing this game.

pdonis• 12 years ago

Clever technique to draw attention away from what is most likely a contractual dispute.

Oh, please. If that's really what's stopping this from being fixed, then I would be hard pressed to find a better illustration of the need for regulating "last mile" providers as public utilities. We're talking about an investment that is not even rounding error in Verizon's accounting, to connect Level 3's network to a Verizon network that, by Verizon's own admission, has plenty of excess capacity that is not being used. So the whole idea that, if the traffic isn't balanced, Verizon is somehow incurring huge extra costs that they need to be compensated for, is obviously bogus, which means their contractual terms are nothing less than an attempt to use their privileged position to extract huge monopoly rents in exchange for no added value whatsoever.

jwecker• 12 years ago

In my (admittedly limited) experience with peering I've never seen a contract. Lots of ad-hoc verbal agreements, and occasionally emails outlining a mutual understanding, but because it so often is obviously beneficial to both parties it is almost always handled in a casual way and decisions are deferred to network engineers for engineering considerations. I've never seen legal get involved. If Verizon brought it to a point where it is a contract dispute then it would be just as much of an indictment- it doesn't change the substance of the post at all.

einrealist• 12 years ago

That would be the case when Level3 would like to peer with Verizon on order to send traffic to another peering party's WAN of Verizon.

But it is Verizon customers, who request the traffic and Verizon is not providing the best quality, because they don't want to upgrade their own equipment and the peering with e.g. Level3. In fact, it should be Verizon paying to Level3.

Yver• 12 years ago

There's too much confusion, you're right. What Netflix need is some hard numbers. I suggest they take very detailed statistics at the client level and constantly report them back to their servers, in real time. With all that data available, it will surely ~~artificially balance the network usage and solve the legal aspect~~ allow both parties to make informed decisions.

phkahler2• 12 years ago

I've always been suspect of cable and wireless internet connections. Anything that shares bandwidth with all subscribers is going to get maxed out if a large number of customers start streaming video. Is it possible that Verison (and Comcast and etc...) know that opening up the gate to the outside world will crush their network under the increased load? I like the milk-our-monopoly-on-the-last-mile theory as much as the next guy, but is it even possible for the technology used to deliver the necessary bandwidth if they open up the pipe?

shimshim4• 12 years ago

Even so, as a paying customer I would expect them to calculate for this and know how to scale.

phkahler2• 12 years ago

My point is was that spectrum doesn't scale. There's X gigabits per second available over the air, and a similar X over a cable. For cable companies they can push equipment closer to the home and have fewer subscribers on the same line. I suppose Verizon could do the same with more towers, but imagine doubling the number of towers to support double the bandwidth (or something like that). As people have pointed out though, the congestion is specific to connections that carry netflix.

ktsmith5• 12 years ago

Oversubscribing lines has been one of the ways to keep ISPs profitable for years. Streaming video absolutely cuts into the ability to do this of course but I doubt it's the primary issue here. Verizon has every incentive to not add more bandwidth to their peering points with Level 3 and try to get more money from Netflix directly.

roc• 12 years ago

Net Neutrality isn't about usage, it's about destinations.

Even with Net Neutrality it's fully within an ISPs right to say X GB/mo for $Y and throttle or block over-use. The problem is an ISP deciding that, regardless of what is going on in the network, regardless of what a particular user has used or is trying to use, traffic to Netflix will be slower than traffic to Vudu.

Verizon, Comcast, et al have plenty of available tools to prevent congestion without violating Net Neutrality.

They're intentionally trying to confuse the issue with usage, because it's an easier PR sell.

jakejake• 12 years ago

Verizon could have legitimately argued artificially throttling at the Level 3 connection is necessary to protect their network. But they didn't say that at all. They basically bragged that their own network is running well under capacity. The insinuation is that Verizon is ready to handle all the bandwidth that anybody can send, but that Level 3 was over capacity.

But, assuming they really did need to throttle traffic in order to protect their own network. For one it wouldn't make sense to control your throttling by running your routers at 100% capacity. But also any throttling should be equal among all of their peers and not selectively chosen based on Verizon's business interests.

kristopolous• 12 years ago

nope. if you look carefully, the math doesn't work out. verizon will be fine. They want to charge netflix and have people pay for tv programming. The current legal way to do this is to claim they don't have the capacity (between what and where is unspecified, and by refusing to add it - even when it's 5 minutes and free).

mreiland2• 12 years ago

How could it possibly crush their network? They can simply charge more from their customers, and build out their network.

By definition, the only way they can be 'crushed' is if they're not using the funds they get from their customers for maintenance of their network.

bcRIPster2• 12 years ago

Let's not forget that those providers that Netflix chooses to use are global edge service providers for high bandwidth streaming content services. Cogent, Level 3, Limelight, etc... Their whole business is providing high capacity back-end network transit which they do well, otherwise they wouldn't be in business for very long. High-five Level 3 for calling B.S. on this!

MichaelGG6• 12 years ago

Funny you'd mention Cogent and L3 together, seeing as L3 has done worse to Cogent (depeering) for the exact same reason Verizon is using (unbalanced traffic).

bcRIPster2• 12 years ago

Oh I didn't say they weren't bastards. That's a whole other discussion.

mikecb14• 12 years ago

Is it just me or is this not at all inconsistent with Verizon's arguments? They've been saying the whole time that this link is meant to be balanced with roughly equal upstream and downstream, and they showed it was not. What am I missing?

Crito2• 12 years ago

Residential ISPs have pretty much never, in the history of ever, not had asymmetrical traffic. Netflix may have aggravated that, but they did not change that.

RKearney• 12 years ago

What made me laugh the most was when Comcast posted about a peering dispute with Level3[0]

Q8: Comcast says that Level 3 sends it 5 times the traffic that Comcast sends Level 3. Is that true? If it is true, why shouldn't Level 3 pay for the traffic it sends to Comcast?

Comcast goes on to argue that Level3 pushes 5X more data onto the Comcast network than Comcast pushes onto the Level3 network. What I find ridiculous about the whole claim is that every single one of Comcast's residential internet packages has 5X higher download speeds than upload speed. (e.g. 105Mbps/20Mbps, 50Mbps/10Mbps, 20Mbps/4Mbps)

Comcast provides internet speeds that force customers to download 5x faster than they can upload and somehow Comcast is surprised that their customers are pulling down 5X more data then they're pushing to Level3?

[0] http://corporate.comcast.com/comcast-voices/20-qs-with-accur...

mikecb14• 12 years ago

And I don't know how Verizon's backbone is laid out. But they have transit and end-user traffic.

Regardless, if Level3 keeps insisting this is a peering point, and then demonstrating that it's more like a transit to ISP link. Which ISPs pay for.

msandford28• 12 years ago

You've missed the part where they're trying to double-bill for the same network traffic.

Verizon wants to bill it's customers for the bandwidth that they use. And then for some of that traffic, if it's from Netflix, they also want to bill Netflix.

Verizon is the company which is selling asymmetric internet plans so it should come as no surprise to Verizon that their customers are requesting more data than they're sending.

But they use this asymmetry to suggest that what's happening between Verizon and Level3 isn't business as usual (which it is) and that they should be able to double bill.

Verizon is doing a decent job of spinning but if you really look at what's going on it becomes clear that they're really torturing definitions to make their arguments.

mikecb14• 12 years ago

I agree to an extent. But there's a point at which Level3 stops becoming a peer and starts becoming a subscriber. That point is somewhere between equal utilization of Verizon's peering point and (near) total upstream into Verizon's network, just as any Verizon subscriber would have. Where is that point?

I'm a Verizon subscriber and I get apoplectic when Netflix stutters, because I know exactly what's happening. But I also don't know what's the exact right model for peering points, and I think net settlement might be a better model.

Zmetta• 12 years ago

I agree, the fact that they are selling asymmetric internet plans should show that they're clearly aware that the average internet user pulls down far more data than they upload, but (according the Verizon) Netflix's asymmetric traffic is a complete surprise and a burden so Netflix should also pay Verizon for the additional asymmetric traffic being requested by Verizon's customers (who already paid for it).

wmf11• 12 years ago

Following Verizon's argument to its logical conclusion, if the link is unbalanced then Verizon needs to buy transit from Level 3 to make up the difference.

mikecb14• 12 years ago

Surely someone should. Level3 is getting squeezed between its customer (Netflix) and its customer's customers.

jimktrains26• 12 years ago

Thinking only of Verizon the consumer ISP, in what universe are consumers creating as much bandwidth as they consume. Upload speeds for consumer internet are what? 20% 30% the download speed. Verizon the consumer ISP would never generate as much data as they pull.

arg012• 12 years ago

Netflix did offer to have their client generate random upstream traffic to balance the load. Which netflix says was met with silence in the meeting where they proposed it. It has some amusing implications where netflix saturates verizon's network to pay less.

trjordan• 12 years ago

The claim from Verizon that still stands after this blog post is the fairness argument to peering. If you peer between symmetric networks, it's typically free. That's not the case here, so Verizon wants to be paid for the asymmetry.

I actually don't know. Is that legitimate?

bhouston• 12 years ago

Consumer internet providers will always have assymmetry because people generally are consumers of content, not creators or distributors. Remember that standard home internet connections have assymmetric download/upload speeds and also ban consumers from running hosting via their consumer grade internet connection.

davej• 12 years ago

No, it's not. Their customers are paying Verizon money for access to the internet, their customers are accessing Netflix.

wil4214• 12 years ago

Level 3 is a tier 1 provider so in theory they should be peering for "free" with other Tier 1 providers such as Verizon.

From Wikipedia:

"However, the most common definition of a tier 1 network is a network that can reach every other network on the Internet without purchasing IP transit or paying settlements."

http://en.wikipedia.org/wiki/Tier_1_network

masklinn• 12 years ago

Is that legitimate?

Not really, for the consumer ISP side of its business Verizon absorbs more traffic than it produces by definition: consumers don't typically produce much data outside of p2p.

Furthermore, Netflix has offered putting server inside Verizon's network which makes this argument moot.

VZ just wants to double-bill.

rhino3694• 12 years ago

The issue is that Verizon is both a transit provider and a last mile ISP.

For transit providers, asymmetry = Paid.

For local ISPs, they are typically willing to asymmetrically peer because they'd otherwise have to pay transit.

Since verizon is a T1 transit provider, it doesn't want to give it's service away for free.

I think the fair answer in this dispute depends on whether L3 and Cogent are forcing Verizon to provide transit for free or are they just delivering the data the local network for the customer (like a local ISP peering arrangement).

I'm not sure which is going on. If L3 is just dumping Netflix packets on the nearest Verizon peering location, well then Verizon is providing them a transit service. If L3 carries the data all the way to the local network, then it's not transit at all.

I'm not sure what is actually happening. I'd imagine a mix of both.

allenlavoie• 12 years ago

One solution would be for Netflix to modify their clients to send useless and quickly discarded UDP packets back to balance out the traffic. Verizon then has tons of data to push back to Level3 and everyone's happy.

mreiland2• 12 years ago

oh man... the evil in me laughs with glee at this.

But honestly, it couldn't be done. Not with the upload restrictions at the end of the pipe.

ktsmith5• 12 years ago

That wouldn't work though. Verizon sells unbalanced services with larger download capacity than upload capacity. So it's simply not likely/possible that their customers can send matching data for what they are requesting. Additionally, while this might not negatively affect their FIOS customers their DSL customers are on asynchronous connections meaning that the uploaded data will negatively impact their ability to receive data at the same time in effect degrading their ability to stream and the quality of the video they can receive.

voxic11• 12 years ago

Shouldn't it be Verizon paying level 3 to provide the access Verizon customers are paying for?

wil4214• 12 years ago

It doesnt matter what content is coming down the wire, if I pay for internet I expect to be able to access or connect to anything thats not forbidden in the EULA.

With comcast my data cap is 300GB a month, I should be able to use 300GB of whatever I want Youtube, Netflix, hell even torrents or file sharing. Doesnt matter where it comes from I'm paying for 300GB of data at X speed. How hard is that to get?

MichaelGG6• 12 years ago

Cool! Now tell Comcast you want to download 300GB from my server, which I've got connected on a single 300 baud channel. Will they come give me a free 1GE connection?

azernik• 12 years ago

Nope.

Netflix, on the other hand, has paid good money for a very fast and reliable connection all the way to the Level3/Comcast PoP.

For yet another analogy in this thread - if your server was instead sitting in a data center with an open 10GE ethernet port or two, right next to a Comcast router, yes, they should just install the extra NICs and wire the things up without for the price of supplies and installation.

wil4214• 12 years ago

It doesnt work like that because Netflix has already paid level 3 who does most of the heavy lifting. Similar to if you had a 10GE connection at your sever but comcast is still running a 300 baud channel on the connection they have to your server.

eyeareque• 12 years ago

He makes great points. But this is what the verizon marketing team is paid to repond to. They'll come up with something, just wait for it.

staunch2• 12 years ago

This is the entire net neutrality debate in one simple blog post. Verizon is selling internet access to their users but not actually providing real access!

troels2• 12 years ago

One question: There aren't that many large edge providers around. What would Verizon do if Netflix signed a contract with a dozen of the larger ones and routed their streams through whichever had the best connectivity for a given endpoint? If Verizon throttle all of them, they are going to get a lot of collateral - essentially degrading the Internet at large for their customers. Or would they just do packet level filtering then?

josho• 12 years ago

I've wondered this as well. I suspect that it would be cost prohibitive for Netflix to have a contract with several edge providers like Level 3. I'm certain that Verizon's price demands to Netflix would be cheaper than this option.

Think about your own ISP. I pay $80 for service, but to get the same bandwidth shared between 2 ISPs I'd have to pay $120 plus now I'm doubling my hardware costs (routers to each ISP).

wmf11• 12 years ago

I think the costs work out differently at terabit scale. Sending a terabit split across two backbones may be the same cost or cheaper than sending it over a single backbone.

troels2• 12 years ago

True. But I would assume that the contract is largely a pay per usage or at least expected usage. So if they just make an estimate of how many Verizon customers they have and buy that block from someone else than Level 3. Then route all that traffic through the other provider, while keep using Level 3 for the majority of their traffic.

Or perhaps they could ask Level 3 (who are clearly pissed at Verizon) to make the deal for them. These edge providers already have business with each other (peering), so it should be relatively simple for Level 3 to purchase some bandwidth from another large provider and then route some of their packets (e.g. the Netflix ones) through there.

In any case, the edge providers have good reason to collaborate on thwarting Verizon on this, so why don't they?

gcb0• 12 years ago

i recently gave up my netflix and amazon prime because i was tired of streaming hanging up in the middle of an episode/movie. it was just annoying and embarrassing if i had guests.

now i just torrent everything so i can watch uninterrupted when i want. I consider i am still legal since i'm paying for both subscriptions yet and they pay the content. if not, not my problem.

but that raises a question... why the RIAA/MPAA is not using its bullying powers to harass ISP like verizon?

as i just showed, they are the reason lots of people are going back to torrents that do not make money to the studios like streaming services does. Why do they rather sue john does instead?

FireBeyond• 12 years ago

"but that raises a question... why the RIAA/MPAA is not using its bullying powers to harass ISP like verizon?"

Because it presupposes that whether or not in fact, streaming via Netflix is profitable or not, the studios didn't have to be dragged kicking and screaming to allow it at all.

ninv• 12 years ago

Greed! It would not stop here. Today it is Netflix and youtube, tomorrow it will be image heavy sites. Then they will go after google, mail provider and bank sites etc.

ajmurmann• 12 years ago

The Internet is a crucial part of our infrastructure and a important backbone of our economy. If there is any indication that someone like Verizon in this case is tempering with it intentionally, they should be charged with terrorism. This is no different than someone going out and sabotaging Hoover damn on purpose. In fact it's worth!

BryantD2• 12 years ago

Point of information about Verizon as a purely residential carrier: Verizon owns MCI/Worldcom and Terremark. They're not just a residential carrier. I don't have any idea what the ratio of residential to business traffic is, of course, but I think Verizon is a real Tier 1 carrier. Happy to have someone with more definite data chime in if I'm wrong.

msandford28• 12 years ago

They are but the residential and commercial sides of the business are VERY different: https://news.ycombinator.com/item?id=8050089

BryantD2• 12 years ago

Do they use different backbones? Which side of the business runs the interconnects?

adamman• 12 years ago

Doesn't Verizon realize they lose customers for crap like this? My phone contract with Verizon is up in November. I can't wait to be done with them.

revelation• 12 years ago

Theres a common theme in this discussion. We're stuck discussing analogies. Instead, we should be reasoning from "first principles" as Musk suggests.

If we do that, we find that physical link speeds of any variety have improved by many many magnitudes over. As this statement suggests, 10GBit is absolute jellybean in the networking world. Yet, last mile bandwidth available to customers has stagnated. Imagine a perfectly paved autobahn, and here we are discussing if we should be able to drive 20 mph on it, and who should pay for the extra damage that will cause to the surface. Incredulous.

There are some regulatory systems, where in a goldilockes moment, these basic truths are recognized and acted upon. Consider the EU drastically cutting roaming charges. What was so blatantly obvious that even the senile, crooked decision makers at the EU couldn't miss it? All the providers already roam over the internet, at basically zero cost for both parties if you consider that speeds are limited by the over-the-air interface at either end. Yet they kept charging customers the price for a specially erected microwave link from 1960.

bitJericho3• 12 years ago

I'm pretty sure that was an analogy! (A good one though)

knodi• 12 years ago

If only we have a government body that did something about stuff like this... o wait FCC... o wait :(

abalone3• 12 years ago

Summary: The bottleneck is the the interconnection point. Netflix is saturating the interconnection point, and Verizon wants them or their transit provider to pay more to "ensure a level of capacity that accommodates their volume of traffic".

But Level 3 claims it would be very inexpensive for Verizon to increase capacity there. Just a few port cards and a few thousand bucks for a major city like L.A. They even offered to pay Verizon's costs.

msandford28• 12 years ago

Nope, Verizon's customers are causing the interconnection point to be saturated. Netflix doesn't send data unless Netflix's customers request it. If they sent data without requests Verizon could (and probably would) interpret that as some kind of a network attack.

This is Verizon wanting to double-bill for traffic. Plain and simple.

Verizon wants them or their transit provider to pay more to "ensure a level of capacity that accommodates their volume of traffic".

That's Verizon's job. To go out and get by any and all means necessary enough bandwidth to satisfy their customers, namely the residential customers who seem intent on using Netflix. At no point do they have a moral imperative to hold their customers hostage to extort money out of someone else that their customers are trying to access. It's already been paid for!

res0nat0r9• 12 years ago

Nope, Verizon's customers are causing the interconnection point to be saturated. Netflix doesn't send data unless Netflix's customers request it. If they sent data without requests Verizon could (and probably would) interpret that as some kind of a network attack.

All of your comments keep ignoring or misunderstanding what peering is. It is supposed to be roughly equal data traffic for both sides. Once it is lopsided in one direction, it is no longer peering.

It doesn't matter who requested what, the fact that L3 traffic is consuming a majority of the bandwidth now is the issue, and Verizon wants L3 to pay to fix that.

abalone3• 12 years ago

What's up with the downvote? Is that not a fair summary?

The haggling over semantics here doesn't really change the substance. Ok, "Netflix" isn't saturating the interconnection.. "customers using Netflix" are saturating the interconnection. And it would still reportedly cost Verizon very little to fix it.

I think that points the finger pretty strongly at Verizon.

drderidder• 12 years ago

How did people think this wouldn't happen when ISP's are allowed to be content providers, too. Of course Verizon wants to choke Netflix and promote their own service instead. Letting cable and content providers be ISPs was always a bad idea.

macspoofing2• 12 years ago

Great post. Frankly Level3 should be pissed off. Verizon essentially called them out for providing crap service when in fact they are the ones responsible.

slyrus• 12 years ago

More like an accidental "Fuck you" than an accidental "mea culpa".

brownbat• 12 years ago

What would happen if Netflix stopped allowing access to its service to Verizon customers in areas where there's another provider? Netflix could argue that it doesn't want to provide an inferior product if there's a choice, and maybe leave a number on the homepage for the cable company in your area.

I wonder if Netflix or Verizon would lose more customers...

wpietri4• 12 years ago

Then a) Netflix becomes the bad guy, b) they lose money from those customers leaving, and c) they no longer have the opportunity to influence those people.

They should continue to allow access, but they should also keep saying when the problem is with the network provider. Customers love Netflix (NPS of 54) more than Verizon (NPS of 32), so they'll be more inclined to believe Netflix.

ck2• 12 years ago

This congestion only takes place between Verizon and network providers chosen by Netflix.

The providers that Netflix does not use do not experience the same problem.

This needs to be put in front of the FCC since it is the bare minimum language they should be able to understand.

exabrial• 12 years ago

I think a natural solution to this is to have Netflix charge all Verizon customers a $5/mo surcharge to pay for 4 more 10gbps ports.

But after reading this... am I the only one that thinks peering agreements are done bass ackwards? If Verizon's customers are _requesting_ Netflix access, shouldn't Verizon pay Level3, not the other way around?

Verizon's accidental mea culpa

Related Stories