Netflix Open Connect Content Delivery Network

https://www.netflix.com/openconnect

by nkurz • 12 years ago

99 74 12 years ago

netflix.com

mey• 12 years ago

https://www.netflix.com/openconnect/hardware https://www.netflix.com/openconnect/software

It looks like they are heavily redundant boxes with the goal of reducing field maintenance, running FreeBSD as the OS.

Alupis11• 12 years ago

running FreeBSD as the OS.

Presumably for ZFS

howeyc2• 12 years ago

Actually it turns out they don't really have a need for most of the features of ZFS. So they don't use it.

http://www.youtube.com/watch?v=FL5U4wr86L4

Duckpaddle2• 12 years ago

Very interesting to see what hard drives they "blessed".

Alupis11• 12 years ago

Google uses standard off-the-shelf sata drives too.

Too many disk failures to be worth spending money on the expensive "enterprise" drives when a standard SATA drive will due the job just fine (especially in arrays like this -- plenty of spindles to get performance)

danudey• 12 years ago

Backblaze also uses off-the-shelf HDDs (including, when HDDs were in short supply after the flood in Thailand, buying USB drives and tearing the case off). They publish a lot of reliability numbers on drives as well (which match up quite well with my own experiences), as well as cost/GB numbers.

nacs• 12 years ago

The hard drives used are listed here: https://www.netflix.com/openconnect/hardware

  Hitachi Deskstar 5K3000 3TB
  Seagate Barracuda 7200.14 3TB

ihsw• 12 years ago

The following system was developed and first deployed at the end of 2011.

This is very out of date. Their latest OpenConnect appliances use SSD drives, have higher capacity, and use less power. They're heavier, too.

blerk3• 12 years ago

For the operating system, we use FreeBSD version 9.0. This was selected for its balance of stability and features, a strong development community and staff expertise. ...or due to the less restrictive licensing, it's ok netflix everybody does it...

Alupis11• 12 years ago

FreeBSD is pretty standard for most storage servers, be it CDN's, file hosting, etc. When they say "BSD powers the internet", it's not really that much of an exaggeration.

wmf8• 12 years ago

AFAIK Netflix contributes back all their changes to FreeBSD, so I don't see how the license matters.

My impression is that companies that don't like the GPL either deal with it (Android) or just violate it (almost all embedded devices). Nobody is actually willing to switch to BSD for licensing reasons.

blerk3• 12 years ago

Sony switched over from Linux (ps3) to kfreebsd (ps4) due to licensing. Sure they (netflix) contribute back to FreeBSD, I don't doubt it, but don't act like licensing doesn't have anything to do with it.

justincormack3• 12 years ago

They did mention licensing as a benefit at the talk I went to.

curun1r• 12 years ago

The one thing missing from all the articles about OpenConnect is any way for Netflix subscribers to see which ISPs are participating.

I'd like to know if my ISP participates and, if not, be able to call in to suggest they do. If they still don't join in a reasonable time frame, then I'd at least have the option to switch providers to someone that participates.

A program like this will be more successful if they allow for consumers to pressure ISPs to participate.

Alupis11• 12 years ago

Unfortunately, as we've discovered from the whole net neutrality debate -- netflix has offered these boxes for free to all major ISP's... yet the Big5 ISP's have all turned them down.

Thaxll3• 12 years ago

Yes because those box aren't sufficient for 1M+ sub. You're better off using peering rather than cache boxes in your network if you're a large ISP.

FireBeyond• 12 years ago

They at least used to have something on their SuperHD page that would tell you whether your ISP participated ("Sorry, you can't view SuperHD content because your ISP has not opted in to our OpenConnect program"), but I think since they opened that up, there's no obvious list.

filmgirlcw• 12 years ago

The only major ISP I know that definitively is part of OpenConnect is Optimum (Cablevision). That's why they have the fastest Netflix speeds (only after Google Fiber, and even then it is close) on the ISP Index Netflix publishes each month.

As an Optimum subscriber, I can say there is a massive difference between the Netflix experience at my apartment in Brooklyn compared to Netflix at my office in Manhattan.

rev_bird• 12 years ago

The Security Now podcast had an indie ISP owner on the show back in June (10, I believe). Most of the show was Leo Laporte calling him as asshat, but one thing he said that was compelling was that his customers require Netflix -- if he doesn't give them Netflix, they don't sign up. The bandwidth was killing him, so he told Netflix he wanted the OpenConnect, and that he'd pay for installation and maintenance. They told him no.

wmf8• 12 years ago

Brett Glass is a bit of a crank, though. His ISP is unviable but rather than just going out of business honestly he's looking for others to subsidize it. It was discussed extensively on NANOG: http://mailman.nanog.org/pipermail/nanog/2014-July/068290.ht...

mmalone6• 12 years ago

If you're a net neutrality advocate you should seriously consider whether this sort of peering arrangement should be tolerated. This kind of peering is a de-facto QoS arrangement that gives Netflix a performance and cost advantage over other video providers that aren't big enough or rich enough to justify and pay for direct peering with ISPs. It also gives bigger ISPs (that do more than 2Gb/s of traffic to Netflix and agree to peer, apparently) the same advantage over their smaller competitors. Isn't that exactly what we're trying to avoid with net neutrality?

This is not a new practice and I'm surprised it doesn't come up more frequently in the net neutrality debate. It's been going on for years -- broadcast.com did it, YouTube and Google do it, and I'm sure others do as well. We've never really had net neutrality -- peering agreements have been happening forever in shady back room deals at NANOG. At least the more recent shenanigans are happening out in the open.

Alupis11• 12 years ago

I think you miss the point of open connect.

It's no different than if Netflix was paying Comcast (or whover ISP) to colo a server(s) in their data centers (you can do this as a normal citizen too).

The only difference, the idea is Comcast wouldn't charge Netflix for colo space since Netflix is "earning" it via cost reduction for the ISP. So they are still technically paying for the colo space.

Perhaps I missed your point though -- because this seems to be a much better way to a neutral net than having ISP's charging both sides of the pipe (which the small guys can't afford even more)

mmalone6• 12 years ago

Unless I'm reading this incorrectly this is a peering arrangement, which is qualitatively different than just hosting a box in a DC run by Comcast (which doesn't exist, as far as I know, so I can't do it as a normal citizen). I could colo a box "near" Comcast, but I'll be more hops away (worse performance) and I'll have to purchase transit from some ISP that does peer with Comcast (more expensive).

tzs• 12 years ago

Is colo space at the ISP an unlimited resource? If not, then it will end up priced out of the reach of the small guys.

I suppose the ISP could offer some kind of "virtual colo", where instead of putting your servers in the ISP data center for fast access they offer some kind of faster connection to your data center. This would have two big advantages over physical colo. First, it would not have the space limitations so it could be more widely available. Second, the price could be based on how many customers you are serving at the ISP so that even small content providers can afford it.

This now sounds similar to "fast lanes", which have been the major focus of opposition to the proposed approach to net neutrality of trying to regulate under section 706.

The problem I see is that in both the company specific CDN approach and the "fast lane" approach it comes down to a content provider is making an arrangement with an ISP to make that content provider's content better for its customers on that ISP.

Is that against net neutrality? If the answer is "yes", then I don't see why it should matter if it is done via physical boxes placed on the ISP's network or by prioritizing traffic from that content provider or by purchasing a higher capacity interconnect to the backbone providers that content provider's traffic comes through. This seems to me to be one of those things where it should be looked at as a black box, and net neutrality should be concerned with the externally visible behavior of the box, not the mechanisms within the box.

rhino369• 12 years ago

But charging some and not others is pretty clear discrimination. Colocation should definitely be allowed, but Netflix should pay the same rate a start up would.

mobiplayer• 12 years ago

What? No. This is the same situation as the company hosted in a DC that grows and prefers to pay for a 1Gbps connection than the previous 100MBps.

Net neutrality means treat all the traffic the same, doesn't matter where it comes from or what protocol it is. That's why "this" doesn't come up more often in the net neutrality debate, because it's got nothing to do with it.

JoshTriplett• 12 years ago

Alternatively, if you're a network neutrality advocate, you should consider whether this is really the kind of thing you're advocating against. It's one thing to complain about ISPs intentionally throttling or dropping traffic, or shaking down services to get paid twice for the same traffic. It's something else entirely to complain about services going out of their way to establish CDNs near their customers and reduce overall network load.

mmalone6• 12 years ago

Comcast is throttling / dropping traffic because they want Netflix to pay for bandwidth. You're right that they'd be getting paid twice for the same traffic, but ISPs already do this -- if Comcast and Netflix are connected via Level 3 they both have to pay Level 3 for the same traffic.

What if Comcast offered to host Open Connect devices for a fee -- would that be a violation of net neutrality? To me that just seems like the free market doing what free markets do (I'm ignoring for a moment that Comcast is basically a monopoly which is a legitimate problem).

What if Comcast throttled everyone _except_ Netflix. Is that really different than Comcast allowing _only_ Netflix to peer?

mmalone6• 12 years ago

Wanted to add a little bit more to my previous comment since I've been thinking about this for a while. The story I heard about broadcast was that they pursued this sort of peering arrangement with smaller ISPs so they didn't have to pay for transit -- basically to reduce their bandwidth bill. It was really a brilliant move on their part. Since then lots of other large bandwidth consumers have done the same.

I'm not against net neutrality but I do think the issue is more complicated than people make it out to be. Once you understand that this is common practice the position of Comcast, et al. does make more sense. If Netflix is saving a shit-ton of money and providing a better service to their customers by peering, and Comcast has less upside, why shouldn't Comcast charge Netflix for the privilege?

RyJones• 12 years ago

I worked at MSN in WAN-OPS in the 90s. We did exactly that: find out what ma-and-pa ISPs we were paying transit for, offer to run a T1 directly to them for all MSCOM/MSN traffic. Easiest sales call ever.

wmf8• 12 years ago

If Netflix is saving a shit-ton of money and providing a better service to their customers by peering, and Comcast has less upside, why shouldn't Comcast charge Netflix for the privilege?

I don't know if we can accurately estimate who's saving how much, and I definitely don't believe anything coming from Comcast. Given that there is no way to reach Comcast customers other than Comcast and both parties have an incentive to lie about their costs, I think zero is the fairest price for such peering.

rmrfrmrf2• 12 years ago

I am a net neutrality advocate in the sense that I'm against artificial traffic shaping. Netflix offering peering and cacheing solutions to offload massive amounts of traffic is not the same thing, and, in fact, is an attempt to prevent ISPs from shaping its traffic.

mseebach• 12 years ago

Net neutrality was NEVER about making all content equally available to every consumer regardless of geography and network topology. If you're European or American serve customers in Australia, you should put a server there at your own cost, or your Australian users will suffer, net neutrality or not.

Netflix OpenConnect is just doing that on a much larger scale.

13throwaway2• 12 years ago

Because Netflix uses something like 45% of North American internet traffic, they need to do something like this in order to efficiently deliver video. Of course this is going to give Netflix better ability to stream, but I don't think it is unfair.

It would be unfair if Netflix was paying to "speed up its traffic" (read as: slow down competitor's traffic).

condiment• 12 years ago

For all the hand-wringing about ISPs creating an "internet fast lane", OpenConnect sure seems like a step in that direction. If Netflix is allowed to colocate their server in an ISP's facilities (effectively eliminating the cost of their egress bandwidth), how does that not give them an unfair advantage against other internet video operators who lack the leverage necessary to broker this kind of deal?

OpenConnect isn't the first attempt at this sort of business model, either. Back in the 90s, Akami tried this same thing with their "FreeFlow ISP" service model - putting their servers in the ISP's point of presence in order to eliminate their bandwidth costs. Then, as now, major ISPs refused.

StavrosK• 12 years ago

Can someone explain what this is? I don't see that mentioned anywhere.

rmrfrmrf2• 12 years ago

I read that entire OpenConnect Deployment guide PDF, and I have to say that it looks like a really great, well-thought-out program. I'm impressed with how many moving parts there are and the time it must have taken to develop a system like this; they don't even touch on the details behind the Cloud Control Plane, and I'm sure that must have taken a while to develop. Netflix has said in the past that they only want the most skilled employees possible, and it certainly shows here.

swasheck• 12 years ago

it seems to me that calling this an "internet fast lane" is a false equivalency. my understanding was that the internet fast lane concept was something that users would pay more to use, which leaves the isps free to render the "regular lane" unusable so as to bump the pricing. additionally, there was the whole notion of the "bundles" of websites available at different cost tiering.

openconnect appears, at least to me, like it's a collaboration between providers of different services to enhance the value for their shared customers.

edit: directed at both mmalone and condiment

whoopdedo2• 12 years ago

What's to stop Netflix from later on asking the ISPs to pay for the cost of OpenConnect? The way that satellite TV networks charge a subscriber fee to the cable operators. Netflix has stated that they want to become like HBO. I don't imagine they're unfamiliar with how HBO become so successful.

wmf8• 12 years ago

What's to stop Netflix from later on asking the ISPs to pay for the cost of OpenConnect?

The fact that ISPs will unplug it five seconds later.

The way that satellite TV networks charge a subscriber fee to the cable operators.

HBO only charges each customer once (via the cable company). If Netflix tried to double-dip by charging the customer directly and also charging the ISP they'd get rightly flamed.

whoopdedo2• 12 years ago

The fact that ISPs will unplug it five seconds later.

At the risk of angering the many people who watch Netflix.

I don't get the double-dipping argument either. We already have that with television, newspapers, and radio being paid by advertisers yet still charging a subscription to viewers.

Touche• 12 years ago

They already are paying. These servers require power and space.

13throwaway2• 12 years ago

I wonder what measures they have in place to protect against someone cracking this open and dumping the entire video library.

nodata• 12 years ago

Funny name: is it open to other CDNs too?

rdl• 12 years ago

I wish I could buy these custom chassis; I didn't see a link in the article.

mbell• 12 years ago

I would think in low volume a supermicro chassis would be more cost effective, they have many 4U models that support 36+ 3.5" drives.

Netflix Open Connect Content Delivery Network

Related Stories