Netflix begins the deployment of their own content delivery network

http://www.zdnet.com/blog/datacenter/netflix-delivers-using-commodity-hardware-and-open-source-software/1409

by fendrak • 14 years ago

76 43 14 years ago

Cool, maybe they can deploy it at comcast and deliver content via the same magical content pipe comcast uses for their xfinity service.

mikeryan3• 14 years ago

Most of the large CDN's already deliver straight into Comcast's network. Last I heard Netflix was already paying Level3 to do this. Xfinity doesn't have a magic pipe, all the content and deliver is internal to the Comcast network so its its own "CDN".

eggnet• 14 years ago

It was a joke, the hint was the use of the word magic.

wmf5• 14 years ago

More info on Comcast's DiffServ magic: http://ber.gd/post/23025893856/comcast-traffic-prioritizatio...

seanp2k2• 14 years ago

I think Comcast would be pretty opposed to let a competing product into their not-for-sale DC space. The real benefit here is that it saves ISPs on peering charges. Comcast, AFAIK, does a lot of their own peering and has financial motivations to NOT let Netflix into their DCs (onDemand.)

We will see how this plays into the me neutrality battle...

squeed3• 14 years ago

Comcast does not operate their own video delivery infrastructure. They contract that out to Akamai. Furthermore, Comcast will happily sell non-transit, A.K.A. paid peering, to all who come. They don't (and never have) sell colocation (DC space). Presumably, Netflix is just going to buy access to the Comcast network, same as every other CDN has done.

http://www.comcast.com/dedicatedinternet/

cluda01• 14 years ago

Hopefully they should be able to use these cost savings to purchase more content and get it into their system. Does anyone have any informed speculation about how much they could potentially stand to save? I'm unfamiliar with how peering agreements work and or the cost structure.

biot• 14 years ago

It's interesting from a bit of a game theory standpoint. The optimum outcome might be achieved if Netflix hands over the appliance and the ISP installs and keeps it running at no cost to either party, except for maintenance costs (hardware replacement, etc.) which Netflix bears. Both parties save.

If Netflix wanted to get clever, they could calculate the cost an ISP would likely pay for the bandwidth their users consume, and charge the ISP say 40% of this amount to host the appliance. Netflix saves 100% of its third party CDN bandwidth costs for that ISP and the ISP saves 60% of their peering bandwidth costs.

If the ISP wanted to get clever, they could calculate the bandwidth that Netflix is paying a third party CDN for the traffic consumed, and offer to host the appliance for 40% of this amount. The ISP saves 100% of its peering bandwidth costs for Netflix traffic and Netflix saves 60% of their CDN cost.

What's likely to happen is a bit of cat and mouse over the costs. The ISP peering is likely a lot cheaper than Netflix's CDN costs, so this could work out as:

    Netflix CDN cost
  - ISP peering cost
    ----------------
  = Cost to Netflix

This is all pure speculation with made-up percentages, of course.

rdl2• 14 years ago

comcast is actually charging people more than transit rates to get access to their network -- which is why (except for level3, who has a special deal) video on comcast sucks even with large comcast pipes.

wmf5• 14 years ago

Netflix already said the box is free (as long as you're big enough to actually need it).

muhfuhkuh• 14 years ago

I think the content question will never be solved as long as

a) Cable companies put the squeeze on content owners to give them exclusive rights

b) Netflix religiously clings on to both commercial-free and buffet-style pricing

It's tricky. You have a movie studio that pumps out AAA features that cost upwards of $200 Million to make, then Netflix comes along and says "hey, you mind taking this really low price for your content, hopefully in perpetuity?" What would your response be? So, it may be awhile before Netflix gets the big-budget blockbusters quickly. That is, unless, their subscriber base goes over 50 Million, then by ALL means game on.

I think that, in the interim, a dynamite move for Netflix would be to acquire a music streaming company like Pandora or Spotify, then a streaming game company like Galkai or OnLive. Push it out on a set top box, a la the OnLive console, charge $19.99/month with the opportunity to rent premium newly-released blockbuster movies, hot records, and AAA games on the day of release for the same prices that redbox or gamefly charges to rent them per day.

mef• 14 years ago

From https://signup.netflix.com/openconnect/software:

Operating System

For the operating system, we use FreeBSD version 9.0. This was selected for its balance of stability and features, a strong development community and staff expertise. We will contribute changes we make as part of our project to the community through the FreeBSD committers on our team.

Web server

We use the nginx web server for its proven scalability and performance. Netflix audio and video is served via HTTP.

Routing intelligence proxy

We use the BIRD Internet routing daemon to enable the transfer of network topology from ISP networks to the Netflix control system that directs clients to sources of content.

patrickgzill2• 14 years ago

They are probably using ZFS also. (my guess)

amalag• 14 years ago

Is ZFS stable enough on FreeBSD? Seems to be an unofficial port.

wanderr• 14 years ago

Why would they use ZFS on a CDN? IIRC, ZFS has some performance characteristics that would make it a poor choice for something like caching data on a network edge when it's really designed for ensuring data integrity.

mikeryan3• 14 years ago

Makes me wonder how long until Netflix looks at doing managed services for video delivery along the lines of Amazon's AWS services.

filmgirlcw2• 14 years ago

But would that have great business potential to them? For Amazon, having that infrastructure helped create not just future products (Instant Video, Amazon Music Cloud, Amazon Cloud Drive, the Kindle and Kindle Fire backend systems), but was aligned as an additional revenue source.

You'd have to wonder if enough people would want to use Netflix's technology stack in a way that didn't include distribution on Netflix.

ketralnis• 14 years ago

The entire television/sports/web video/publishing industry may be able to find a use for such a thing

wmf5• 14 years ago

Netflix has a very different attitude than Amazon; they wanted to spin off DVD rental because it was distracting from their real business.

ChuckMcM2• 14 years ago

Interesting box. With an AFRR of 5 - 7% the box would nominally 'lose' a hard drive or two a year. I wonder how they deal with transient failures (which would be more like 2 - 3 month). If they do the 'triple replica' thing they only get 36T effective per box, but I suppose at about 2G / movie in MPG2 thats still about 18,000 movies 'online' as it were. Now if they did something wild and crazy like a form of RAID6 that would be even cooler (and give them like 100T of actual space or 50,000 movies. Serious video, but really hard to stream to multiple people.

The old SGI/TimeWarner Video-on-demand nightmare seems so quaint now. That had a full cabinet (42U) of storage and servers and it poorly served about 50 streams of a couple of thousand 'titles' and cost > $2M each if my aging brain remembers all the specs. Hmm these guys [1] peg the cost for the "Full Service Network at $100M which sounds about right.

[1] http://www.vizworld.com/2009/04/what-led-to-the-fall-of-sgi-...

ropiku• 14 years ago

Since it's a cache I don't think there's a big problem in losing a hdd and some cache capacity. I would expect it to be more efficient to use as much storage as possible even if a hdd will fail.

ChuckMcM2• 14 years ago

I agree the cache nature makes it easier, but consider that a 3TB drive, at 2GB per 'movie', holds 1500 movies. So when you lose a drive, if the only content you have is on that drive you need to seamlessly switch from sucking it off the drive and sucking it down the back haul network. I can think of a number of ways [1] that might be done with varying degrees of 'hiccup' on the user end, but it is a big chunk of cache to lose at once. Note that 3TB is also 50 minutes of full on 10 GbE pipe bandwidth. But as you say it isn't as if you lose data, since its still there back at the head office, but you lose that copy of it.

[1] Ways that come to mind are; multiple boxes with a diversity of the content amongst them so that another box can 'pick up' the stream, keeping a stream to the home office open but not pulling data until you lose the local source, or even doing a 'buffer' to disk and stream from the buffer then switch disk targets when your disk fails.

fendrak• 14 years ago

Here's the accompanying Netflix blog post:

http://blog.netflix.com/2012/06/announcing-netflix-open-conn...

ghshephard• 14 years ago

I've often wondered if CDNs are going to go out of business in the streaming world, and no longer be relevant, in the world of (rapidly) falling transit costs.

Once transit costs drop below $1/mbit @95th percentile (circa 2014 [1]), it's costing you $1 to transfer approx 300 gigabytes/month. At that point, the management costs associated with all of those distributed CDNs becomes greater than your transit costs. This is somewhat offset by continuously dropping prices of CDNS [2] which is as low as $0.02/Gigabyte on a 500 TByte commit.

What this does tell me is that NetFlix thinks they have an opportunity to save money on CDNs and can beat that $0.02/Gigabyte.

Bandwidth Delay Product(BDP) - keeps CDNs relevant for _downloads_ (iTunes, Software, etc..) - because everybody likes to max out their 100 mbit comcast connection, but you don't need much bandwidth to stream a show. Maybe some value for CDNs in HD or live event streaming.

[1] http://drpeering.net/AskDrPeering/blog/articles/Peering_vs_T...

[2] http://blog.streamingmedia.com/the_business_of_online_vi/201...

dsl• 14 years ago

The minimum for deployment of a Netflix node is 5 Gbps within a metro area. You must also peer with Netflix at a common interconnect so that "loading" is free for them.

By your math they are saving a minimum of $5k per month per deployed device.

Any network not meeting these minimum requirements will be served under existing Level 3/Limelight/Akamai contracts.

jedberg• 14 years ago

Your calculation doesn't account for the artificial barriers that some ISPs put up by charging the CDNs to peer with them (see Comcast and Level 3).

wmf5• 14 years ago

Isn't that orthogonal to the choice between serving from the edge or from a single mega-datacenter?

patrickgzill2• 14 years ago

Cogent is already $1/mbps at "retail" pricing, Level3 (better quality) at $3/mbps or less in some cases.

donavanm• 14 years ago

Pretty sure CDNs will be around. The value add of a cdn isn't their cheaper commit rates. The value is in location diversity and capital investment in infrastructure and ports in those locations. In new speak CDNs are "bit delivery as a service". The customer pays a premium over raw transit to have the cdn make those capital & management investments.

Even for Netflix or another DIY delivery case a CDN will continue to have value. A couple cases immediately come to mind: Lower first byte latency on first video blocks, then a hinted handoff to the openconnect device. Resiliency/recovery of network events, in a similar manner. "long tail" content that doesn't fit in the open onnect cache width.

A key point of these devices is IIRC they're only available to peered ISPs in one those NA IXs. Netflix is essentially converting 5-8gbs of CDN/transit expense in to $50,000 (WAG) of capital. That's what's really brilliant to me.

thelastknowngod• 14 years ago

I have worked on Netflix's servers first hand. After seeing how large of a presence they have in their current CDNs, I can say that they have a hell of a lot of work ahead of them for that to become viable. They are enormous.

ComputerGuru• 14 years ago

Completely off-topic, but I love the redesigned Silverlight player for Netflix on PC. It's gorgeous.

Netflix begins the deployment of their own content delivery network

Related Stories