How Netflix Reverse Engineered Hollywood

http://www.theatlantic.com/technology/archive/2014/01/how-netflix-reverse-engineered-hollywood/282679/

by coloneltcb • 13 years ago

336 129 13 years ago

mbillie1• 13 years ago

There are so many that just loading, copying, and pasting all of them took the little script I wrote more than 20 hours.

I want to see that script.

lstamour3• 13 years ago

It ran on a netbook and was actually controlling a browser and copying and pasting. It's not optimized, but it got the job done, so I'll applaud anyway. Great article, I expect we'll see more like this soon. ;-)

lstamour3• 13 years ago

Oh and if you're nice on Netflix's servers, perhaps at 1 request a second, it would still take 25 minutes to run through 90,000 requests without the overhead of the browser. Sounds like they averaged one request every 24 seconds.

tonfa• 13 years ago

It actually looks pretty wasteful, since it probably needed to load the full webpage instead of just the html.

jerf• 13 years ago

This sort of thing is generally pretty easy. For instance, this would be a good start:

    for i in {0..10000}; do
       wget http://whatever.invalid/genres/$i
       sleep 1
    done

From there processing the results to pull out the genre would be a regex away; a single properly formed grep could probably do it.

Determining the top number would have to be done by hand; having holes in the sequences makes it more complicated but you could still use a sort of basic binary search.

pravka• 13 years ago

That would be a good start on a site that doesn't require authentication. :)

visarga2• 13 years ago

I laughed at that line. Here is someone who never scraped a website or run a spider to collect data from the net. It's a magical feeling next day, when you see your data.

scott_s• 13 years ago

To people who do their work on computers, but have never programmed before, programming is like a superpower.

eli3• 13 years ago

Haven't people gone to jail for scraping a URL and enumerating its possible values?

res0nat0r3• 13 years ago

Yes. When they knowingly are collecting data they know they shouldn't have access to, unlike the data in the article.

eli3• 13 years ago

Not defending the law (which I think is pretty terrible), but that seems like an incredibly fine distinction. I doubt Netflix would agree that I'm supposed to be able to download a list of every possible genre.

pdubs• 13 years ago

Netflix cooperated with my quest to understand what they internally call "altgenres,"

eli3• 13 years ago

Not sure how I missed that -- thanks. Does seem like they scraped first and asked later, though.

justinph• 13 years ago

Usually they don't have the lawyers that The Atlantic probably has. You can get away with a lot under the guise of journalism (and you should be able to).

msg• 13 years ago

At the top of the article is a Netflix genre generator. That is worth the price of admission all by itself.

But then there's a fairly entertaining look into what happened to content at Netflix after the million dollar challenge.

visarga2• 13 years ago

It's simple. They payed people to tag films with more useful features for their machine learning algorithms.

abjorn• 13 years ago

My favorite so far is "Viral Plague Gay & Lesbian Movies". For a random generator it sure seems to have an agenda.

clarkmoody• 13 years ago

  Rogue-Cop Viral Plague Tearjerkers Based on Real Life Set in the Edwardian Era

girvo• 13 years ago

I'd actually watch that.

hershel• 13 years ago

There's also jinni.com which has a similar system, not limited by UI issues and that can be used globally. Usually i get great recommendations from them , and they're fun to play with.

chrisgd2• 13 years ago

I signed up for that a year ago and everytime I am reminded of its existence, I kick myself for not using it more often. Especially a time like now when the TV shows my wife and I used to watch are not on and we are looking for something new.

Just finished Broadchurch over the holidays, very entertaining for those interested. BBC murder mystery

pyre2• 13 years ago

Broadchurch was aired on ITV[1], not BBC. (And in typical fashion, when bringing it to the US, it needs to be a remake because US audiences are apparently too stupid -- according to the television network execs -- to watch British television without heavy localization -- by Fox Networks in this case).

[1] http://en.wikipedia.org/wiki/ITV_%28TV_network%29

danielharan• 13 years ago

Netflix's data allows it not only to recommend movies, but also to finance original productions.

Lots of businesses want "recommendation engines" to appease their cargo cult gods, few ask what possibilities their data really creates.

Sometimes data can make you better at delivering your service. Other times you can optimize inventory, enter entirely new lines of business or even obsolete your competitors.

objclxt• 13 years ago

So audience data when you're talking about original productions can be a blessing and a curse. We don't need big data to tell us that the top four favorite subjects of Netflix users are marriage, royalty, parenthood, and reunited lovers. We've been telling stories about those things for hundreds - if not thousands - of years.

The data can also be misleading, because sometimes what audiences want isn't what's good for dramatic effect. Both Dexter and Homeland suffered from audiences reacting positively to the main character, which caused Showtime to stop the writers from making creative decisions that yes, would have alienated the audience initially, but on the other hand would have made for shows that would have in their later seasons been better received.

Another example: I'm pretty sure that if you looked at the Netflix data it would show that people liked to watch movies and TV shows that either had happy endings or twist endings. The former makes you feel good, the latter makes you appreciate the writing. What people really don't like are ambiguous endings. If we based creative decisions purely on audience data a show like The Sopranos wouldn't have the ending it does. The upshot of all of this is that many writers end up 'cheating' the data to get their shows made. Take Orange is the New Black, where the creator Jenji Kohan has gone on record saying she basically used the main white, engaged protagonist as a trojan horse to get the show commissioned. What I'm basically trying to say in this rant is that using data to inform what shows to commission is one thing, but using it to direct the shows as they progress is another (and something I'm against).

graylights• 13 years ago

You're right that audience data is going to be full of the obvious. But it's also good for watching trends, before they go bust. A number of genres (e.g. superheroes) have cycles of growing, being massively popular, then dying off for a couple decades.

If you're in the business of making content, you want to join the trend early and get out before everyone realizes it's a fad. A truly great movie can buck trends, or even change them, but there are few truly great movies. Or you could just ignore this all and make the perennial favorites, generic romance or action movies.

saraid216• 13 years ago

We don't need big data to tell us that the top four favorite subjects of Netflix users are marriage, royalty, parenthood, and reunited lovers. We've been telling stories about those things for hundreds - if not thousands - of years.

I'm really irritated at this attitude. Things that are obvious to you aren't obvious to other people. Confirming things using hard data is useful.

zheng• 13 years ago

What would be really cool is if this list of genres was open-sourced somewhere. I can see Netflix not wanting that, but it would really save time for however many hackers read this article and decide they want the same data.

joshdick• 13 years ago

I was just thinking how sad it is that Netflix has all this great data on movies (as Pandora does for music), and it's locked away from us.

shawnc• 13 years ago

I find the part at the end about the Perry Mason aspect very interesting, and actually my favourite part of the article.

And the final sentence, feels like the real reason this was posted to HN: "And sometimes we call that a bug and sometimes we call it a feature."

Edit: Also, the 'Gonzo' genre of Post-Apocalyptic Comedies and Friendship seems it's got its first one in "This Is The End".

peter303• 13 years ago

I wonder how often there are "new generes", that is a complete creative movie that doesnt fit an existing one. Someofthose quirkly little movies that seem to win oscars may be such.

alvare• 13 years ago

What about the Perry Mason thing? That's scary shit.

firebones• 13 years ago

Theories:

1) Assuming there's some clustering algorithm in use, it could just be some tradeoff edge case that isn't worth optimizing away.

2) It could be that one of their reviewers had an overly-aggresive and non-standard genre-tagging approach for Perry Mason and classified a bunch of shows in such a way that the source data was polluted. This could be something stupid like most other movies having many reviewers giving higher confidence, while a large number of Perry Mason shows and DVDs only had a single reviewer who was either through a bug or through overweighting, given too much influence. This seems to be the most likely cause of the bug--some skew or amplification in the source data.

3) Intentional poisoning of the results, like cartographers putting in bogus features, or data sellers seeding their data with watermarks, etc.

sriramk• 13 years ago

Sounds like some data cleanup issue. Definitely an outlier that should probably be ignored.

discardorama• 13 years ago

How is this any different from that Pandora did with music?

JumpCrisscross• 13 years ago

The only semi-similar project that I could think of is Pandora's once-lauded Music Genome Project, but what's amazing about Netflix is that its descriptions of movies are foregrounded. It's not just that Netflix can show you things you might like, but that it can tell you what kinds of things those are. It is, in its own weird way, a tool for introspection.

I have both Pandora One and Netflix, and am very satisfied with both. When I read why Pandora served up a particular song I have sometimes gone "hmm," but not much more.

But I can still remember pausing when Netflix put "critically-acclaimed cerebral dark thrillers" or "visually-striking foreign sci-fi & fantasy" in front of me. Not only did it map my preferences to films I hadn't seen, but it told me something new about my tastes and thus myself.

swanson• 13 years ago

Just an FYI if you didn't know this, but Pandora will show you the same kind of information if you hover over the song (web) and pick "Why was this track selected?".

"Based on what you've told us so far, we're playing this track because it features electronica roots, house influences, danceable beats and vocal samples."

skybrian• 13 years ago

Sure, it's the same basic idea.

mixmastamyk• 13 years ago

Meanwhile, their client still can't separate my daughter's kid shows from mine. It took them several years to implement profiles on iOS and then another to do it on Android.

Now implemented, "My Top Picks" last night were still dominated by My Little Pony.

Also would like to choose which shows she can watch, but the client doesn't support that. </complaints-over> ;)

mathattack2• 13 years ago

I get a kick out of the genre names. My wife and I both use the same account, and each rate movies ourselves. Whenever something comes out of left field, we say, "Look at your movies..."

I wonder if Netflix can tell if multiple people are rating movies. Does it think we are one confused person, or two distinct personalities?

ryen• 13 years ago

Have you tried their multiple profiles feature? https://movies.netflix.com/EditProfiles

officemonkey2• 13 years ago

We have multiple profiles now, but before that there were 10 years of Netflix and five of them dominated by a child who enjoys "Curious George" and "Super Hero Squad."

It's going to take a while to get back to normal.

arek2• 13 years ago

76k micro-genres seems much. For my website http://5000best.com/movies/ I created 40 main genres using IMDb tags together with the 100 million ratings from the Netflix Prize data (I was 43rd in that competition).

Additionally, earlier I extracted and named 12 new genres (those ones on the right) from the Netflix ratings alone - I described the process here: http://arek-paterek.com/book/predict_sample.pdf

KaoruAoiShiho• 13 years ago

What do you mean much? When working like this the more the better. There's an opportunity for an open source variant of this technology.

xacaxulu• 13 years ago

Anyone else wish that "Taken 3: Lil Bub" was a real movie?

jasimq• 13 years ago

I wonder if NSA would find that data useful. You know, to profile netflix users.

agibsonccc2• 13 years ago

For those who are data curious: https://gist.github.com/agibsonccc/8230583

I cherry picked this from the source for those who might want the generator. I "think" that's everything, correct me if I'm wrong there. I didn't really test it, just took a few seconds to grab what I saw for later.

refurb2• 13 years ago

A friend of a friend works at Netflix and told me how they use some of this data.

House of Cards was basically a data driven production. Based on Netflix's customer preferences, they knew that a political thriller, starring Kevin Spacey and directed by David Fincher would maximize the number of views based on the habits of its customers.

It would appear the data was correct!

thetylerhayes• 13 years ago

Is there a post somewhere where they describe this?

yonasb• 13 years ago

http://blog.kissmetrics.com/how-netflix-uses-analytics/

ImpressiveWebs2• 13 years ago

Good link by yonasb.

But the comment by refurb sounds a bit misleading. As I understand it, they did not produce this solely because of the data (as if the show was born out of the data). The idea, the actors, and the production staff were already chosen before Netflix's involvement. Netflix looked at those things as they had been proposed and realized it was a good gamble to bid on the show.

smsm422• 13 years ago

For all the high praise that gets heaped on Netflix for their brilliant technology, I have a feeling there is some other Netflix that is concealed from me.

I have been Netflix customer for years. I thought the idea was brilliant - super-cheap movies arriving whenever you want, what could be better?! I loved Netflix. Then I slowly discovered Netflix is running out of movies I want to watch - up to now where about 95% of movies I want to see are out. Then there was that streaming vs. DVS fiasco - and I stayed with streaming. But then I discovered there's nothing for me to stream. I thought maybe my tastes are weird - so I went to wikipedia and IMDB and looked "top X movies" - and most of them, of course, can't be watched on Netflix, except for those few that I've already watched long ago.

And that million dollar recommendation system? I've over 800 ratings, and I have hard time remembering last time their system suggested me something useful. In fact, the only reason I am keeping the subscription is because my wife has some series on her sub-account that she's watching. For me, Netflix has become almost 100% useless. So I wonder, with all the high praise to their brilliant data usage and innovative technology - am I doing something wrong? Am I missing some important part of Netflix that everybody else is seeing?

coliveira• 13 years ago

Yes, you're right. What is making Netflix gain market share are the series. Basically most people are using Netflix instead of TV, not to get Hollywood movies. The same pattern happened at my home: there are no good movies to see, but plenty of activity around TV shows and documentaries. And from a business point of view this is even better for them, because TV series make people stick around longer than a single movie.

ghaff2• 13 years ago

Right, you basically still need to get the little plastic disks--from Netflix or elsewhere--if you want to watch quality movies, especially recent ones. Someone I know there told me a year or two back that people get streaming with the intention of watching movies but they end up watching TV shows instead.

This doesn't quite apply to me because I quit streaming when they did the whole splitting of physical media from streaming thing. And signed back up when House of Cards came out. But I do view streaming today as a source for TV shows and any movies I want to watch on it are basically lagniappe.

Ntrails• 13 years ago

If they had the same films available to stream as they do to get on DvD, I'd pay for netflix.

I cancelled the free trial in about 30 minutes for this reason. I still think netflix owns for series, but I didn't need it for that.

dave84• 13 years ago

I think streaming Netflix requires a slightly different approach in how you watch them.

For ages I found myself ignoring Netflix recommendations based on the descriptions, but when I watched them I actually enjoyed them. If I wanted Netflix for well-known movies and safe choices I'd be disappointed, but as a way of watching unknown (to me) movies, I find it great.

talmand• 13 years ago

Your complaint is misguided. This is not the fault of Netflix, they can only show what they are licensed to show. If your favorite movie is not on Netflix, if the movie you really want to see is not on Netflix, if the latest movies are not on Netflix fast enough; it's likely because of the copyright holder not making a deal with Netflix.

For example, take a look at newest releases in Netflix in a browser. Now use a proxy to change your country and look again. Likely, depending on where you actually are and where you say you are, you will see a different list. That's because of licensing.

I would imagine that Netflix would be more than happy to have in their catalog every bit of movie and television products that have ever existed and will ever exist as fast as possible for everyone to see regardless of where you happen to live. But as long as the copyright holders feel they will make good money off of DVD sales or other avenues before offering it to streaming, then that's the environment streaming will always be.

If there's something you wish to see on Netflix and it isn't there, complain to the distributor of that product and not Netflix.

smsm422• 13 years ago

It's not the question of "fault", it is the question of Netflix not being a useful service to me. What I'm saying is that maybe that recommendation engine is excellent technically, I don't know - for me good recommendation engine is the one that can recommend me something I could use, and Netflix one can not. Whose "fault" it is - why would I care? I certainly wouldn't start a personal crusade to find out who in the byzantine network of IP rights is to blame that I can't watch anything of interest on Netflix. For me it is a failure of Netflix as a service, and the reason for this failure is not for me to investigate - it's for them if they want to become a useful service.

eruditely• 13 years ago

Maybe you are wrong, and if you were it would be great because then you could enjoy it right?

However, maybe this product just isn't for you.

afterburner• 13 years ago

Royalty is America's second favourite topic?

smacktoward• 13 years ago

Nobody loves monarchs more than someone who has never had to live under one.

How Netflix Reverse Engineered Hollywood

Related Stories