Netflix’s Secret Special Algorithm Is a Human

http://www.newyorker.com/business/currency/hollywoods-big-data-big-deal

by donohoe • 11 years ago

214 59 11 years ago

As much as people (especially here) think that everything can be solved by Big Data/Machine Learning, human experience and intuition are still very important.

No, you can't A/B test your new logo, or the design of your page from scratch.

Big Data can't tell you your product sucks. Option A may be better than Option B but this is in the context of both options (and not considering all other possibilities)

I see companies firehosing every tiny bit of consumer data hoping to be able to make sense of it all and find something there. Meanwhile they're missing whatever their competitor is doing and what their consumers are liking about it.

jobposter1234• 11 years ago

Those companies are just cargo-culting into the Big Data cult, like everyone else. It doesn't mean they aren't getting useful answers from their black box.

Heck, sometimes it's even useful to have a Magic 8 ball make a decision for me.

scott_s3• 11 years ago

In other realms, we don't call this data but intelligence. Humans will still make the decisions, but in more and more places, the decisions are more likely to be informed by data (intelligence).

Consider that Netflix's data is tiny compared to the amount of data that any government must sift through. Algorithms don't make government decisions, people do. But they (hopefully) base those decisions based on intelligence (some of) which was gathered and filtered by algorithms, then synthesized by humans.

SquidMagnet• 11 years ago

I am not entirely sure what you are saying here, but the way I have always understood it is: data is knowledge; the ability to apply knowledge is intelligence.

Netflix has a considerable amount of data (knowledge) and its algorithms exemplify some efforts to apply that knowledge (intelligence). As it stands presently, though, humans still tend to be more intelligent than any algorithms we have created. (Generally speaking, of course.)

I think we are trying to say the same things here, right?

muaddirac• 11 years ago

I assumed they were using the other meaning of intelligence, as in "signal intelligence"

scott_s3• 11 years ago

I'm trying to reframe what Netflix does as something that is already done - we just tend to use different names for it in those other domains. What's new is that we're doing this old thing - processing massive amounts of data, extracting the relevant facts, synthesizing those facts into a coherent story, and presenting that story to decision makers - in new contexts.

In the Netflix context, what is mostly called "data" is called "intelligence" in, say, government decision making.

digisth• 11 years ago

This is another facet of the "amplified-teams" trend that's been happening over the past few years. This book review of 'Average is over' has some good information about it:

http://ieet.org/index.php/IEET/more/searle20150109

"In his vision intelligent machines will revolutionize everything from medicine to education to business management and negotiation to love. The human beings who will best thrive in this new environment will be those whose work best complements that of intelligent machines, and this will be the case all the way from the factory floor to the classroom."

Very interesting times ahead.

scott_s3• 11 years ago

Perhaps another way of stating that is: The people who will best thrive are those whose work cannot be automated. Which, in some ways, has always been true.

tel• 11 years ago

In Health Informatics at least this is taken even further

    Data
      X of Y people have some disease
    Information
      Based on data, I can predict disease likelihood
      given some environmental and personal factors of
      a patient
    Knowledge
      Using informed predictions, I make good inferences
      about how to proceed with diagnosis
    Wisdom
      Using knowledge and experience I choose the right
      approach for treating and diagnosing a patient
      which is efficacious, healthy, and works with the
      patient's actual needs

It's easy to draw these lines in other places or to call the tower a lot of woo woo able to be reduced into inferences atop raw data all combined correctly... but it serves to remind just how difficult it is to combine the right data in the right way to make the right decisions at the right times.

It also serves as a sharp counterpoint to the idea of, say, machine learning patient diagnoses. It turns out that diagnostic accuracy is terrible, but not because people are directly bad at it (even if they are) but instead because knowledge/wisdom dictates that perfect accuracy isn't that valuable---perfect care is and that can involve chasing down treatment and care avenues that would never be predicted or acting on information that is not currently in your model.

ljd• 11 years ago

I see this argument akin to something along the lines of, "A website is really a human because someone needs to maintain it and make decisions on it's appearance."

You are describing a tool. People don't say hammers are humans, either.

Netflix's special algorithm is indeed an algorithm.

abandonliberty• 11 years ago

I was really hoping Netflix had a secret warehouse filled with hundreds of workers whose job was to provide recommendations based on what you like, a la 'invisible boyfriend' [1]

Sadly, they just have people on the quest for the perfect hammer.

[1]http://www.washingtonpost.com/news/the-intersect/wp/2015/01/...

stillsut• 11 years ago

I see how netflix's dataset helps them produce an addictive serialized drama like House of Cards. There are many examples to learn from, and the shows are basically built out of a formula.

What I don't see is how this would help you make a critically well received documentary about Nina Simone. Sure, you might be able to predict a lot of people will stream it, but how does that make it well received at Cannes? Nothing in their dataset is telling them about the art of film-making.

Bluestrike2• 11 years ago

No, but their data allows filmmakers to gain a better idea of what the consequences of certain decisions are likely to be. It's not so much as allowing algorithms to dictate creative decisions as it is enabling those decisions to be better informed. As for targeting film festivals in particular, I've no doubt that Netflix has sufficient data to look at sort of festival-goers (and those like them) in particular. While it can't provide a guaranteed formula for festival success, it should still give a filmmaker another tool that helps them do their work.

smacktoward• 11 years ago

The argument of the article is that Netflix's dataset doesn't really help them make decisions like what the content of their next documentary should be. It argues that what happens instead is that Netflix's programming honcho, Ted Sarandos, picks projects based on the person who's putting them together, looking for creators with a solid creative track record and an existing cult following:

I began to sense that their biggest bets always seemed ultimately driven by faith in a particular cult creator, like David Fincher (“House of Cards”), Jenji Leslie Kohan (“Orange is the New Black”), Ricky Gervais (“Derek”), John Fusco (“Marco Polo”), or Mitchell Hurwitz (“Arrested Development”)... I do think that there is a sophisticated algorithm at work here—but I think his name is Ted Sarandos.

So take the case of the Nina Simone doc. It was directed by Liz Garbus (http://en.wikipedia.org/wiki/Liz_Garbus), whose previous films have been Sundance- and Oscar- fare for a few years now. So it could just be that Sarandos looked at her track record and decided that Garbus was an up-and-coming talent, and that mattered more than the specific content of whatever her next project was going to be. He was just betting that a Liz Garbus documentary was going to be great, no matter what story it told.

JeremyMorgan• 11 years ago

Hollywood Video's corporate office had a guy, one guy whose job it was to just watch movies. He'd watch them, make sure it was categorized properly, then create associations with other movies that "you might like" if you like this movie, and vice versa. All kept in a spreadsheet that we later plugged in.

We also thought there should be an algorithm, but he was pretty dang good at what he did.

boomzilla• 11 years ago

It's simple ROI. There is not that many movies this is a task that one doesn't need to scale with computers. Hiring a guy is likely much cheaper than investing on the software. Plus I am sure there are plenty of people who would like to watch movies for their day jobs.

On the other hand, I am pretty sure there is at least some degree of automated video classification at Youtube.

RogerL• 11 years ago

I think it does need to scale.

What happens when after a bus event? Someone else has to watch every film in existence? It is not enough to just watch the new films - you have to have a memory of all other films in order to make that association.

tessierashpool• 11 years ago

    I am pretty sure there is at least some
    degree of automated video classification
    at Youtube.

there's definitely collaborative filtering.

scmoore• 11 years ago

This reminds me of Derek Sivers' album recommendations for CD Baby:

http://sivers.org/hi

Basically Sivers just listened to every album and did the same thing, and when he couldn't get to all of them he hired someone to do it full time.

How many hours of new programming do they add a day? How many people would it take?

72deluxe3• 11 years ago

Very interesting. Did he enjoy it or did he get bored? Did he have to watch some horrific films? Did he feel like his brain was being polluted as more degenerate films come out? Did it affect his perception of reality?

Just interested.

thret• 11 years ago

Only a few times have I watched a full three movies in a day, and I found it exhausting. I think having to watch movies all day every day would be much harder than it first sounds.

cmdkeen• 11 years ago

Spare a thought for the people at the BBFC in the UK who in addition to having to watch every feature film to rate it also have to watch all the pornography videos to rate and check for legality.

I attended a university debate once between a BBFC censor and a female porn director. Watching the two of them talking about "counting knuckles during a fisting scene" to check that the UK requirement that at least one knuckle is always visible was a truly bizarre experience. The BBFC man had the look of an individual who has stared into the heart of darkness for a little too long. He had some excellent stories though!

If anyone does organise debates, talks, conferences etc in the UK on associated topics I can highly recommend getting in touch with the BBFC. They're really keen to engage with the wider public rather than just be two signatures on the screen just before a film comes on.

antoncohen• 11 years ago

That is basically what Netflix does. They have teams of people trained to watch movies and tag them with lots of metadata -- really precise metadata. Then the algorithms take over and do the actual recommendation.

http://www.theatlantic.com/technology/archive/2014/01/how-ne...

minthd• 11 years ago

This article looks at single point in the industry with regards to automation. But i believe the important point is whether the addition of data provides more efficient movie production, with less failure , and by how much.No need to automate a movie you don't produce.

dtjones2• 11 years ago

Even Google, the champion of algorithms, employs substantial human adjustments to make its search engines perform just right.

The author doesn't have a solid grasp on machine learning.

The 'human adjustments' provide feedback to the algorithm, which the algorithm then uses to update and improve performance. His tone implies its a bad thing to use human feedback.

mostly_harmless• 11 years ago

Ideally, it would be a completely automated system. You don't need to understand machine learning to know that it would be better for the computer to do everything itself without human intervention

dtjones2• 11 years ago

It is completely automated. It automatically incorporates human/user feedback into ranking.

DannyBee2• 11 years ago

?? It doesn't require human intervention.

Simplified: Machine learning algorithm constructs page to show human.

Humans click on this or that on page.

Click data is fed into algorithm.

Algorithm uses this data decide how to show better page to humans

You are taking human input. It is still completely automated, it requires no human intervention.

CHY872• 11 years ago

How so? If the cost of making the computer able to do everything itself is greater than the cost of an equivalent system that takes human input, then you should use the human input version every time.

j2kun2• 11 years ago

featuring the television star Melissa Raunch

Whoa there! It's Rauch. Kind of a Freudian slip there, considering the actress is being named in relation to a sex scene.

ScottBurson• 11 years ago

Here's another:

Television studios have Nielson ratings

It's "Nielsen".

j2kun2• 11 years ago

Looks like they've fixed it. But seriously don't they have editors reading this stuff before it goes live?

zkhalique• 11 years ago

In other news, Pandora is built on human input, among other things. The Music Genome Project.

ctdonath• 11 years ago

Drive such a thing thru Amazon Turk perhaps?

Might just be the "killer app" for Turks: get paid to watch movies!

sireat• 11 years ago

MTurk has already been used to classify and tag videos.

I spent a month as Turker (as an experiment) a few years ago.

The best paying HITs were provided by a porn company to classify and tag their videos. You were given a sequence of images from the video presumably to save bandwith.

I even debated scripting a helper tool which would take a few keystrokes (for example dvda) and generate a suitable description.

domdip• 11 years ago

What a weird strawman article. Did the author really think Netflix produced television shows in the same way that Siri suggests gas stations? Any blockbuster-finding algorithm (or any other component of this decision) is going to have false positives and require human intervention.

wallflower2• 11 years ago

See also Shazam's Secret

The hunt keeps Mr. Slomovitz on his toes. Every morning, he skims dozens of music blogs, checking for new releases he might have missed, as well as the iTunes, Amazon.com and Billboard charts, and blog aggregators like the Hype Machine.

Most weeks he also goes to local record stores to see if there is something in stock he has not heard of, or if older albums are being remastered or reissued. And he listens to local radio stations, especially near universities.

"Shazam’s Search for Songs Creates New Music Jobs" http://www.nytimes.com/2011/02/14/technology/14shazam.html

skrebbel• 11 years ago

Shazam actually has 2000 music geeks in a call center who listen to tiny audio fragments and guess which song it is. The algorithm just makes sure the rock geek gets the rock songs, and so on.

afro88• 11 years ago

Really? I'm genuinely interested in this. Do you have an article or interview or something with more info?

hotgoldminer• 11 years ago

Dao Nguyen @Buzzfeed, "Data should not dictate your strategy," Nguyen says, "But you should understand what data tells you and also what its limits are."[0]. Heard this a few weeks back and still resonating with me.

0 -http://www.marketplace.org/topics/tech/buzzfeed-wizard-who-c...

karmacondon• 11 years ago

The lesson here isn't "Human Judgement > Algorithmic Judgement". It also isn't "Humans are good at some things and computers are good at others". It's that there isn't a good reason to make the investment in solving this particular problem algorithmically (yet).

Algorithms are great when you need scale, especially in situations where a 10% improvement in prediction accuracy can make a big improvement in the bottom line. Netflix and other studios might greenlight several new shows in a year, out of dozens that receive consideration. And the Pareto Distribution is in full effect. Most of the profits and awards come from one or two big hits. Algorithmic decision making just doesn't make a lot of sense in situations with a small sample size and uneven reward structure.

It doesn't mean that it isn't possible, though. If someone were to make the massive investment necessary to do a more thorough analysis of the content creators, the actors, the scripts and potential audiences and all of the other possible inputs then algorithms could probably do as good a job as humans, if not better. Netflix and others have only taken baby steps in this direction, working with data that is readily available and using predictive techniques that are well tested and understood. Given the nature of the problem, it doesn't make sense for them to approach it any other way at this time. But when it comes to making billions of recommendations to millions of people per day, they still rely heavily on data and algorithmic prediction. There's a time and place for everything. The time and place for algorithms in our daily lives is changing and expanding, but very slowly.

rurban• 11 years ago

I also rather trust the algorithm to see what is a good movie and what is not. In my case the collected ratings of the experts, not the users. Current example: http://cannes-rurban.rhcloud.com/Sundance2015

There are some outliers, typical "festival hits", which only relate to festival specialists, but they are easily detectable, with a "10%" human bullshit detector. Like last years Godard at Cannes, 2012 Leos Carax, 2011 the Kaurismaki and 2010 both the Godard and the Jury winner Weerasethakul. Those outliers are even statistically detectable.

One thing is clear, you can trust the collected experts more than the juries. So I can fully confirm the story.

cpeterso• 11 years ago

Rotten Tomatoes' rating, mentioned in the article as scoring THE BRONZE as 10% (though the RT now says 18%), is also a collection of professional reviewers' ratings.

abruzzi• 11 years ago

The problem I've found is the algorithms for detecting taste seem to struggle greatly the further ones taste deviates from the norm. I find Netflix's recommendations to be fundamentally useless and is always reccomending things in which I have zero interest. But I've also found Rotten Tomatoes to be borderline useless as well (the second worst film I've seen in the last decade--Snowpiercer--has a very high RT score...go figure.)

keypusher• 11 years ago

Do you actually rate the things you watch on Netflix? Or go back and rate a bunch of popular movies you loved and hated? Because if you don't, you will just keep getting the same mainstream recommendations.

otakucode• 11 years ago

It always amazes me how the people in the content creation industries keep their jobs. Even though comprehensive studies have shown that the public responds randomly to entertainment products, they still skate by claiming to 'know what the public wants.' Analysis of the decisions of past executives at movie companies showed that executives who were said to be "on a hot streak" were mostly benefitting from the results of projects started by their predecessors, and then when their "streak" ended and they left, their replacement got the same benefit from the projects they left in the pipeline.

The book 'A Drunkard's Walk' analyses various different industries and situations and shows where randomness shows up. The success of media is one of the strongest ones. No factor correllates with success. Not budget, star-power, genre, directors, nothing. For every runaway hit there are exactly as many abyssmal failures. Before Titanic hit theaters, film critics and industry insiders were dead certain that it would prove to be the most monumental theatrical failure in history, making Kevin Costner's Waterworld look like a walk in the park. Of course, they were wrong. They will always be wrong as often as they are correct. Their entire careers are built on, quite literally, absolutely nothing. These executives choose what gets produced, and they are incompetent at it. Yet they are paid millions of dollars. It's astonishing that they get so far with unmitigated bullshit.

This is why large movie production houses will die. Their performance has always been random in terms of producing successful content. But they always made up for it by having total control of the distribution. Now, distribution is worthless. Any 12 year old with an Internet connection can distribute media better than large corporations can. Take away that control and profit from distribution, and those companies will end up simply fading into bankruptcy after enough failed projects pile up.

marketforlemmas• 11 years ago

You might also be interested in Duncan Watts' book "Everything is Obvious", which explores this tendency we have to assume that observed success equates to some skill. Its a nice read.

roasm• 11 years ago

Large movie production houses won't die in the near future, because predicting a financial winner isn't as hard as you make it sound. We currently have a movie economy that is driving the big producers to make big budget comic book sequels that do well internationally (generally based on the universal appeal of liking suspense and action). They've boxed themselves into a very profitable corner of funding high budget movies that independent producers can't fund and then profiting internationally. They make a movie for $200M then make $300M+ in sales.

I do agree that more independent (but still highly funded) producers are rolling dice, then manufacturing winners with their marketing power.

redRobo• 11 years ago

The title is completely misleading, seems like it's talking to the recommendation feature on Netflix, but it isn't. Everyone with common sense knows that big executive decisions aren't made solely by computers, yet the author seems amazed that big executive decisions are ultimately made by big executives. Talk about clickbait, huh?

Netflix’s Secret Special Algorithm Is a Human

Related Stories