Netflix: Lessons in Experimentation

https://www.aakashg.com/2022/01/18/netflix-experimentation/

85 35 4 years ago

We’re hiring to do more of what’s in the article: https://jobs.netflix.com/search?q=Experimentation

weq• 4 years ago

Yes, someone with some basic recommendation engine knowledge please apply! A basic rip-off website streaming site has a better recommendation engine AND ux then this company. Was given a subscruption as a present and it dumped it in the bin a few weeks later after. It seems like the algorithm is a basic highlight whatever scam show is pumped by the bingers at the time. Not even the show i last watched (consitantly everyday) is easy to navigate back to if u close the app. I swapped back to the streaming sites and found more NetFlix content immedataly.

Thx for observables though!

rawoke083600• 4 years ago

Yup like https://www.wepickflix.com :p - I tried to implemented a "group recommendation" - since watching "tv/shows/movies" is many times a "group activity".

dmitriid3• 4 years ago

Do create a button called "wipe my recommendations and start anew"

tedsanders4• 4 years ago

There already exists button for wiping your recommendations and starting anew: It's labeled 'add profile'. :)

(Otherwise, you can scroll through you watch history clicking delete a zillion times.)

laxatives• 4 years ago

I would be a lot more interested in a technical article than hearing about the history of Netflix.

rwilson4• 4 years ago

You might be interested in Trustworthy Online Experiments by Kohavi et al then. I enjoyed the historical progression!

travisjungroth3• 4 years ago

We've recently done a large blog series on experimentation. https://netflixtechblog.com/netflix-a-culture-of-learning-39...

Kinda covers everything and links out to posts that cover the software engineering in more detail and papers that cover the math in more detail.

gordon_freeman• 4 years ago

the blog posts series on rec system is interesting. But when I watch Netflix I still run into simple issues. can you please take care of some basics first, such as if I press 'thumbs down' then Netflix would never recommend that title again or similar to such titles.

smartbit• 4 years ago

If their experiment are zo good, why are the recommendations so bad that I had to turn off the feature that adds items to My List?

Several series I’ve enjoyed watching several years ago now appear on Netflix. Too late for me on Netflix, but as I liked them I rate them with thumbs up, hoping that Netflix would give me better recommendations. Instead of recommending other movies or series, to my dismay Netflix starts recommending the series I’ve already watched elsewhere. As I can only rate a series I’ve watched, why on earth does Netflix recommend something I’ve already watched?

The diversification of Netflix is great, but most of it is too little or too late for my taste. Still better than Disney or Apple TV+ where I could find 1 or 2 shows of my liking. Looking forward to a couple of months of HBO Max once it is available in my region. HBO Europe made some award winning documentaries and I’m hoping to find more that I haven’t watched.

Luckily for me there is an alternative for the streaming platform as for the past several decades every week my local TV stations have 1 or 2 great movies and maybe a new serie, selected not by experiments but by real people of flesh and blod.

ajmurmann• 4 years ago

If you want something selected and curated by actual humans, I recommend checking out Mubi. Lots of odd ball, but phenomenal movies on there. They started out having thirty movies on there and replacing the oldest every day with a new movie. It's often times a few in a row by a specific director or around a theme. They now also a have decent library in addition.

Almost every movie I find truly memorable in they just few years came to me via Mubi.

onlyrealcuzzo• 4 years ago

Recommendations aren't really chosen by "experiments". And certainly, they aren't A/B testing budgeting and development of shows.

Recommendation engines can be improved with experiments. And they - along with budgeting and show development (at least at Netflix) - are "driven by big data".

Further, it doesn't matter if you - personally - like a feature. If that feature is moving the numbers and passes the A/B test, it will make itself into the app.

They run thousands of experiments. Of course there are going to be several that you - personally - do not like.

dewey• 4 years ago

why are the recommendations so bad

Without knowing anything about how it works internally I'd guess that the recommendations are not simply about what's best for the user but also what's best for Netflix (pushing Netflix originals).

kevin_thibedeau• 4 years ago

They used to have good recommendations for their DVD service with star ratings. The rental catalog was huge and varied then so a lot of interest areas were well covered and easy to extrapolate from your ratings and your common cohort. I suspect the problem they have now is most of the top tier content is gone and it's 90% filler that isn't compelling to watch.

Semaphor• 4 years ago

I can only assume that the majority of people like a) rewatching things they’ve already seen, b) like watching things they said they don’t like, and c) watch things from categories they don’t like.

Because that’s what Netflix has been doing to me pretty much since they became a streaming service.

WhyNotHugo• 4 years ago

If their experiment are zo good, why are the recommendations so bad that I had to turn off the feature that adds items to My List?

They've gotten very good at grabbing attention with cover art, but then you click on it and quickly realise its crap.

In other words, they've put appealing covers on all the boxes, so now you pick ones that are crap only to abandon them 20 minutes later.

I wonder if abandonment rate is a metric too.

wheelinsupial• 4 years ago

I wonder if abandonment rate is a metric too.

Apparently they categorize the type of viewer you are based on when you abandon the content. [1] mentions starter, watcher, and completer as viewer categories based on how much of the content they view.

[1] https://www.fool.com/investing/2019/12/06/how-netflix-decide...

nerdponx• 4 years ago

I think the recommendations are not good on Netflix because the content is mostly not good. Lots of Netflix Original stuff that lacks staying power and seems designed to pander and not to actually be good. Fortunately, as time marches on, Turner Classic Movies is starting to show films from the 70s and 80s (I guess something can only be a "classic" if it's at least 40 years old), so I imagine that TCM will get a better and better selection in the coming years. If you have a cable subscription, you can stream their rotating selection of films online.

I also find that Hulu generally has better options for TV series than Netflix. You have to deal with ads unless you pay extra for the premium subscription, but it's not like they aren't harvesting your data anyway, and it's easy to just mute the ads (I always muted them on cable, too) and go do something else. It's also still a lot fewer ads than on cable TV.

deathanatos• 4 years ago

Netflix abandoned the ‘large catalog’ strategy for a ‘right for you, personalized catalog’ strategy.

The "right for [me]" catalog seems to contain a lot of "get the viewer hooked on the first two seasons of a show; don't catalog the latter seasons".

jph2• 4 years ago

Netflix experimentation seems to have suffered in the past year IMHO, with worsening recommendations and no simple way for users to provide feedback. And bugs are creeping in-- this week I've hit two different bugs in the Netflix recommendation engine, so something's amiss.

The experiment I would love to see is an simple icon that means "I don't want to watch this".

john___matrix• 4 years ago

It's been pretty apparent for me that their who focus changed a couple of years ago from "here's some stuff you might enjoy watching" to "here's the stuff we want you to watch" or much like Google ads, "here's something you've already watched, watch it again"

I'm almost at the point of cancelling my Netflix subscription simply because it's so much effort to find something of interest now.

tedsanders4• 4 years ago

There already exists a simple icon for "I don't want to watch this": the thumbs down button. Clicking it is supposed to remove the show from your recommendations (after a day or so, to clear the caches).

jph2• 4 years ago

It turns out that Netflix thumbs down conflicts with thumbs up.

For example, pick a movie that you like, yet don't ever want to watch on Netflix; perhaps you've already seen the movie on a different service. Do you click thumbs up ("I like this") or thumbs down ("Not for me")?

Next, if you watch some of the movie, then it shows up in your "Currently Watching" list, with icons for thumbs up ("I like this but don't want to continue watching"), thumbs down ("I don't like this"), and a broom ("Just cleaning up").

What I am suggesting as an experiment is a choice that means "I like this but don't want it on my main page".

Consultant32452• 4 years ago

For the love of God, can we please get an "Only show me things in my native language" option?

birdyrooster• 4 years ago

The only reason Netflix is in FAANG is because without them, people would be saying a very naught word.

thaumasiotes• 4 years ago

The reason they're there is that they are known primarily for the high salaries they pay.

anarazel• 4 years ago

I find the recommendations rarely useful. But what really bugs me is that the same series / movie typically shows up in all kinds of lists. Over and over and over. Besides the categories - often filled with stuff not matching the label - I see the same stuff repeatedly in "trending now", "top 10 today", "only on Netflix", "popular on Netflix".

watwut2• 4 years ago

The Netflix recommendations for me are quite bad. I literally aost always end up watching YouTube.

It makes it is super difficult to discover movies, even if you have pretty good idea about what kind of movie you want.

Netflix: Lessons in Experimentation

Related Stories