Analysis of whether IMDB ratings suffer recency bias

by eoinmurray92 • 7 years ago

72 45 7 years ago

minimaxir2• 7 years ago

Recency bias came up a few days ago in a previous HN submission by the OP, and the OP also commented in: https://news.ycombinator.com/item?id=20099023

In that subthread, I posted a heat map data visualization I made of IMDB rating vs. release year, which is suspiciously similar to the one OP made: https://news.ycombinator.com/item?id=20099344

It's not plagiarism, but if the OP got the idea for checking for recency bias from the HN thread, they should explicitly credit that.

mykowebhn• 7 years ago

I agree with you.

Does the opening sentence count? "I've read a lot recently about the recency bias of movie ratings, IMDB ratings in particular."

KyleOS3• 7 years ago

Hey - not OP but I wrote the article. There was a lot of discussion about recency bias in that thread, including the graph you shared and I decided to expand on the analysis. Note there are a few other graphs in my post. I did just draw this up quickly yesterday. But you're right, sorry - I should have cited your comment in my post. Adding it now.

minimaxir2• 7 years ago

Part of the reason I make data vizzes is so people can expand on the ideas, so there's no issue there! (as long as proper credit is given to the original work)

bscphil5• 7 years ago

Hi, there are what seem to me (not a statistician) to be a number of serious errors in the article. The most egregious is the first conclusion, which threatens to undermine the whole thing.

There does not seem to be any strong connection between number of votes and a movie's IMDB rating.

Well, this just isn't true. I happen to know this because I've looked at this data before myself. The issue is that you're cutting off the graph at a maximum of 60k votes with no explanation or even pointing out that you're doing it! I'm sure this is just an honest mistake, but it cuts out basically every movie that is actually popular!

This slowly ate at me until it was enough to get me out of bed to redo this graph myself. I've uploaded my quick and dirty result in R here: https://i.imgur.com/TTuCFEL.png

As you can see from the graph, there's a direct and obvious link between the number of votes and the average rating. The reason I say this threatens to undermine the whole thing is that I also know from previous experience that more recent films tend to have a lot more ratings on average. This (naively) suggests to me that we should see some recency bias if only because more recent blockbusters get a much larger number of votes than others and the sort of people who vote on blockbusters have different standards for what makes a great movie. (Avengers: Endgame was briefly the #1 movie of all time on the top 250 list.)

I don't have time to plot all the different graphs you did, but perhaps you can recheck your results after fixing at least this issue, and get back to us?

djtriptych• 7 years ago

I was recently thinking about this as Chernobyl (which was excellent to be fair) quickly became the highest rated TV drama ever.

KyleOS3• 7 years ago

A recent post on the same topic: https://news.ycombinator.com/item?id=20098900

watwut• 7 years ago

I mean, if you can suggest other similar shows I will be only glad.

iforgotmypass• 7 years ago

I feel like author is missing the point. Shouldn't the recency bias show up in recent votes (while the movie is "hot" and trendy) vs. later votes?

eoinmurray922• 7 years ago

I think it's been interpreted as recent movies not recent votes.

PaulHoule• 7 years ago

It's well known that movie rankings do. This was key to the resolution of

https://en.wikipedia.org/wiki/Netflix_Prize

The time series of reviews of movies like

https://en.wikipedia.org/wiki/Sucker_Punch_(2011_film)

and

https://en.wikipedia.org/wiki/Batman_v_Superman:_Dawn_of_Jus...

tend to have many early bad reviews by mainstream people who can't get past things that they find squicky and the lack of the usual "Hollywood Happy Ending" but that get better reviews from "true fans" later on who enjoy the fact that the directors tried to make something different even if it was flawed.

deanCommie• 7 years ago

Interestingly, the opposite of what happened with Star Wars: The Last Jedi.

Casual fans and critics loved the subversion of genre tropes, story roles, character archetypes, and plot points.

"true fans" reviled that the direction align with their vision for the franchise, and it sounds like Episode 9 will be a return to the comfortable and familiar.

slyall• 7 years ago

A quick Google shows a few sites that keep history of IMDB ratings. Unfortunately they seem to concentrate on movies actually in the IMDB 250

Some interesting ones:

Avatar - http://top250.info/movie/?0499549

Se7en - http://top250.info/movie/?0114369

Heat - http://top250.info/movie/?0113277

Star Wars: Episode VII - The Force Awakens http://top250.info/movie/?2488496

Terminator 2 - http://top250.info/movie/?0103064

Note: There seems to be a big shifts in movie ranks in May 2005 and July 2012, probably due to a change in the ranking formula.

olalonde• 7 years ago

Doesn't this analysis rest on the assumption that movies are equally good each year? Maybe I suffer from nostalgia (or survivorship bias) but I'm not sure this is a good assumption.

drollcrops• 7 years ago

Good is relative. And you could easily rabbit hole on this. Do people have the same sense of quality? Have people changed their rating scale based on production/effects quality? Etc etc. Probably best to just take the subjective rating.

jdsully• 7 years ago

The problem with past movies is you only see the good ones. I've spent some time watching the bottom barrel movies from the past and they seem even worse than Today's movies - because while both suffer from horrible plots usually modern movies at least have interesting visuals.

However for certain categories the past was most definitely better. Dramas being the main one. Without a lot of fancy effects the plot was the only thing you had to carry the movie. They also don't beat things over your head and can embrace subtlety in a way modern movies do not.

I also really liked the 70s trend of movies that didn't really have a satisfying ending, you just get to experience the lives of people who don't really improve. 5 easy pieces or The Last Detail fit that category very well.

EugeneOZ4• 7 years ago

So you want to say that Shutter Island is a bad example of drama? It's all just old-farts rants, nothing more.

Everything is evolving.

In old movies actors play often too dumb, just few of them have good examples of how to play a role.

Some "legendary" movies from the past would fail in theaters nowadays without their legendary reputation.

We have more movies right now, more choices, so our demands are higher - otherwise we will not have enough time to watch them all. Also, we have much more choices of how to watch them - age of digital distribution.

Higher demands give higher quality, movies evolve both in visuals and in acting art.

jedberg• 7 years ago

I've spent some time watching the bottom barrel movies from the past and they seem even worse than Today's movies

That's because back then we didn't have the internet. No matter how bad a movie was, people would go see it for at least a week before the word got out how bad it was. Looking up movie times was a pain, you had to plan ahead and check the newspaper. If you went on a date Friday/Saturday night, the most likely thing to do was show up the movie theater after dinner and buy a ticket for the movie that started next, since you probably hadn't heard of any of them, or you would buy a ticket that you'd seen the most advertising for.

There were no ratings to check, no reviews to read, unless you happen to read them in the paper that morning.

Nowadays, they know they have to produce at least something minimally good, because word will get out very quickly that something is bad, and it's super easy for people to check rating when they arrive at the theater, or even before they arrive.

baddox• 7 years ago

It seems pretty reasonable to assume that quality is independent of release year.

fileyfood500• 7 years ago

I see from one of the charts that the average movie rating from 2019 is over .5 higher than the previous 5 years, in what appears to be a big jump. Is it possible that recency bias is only relevant on a shorter timescale? It looks like ratings are higher for movies that are a few months old, based on the data. I wonder if this is generally true, or if early 2019 happens to have higher rated movies.

ac29• 7 years ago

Sure - for example, fans of a particular actor, director, or series are more likely to go see a movie soon after release in theaters as opposed to later via rental or streaming.

Fans are likely to rate higher than the general public, so over the first, lets say, 3-12 months, ratings often trend down.

justadog• 7 years ago

Fake reviews and ratings must be a factor. For big movie releases it is hard to find IMDB reviews that seem genuine. I assume good portion of the ratings are fake as well.

It must have an effect given how common it is.

in3d• 7 years ago

I wonder if recording ratings now and then every month or so would show a decline in ratings. IMDb must know already.

dahart• 7 years ago

Number of Movies By Ratings

I’m dying to see all of these as percentages rather than absolute numbers! I was even thinking that while scrolling down through, and then there the totals are right at the bottom. @KyleOS are you up for normalizing them?

mynewstartup• 7 years ago

In terms of shows, Sopranos has long been dethroned, possibly due to this effect. Then again, production values will only get better over time, so maybe the latest shows are of the best quality overall.

Best line:

Fucking Queeahs! (Paulie throws chair at his murder victim spirits)

bscphil5• 7 years ago

Incidentally, in case the creator sees this, this site is borderline unusable for me on mobile Firefox. Text gets cut off in strange places, I can't zoom out properly to see all of the animations. "Request desktop site" seems to fix some things but break others - at least I can zoom all the way out but the text becomes hard to read.

Situation might be the same on Chrome (didn't check), but regardless it could use some work since it's pretty broken as is.

eoinmurray922• 7 years ago

Hi - founder of Kyso here, thanks for your feedback we will get to work fixing those issues now

bscphil5• 7 years ago

Thanks! After playing with it a bit it's clear that part of the problem is that the charts are getting generated on the fly and that's very performance intensive. My brand new Android phone is just choking on the page, to the point I can actually feel the phone overheating. Perhaps static images would be an improvement for mobile devices, with the option to load the full data on request.

8innovate• 7 years ago

This is definitely not the 'recency bias' I expected.

I've always noticed that when a show is released it gets WAY higher ratings in the first week or so of the release date. I've learned not to trust the ratings until at least a 2 weeks after a release.

It looks like this analysis focuses on the year a film was released. This seems like it would be a lot harder to determine because film quality changes (subjective) and the audience writing reviews is probably changing over time.

ajhurliman2• 7 years ago

An alternative way of reading the same data: "Analysis of whether movies are getting better over time"

richardhod• 7 years ago

The problem with your formulation is that 'better' is perhaps the most subjective word there is. Define it in measurable terms, and perhaps you have a reading there. There are many factors to consider which may lead you to find things less clear-cut, for example here's one or two, with some sub-factors - Modern audience Preference for familiar patterns in: (eg) -- acting styles -- music used -- visual tricks and norms -- special effects styles -- acting styles -- soundtrack loudness -- colour and high resolution -- High Concept movies -- Fantasy and Comic-related films -- modern movie themes and topics - Internationalisation of movies, so that fans from abroad prefer modern films which are more universalised and less American-specific (see literature)

ajhurliman2• 7 years ago

Right, I'm not asserting that my alternative title is the correct one, I'm lampooning the assertion that IMDB is "suffering from recency bias" as opposed to a more neutral claim like "ratings for more recent movies are higher."

Answering the question of _why_ they went up is a lot harder to prove.

ddffre• 7 years ago

Of course, there are plenty of examples of that. Like the assassination of the NK leader movie, which at first received 10 star rating. Now look at it.

Sane goes with all the new blockbuster movies, you should usually wait awhile for the real ratings to show.

z3t4• 7 years ago

What I find most amusing is ratings (and also awards/nominations) before a movie has been released.

IAmEveryone• 7 years ago

Series tend to be rated higher than movies, probably because people that don't enjoy them stop watching and rating after one or a few episodes.

That effect may well be spilling over into movies, considering far more movies are now sequels.

ravenstine• 7 years ago

Great job! I'd love to see a similar analysis for Rotten Tomatoes since there's separate metrics for critics and audience, and I find RT ratings to be more extreme than IMDb. I'm often surprised to see how many bad movies get a rating of ~6 on IMDb.

Analysis of whether IMDB ratings suffer recency bias

Related Stories