A better way to rate films

http://blog.goodfil.ms/blog/2011/10/07/a-better-way-to-rate-films/

by bootload • 15 years ago

79 40 15 years ago

blog.goodfil.ms

jonkelly• 15 years ago

I definitely liked the personality that came through the writing. "On the other end of the scale, I think Black Swan is a terrific film, but I’d sooner pull my nails out than watch it again." - well said and a good illustration of the value of the dual scale.

phzbOx2• 15 years ago

Interesting post even though I'd admit that I'm not a huge fan of "I'd watch it again". For instance, some movies are really hard (huge drama, etc.), or particularly long, or even have a huge punch at the end. These movies would have a "rewatchability" really poor.

One might also watch a very simple movie.. nothing extraordinary that you wouldn't necessarily watch again; that doesn't mean it's not a good movie to watch for the first time.

Personally, I think the best factor to "guess" if I would watch a movie would be to anonymously analyse a huge set of data and find people who like the same stuff as I do. So, he likes x,y,z (as I do) and hate a,b,c as I do, good.. we mostly like the same stuff. Now, he really enjoyed K; I'd probably also like it.

phzbOx2• 15 years ago

(Note: This is different than what Amazon does with the "People who bought this item also bought these items".)

ph0rque• 15 years ago

One thing that movie ratings don't take into account, including this one, is: what am I in the mood for? I'd like to type into netflix, "I'm in the mood for James Bond meets Spirited Away", have it perhaps give me a few yes/no choices, and make recommendations based on that.

westicle• 15 years ago

I can't imagine too many people who wouldn't like that system. If only there was some way to introduce this idea to some kind of tech start-up community.

socratic2• 15 years ago

This is the main thing Jinni claims to do.

Have you tried it?

http://www.jinni.com/

teamlaft2• 15 years ago

Thanks for this - never had heard of jinni and after playing around for a few minutes I already found some awesome new movies that I wanna check out

daeken• 15 years ago

I also strongly recommend http://tastekid.com/ . The recommendations are great, although it tends to give them very narrowly (very similar movies, rather than related movies you might enjoy but are in a different style). The name is also very unfortunate, but I love it.

robryan• 15 years ago

Well I think in this case, all you would need are some categories or tagging in goodfilms. Tagging would be kind of cool going past genres to things like mood, like a "easy watching" tag or something.

seanalltogether• 15 years ago

I've thought about this problem a lot and I'm glad to see there is a site out there trying to tackle this problem. For me, I always thought about movies in terms of investment with 4 options, "I'd buy it", "I'd rent it", "I'd watch it for free", "I'd avoid it". I think dollars speak louder then opinions. The problem is that nowadays in the world of streaming video, this scale gets completely trashed as the friction to invest in a movie is lowered considerably.

I do like the idea of production value vs replay value, I remember when I saw Passion of the Christ I told a friend it was an amazing movie i never want to see again.

socratic2• 15 years ago

I really want a good movie recommender system. (Which I suppose makes this a good idea to work on.) However, focusing on these sorts of issues seems like exactly the wrong way to go about it.

I understand that people overwhelmingly tend towards the extremes of rating scales. But who cares? There's so much noise anyway.

Here's what I want: (1) include (all) films and television series and (2) include top 1000 lists for as many niche categories as possible (foreign films about relationships, teen comedy, and so on).

I don't even care if it is personalized, because the personalization is always horrible. (I've tried Netflix, Jinni and others and they're all terrible.) Just having the average for the movie is actually amazingly accurate on many measures (within 10--20%).

Is there something like this? The closest I've gotten so far is finding a good IMDB user list or two.

covercash• 15 years ago

While not exactly what you're looking for, I find myself going back to http://nanocrowd.com quite often for movie recommendations.

They don't do TV shows, but their niche categories are usually pretty solid.

It works well if you can give them an example of the type of movie you feel like watching. For instance, I wanted to watch a cheesy, nerdy, thriller earlier tonight, something along the lines of Hackers and it suggested The Net, Antitrust, TRON, Live Free or Die Hard, The Matrix, Swordfish, etc. Exactly what I was in the mood for.

thirdsun• 15 years ago

edit: I just noticed that you already know jinni. sorry.

I use jinni.com - it is basically last.fm for movies and tv shows. The site could need a better user interface, but the movies are very well tagged, which enables you to dig deep into niche categories. For example you are not only able to filter by genre (drama, comedy, etc.) which is very general, but instead there are filter options for the plot too (corruption, love, drugs, mind-bending,...).

Also their recommendation engine works quite well and gives you accurate suggestion which are not that obvious or popular.

So, while not cutting-edge, jinni.com is probably the closest to the movie discovery service i wish to exist.

Pahalial• 15 years ago

This seems useful, yet awfully much like another step on the slow path to legitimizing 'poor taste'.

That's a bit harsh, so let me qualify it. I actually rather agree with the division inherent in this and think it will resound with people, but I honestly believe that this is yet another step towards turning our society into idiocracy. It's the slow legitimization of appreciation for content that we acknowledge as garbage but consume anyway. I firmly believe that there is inherent social value in this garbage content remaining quasi-taboo - a social pressure not to like such things. Sure, most people will watch some anyway, or play the latest mindless shooter, or what-have-you, but so long as it's understood that most people will mock you for publicly liking this, the norm does not drift towards mindlessness - you still need to consume some legitimately good content, even if only to have something to talk about.

I don't mean to attack the site; I think it's a great way to slice into the recommendations game, and I do think a lot of people will get value out of it, and it seems like it may well succeed. I simply am afraid it's a net negative for our society in the long run (then again, of course, not launching is hardly going to reverse such a trend.)

Edit: Make that symptomatic of a net negative. I've thought some more and this is only giving an online portal to express the shift I talk about above, which has already been happening aplenty.

recursive• 15 years ago

I think the alternative is people pretending to like things they do not. You can't force people to appreciate your brand of high culture by mocking them for not doing so. At best, the people you're trying to win over will publicly pay lip service. At worst, it will become more difficult for people like you to find things that you think are worthwhile, because the space will become polluted by people who don't get it. Let people like what they like.

I, for one, am not ashamed to say that I like stupid things.

thefool• 15 years ago

I disagree, I think there are plenty of social groups where this type of media isn't looked down upon at all. Same with shooters... ect.

If your goal is to chase immediate gratification (as many a person seems to do), you'll be better served by metrics that reflect this aspect of things, instead of deciding to spend time doing something you think will be immediately gratifying, and getting something else in return.

Good for society? Decidedly no.

Unavoidable because it better serves demand, I think probably.

At the same time... I think if these sorts of metrics become ubiquitous, people will become more acutely aware of the distinctions between quality and gratification value... so I don't think these things are only a symptom.

wanorris• 15 years ago

I have a job that I put extra time into, a longish commute, and two enthusiastic boys. When my wife and I get time to watch a movie at all, we want to watch an entertainment, not high culture. I'm pretty sure that historically, the primary consumers of high culture have been the independently wealthy and people who don't have young children that they raise themselves.

I have an Ivy League liberal arts degree, I know what high culture is, and I'm actually quite fond of it, on the whole. But I won't apologize for what I feel like watching these days, and if you try to stigmatize it, I'll laugh in your face and carry on doing what I feel like.

wvl2• 15 years ago

I really disagree with the premise of this post. Emotional impact (United 93), psychological impact (Black Swan), intellectual demands (Primer) all equate to a lower rewatchability score, since watching the movie takes a lot out of me. However, the more of an impact a movie makes, the higher my opinion of a movie. Because something is light and fluffy doesn't make that movie "better", it just means it's easier to watch.

Rating films this way may do a good job of the "so bad it's good" camp (Snakes on a Plane), but will do an incredible disservice to movies like Black Swan.

glen_goodfilms2• 15 years ago

This is a really important point, and it'll probably justify its own blog post soon, but a quick response would be this:

I'd class United 93 and Black Swan as being, sort of, emotionally exhausting. And yes, I'm not going to rush out see them again. But that's very different to saying they're a bad film. I would suggest that, at high levels of quality, low levels of rewatchability makes it actually a better film, in a way.

Or, at least, it differentiates them from another terrific film like The King's Speech. If you're the kind of person who likes watching beautifully constructed but largely enjoyable films, your tastes will tend toward the top-right of the graph. But, if you enjoy more difficult or harrowing films, your favourite films might be lower on the rewatchability score, on average.

My point is, while quality is quality, a measure of skill and talent, rewatchability is personal, and groups similar films together in interesting ways. Where your favourite films come out on the graph is really not important to anyone but you, when you're looking for more stuff you like.

You may still disagree, but thanks for taking the time to comment :)

-glen.

wvl2• 15 years ago

I guess then, it depends on execution. If you're combining both of these numbers into one single metric of "goodness", then you'll just be accomplishing Pahalial's point of "legitimizing 'poor taste'". If, however, you can expose this information separately and well, then perhaps it will add value.

Personally, I'd like to break down the "rewatchability" metric into more components, to really get at the heart of the matter, like others here have commented. Pace, quality, fun, emotional impact, length, genre -- these are the things that really matter. But now we're getting into movie geek territory.

Speaking of movie geek -- I can't believe you left out the most interesting thing in that blog post. Where can we see a breakdown of what movies are where in that scatter plot? I want to know what those outliers on the high quality / low rewatchability scale are!

teamlaft2• 15 years ago

Glen it basically comes down to a genre by genre breakdown I think. If you're looking for a comedy which should inherently have high rewatchability, if any are rated as low then I can assume I should avoid them. For example, in college my buddies and I alternated watching 40 Year Old Virgin and Grandma's Boy for an entire semester. Neither would have the best quality ratings, but they were fun and sure had high rewatchability.

Now take the other end of the spectrum of film like Black Swan or even some slower paced films. I love 2001 A Space Odyssey but am not rushing to rewatch soon (though it has top quality) because it is way too slooow for me. Or take some dramas that while interesting on first glance don't really merit another watch.

Basically I think you should start with a top down approach by genre, and if something like comedy then the dominant factor is rewatchability but then for others it may be "quality" (which itself could be subdivided further into cinematographic elements for an auteur film, the musical score for a certain type, etc). But I feel like maybe you end up going down a rabbit's hole, so perhaps I'm asking more questions than I'm answering.

eridius• 15 years ago

I think this is why they have both ratings. A quality rating and a rewatchability. As the site appears to be in private beta I can't play with it directly, but I would assume both ratings are exposed when you look at a movie, which will help you decide if you want to watch it. For example, if I come home on a friday night and I want something light and enjoyable to watch, I definitely don't want to watch Primer no matter how good it is. I'd instead go for something with a high rewatchability score. But if I'm in a mood for a "truly good" film then I can go for the quality rating score and ignore rewatchability.

tryitnow• 15 years ago

I think that's certainly true for some people (e.g. you), but just the opposite is true for me.

I would like a meta-rating system where I could select the rating algorithm that would work for me. That would be nice.

natesm• 15 years ago

I think a problem with "rewatchability" is that it can vary drastically with the amount of viewings. I'm going to substitute Aliens for Transformers, because "watchable" is not a word I would use to describe Transformers.

Aliens is very rewatchable. It has tense but predictable action (the aliens kill everyone, "get away from her", etc.). Sure, it's best the first time, but Aliens only ceases to be rewatchable when you get tired of watching Aliens.

Films like Primer, Fight Club, and The Usual Suspects, are very rewatchable once, with a sharp decline afterwards. The Soze/Durden effect is incredible once. The next time, you want to watch very closely to catch all of the obvious hints that you missed. You might still miss some, but not many. By the fourth time, it's almost pointless.

Natsu• 15 years ago

It's funny, because it makes me realize that for me, "rewatchability" is in large part all about catching all the details I missed the first time around. But I'm not sure if it's like that for other people or not. And it doesn't necessarily mean that I didn't enjoy it at least once. Some things are only good once. After that, there's just no point in watching them again.

InclinedPlane2• 15 years ago

Indeed. I could rewatch some movies effectively an infinite number of times. But that doesn't mean I think they are the best films, or the films I would put at the top of the list of recommendations for a friend.

nostromo• 15 years ago

Another issue with five star ratings is that visually you would suspect that a 50% rating is 2.5 out of 5 stars.

However, the 50% rating is actually 3 stars. This is obvious when you list out your star options: 1,2,3,4,5. 3/5 stars visually appears to be better than 50%, but is actually exactly 50% due to the inability to choose 0 stars.

(This seems unresolvable as most people now recognize the image of 5 unfilled stars as null rather than 0%.)

cpeterso• 15 years ago

The graph of film "quality" versus "rewatchability" looks pretty linear to me. Why maintain two rating criteria if data shows they are practically correlated 1:1?

ComputerGuru• 15 years ago

I posted this on the site, posting it here too:

I would say a better metric would be "How much did you enjoy this film?" vs "How good is this movie?"

I've been grappling with this problem since subscribing to Netflix, wondering how I should rate a movie I personally did not enjoy that I also felt was incredibly well made. For example, "A Clockwork Orange" is an incredible movie by all means and I would give it 97 stars out of a hundred.... but I hated every single second of it. It's gruesome, sadistic, dark, miserable, and disgusting. But it was undeniably well made.

Rewatchability sucks in the aspect because many movies can't be rewatched, while still being both splendidly produced and enjoyable. Let me make this clearer with examples: "The Sixth Sense," "Rear Window," "The Accused," "The Diving Bell and the Butterfly," "12 Angry Men," "Apollo 13..." The list goes on and on. There are great movies that just don't rank so highly because they're SUSPENSEFUL and that suspense doesn't translate will into rewatches. Rewatching suspenseful movies will make them seem boring and long-drawn.

Rewatching feel-good movies ("Seven Pounds," "How To Train Your Dragon," "Blind Side," etc.) however, is just as good the second time if not better.

tl;dr the rewatchability index will be heavily skewed towards feel-good favorites and epics, and away from dark, gritty, mysterious, and suspenseful. Not a good index.

mkolodny• 15 years ago

I think that "rewatchablility" is an important metric, but it's just one of so many different metrics that people use to judge what a great movie is.

As you said, there are plenty of terrific movies for which I wouldn't be upset if I never saw them again. Yet, would I recommend them? Yes.

Movies are so wonderfully complex that there is no single metric by which we can judge their credibility. We can only state our opinion of them, and leave the judgement to the next viewer.

wernercolangelo• 15 years ago

Discolsure: I am a co-founder for a movie recommendation site called tellmetwin.com so I am biased towards our own rating system...

I have never considered the rewatchability criteria as a potential factor in rating a movie, but must admit that is has appeal. But thinking about it further, it is a very personal metric so would not be very useful in determining the overall watchability of a movie. I.e. if you consider a movie rewatchable it might not correlate with me considering it to be watchable.

I would love to be able to see a plot of individuals and how their ratings cluster (some interesting visual way of adding and removing individuals would be cool) because I would guess (based on our own data) that these plots would not be uniform. And if they are not, then an individual's movie recommendation, based on rewatchability, would have less value to me.

rickmb• 15 years ago

This is purely a way to rate movies for entertainment value. Which means although the method may be better, the end result is very likely to barely differ from the current abysmal state of IMDB ratings.

InclinedPlane2• 15 years ago

It's still broken, just differently (maybe better) broken.

Films, music, books, etc. are not about some abstract universal gradation of quality that a large enough, good enough statistical sampling of viewers will reveal with increasing precision. No, they are art. Enjoyment and appreciation of art is a matter of personal taste, which is not, will not, and cannot be universal for everyone.

That's why up/down votes are decent enough ratings systems despite their crudeness, all they do is tell people whether or not taking the effort to find out if a piece of art suits your personal taste is likely to be worthwhile.

Ultimately what ratings are trying to get at is recommendations. The simplest idea being to measure the average opinion of everyone and present the highest rated works as recommendations. However, more sophisticated systems which try to find people with similar taste and recommend to you things that they seem to like but you haven't been exposed to are likely to be more successful, though significantly more complex to implement. It seems unlikely that even that represents the limit of improvement for the problem, but nobody's done much better yet.

billmcneale• 15 years ago

Any rating system works great if all the ratings come from people with similar tastes as yours.

tellmetwin2• 15 years ago

spot on bill, in a nutshell.

tellmetwin2• 15 years ago

I like the term "rewatchability" but I don't agree with your logic.

Why would you limit your pool of advice-givers to your friends?

They are good for service advice where you need someone you can trust to give you the whole and honest story.

In movies it's beyond that, it's much more important to find people with the closest match to you in movie taste as possible. The more people the closer taste match you will get (goes without saying).

I also don't see the point of giving "quality" scores. I really don't care if a stranger thinks some movie was of high or low quality. I'd much rather know if my taste-twin loved it or hated it. I for example thought Lord of the Rings was super well done and yet super boring. But this opinion of mine only matters to someone who resembles me in movie taste, who hates and loves the same quirky things as I do about movies. And of the two opinions i mentioned (well done, boring) its really only the latter that matters to my twin, he/she should pay most attention to that when deciding if to go and see the Lord, not if I thought the movie was of high or low quality.

But having said that, I like "rewatchability" as a concept to study further. I too would rip of my nails before seeing Black Swan again but I still thought it was a masterpiece. However, I have seen Requim for a Dream (Darren's Arrenovsky's previous movie) again and again although it was also very emotionally intense like the Swan. Just in a different way, less depressive, but killer strong in an exhaustive yet positive way. And then.. yet and still.. I would recommend both movies to everyone I meet. I think both are a must see even though one of them I would never want to see again.

pbreit• 15 years ago

Surprisingly doesn't mention IMDB which has by far the best general purpose ratings.

Curious if they considered "Would you watch this movie again? [yes/no]"

I also wonder if NetPromoter could be adapted for rating things like movies. It uses the question "Would you recommend..." and has a much harsher formula (+1 for 9/10, -1 for 0-6).

dexy• 15 years ago

I think rewatchability is a great metric for one thing: whether or not someone is going to want to rewatch something...

Just because the data better fits your desired skews, doesn't mean it gets at what the users actually want.

I think a major issue at hand is that people often rate things as they expect a critic to rate them. If you would give Transformers 2 a 5-star rewatchability rating, you should be giving it at least 3 or 4 stars for quality, because obviously you enjoyed it very much. Instead, many people pretend to be film critics, whose jobs are very different, and assign an 'objective rating'. A film critic can not give personal opinions because he's supposed to speak for the masses--to appeal to some higher taste he aspires to have and that he hopes society would have. If you loved Transformers, rate it 5 stars--period. The rating isn't meant to be read by others, it's not meant to appeal to an idealized world, it's just how you felt about the thing. Convince your users to rate intelligently with that mindset and you'll start to see good data.

All that said, more data is better than less, and if you could convince your users to double their average rating-time investment and give you ratings for both, awesome. I'm skeptical that users would bother to rate two metrics for everything they've seen...but I think it's certainly possible they would. Be interested to see.

A better way to rate films

Related Stories