When the 'Wisdom of Crowds' turns on itself: IMDB Edition

http://news.cnet.com/8301-13515_3-10000650-26.html

by nickb • 18 years ago

39 62 18 years ago

news.cnet.com

Tichy• 18 years ago

Maybe the majority of people would indeed get more entertainment out of "The Dark Knight" than out of "The Godfather"?

william426• 18 years ago

You're right.

However, if "The Dark Knight" bumping off "The Godfather" was all that was happening, that wouldn't be a problem.

What's happening, though, is that the release of "The Dark Knight" made "The Shawshank Redemption" have a higher rank than "The Godfather". This means that the ranking system has lead to the rating system being used strategically.

The implications of this for approval voting would be an interesting corollary. The polls could influence themselves.

wmf3• 18 years ago

If you allow people to vote after they have seen the rankings, you have the wisdom of mobs. The only way to get the wisdom of crowds is to ask people before they have seen other people's opinions.

breck3• 18 years ago

But it only takes 1 spoiler to publicize the results to ruin it for most everyone else.

Hexstream2• 18 years ago

I upmodded your post because it already had +9!

cstejerean• 18 years ago

I wonder what would happen on this site if the comments and stories were still sorted by points but the actual number was hidden (or perhaps only hidden until after you vote but I don't see why they need to be there period).

noonespecial• 18 years ago

It seems like in this case we have a temporal error. A popular movie generated buzz now and skewed a result that people pronounce loosely as of all time.

Batman might be the best movie out recently but in 2 years time, it will be a dim and fading memory while the Godfather will retain its appeal as it has already proven that it can.

IMDB just needs a temporal filter to keep the latest trend from skewing the universal result.

breck3• 18 years ago

This mob mentality, brought on by a significant event (in this case, the release of The Dark Knight), causes a total breakdown in a system that is otherwise fairly accurate, useful, and trustworthy.

I wouldn't call this a "total breakdown in the system". I would call it a minor blip. I mean, ultimately the system recommends what movies to watch. If someone isn't going to watch the Godfather now that it's ranked #3 instead of #1, then the person is flawed, not the system.

Otherwise, this was an interesting post. I have been astonished that TDK is #1. Didn't realize people were also downvoting other movies to keep it there.

william426• 18 years ago

The flaw in the system isn't that The Godfather was bumped down. The flaw is that people are rating the Godfather down to the point at which it changed its ranking with respect to other movies.

And you're right, it's not a major flaw for a movie ranking system. But let's suppose something similar to the IMDB system(which would be an approval/range voting system) is used in an election, and the first place gets President, the second choice getting Veep. With an election, you don't have real-time rankings like IMDB, but you have something similar--polls. And that's even worse, since there's a few points of inaccuracy. Strategically worded polls could change the results of an election.

unalone5• 18 years ago

But that change is a blip in and of itself. Godfather has been bumped down before. Happened with Lord of the Rings. It climbs back up after the initial wave.

brokenladder• 18 years ago

Score Voting (aka Range Votiing) and Approval Voting are actually the best voting methods, especially when you take strategic voting into account. http://rangevoting.org/StratHonMix.html

In this case, even if the election result changed, you'd still get a candidate who was rated relatively well by the average voter, so the results would not be that much worse. Now consider that other voting methods react much worse to strategic voters, _and_ that score voting is as good with a great number of strategic voters as other systems are with 100% honest voters, and this whole argument falls flat.

nickb2• 18 years ago

If it's a minor blip, how do you propose they now fix the problem? I see it is as so systemic and flawed that the only solution is to scrap it altogether.

Harj• 18 years ago

so a system that has worked fine for over a decade should be completely scrapped because of one unprecedented event?

attack• 18 years ago

The point of top-k rankings is to put items in the proper order. As far as the crowds are concerned, this is no failure. Sorry your favorite movie lost its spot.

bprater3• 18 years ago

It is fine for "FanBoys" to push Dark Knight into number one. But downvoting Godfather to put your movie into the #1 spot is plainly wrong.

unalone5• 18 years ago

But the Godfather people did it first. Look at the spread of points on The Dark Knight. Look at the 1s. Those came first.

andreyf• 18 years ago

That's funny, because the book which coined the term is about exactly this - how to make Wisdom of the Crowds work.

wmf3• 18 years ago

The problem is that people discuss these concepts without reading the book and thus miss out on the subtleties. The long tail is another common victim.

Disclaimer: I didn't read the book.

davidw• 18 years ago

I'm (slowly) working on the summary here:

http://www.squeezedbooks.com/book/show/23/the-wisdom-of-crow...

Contributions welcome!

greenagain• 18 years ago

IIRC, the same thing happened at the releases of Spiderman and the Royal Tenenbaums. Hype does funny things to men.

pg2• 18 years ago

Have they ruled out the possibility that the people marketing the movie are gaming the system? Even if voting was restricted to established IMDB users, surely a reasonably well connected Hollywood marketing person could round up a few hundred of those among friends and friends of friends.

rms• 18 years ago

http://www.imdb.com/title/tt0468569/ratings is the vote breakdown.

pg2• 18 years ago

Hmm, those numbers are definitely too big to be influenced by anyone's network. Judging from the other top-ranked films, it looks as if IMDB's userbase is in the middle of changing.

wheels• 18 years ago

IMDB's ranking system is one of the least sophisticated out there, which is surprising given that they're owned by Amazon these days. I assume it's still just the same couple of guys that started it left to run it in relative independence. You really don't need to game their system much; it's just bad technology.

iamelgringo• 18 years ago

I think that the problem is similar to the eternal September problem. I'm sure that the people who had been voting and contributing to the ratings on IMDB had been involved on the site for a while. Now, there's a big influx of people on IMDB that are voting for their favorite movie. I bet that many of the people who are voting Dark Knight up have never seen the Godfather or Shawshank.

Same thing happened to my wife in film school. She went to film school because she loved the classics of the 30's - 50's. She was stunned to find out that the kids she was going to school with thought that Austin Powers was the best film ever.

ericb• 18 years ago

Wisdom of Crowds was a good book. The sequel, Attention Span of Crowds, was significantly shorter.

bprater3• 18 years ago

Perhaps this is an advantage to Y!Hack not having a downvote button.

Hexstream2• 18 years ago

I'm much less worried about that happening here because of the transient nature of the site. I don't think potential manipulators would find it worthwhile.

bprater3• 18 years ago

Should be easy to cull these votes from the system. Anyone voting Dark Knight at 10 and Godfather at 1 should get nixxed from the system.

In fact, anyone voting Godfather less than 8 should be nixxed.

delackner• 18 years ago

This reminds me of the recent articles about the netflix optimization challenge. Voting systems like imdb might be improved with some simple heuristics that take into account the absurdity of some people's votes.

If a user makes votes that are < 2.0 for a movie that the entire community for years has rated above 9.0, then probably that user's votes should all be ignored, because they are totally uninformed.

william426• 18 years ago

Because having unpopular taste is bad?

thwarted• 18 years ago

What if you assume everything is average and you give more weight toward the finally rating for those individual ratings that are further from the norm? This means people couldn't game it by following the crowd, following the crowd gives the entire crowds rating less weight.

unalone5• 18 years ago

See, but it's all about opinion. Some people honestly like Dark Knight better, because it's far more gripping and easier to get into. And those people have every RIGHT to give Godfather a 1, if people that gave Godfather a 10 are giving The Dark Knight a 1 in return. The Godfather people started this days before the Dark Knight people came in.

dangoldin• 18 years ago

Well anyone voting Godfather at 10 and Dark Knight at 1 should also be nixxed then.

Why not just let the two cancel each other out?

bryanlanders• 18 years ago

If you think of this system as a game and in this instance a number of users have figured out a way to act within the rules of the game to achieve an outcome inconsistent with IMDB's intended one, then IMDB can simply change the rules to get what they think is best!

It wouldn't be too difficult to observe the behavior of the offending members (vote for TDK, within x hours vote down Godfather...) and simply setup criteria to disallow these actions. You could also set up an algorithm to detect screwy behavior to alert a developer or block that action for a duration.

And hey, we throw error messages for repeated button clicks and such so that click-happy users or spam bots don't take down the system...can't that kind of precaution be extended if necessary? "We know you're in love with TDK, but do you really hate The Godfather...come back in a month if you still do!"

william426• 18 years ago

Soon you get websites with people saying "Rate The Godfather 1 before rating TDK 10."

MikeCapone• 18 years ago

AFter the first weekend, Dark Knight had 9.7 average with almost a hundred thousand votes. That's just crazy.

unalone5• 18 years ago

Nah. 9.7 was with only 3,000. 100,000 only hit a few days ago.

rrf• 18 years ago

The problem is not the Dark Knight fans, so much as the algorithms being used to rank the movies. How can you take advantage of crowd wisdom if you don’t control for spoofing, spam and mob voting? It’s surprising that there wasn’t already in place a process for dealing with such a common phenomenon.

Erwin• 18 years ago

Blockbusters popular with the avid Internet users have always started at incredibly high ratings only to slowly fally to the right place. The LOTR movies were in top-10 once too.

I suppose IMDB might try to account for the initial voting wave being raving fans and dampen the initial votes, but why would they? Personally I've reloaded the "Dark Knight" IMDB page half a dozen times to see if it was still at #1.

The most popular topic on the IMDB message boards is "This does not deserve to be in top 250" or "Why is this not in Top 250 why some-other-movie is not?". People like to keep an eye on rankings and to comment on them.

The main practical use for the ratings IMHO is to get a list of movies within each genre sorted by the rating, to see if you missed something people thought was great.

ragaskar• 18 years ago

exactly -- imdb has an internet and populist skew. As long as you take this into account, their ratings are fine and predictable. Gaming rankings is practically an official internet pastime, so it's not unexpected that this event is taking place, although I'm surprised it built up enough steam without (presumedly) official co-ordination to drop the Godfather a few places -- that is notable. I'm sure politicians are taking notes as we speak.

johnyzee• 18 years ago

I don't see why it wouldn't be possible to build into the system some of the same logic that obviously makes it clear to everyone that there is a problem with these votes.

A computer might not be able to appreciate the goodness of The Godfather, but it can be taught that a movie that has consistently been voted great for decades, does not suddenly fall out of favor in a month.

flipbrad• 18 years ago

over time it should return to a 'fair' position, once temporary disequilibrium in the 'crowd' (induced by marketing, hype, whatever) is resolved. It's a shame people vote 'strategically', not to reflect their true sentiments - but whatever. Over time, with a diverse enough crowd, 'wisdom of the crowd' systems are effective and reasonably unbiased.

doubtless IMDB will enjoy the free publicity.

Eliezer• 18 years ago

I've seen The Godfather. I've seen The Dark Knight. Now, maybe I'm just missing the point of that Godfather thing, but I'd say The Godfather had more dramatic tension and The Dark Knight had more philosophical depth.

No, stop laughing, I'm serious. Tell me what point The Godfather makes.

snorkel2• 18 years ago

Leave the gun. Take the cannoli.

pageman• 18 years ago

Michael: Fredo, you're my older brother, and I love you. But don't ever take sides with anyone against the Family again. Ever.

snorkel2• 18 years ago

I'm astonished. The internet hordes usually have impeccable taste in film and music. "Beverly Hills Chowahwah" is not even in the top 20?!!

When the 'Wisdom of Crowds' turns on itself: IMDB Edition

Related Stories