Grand Prize Awarded To BellKor Pragmatic Chaos

http://www.netflixprize.com//community/viewtopic.php?id=1537

by physcab • 17 years ago

85 18 17 years ago

netflixprize.com

mildweed• 17 years ago

Did they figure out the Napoleon Dynamite conundrum?

physcab• 17 years ago

I did a search on Google, but couldn't find the reference. Care to share?

arfrank• 17 years ago

From what I remember reading about the contest, the problem was that when comparing movies and using movies that were similar to others as a way to suggest movies people either hated or loved Napoleon Dynamite and it didn't match up at all with their previous movie ratings. Thats how I understood it at a basic level.

bayes• 17 years ago

The NY Times had an informative article: http://www.nytimes.com/2008/11/23/magazine/23Netflix-t.html?...

robg• 17 years ago

Papers here:

http://www.netflixprize.com/assets/GrandPrize2009_BPC_BellKo...

http://www.netflixprize.com/assets/GrandPrize2009_BPC_BigCha...

http://www.netflixprize.com/assets/GrandPrize2009_BPC_Pragma...

tmsh4• 17 years ago

so why didn't the ensemble win? in the rules it says, 'At the end of this period, qualifying submissions will be judged (see Judging below) in order of the largest improvement over the qualifying RMSE on the test subset.' this isn't the icfp. i assume the person on the leaderboard is in fact the leader.

the difference between the 'test' and 'quiz' sounds b.s. to me. at this rate, i know i won't even be contemplating netflix prize 2. at the very least, netflix owes it to the community to explain why they (at this point, seemingly corruptly) made the decision they did.

i suppose they say they'll post the final 'test subset' scores on the leaderboard. it doesn't appear to be a very 'open' contest if they don't actually publish exactly what the test subset is and how the rankings are determined. when it's that close, everything should be shown to be exactly done within the rules. otherwise, they really risk delegitimizing the whole contest, imho.

pmjordan• 17 years ago

I'm not familiar with the details in any way, but it sounds like you could probably produce an algorithm that produces a perfect result for the rest if you give it a subset of the publicly available data. That doesn't mean your algorithm is good in general, that just means you essentially encoded the test data set into your algorithm. The real test is always going to be how well you do on data that you've never seen.

As for why the intermittent leaderboard shows results based on the public dataset: using a specially crafted algorithm, you could leak information about hidden data through any feedback you get from the test environment. Such information could be used to create an aforementioned algorithm that does well in this specific case.

tmsh4• 17 years ago

i see. but if they don't make their private dataset public, what's to prevent netflix from fitting it to a particular entry?

AGorilla• 17 years ago

This is great and all, but it still thinks I'll love Once: http://www.imdb.com/title/tt0907657/

pronoiac• 17 years ago

I'll repost from another thread about the NYTime article, http://news.ycombinator.com/item?id=834681 : The next contest info surprised me:

"The data set of more than 100 million entries will include information about renters’ ages, gender, ZIP codes..." It's enough to identify 87% of the people, apparently: http://www.freedom-to-tinker.com/blog/paul/netflixs-impendin...

I hadn't realized that someone identified some of the raters in the previous contest with their imdb ratings: http://www.cs.utexas.edu/~shmat/netflix-faq.html

Also, why this matters, page 44 of a research paper (PDF): http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1450006

Grand Prize Awarded To BellKor Pragmatic Chaos

Related Stories