An open letter to Netflix from the authors of the de-anonymization paper

http://33bits.org/2010/03/15/open-letter-to-netflix/

by randomwalker • 16 years ago

89 65 16 years ago

prakash• 16 years ago

fyi: randomwalker is Arvind Narayanan.

bmickler• 16 years ago

P.S. - BTW, when do you expect to allow your linux-using, paying customers to watch your instant streaming movies online?!

(off topic rant, I know. goodbye karma!)

nkurz6• 16 years ago

I'm saddened to see someone gloating at having helped to prevent the release of a dataset that I see as beneficial. Netflix offered an unprecedented corpus for research, and now someone is proud about helping the lawyers to lock it up. I think I just have a fundamentally different sense of privacy than the author. I think this comes out most clearly in their FAQ:

  Furthermore, even if the algorithm finds the "wrong" 
  record, with high probability this record is very similar 
  to the right record, so it still tells us a lot about the 
  person we are looking for.

So the "violation of privacy" occurs even we don't actually reveal information about the individual, even if we only provide a framework for making predictions? So if I publish a study (with backing data) that says that 38 year old males are likely to commit adultery, I've "violated the privacy" of all 38 year old males?

Could someone who shares the author's worldview try to explain it? I've tried, but I just don't see it.

randomwalker3• 16 years ago

'Gloating' is so diametrically opposed to the view expressed in the article that I have no idea how to respond.

As for that part of the FAQ, it is intended as an explanation of some of the theorems proved in the paper and is a response to some of the theoretical objections we face from the data privacy community. It is not an issue that arises in practice.

pbh4• 16 years ago

I think the parent is referring to an element of we-told-you-so in this letter. I think around the "instead, you brushed off our claims" part of the letter. It sounds a bit confrontational, though perhaps it's too late to rewrite.

dasil0032• 16 years ago

This is the part that did you in:

Instead, you brushed off our claims, calling them “absolutely without merit,” among other things.

Now even if you are all emotionless, 100% objective researchers with no interest other than the greater good, this very sentence will make it impossible for humans to read your letter without implying a certain level of snark.

If you want to project pure motives for the letter then it would have been best to leave their reaction to your original research out of it, or, probably more appropriately, don't publish an open letter at all—contact them directly.

nkurz6• 16 years ago

Ironically (and I think this is a legitimate use of the term) I hadn't realized when writing my comment that 'randomwalker' and 'Arvind Narayanan' were one and the same. But I apologize for my misinterpretation. I've read it again, though, and still have to fight the same reaction. Perhaps I'm just prejudiced against the 'open letter' format, as I often see it used in that way.

Still, if you can overlook my misreading, I would love your response. I understand that the example from the FAQ is not central to your argument, but I do think it's important in understanding your worldview. If I provide a tool that can be used, erroneously or not, to create or enable prejudice, at what point have I crossed the line into 'violation of privacy'?

earl4• 16 years ago

Well, I can say this: congrats on getting one of the best industrial data sets locked up. I hope you're proud of the work you did.

As for the possibility of netflix running a contest like this in an online fashion, well, maybe, but the benefits of having access to the data are enormous, plus you've now moved to a model where only the privileged few are allowed access via NDA, or Netflix has to provide computing resources to all researchers, etc. I don't see it happening.

whyenot• 16 years ago

Instead, you brushed off our claims, calling them “absolutely without merit,” among other things. It has taken negative publicity and an FTC investigation to stop things from getting worse. Some may make the argument that even if the privacy of some of your customers is violated, the benefit to mankind outweighs it, but the “greater good” argument is a very dangerous one. And so here we are.

In other words, "we showed them!" -- seem an awful lot like gloating to me.

It also doesn't appear that you ever seriously considered the "greater good" argument, beyond asserting that it's "dangerous." This leaves me with the suspicion that you chose to take the easy way out instead of weighting the actual costs and benefits of what you were doing.

raganwald3• 16 years ago

> Could someone who shares the author's worldview try to explain it?

Well, I can give you my worldview. Did you see the article "70% of HR managers turn down job candidate based on online reputation?" Here's the HN discussion: http://news.ycombinator.com/item?id=1192996

So. I rent a bunch of movies with strongly opinionated themes. I dunno, maybe every Michael Moore movie plus travel documentaries about Cuba. I also rent the entire Star Wars sextilogy and The Godfather Trilogy, which I review on IMDB.com. Some enterprising hacker ties my reviews on IMDB to my anonymized rental record. How many 47 year-old males are there in my neighborhood who watched those exact movies?

The hacker publishes his "findings," of course. Now I go looking for a job and someone in HR decides that my political views are too risky, so I don't even get an interview.

The author's views are that Netflix can provide the benefits you desire without compromising the privacy I desire. I think the debate should be around whether the authors are correct.

If the authors are correct, then the problem isn't the authors preventing the release, the problem is Netflix failing to learn from their prior mistake.

If the authors are incorrect, we should simply point that out.

Jun83• 16 years ago

The important point is that just renting out the movies isn't enough to reveal your identity, you also have to review them (anonymously) on IMDB or some such site. So the solution is simple: If you rent a movie that you would object to being listed in, e.g. your public Facebook profile just don't review it. Is this so hard?

nkurz6• 16 years ago

  Now I go looking for a job and someone in HR decides that 
  my political views are too risky, so I don't even get an 
  interview.

It's not ideal, but basing hiring decisions on real information (even if 'private') strikes me as better than using uninformed prejudice. In the example you give, is it better or worse to have the HR person say "Homoiconic? Sounds gay to me!"? I don't see much increased risk in offering a deeper pool of inherently unverifiable information, even if this information might be used badly.

  The author's views are that Netflix can provide the 
  benefits you desire without compromising the privacy I 
  desire. I think the debate should be around whether the 
  authors are correct.

This is a good debate, but for me it's a secondary one. Even if a privacy-safe solution is theoretically possible, I fear the only likely outcome will be that Netflix buries a 'release authorization' somewhere deep within their terms of service and is dissuaded from offering similar data sets in the future.

I also think that for my needs 'anonymize and release' is the only feasible solution. While online systems might be better than nothing, the approaches I'm interested in (kNN on GPU's, massively parallel cross-validations) really require local data and full control over data layout. But perhaps there is indeed a solution that works for everyone.

tedunangst3• 16 years ago

Except the Netflix data didn't contain your age or your sex or your neighborhood.

imurray• 16 years ago

I'm saddened to see someone gloating at having helped to prevent the release of a dataset that I see as beneficial.

I'd encourage people that usually skim just the comments to read the post, which was not gloating at locking up the dataset. The thrust of the post is about what a shame it is that the contest was canceled and how to make sure future contests can work.

[Note one: I have nothing to do with either side. Note two: I guess there is a gloating interpretation, with the paraphrase "you ignored us, but the FTC said we were right, nah nah nah" — but this isn't a useful or constructive way to continue the conversation.]

lvvlvv• 16 years ago

"you ignored us, but the FTC said we were right, nah nah nah"

That how I read their open letter. And my interpretation of "you should have worked with us" is as "you should have hired us as consultants"

gwern2• 16 years ago

> So if I publish a study (with backing data) that says that 38 year old males are likely to commit adultery, I've "violated the privacy" of all 38 year old males?

Yes. The difference between 'John Elks of 7 Arborview is a rapist' and 'there is a 0.5 correlation between living in Pleasantville and being a rapist' or '38 year old males commit 10% more rapes than average' etc. is solely one of degree and not kind.

Suppose I have a set of datapoints like that. And let's say each datapoint applies to only half the population. How many datapoints before I have broken your privacy and linked you to the furry porn you like to rent? Well, I'm guessing you're an American male. The US population is ~300 million, and roughly half of that is male, so 150 million. The first datapoint pins you down to within 75 million. The second, down to 33 million. The third, down to 16 million, the fourth, 8 million, the fifth 4 million, the sixth 2 million, the seventh 1 million, the eighth 500,000 (starting to feel nervous yet?), the ninth 250,000, the tenth 125,000, the eleventh 75,000, the twelfth 30k, etc. until the 25th or 26th specifies just 1 - you.

Now, tell me: Where in this slippery slope did it suddenly flip from not being a privacy violation of some degree, to being a privacy violation?

Was it at the 5th bit of information? Are you damaged at the 12th bit of information? Or did it take until the 24th or 25th bit of information before it magically flips from being good science to bad privacy violation?

Is it fine just so long as it might also be your neighbor down the street, even though most people would shun you based on far less than a 50-50 chance of things like being a child rapist? (An employer on the bench might regard a 10% chance of you being objectionable as being too much; that only requires, what, 18 bits of information?)

Predictions embody a great deal of information. That's how Bayesian statistics and statistics in general work, after all.

tedunangst3• 16 years ago

I'm very curious about what bits of information you think exist that so precisely bisect the population.

foldr2• 16 years ago

The difference between poking someone and shooting them is also "one of degree rather than kind", but we choose to make a categorical distinction between the two cases nonetheless.

madair6• 16 years ago

Netflix has a lot of subscribers, but it doesn't have that many, and in particular, in your neighborhood, does it have that many who are 38 year old males who like action movies and Michael Moore? It's a question partly of scale. It's big, but not that big that we can't get dangerous results with demographics & statistical analysis.

It comes back to the contemporary problem of statistics: We, programmers included as evidenced by many discussions of the sort, have a hard time getting it. We are only at the cusp of understanding what statistics can do with large data sets.

Now, we can either work with researchers and organizations to deal with the very real concerns, or we can simply refuse to believe that giraffes don't exist because we never saw an animal with such a long neck before. It's the basic problem of progress.

pbh4• 16 years ago

It seems to me that any data at all will necessarily reduce the entropy of the probability distribution of members' preferences, likes, dislikes, habits, and so on (i.e., their privacy). The authors seem to brush off the "greater good" argument here, but I don't understand how any large scale data release can happen without at least some reference to such an argument given that context. Given that, the authors seem to be making a fairly strong claim here: that no large scale "anonymised" data release should ever happen. Is that helping anyone in the context of movie viewing? And is it hurting anyone other than academic researchers, given that companies share more sensitive data anyway?

raganwald3• 16 years ago

The authors mentioned two alternatives to the current form of large-scale release. First, opt-in. Second, contestants submit programs that run on the anonymized data but the contestants do not have access to the data itself. Could either of these approaches contribute to the greater good without compromising privacy?

pbh4• 16 years ago

I do not have any data regarding opt-in, but my impression was that as a rule, no one opts-in, and no one opts-out of basically anything (excepting the notorious cases, e.g., Real Player). If it worked, and somehow gave an at least somewhat unbiased sampling of the data, opt-in would obviously be best.

I am completely unsatisfied with the submit-and-run model. Feature engineering seems to rely on knowing your data really intimately, and that does not seem possible in a submit-and-run model.

stevoski• 16 years ago

IIRC, in some countries census data is sometimes released with small, intentional errors to prevent the ability to locate specific individuals. Make a 36 year old sometimes a 37 year old or a 35 year old. Make a 180 cm person sometimes 182 cm or 178 cm. Small enough errors not to make the aggregate data invalid, but enough to make it hard to identify individuals from the data.

Perhaps this is a partial solution for the Netflix dataset.

nkurz6• 16 years ago

This is the approach that Netflix took with the initial data. The paper referred to shows that this is insufficient, and does little to ease privacy concerns. The general problem is that if you 'fuzz' up the data enough to make identification impossible, it's no longer useful as a dataset.

wooster• 16 years ago

The Census obfuscations have apparently screwed up a variety of research findings:

http://freakonomics.blogs.nytimes.com/2010/02/02/can-you-tru...

randomwalker3• 16 years ago

I'm a little taken aback by the tone of some of the comments here, so I thought I'd offer a few points of clarification.

* For the longest time we mostly stuck to doing the math. We certainly didn't call for the contest to be cancelled, and we had nothing to do with the lawsuit. But when some people implied that we were responsible for the mess that ensued, we were kind of pulled into it. We posted this as a way of explaining our point of view and reaching out to see if there's a possibility of collaboration.

* The sadness we expressed is genuine. The reason we brought up Netflix's response to our paper wasn't "snark" or "gloating." Rather, we were pointing out that the cancellation of this contest was rather needless, because if they had acknowledged the privacy risks back when we published the paper, they would have had more than enough time to deploy an opt-in system for this contest. I think it is really unfortunate that that didn't happen.

* Someone wanted to know exactly what I thought of the "greater good" argument. Well, I'll tell ya. I'm vehemently opposed to it and I think it's a dangerous slippery slope. I think this point of view is enshrined in the ethos of this country -- "better let ten guilty men walk free than to convict one innocent man," etc. I don't think anyone has the moral authority to decide that the privacy concerns of a few can be sacrificed.

* There is a specific reason we chose the open letter format rather than communicating with Netflix directly. Acutally, two reasons. First, there are many data privacy researchers who are at least as qualified as we are for this role. We wanted to make sure the community had the opportunity to participate in whatever ensures, rather than just us.

Second, I'm sure there are many companies other than Netflix who have a similar need for privacy preserving data mining. If Netflix doesn't take us up, perhaps one of the others will. Bottom line, since there are multiple parties on both sides, and we don't really know who they are, we felt it is better to have this dialog in public.

* Finally, it is understandable when something like this happens to want to find someone to blame. But think twice before shooting the messenger.

Aron• 16 years ago

I am gonna make a note on only a small part of this, which is that 10 guilty men is different than 'an uncountable number of guilty men'. In other words, all principals have soft margins in practice.

earl4• 16 years ago

Please. You directly enabled the lawsuit. At least have the integrity to acknowledge your part -- your pretense "Oh wow, a lawsuit just happened! But I had nothing to do with it!" -- is pretty stupid.

I'll bet money the most likely result of this lawsuits -- and your actions are a big piece of it -- is Netflix, et al, will release no more datasets to the public. Instead, only researchers under NDA will be allowed to work with the data, as was basically the tradition before this.

Good job.

jvdh• 16 years ago

Please. You're acting like Netflix did nothing wrong in releasing the dataset when they knew that they could be pretty easily de-anonimized. Thus providing a privacy risk.

greendestiny• 16 years ago

I'm not actually sure it was ethical doing this research on a set of users who had no desire to be de-anonymized. While the data was out there, and anything you came up with on other data could certainly have been used to de-anonymize the netflix data, specifically connecting the two seems like a mistake.

In fact in using this dataset you've kind of done exactly what you set out to stop - used peoples data in ways they haven't consented to.

houseabsolute• 16 years ago

> Rather, we were pointing out that the cancellation of this contest was rather needless, because if they had acknowledged the privacy risks back when we published the paper, they would have had more than enough time to deploy an opt-in system for this contest.

And get 0.1% participation.

hooande• 16 years ago

I looked at the methods used in the paper and it's clear that my definition of "privacy" varies greatly from the author's definition.

Essentially they are saying that if you know what rating someone gave 8 movies and the date that they gave those ratings, you can find a sample of their rating list (or something very similar to it) with 99% accuracy. So freaking what?

"Evidence" that flimsy wouldn't stand up in a local bar argument, much less a court of law. The records are still completely and totally anonymous. No names, no addresses, no way to identify anyone...nothing but a strong statistical correlation to a set of ratings in a database.

It sounds like they have a problem with the power of predictive modeling and not with the handling of anonymous data. Esstentially what they are saying is "if we know a little about you, we find can out things that we didn't know with a very high degree of accuracy, but no certainty.". Duh. That's what the whole netflix prize was about...using known data to make strong predictions about unknowns.

They had some interesting methods (especially in their similarity calculations) but this has nothing to do with privacy.

An open letter to Netflix from the authors of the de-anonymization paper

Related Stories