Facebook and OkCupid Broke the Law When They Experimented on Users

https://medium.com/@JamesGrimmelmann/illegal-unethical-and-mood-altering-8b93af772688

by jboynyc • 12 years ago

33 49 12 years ago

vonklaus• 12 years ago

Maybe they were illegal. The point is they are companies and can do whatever they want, largely without scrutiny. I mean this in the sense that it is difficult to bring charges against them, as evidenced by basically the entire financial sector.

When things like this are published it just serves to dissuade them from PUBLISHING the results of the experiments but not from actually CONDUCTING THEM.

I would much rather know when they do things like this than have them continue to conduct them in secret.

greglindahl• 12 years ago

Basically, they're claiming that the law says that any experiment on people, by anyone, requires informed consent. Every business with customers breaks this law. "Let's see if substituting medium roast espresso beans raises sales!" Sorry.

thrownaway2424• 12 years ago

It's also ridiculous because it presupposes an objectively correct way to rank results. The way Facebook places items on the page isn't the result of some natural law, it's the result of code they wrote. Nobody can seriously say that That Way of ranking results is superior to This Way, and certainly not that This Way is somehow illegal.

scarmig3• 12 years ago

I wonder if a retailer hiking or cutting prices would fall afoul of the legal idea being pushed here. After all, it amounts to an experiment on people to discover the ideal price of a good.

pdkl952• 12 years ago

Sure there is - you provide the service that customers signed up for. To most people that would be showing them all the results they expect. The research at Facebook selectively left some things out, which is clearly not what the customer signed up for.

Even better, the entire issue can be avoided by placing the decision in the customer's hands. Let them decide what kind of results they want. You can provide all the tools you want when it is the customer that is voluntarily choosing to use them. You might even learn something interesting from that choice, too.

/* now, to the thread in general, and not specifically the above comment /

What so many here seem to be missing is that this is about respecting the customer and not trying to go behind their back. They chose to use a service for some reason(s), and suddenly changing that without notifying them or asking for their input* is at a minimum highly unethical (and plain rude). As this article points out, sometime it is also illegal. Depending on the laws, contracts, and/or monetary transactions involved (if any) this kind of unilateral modification to the relationship may conflict with contract law or various business laws as well.

As some may people seem to have not read the article, a key quote:

"The central pillar of modern research ethics is that in most cases researchers don’t get to decide for themselves whether an experiment is worth it. Unless the risks are minimal or nonexistent, that decision belongs to the participants, not to the researchers. That’s what the Common Rule does: it systematically takes these decisions out of researchers’ hands, and gives them to participants and IRBs."

Yes, whenever humans are involved, the decision to modify ("experiment") on humans in any way cannot be made by the experimenters. Which bring me to the thing that I find most damning about this whole situation: the Facebook[1] and OkCupid experiments was fairly trivial. In in my not-an-IRB opinion, there is little risk to any of the participants; it's trivial compared to experiments like drug trials or other medical research. If they asked an IRB first, they could have gotten their approval quite easily. If, due to the specifics of the experiment, proper Informed Consent would have made the experiment useless, the IRB could have easily waived some of those requirements.

So no, various testing is not "illegal". You can try new things and see how it affects your customers. We have simply decided as a society (and try and ensure with various federal and state laws) that if it involves humans, you need a neutral 3rd party to make that decision. Yes, this slows[2] you down a bit, but again approval for an experiment like those done by Facebook and OkCupid should be fast and easy. Be glad you're not doing medical research with nasty risk.

Traditionally, if there is a question as to if approval is needed you should assume you do need it. If you are going behind the customers back to change something from the expectations[3] they initially signed up for without their approval, you absolutely need IRB approval.

[1] As I understand it, it may be Cornell that was the initially at fault for not getting IRB approval, so they deserve some of the blame in that case. Regardless, as the researchers Facebook has a duty (and sometimes, a legal requirement) to make sure that approval was done properly before they proceeded with the experiment. As an analogy, consider that before you do any penetration testing it is important make damn sure you have all your contracts and approvals properly signed. You don't rely on "but the CEO said so" or "We just tried some common passwords and did a few nmap scans - they shouldn't even notice", you get explicit approval first.

[2] If getting IRB approval too slow for the tech industry, they could try working with existing IRBs to try and come up with some new way of fast-tracking some types of experiments or they could create some sort of new mechanism that better fits the types of experiments would like to do regularly. The important part is that a 3rd party - not just the researchers - has vetted the experiment. The methods to get that approval can probably be modified.

[3] While it's not related at all to the topic of human testing and the IRB, HN readers my find the talk given by Joe Peacock at NOTACON 8 about how they (fark.com) screwed up their website redesign in the infamous "you'll get over it" incident. The point of the talk is about the dangers of violating the expectations of your users. Not only is it an ethical/legal issue in some cases, the attitude where the customer was not even involved in the change (even trivial) can cause them to leave you surprisingly fast. Simply asking or otherwise involving you users is the only real way to keep them in the long run. (it's fark, so mild NSFW language warning) https://www.youtube.com/watch?v=YnVeysllPDI

colbyh2• 12 years ago

And that's not to mention the fact that each user had effectively "signed" a TOS agreement that no doubt covers these sorts of scenarios. Informed consent (as I understand it, might very well be wrong) is given if the user in question had the materials made available to them for a sufficient enough time frame to understand both scope and intent.

You could probably argue that legalese pushes the boundary of "understanding scope and intent", but included in every TOS is a contact link for a user to learn more and ask questions so I'd be surprised if there was legal footing for that complaint.

The ethical nature of what they did is certainly in question, but I just don't see how one could declare either action "illegal".

EDIT: s/signs/signed/

revelation• 12 years ago

A TOS isn't a magic legal document you can fill with whatever you want. It might be in the USA, but in Europe any out of the ordinary clauses are regularly laughed out of court.

desdiv2• 12 years ago

I concur with your point, but your example is flawed. Food tasting studies are exempt from the Common Rule by 45 CFR 46.101(b)(6)(i):

  (b) Unless otherwise required by department or agency heads, research activities in which the only involvement of human subjects will be in one or more of the following categories are exempt from this policy:
    (6) Taste and food quality evaluation and consumer acceptance studies, (i) if wholesome foods without additives are consumed

A better example, and one that's more familiar to the HN audience, would be A/B testing of website front-ends.

dragonwriter4• 12 years ago

The Maryland law at issue, while it incorporates the federal rules generally by reference, also specifically disclaims any of the limitations in the federal regulations in terms of applicability in favor of making the rules applicable to all research involving human subjects without limitation.

nostromo• 12 years ago

A/B testing is also illegal by this standard.

jboynyc• 12 years ago

It is not, because it is not "designed to develop or contribute to generalizable knowledge" in the sense that it seeks to communicate to wider scholarly publics.

xux• 12 years ago

So.... are A/B testings now banned?

nernst• 12 years ago

The Common Rule AFAIK only applies to places that get government funding, so I don't see how this applies to research FB does on its own.

scarmig3• 12 years ago

Two Cornell researchers were collaborators in the experiment. So it's possible that its IRB screwed up somehow. Hard to see how Facebook or OkC is at fault for that, though.

terravion• 12 years ago

It seems like this "law" as interpreted the authors to ban A/B testing would violate the 1st Amendment rights of companies to uncensored speech with their customers. I hope that they challenge FB in court so it can be overturned.

smoyer• 12 years ago

The typical TOS these days turns users into corporate serfs ... I'll bet they respond to these complaints by alleging the users had granted consent.

dang• 12 years ago

We changed the url from http://laboratorium.net/archive/2014/09/23/facebook_and_okcu..., which points to this longer piece.

scarmig3• 12 years ago

"from the horrors of Nazi medicine to the deliberate withholding of care from syphilis victims in the Tuskegee experiment"

This is a deeply offensive comparison. Reading the complaint letter makes me wretch.

MetaCosm• 12 years ago

Yeah, it was horrible to read. It is a rare mix when someone is -- self-important, offensive, long-winded, ill-informed, and manages to invoke Godwin's Law.

For awfulness: 5/5

Someone12342• 12 years ago

"This is why we cannot have nice things."

OKCupid's blog is super interesting but they might have to discontinue it if it is going to generate legal challenges like this one. The only reason we know about their A/B testing is because they openly told us about it, which if they get prosecuted for it might be a mistake.

I kind of liken it to the "never apologise" rule many companies now have. It is when common sense/public interest run into cold legal restrictions.

namecast• 12 years ago

This isn't the law; it's a set of guidelines for federal agencies to follow. It'd be illegal if the department of health and human services or the FDA pulled this type of stunt, because they've voluntarily agreed not to do that by amending their rules and regulations to comply with the common rule that this blog post refers to (http://www.hhs.gov/ohrp/humansubjects/commonrule/)

Facebook and OKCupid are, at least not at the moment, part of the federal government.

jwise0• 12 years ago

I recommend reading the rest of the article; the author describes multiple state laws that apply in this case.

j2kun4• 12 years ago

This is actually not so clear to me. The law in question [1] specifically states that it defers to the federal law [2], which clearly states in section §46.102(e):

"Research subject to regulation," and similar terms are intended to encompass those research activities for which a federal department or agency has specific responsibility for regulating as a research activity.

Furthermore, the definition of "minimal risk" is given in part (i) of the same section,

the probability and magnitude of harm or discomfort anticipated in the research are not greater in and of themselves than those ordinarily encountered in daily life or during the performance of routine...psychological examinations or tests.

The argument against Facebook and OKCupid are pretty slim here... How could you experience significantly more discomfort in daily life if OKCupid slightly changed their rating system? Any reasonable interpretation of this risk is negligible.

[1] http://mlis.state.md.us/2002rs/bills/hb/hb0917t.pdf [2] http://www.hhs.gov/ohrp/policy/ohrpregulations.pdf

acqq3• 12 years ago

Here

https://medium.com/@JamesGrimmelmann/illegal-unethical-and-m...

the author explains more clearly his position:

"Federal law — primarily the so-called “Common Rule”— ( http://www.hhs.gov/ohrp/humansubjects/commonrule/ ) regulates research on people in the United States. The details are complicated, the gist simple. If you engage in “research involving human subjects,” you must have two pieces of paper before you start. You need a signed informed consent form from the person you’re experimenting on, and you need approval from an IRB (short for “institutional review board”)."

Someone12342• 12 years ago

The Common Rule linked doesn't apply to private businesses at all. So therefore I don't understand the author's point here?

acqq3• 12 years ago

That's what I also wanted to point to, you were faster.

The "Common Rule" appears to be a rule for the Federal agencies, and it is a rule, not a law.

Long jump from that to the illegality? But I'm certainly not a lawyer, whereas the article author seems to be "James Grimmelmann, Professor of Law, University of Maryland."

Strange.

jawns• 12 years ago

The letter that the authors submitted to Maryland's attorney general (http://james.grimmelmann.net/files/legal/facebook/MDAG.pdf) point out that Maryland's HB 917 -- which essentially applies the federal Common Rule to private companies who do business in Maryland -- defines "research" as "a systematic investigation ... designed to develop or contribute to generalizable knowledge."

That definition is so vague that practically any sort of interaction with a person in which any information about the interaction is recorded and analyzed can be construed as research. If a Girl Scout troop makes a graph of which neighborhoods buy the most cookies ... boom, that's research on human subjects.

I'd be interested to know whether any entity has been prosecuted for violating HB 917. I can't imagine that the extreme vagueness of the definition would hold up against a determined challenge.

What puzzles me is why James Grimmelman, J.D., and Leslie Meltzer Henry, J.D., think that the ultra-vague definition of "research" is legally tenable. Like, can't they anticipate the objections to the definition -- that it effectively bans all sorts of everyday actions that no reasonable person wants to subject to these sort of legal hurdles?

j2kun4• 12 years ago

More likely they want exposure and a high profile win rather than that they really think this is as immoral as they claim (via their comparisons to Nazi experiments). If they can get away with it and boost their career, why should they care about the impact?

disjointrevelry• 12 years ago

If some girl scouts' dad came to me, and told me they were selling mint chocolate chip cookies which I buy 10 packs of, but gave me butter almond instead to 'test a hypothesis'... there would be problems and repercussions of the likes unheard of.

barnabee• 12 years ago

This makes no sense to me. The algorithms employed by Facebook, OkCupid et al. are not published and presumably changed regularly both to better serve their users and to advance commercial imperatives - apparently this is OK but if you measure the outcomes and call it an experiment it's not?

I don't buy this, as long as all representations made to customers about the service are met, it seems perfectly reasonable to tweak the already unknown algorithms behind the scenes for whatever reason.

pdkl952• 12 years ago

as long as all representations made to customers about the service are met

Exactly this. The thing that some people seem to be missing is that there is a (very) good chance nothing is banned. It's just that you have to actually ask the customer. And IRB could waive a lot of the usual requirements (that are primarily intended for experiments with a LOT more risk).

The Common Rule, in these cases, is primarily about enforcing that stuff like a boilerplate EULA does not count as properly informing the customer and getting their approval. When humans are involved you have to get explicit approval each time as a precaution, which isn't actually that hard.

hyperpape• 12 years ago

The phrase "generalizable knowledge" seems crucial. I am not a lawyer, and have not done human subjects research, so hope someone else can comment.

However, looking around, it sure sounds like this would not normally cover A/B testing, a business changing its prices and tracking results, or similarly "local" investigations.

Here is one IRB's interpretation of that term: http://www.irb.wsu.edu/definitions.asp. Others seemed similar.

michaelochurch• 12 years ago

OP's position is extreme, and I feel like any negative publicity to Facebook or OkCupid over this is going to ultimately have the effect of punishing them for saying that they ran these experiments.

Changing the way content is ranked by your private service is not in the same ethical territory as the Tuskegee Syphilis Study. It's just not.

That said, there is an interesting discussion here, which is: where is the line? Netflix runs an experiment (and whether it chooses to collect data) every time it changes its ranking algorithm, but I think we'd agree that it's ethically OK in that case. What about those sleazy mobile game companies that use data science and heavy experimentation to find and target "whales" who have an in-app-purchase addiction problem? Not ethically OK. I don't know where the line is, but it's clear to me that some tech companies are on one side and some are on the other.

j2kun4• 12 years ago

I encourage people to read Section 46.102 [1] (page 4) of the federal regulations that Maryland is enforcing. It lays out the definitions of all these terms (as a mathematician, I put the highest weight on definitions), and in particular section (i) on "minimal risk" is clearly in Facebook/OKC's favor.

[1] http://www.hhs.gov/ohrp/policy/ohrpregulations.pdf

dalke4• 12 years ago

The Maryland House Bill 917 is at http://mlis.state.md.us/2002rs/billfile/hb0917.htm . The gist of it is:

(a) Compliance with federal regulations. -- A person may not conduct research using a human subject unless the person conducts the research in accordance with the federal regulations on the protection of human subjects.

(b) Scope to include all research. -- Notwithstanding any provision in the federal regulations on the protection of human subjects that limits the applicability of the federal regulations to certain research, subsection (a) of this section applies to all research using a human subject.

My problem is, I'm not sure what MD 917 (b) means. I'll start with §46.102 (a) (1) at http://www.hhs.gov/ohrp/humansubjects/guidance/45cfr46.html#... :

(a) (1) Research that is conducted or supported by a federal department or agency, whether or not it is regulated as defined in §46.102, must comply with all sections of this policy.

Is MD 917 (b) supposed to mean that §46.102 (a) (1) applies to all human research in Maryland? I believe that's the case, because §46.102 (a) (2) only applies to those organizations which are regulated by a federal department, so would already be covered under the law.

Section §46.102 (b) has a list of human research which is exempt. The most relevant for Facebook and OkCupid appears to be:

(2) Research involving the use of educational tests (cognitive, diagnostic, aptitude, achievement), survey procedures, interview procedures or observation of public behavior, unless: (i) information obtained is recorded in such a manner that human subjects can be identified, directly or through identifiers linked to the subjects; and (ii) any disclosure of the human subjects' responses outside the research could reasonably place the subjects at risk of criminal or civil liability or be damaging to the subjects' financial standing, employability, or reputation. [Note: does not apply to children.]

If A/B testing falls under 'educational tests' or 'survey procedures' then it's exempt.

Otherwise - and this is another tricky part - §46.102 (c) says "Department or agency heads retain final judgment as to whether a particular activity is covered by this policy."

Who is the department head for Facebook and OkCupid? §46.102 (a) defines it as:

(a) Department or agency head means the head of any federal department or agency and any other officer or employee of any department or agency to whom authority has been delegated.

MD 917 doesn't have the information for how to interpret §46.102 outside of the federal context for which it was written.

An alternative interpretation of MD 917 (b) is that no human testing is exempt, including §46.102 (b) (2) which allows surveys. Instead, all testing must go through an IRB.

However, if that's the case then the other exceptions of §46.102 (b) are also no longer allowed, including

(4) Research involving the collection or study of existing data, documents, records, pathological specimens, or diagnostic specimens, if these sources are publicly available or if the information is recorded by the investigator in such a manner that subjects cannot be identified, directly or through identifiers linked to the subjects.

which would require IRB review for every historian working out of previously published materials.

Therefore, I don't think this alternative interpretation is correct.

Which means I don't know if MD 917 really applies to OkCupid or Facebook. (The Facebook research was published in a journal which follows the Common Rule compliance; I presume the journals have a way to resolve the question of what "department head" means when there is no department head - it must have come up before. I only investigated MD 917 compliance here.)

_EDIT_ : The MD attorney general clarified this at http://www.oag.state.md.us/Healthpol/hb917letter.pdf .

Implicit in the federal regulations, and therefore in House Bill 917's mandate for compliance with these regulations, is the requirement that a person have in place a reasonable process for determining whether research falls within one of the exempt categories.

So determining if A/B testing is exempt comes down to if there's a reasonable process for determining if it's exempt. It's not clearly illegal.

dragonwriter4• 12 years ago

My problem is, I'm not sure what MD 917 (b) means.

Probably Annotated Code of Maryland Section 13-1601(b), added by House Bill 917, means exactly what it says. "Notwithstanding any provision in the federal regulations on the protection of human subjects that limits the applicability of the federal regulations to certain research, subsection (a) of this section applies to all research using a human subject." (emphasis added)

§46.102 (b) has a list of human research which is exempt.

But those exemptions are a provision of the federal regulations which limits the applicability of the federal regulation to certain research, and so are expressly excluded from the part of the regulation incorporated into Maryland law by Section 13-1601(b).

Otherwise - and this is another tricky part - §46.102 (c) says "Department or agency heads retain final judgment as to whether a particular activity is covered by this policy."

This, again, is a provision of the federal regulations which limits the applicability of the federal regulation to certain research (to those things not excluded by the relevant funding department/agency head), and so are expressly excluded from the part of the regulation incorporated into Maryland law by 917(b). So there's no need to figure out how it applies -- it doesn't.

However, if that's the case then the other exceptions of §46.102 (b) are also no longer allowed

Note that, if you go through the link you posted and click on the final passed (Third Reading) version of the bill and note the edits from the prior version, that prior version of the bill had express language incorporating the exceptions in 45 CFR § 46.102(b) as exceptions to the Maryland law, and this was stricken in favor of the language now present expressly disregarding any exceptions in the federal regulations. This makes it pretty clear that, whether or not this is desirable, the legislative intent was very expressly not to incorporate the exceptions in 45 CFR § 46.102(b).

dalke4• 12 years ago

I suspect my edit was too late for you to read. I added a link to http://www.oag.state.md.us/Healthpol/hb917letter.pdf , where the attorney general describes exemptions:

Therefore, under the applicable federal regulations, a person may conduct research in an exempt category without, for example, obtaining review by an institutional review board. A Maryland researcher likewise may do so under House Bill 917, because such a person is conducting the research “in accordance with the federal regulations on the protection of human subjects” and, consequently, does not violate the prohibition in § 13-1602(a) of the Health-General Article.

This is dated May 2, which is after the bill passed both houses and before it was signed by the governor.

This seems like the exemptions in §46.102 (b) are all allowed under MD law.

brg• 12 years ago

Would it be correct then that any change in layout or diversity in Maryland newspapers, periodicals, and publications would also be illegal? For instance, asking people they prefer a Sports section prior to Entertainment, or having Politics or Local Interest above the fold? Would it be ok to make the changes given that the effects were never measured?

Facebook and OkCupid Broke the Law When They Experimented on Users

Related Stories