We Experiment On Human Beings

http://blog.okcupid.com/index.php/we-experiment-on-human-beings/?utm_source=rss&utm_medium=rss&utm_campaign=we-experiment-on-human-beings

by dochtman • 12 years ago

367 103 12 years ago

blog.okcupid.com

rockdiesel• 12 years ago

Nice to see a new blog post from OkCupid.

They haven't updated their blog since April 2011.

zabcik• 12 years ago

I hope it sticks around! They're so fascinating.

cschmidt• 12 years ago

Interesting that the author has a plug at the bottom for his new book, which he says he's been working on for the past 3 years. I guess his book writing got in the way of blogging.

jkaunisv1• 12 years ago

It seemed to me the company being acquired by Match.com got in the way of the blogging.

chimeracoder• 12 years ago

I guess his book writing got in the way of blogging.

There were a number of factors (which I've explained elsewhere in my comment history). That's part of it, but another factor was that, in 2010, there were 2.5 people working full-time[0] on doing research for OkTrends.

The blog posts took a lot of work. "The Real Stuff White People Like"[1] took almost two months of my time, plus some from Max and Christian as well. (Much like the product design process, since we didn't start each post off with a clear end result in mind, not all the work was visible in the final product).

I left to go back to school. Max ended up taking on more responsibility for other data/stats work, which slowed the pace a bit, and he left at the beginning of 2012 to do his own stuff.

People asked me for the last three years whether the reason OkTrends hadn't posted since 2011 was because of the Match.com acquisition and whether Match shut them down and I had to tell everyone "No, trust me, they're still around! It's just a coincidence!". Thankfully I no longer have to. :)

[0] 2.5 full-time means: Two of us full-time, as well as Christian, though he split his work time between OkTrends (the blog) and other stuff.

[1] http://blog.okcupid.com/index.php/the-real-stuff-white-peopl...

ShaneOG• 12 years ago

I immediately dropped everything to read this blog post. I have been waiting for 3 years! I am fascinated with the data and science behind human behaviour, and their blog posts were (now are) always really interesting.

heyalexej• 12 years ago

This!

adamzerner5• 12 years ago

Yeah, I just started reading all the older blog posts and they are really interesting!

Ntrails• 12 years ago

I'm still mad they removed the blog post on the success rates of paid dating services a few days after being acquired by Match.com (I think it was)

discardorama2• 12 years ago

They haven't updated their blog since April 2011.

I think that's around the time they were bought out by Match.com .

foolrush• 12 years ago

I suspect that the cavalier sentiment OKCupid has approached the discourse with will not go well for them.

Will be interesting to observe the sociological takedowns of this post.

tomjen33• 12 years ago

Unlikely. Facebook went overboard with it after hiding it from their users. OKC has always done this with their data, although I am more than a little disturbed by them showing deliberately bad data to their users.

Also who the hell though it was a good idea to have a dating site without pictures (even for 24 hours?)

ghaff• 12 years ago

And, probably more to the point, Facebook is Facebook and OKCupid is OKCupid. And even if, as OKCupid says in their post "But guess what, everybody: if you use the Internet, you’re the subject of hundreds of experiments at any given time, on every site. That’s how websites work." the fact that Facebook did it ignited the Internet's collective anti-Facebook sentiment. (Of course, it's also the case that Facebook's experiments were more oriented toward their own advertising than improving the site experience for users.)

Mandelbug• 12 years ago

Unfortunately, while showing what we would consider "intuitively bad data" may seem like a bad idea, only actually empirically testing it can we measure and quantify the exact impact that bad data (either shown purposefully or accidentally, or unknowingly) will have on users.

It may have in fact turned out that what we intuitively think as bad data results in better matches or better experiences. I think experimenting is worthwhile, so long as it is done in the open as they have been doing.

xanderstrike• 12 years ago

I am more than a little disturbed by them showing deliberately bad data to their users.

I actually find this reassuring. It means they're sanity checking their algorithm to make sure that it works better than random pairings, or isn't outright wrong.

danudey3• 12 years ago

I'm willing to bet that Facebook has always done this with their data as well. You have to, in order to determine not just if your algorithms work, but whether they should work and what they should do.

Also, Facebook didn't 'hide it from their users' so much as 'not tell anyone they were doing it'. And for the average OkCupid user, I'm willing to bet that 99% of them have no idea this sort of thing happens. They log onto the site, do some stuff, and then leave.

Personally, I was as surprised by the idea that OkCupid outright lied about compatibility scores to see what was up as I was about the results, but it doesn't bother me, and I'm the type of person who's read all their blog posts anyway.

andrey-p2• 12 years ago

I really enjoyed reading this.

On the flip side (as an OKC user), I've also had moderate amounts of fun trying to figure out their algorithms by trial and error.

For a while, I discovered I could figure out who scored me highly on their Quickmatch [1] feature by visiting it repeatedly - the ones who gave me a high score always came up first. This doesn't seem to work anymore.

[1]: If you haven't used OKC, the idea is: you score potential matches 1-5. You can tell if someone scored you highly (but not who) and if both of you give each other 4 or 5 you get a message "introducing" you both. If you're an A-list (paid) member, you get to see who scored you highly without scoring them.

Edit: tweaked phrasing as per comments.

Chinjut• 12 years ago

I believe you are supposed to get notified when rated 4 or 5 regardless of whether you rated the other person at all.

wldcordeiro• 12 years ago

Didn't they recently remove the 4-5 star system in favor of just a like button? I've been using the mobile version lately but it just has a star now for like and no longer shows a 1-5 rating.

delecti• 12 years ago

You can always get notified that somebody scored you 4-5, and you get notified of specifically who it was if you score them 4-5 back.

andrey-p2• 12 years ago

Fair point - that's what I meant. Let me rephrase that.

tomjen33• 12 years ago

But we took the analysis one step deeper. We asked: does the displayed match percentage cause more than just that first message—does the mere suggestion cause people to actually like each other? As far as we can measure, yes, it does.

Hm, that is unexpected. Wonder how long it will last.

sliverstorm• 12 years ago

Nah, we like people who like us. So if a trusted source tells us someone else either likes or will like us, we tend to like them.

surement• 12 years ago

From Modern Seinfeld [0]:

George swipes right for every woman on Tinder. E:"What if you're not attracted to her?" G:"If she's attracted to me, I might be!"

[0] https://twitter.com/SeinfeldToday

interstitial• 12 years ago

OKC experimented with the matching process -- the express purpose of their site. Facebook experimented with mood manipulation -- something they have no permission to do -- regardless of the fine print. Facebook is a very bad landlord, but people don't want to move.

yarrel• 12 years ago

They experimented through misrepresentation. It's the latter that's the problem, not the former.

If your bank experimented with not completing transactions or your email provider experimented with not delivering emails, the problem wouldn't be that they experimented.

Dylan16807• 12 years ago

'match percentage' is a fuzzy subjective measure. To a large extent it's asking of OkCupid thinks two people having a conversation is a good idea, which is basically impossible to be a lie.

And they're doing it to fight against users being hideously misrepresentative, which is hilarious.

azakai• 12 years ago

Facebook didn't experiment with "mood manipulation" any more than OKC. Facebook experimented with changing the site layout and which content and how much to show. Exactly the same stuff as all sites test.

Facebook did theorize that the changes it was making would affect people's moods in a certain way. That might make it sound like intentional mood manipulation.

But OKC's changes also affected people's moods. More than Facebook's, I would guess.

All these sites are constantly experimenting on humans. That's what changing the site content means. Everything that affects us affects our moods and everything else.

webmaven• 12 years ago

I'd also note that the proxy they are using ("apparent mood of subsequent content") hasn't been shown to correlate with the subjects'/users' actual mood.

All the FB experiment actually shows is that by manipulating the mood of content seen, you can affect the mood of content produced.

Here's a couple of contrarian hypotheses: "when some users see more 'happy' content, they feel worse about themselves in comparison, but post more 'happy' content to pretend that isn't so." Or, "when some already-sad users see more 'sad' content, this does not affect their mood directly, but does give them tacit permission to share how they are already feeling. Subsequently, their mood actually improves."

danudey3• 12 years ago

But Facebook already displays posts based on an algorithm; it doesn't do it chronologically, and it's actually almost impossible for me to find a specific post in my timeline. Especially from the mobile site, it's very common that I see something in the moment between opening the app and my timeline updating, and never being able to see the update again.

Facebook is already making the decision as to whether to show you a post or not, and they make that decision based on what they think you will want to see and what you will engage with, all of which is fuzzy and subjective. They already manipulate things you see to encourage engagement; for example, if their algorithm shows that your engagement is dropping and you're about to leave, they'll show other people your profile and say 'do you know this person?', because someone adding you as a friend boosts your engagement.

So if you think that showing you posts which are happy or sad is manipulative, be aware that Facebook is already filtering your potential-friends list and showing you people solely to boost engagement (either yours or theirs). Looked at another way, this means that it's possible their algorithm isn't currently showing you people you might know because it's not as beneficial for them to make that connection for you yet.

So the purpose of Facebook's site is to get people to interact and generate behaviour, and now they're experimenting with that; which posts do we show? which do we hide? They haven't shown them all for ages, so this is just a tweak to their algorithm that they were testing.

What's really interesting here is how this could actually be used for good. Someone feeling crappy? Show them fewer negative posts and more positive posts. Maybe that will help. Show negative posts less often, and make society in general a little more positive.

rabbyte3• 12 years ago

I don't think the sticky point is whether or not we should be experimenting with people to find what works and what doesn't. It's about respecting users as humans and not simply data points. How does OkCupid know that conversations "went deeper" or how often emails and numbers were exchanged? Were the participants aware of that level of scrutiny or is that tucked away in the ToS? Most people I know aren't even comfortable talking about their OkCupid, they use it as a means to an end and place trust in the service to be discrete. Do these experiments fall in line with user expectations?

I'm not saying you have to find these experiments upsetting just that when people are upset it tends to be about treating users like honorable guests and not about whether or not we should be allowed to tinker with services to find the right solution.

herge• 12 years ago

For the emails and numbers, they can probably search for those in the messages quite effectively without many false-positives.

I'm all for the expectations of privacy, but if you type something into a search box on a website owned by somebody else, well, you get what you pay for.

rabbyte3• 12 years ago

Sure, my comment wasn't to say they're doing anything nefarious. I'm actually not taking a position here either way. I'm just pointing out that people aren't upset that services use data to improve their services which seems to be the tone and premise of OkCupid's post.

adamzerner5• 12 years ago

Is anyone surprised that only 1/5 people who are 90% matches have a "conversation"? I don't use dating sites, but I'd think that people who do want to converse, and who better to converse with than 90% matches?

But I guess there's a lot of matches, and you can't talk to everyone, which would explain the low numbers. Still a bit surprising to me.

Edit: What you guys said makes sense, thanks. My lack of knowledge of dating sites lead to a bad intuition.

cm2012• 12 years ago

It's quite possible to have 100 90% matches - can't contact them all!

Someone1234• 12 years ago

If you want to understand this better I'd recommend going back through that Blog's archives, they talk extensively about this (TL;DR: Women are the minority on the site, and only respond to a small subset of messages, plus matches are personalty not looks and as this article indicates that is only 10% of people's opinions).

javert2• 12 years ago

Women don't respond to most messages, even 90%+. 90%+ match is not that special.

danudey3• 12 years ago

90% matches are common because of the huge number of people on the site. And they seem common because those are the kinds of people you're likely to be shown.

Either that or they're usually rare and I'm just so generic I get along with everyone.

egypturnash2• 12 years ago

Here's a screengrab of my okcupid inbox. http://imgur.com/jsKVyCi

One-liners and no picture are instant 'don't bother replying', and most people with more details tend to get weeded out for other reasons.

adamzerner5• 12 years ago

Is the gray bar "match %"?

searine• 12 years ago

It is kinda fascinating how awful most guys are at pickup lines.

Be brief, be funny, and always as a question. That's how you do it folks.

phantom784• 12 years ago

Someone should make a dating site where the men can't send the first message.

angersock3• 12 years ago

On a date with a nice young lady and had the dubious pleasure of seeing some of the messages she'd been getting--we'd been trading funny stories about dating.

The bar is set really rather low. :(

mkr-hn5• 12 years ago

90% of the messages I get from 90%ers are dull one-liners.

blt• 12 years ago

Heh heh. I recently got a message from okcupid saying "Because of a diagnostic test, your match percentage with <user> was misstated as 31%. It is actually 91%. We wanted to let you know!" I had a feeling something like this might be involved. (I messaged that person anyway, because they had a funny username, so I guess I supported their point.)

zach• 12 years ago

I thought Experiment 2 was a great example of the pain of trying to fit your users' feelings into an "ideal" model.

When we started LALife.com, it was with the idea of making a real estate site with lots of great statistics and quantitative data. We gave grades for how safe things were, how good the schools were, multiple heatmaps, assessor data, census data, nearby amenities -- a buffet of data. And sure, people who also fit in the mold of trivia collectors thought this was amazing.

But the more we talked to people, the more we found out that they really didn't use this data, even when they said they appreciated it. We had really overshot the market -- users almost never delved into the statistics, yet took our grades as absolute authority. What people wanted first of all was just insight about whether it was a "nice place" or not, a maddeningly vague concept.

Ultimately, we trashed countless tables and statistics and scaled it back to one number. Yes, one 0-99 number that shows you, well, how "nice" a place is to live. We were so afraid of generalizing things like this because everyone is different and has different priorities and so on. But trying to make things custom for everyone is a losing game, and it turned out "is it nice?" is something everyone already knew intuitively.

So insanely, we went from having 50 extra data points on a home for sale to having one "superscore". But a funny thing happened, which is it became much more successful in the site's actual mission, to help people understand a home's neighborhood without having to visit in person. And now we're providing an insight that is compact enough that we can put it everywhere and people can digest it instantly.

So although we would ultimately like to give people a more data-conscious mindset, the tool and the user need to agree on that commitment. So we're accommodating the user's actual mental model while we work towards expanding it.

PS: We have our scoring model working nationwide, but as you have seen from the Zillow thread, it's hard to get nationwide home listings. We're working on getting homes for rent on http://www.padrank.com/ so you can sign up there if you want to see how it works throughout the US.

boomzilla• 12 years ago

This phenomenon is actually quite common. In retail, it's called the customer funnel: http://en.wikipedia.org/wiki/Purchase_funnel. The idea is you want to capture as much awareness with something really simple, usually it's a brand name, e.g. Apple. When customers express more interest, by coming in Apple store/website, they are presented with more design/technical details. Finally, only when they show purchase intent, they are shown price/configuration/delivery options.

ssl_love• 12 years ago

Hey OkCupid – How about some SSL Love?

"For the hundreds of thousands of users searching for that special someone through one of the largest free online dating sites, the love fest may be coming to an end. OkCupid is putting users’ privacy in danger by failing to support secure access to its entire website through HTTPS. Every OkCupid email, chat session, search, clicked link, page viewed, and username is transmitted over the Internet in unencrypted plaintext, where it can be intercepted and read by anyone on the network."

https://www.eff.org/deeplinks/2012/02/hey-okcupid-how-about-...

The Heartbreaking Truth About Online Dating Privacy: https://www.eff.org/press/releases/heartbreaking-truth-about...

3pt14159• 12 years ago

Is this really still the case?

handsomeransoms• 12 years ago

Oh, yeah :/ The login page is over SSL, but everything after that is not (!!) This protects your password but not your OkCupid session cookies. It's an improvement over no SSL at all (which was the case for a long time) but still leaves a lot to be desired.

nostrademons2• 12 years ago

When I was still on OKCupid, I once wrote a Chrome extension to hide the pictures. I found that I had, in general, a much better experience with the site - I'd actually read people's profiles, I sent better messages, and I got more responses. I eventually gave up on it when a site redesign changed some of the #ids I was depending on and I didn't feel like revamping the code. But interestingly, my eventual girlfriend had terrible pictures - the grainy, multiple-people-in-the-background sort you're never supposed to put on a dating site. Setting up our first date, she was like "We don't need to exchange numbers, you know what I look like", and I was like "Actually, I'm not sure I do, here's mine." (Okay, I didn't actually tell her her pictures were terrible until we'd been dating for six months or so, but that was the general sentiment.)

I think it's one more example of when people's emotions and desires lead them to suboptimal outcomes. Most of the cues we associate with beauty and sexual desire evolved back in the savanna when health and fertility were very real risks; being able to pick up on which potential mates would be able to carry healthy offspring to term and nurture them until adulthood was very important then. Nowadays, the far greater risk is that you'll hate each other and fight all the time, but this has only been a concern since people started living long enough and in close enough proximity to care.

jessriedel2• 12 years ago

my eventual girlfriend had terrible pictures - the grainy, multiple-people-in-the-background sort you're never supposed to put on a dating site.

This is not what people mean when they say someone looks terrible in their pictures.

The point of the experiment is to withhold information about whether someone is physically attract, not about whether they have good photography skills. The only reason to avoid putting up grainy pictures is because people will assume you're not good looking and are trying to disguise this fact.

nostrademons2• 12 years ago

Previous OKTrends blog posts and their MyBestFace feature have shown that putting up poor-quality photos has the same effect as being ugly. People just assume you're ugly.

icefox• 12 years ago

According to this research done by the okcupid guys here: http://blog.okcupid.com/index.php/page/7/

"The flash adds 7 years"

"The type and brand of camera you use has a huge effect on how good you look in your pictures."

"There are peak times of the day to take a good picture." ala you are more attractive in photos taken at sunrise and sunset.

Between that blog entry and the other ones on the topic the conclusion really is that how attractive you are in real life and how attractive you are in your photo can have very little correlation.

ianstallings• 12 years ago

For the record not every site does these type of experiments. I've been programming websites since the 90s and I've never experimented on my users like this. The most I've done is swap in and out user interface features to see which one has a better response in a typical A/B test. It's not the same as telling a lie to a user no matter how you split hairs about it. They're overstating their case because they happen to be in the business of matching people up.

wcdolphin• 12 years ago

But you never did so with the intention of eliciting a negative psychological response.

That to me is a distinct difference, both in practicality and morality.

krallja• 12 years ago

Doesn't load completely in IE11.

iamleppert• 12 years ago

What a bunch of pointless data and hubris to prove that, yes, people are predominately fueled by the laws of physical attraction.

xj9• 12 years ago

It might be hard to accept, but people are shallow. Everybody has an attractiveness threshold that must be reached before someone will spark their interest. It might be lower or higher, depending on the person, but everybody has one.

Obviously, sustained interest and the development of romance is more complicated and involves aspects of personality, shared interests, &c. However, that doesn't change the fact that if they weren't good looking enough for you in the first place, you never would have made the effort to find out about the stuff you actually care about.

rabbyte3• 12 years ago

Being able to relate attractiveness to a linear scale doesn't mean attraction operates on linear terms. In reality, there is no threshold nor is there a higher or lower. I agree with you in the sense that people will disregard personality or shared interests if they don't find the person physically attractive but there's no answer that's more correct than another.

pervycreeper2• 12 years ago

I'm surprised no one has commented on the ethics of "Experiment #3". Feeding false data to users is not something that should be taken lightly.

spott• 12 years ago

I think the premise you are assuming is wrong: You assume that it is sending false data to them to say "this person (which our algorithm has deemed is a bad match), is a good match". The problem is that until the experiment is done, you don't how good a match that person actually is. So you aren't sending them "false data," because you don't know that it is "false data".

pervycreeper2• 12 years ago

If it is being reported as known, but is actually unknown, that still would be considered as 'false information'. For instance, if I claim to be clairvoyant, and predict that the next coin you toss will come up 'heads', that would be a false claim, regardless of the outcome of the coin toss.

lifeisstillgood• 12 years ago

Basically, people are exactly as shallow as their technology allows them to be.

Just a quote that resonates ... The design and architecture of our online environments will affect our happiness as much as anything else.

jessriedel2• 12 years ago

I found a similar thing: once they got to the date, they had a good time more or less regardless of how good-looking their partner was. Here’s the female side of the experience (the male is very similar).

I'm skeptical it was similar, at least if you were to measure the thing that actually matters: whether or not the person wants to go on a second date. ("Did you have a good time" is probably equating to "Did the person avoid doing something terrible" in the survey.)

It's ludicrous on its face to think that men, if they choose a blind date partner based on conversations and profiles, will want to go on a second date just as often with a beautiful woman as with a plain woman.

bshoemaker• 12 years ago

#clickbait

galfarragem• 12 years ago

I feel very "bearish" towards dating sites. People convince themselves they are saving time meeting more possible "matches" in less time but they forget that meeting online is not the same as in person, "turn-based" chat is different than "real-time" chat.

Some years ago randomly I met a girl online, that most of men (based on her picture) would wife up. I was living abroad at the time and we were only able to meet in person some months later. After few minutes in the date I could see that we would not go anywhere. There was a lack of "real-time" empathy and I hated some details in her personality that I couldn't realise before meeting her (not her look, surprisingly she was even more beautiful in person than at photos). If I had met her in person instead of online, I would have saved months of my life.. I will not do it ever again.

edit: Because of her I met my gf of 5 years, but that was just serendipity working, not because of a dating site.

tragomaskhalos• 12 years ago

TL;DR; people are shallow idiots. Cool article though.

We Experiment On Human Beings

Related Stories