Mechanical Turk Workers Are Not Anonymous

by anandkulkarni • 13 years ago

127 19 13 years ago

firloop• 13 years ago

Shocking that something with privacy implications like this wouldn't have been already caught by Amazon. It almost seems like common sense to not reuse (at least, publicly) the same user ID between a user's public page and a users more anonymous MTurk profile but I guess it slipped through the cracks or something.

anandkulkarni• 13 years ago

This was a curious design choice by Amazon.

By choosing to use Amazon IDs as worker identifiers, Mechanical Turk gets all of the disadvantages of having anonymous strangers do jobs on the web -- lack of accountability, diminished incentives, malicious workers -- but none of the advantages of protecting workers, since at the end of the day, they're not actually anonymous anyway.

This is a non-issue in MobileWorks (and for that matter, oDesk and Freelancer as well). In these systems, everyone's explicitly working under a real name, which improves incentives to perform well, and makes it a whole lot easier to get good results through trust and honesty.

jheitzeb• 13 years ago

Well said and 100% agree. I pressed them on that a few times and never got a good answer as to why they think anonymity is a good design choice. Many others I have spoken to who've used mturk significantly have come to the same conclusion you did.

tellarin• 13 years ago

Well, I know this is not the typical use case, but the need I had for it was to perform some experiments on the quality of recommender systems suggestions.

In this case, as the experiments involved people and the intend was to publish the results, the "workers" had to be anonymous.

obituary_latte• 13 years ago

It appears crowdresearch.org has yet to research how to handle a crowd.

raverbashing• 13 years ago

Also, do they need a whole paper to say that the Worker ID is the same userid used in different parts of Amazon?

est• 13 years ago

site's gone... any cache?

runarb• 13 years ago

Cache: http://webcache.googleusercontent.com/search?q=cache:crowdre...

mrtimo• 13 years ago

here is a text only version that worked better for me: http://webcache.googleusercontent.com/search?q=cache:crowdre...

niggler• 13 years ago

Where in the privacy policy does it state that Mechanical Turk program wouldn't share the ID with people using the service? I don't see any item in the privacy notice https://www.mturk.com/mturk/privacynotice or the Participation Agreement https://www.mturk.com/mturk/conditionsofuse

krilnon• 13 years ago

Static page: http://groups.csail.mit.edu/uid/crowdresearch/mechanical-tur...

nmcfarl• 13 years ago

This ID is pretty much the only thing employers get to ID the workers, and thus track the quality of their work. (Which is particularly important on Turk, as anonymity seems to encourage fraud.) Everything else is push: web service calls to contact a worker, or comment on an assignment. That stuff is all intermediated by turk, meaning it’s all anonymized fairly effectively - and quite constrained as to content and format.

These days Turk goes so far as to handle your 1099s for you - so you don’t know the names of even the workers that have done more than $600 worth of work for you…

Of course most employers in it for the long haul build up reputation databases built on these ids, and many encourage their workers to fill out profiles, direct on the employers website, allowing for more direct contact, and less constrained messaging.

Which is why this is a big deal for the employers: names, pictures, locations, and review helpfulness, can all be shoved into the reputation algorithm or used by survey companies for demographics. So it will be harvested.

And that makes turk a lot less anonymous. Which seems like a breech of trust to the workers, because although they aren’t guaranteed anonymity in the TOS, that has been the bargain so far.

smackfu2• 13 years ago

Are they supposed to be anonymous, or were people just assuming that?

Edit: The closest thing I found was that you can't create tasks that involve "collecting personal identifiable information". I would think that is aimed more at preventing tasks that are essentially selling your info for the price.

LogicX• 13 years ago

> "For academic researchers: worker IDs may have to be treated as personally identifiable information. For example, publishing worker IDs online in public data sets may be a violation of worker privacy, and counter to the requirements of the researcher’s institutional review board."

This is why we created http://SocialSci.com for use by academic researchers: We have better control over PII outside the Amazon ecosystem, and access to more data we can take our own steps to protect or de-identify, enabling higher quality data collection.

In general academic research done on Mechanical Turk has a ton of issues, from response quality to verifiability of responses. Frankly people have no incentive to remain authentic across surveys, and we've seen cases of people just taking on identities or just faking answers, which isn't reliably detectable. The only way we've been able to overcome this is by tracking users' response consistency over time, and keeping them anonymous in the process. It's proved to be a much more reliable method, and at this point based on our experience we can't recommend MTurk for valid academic research.

smackfu2• 13 years ago

It seems like academic researchers would already have been reluctant to publish worker IDs unless they had some assurance from Amazon they were anonymized somehow. Assuming it is anonymized is pretty bad practice.

1337biz• 13 years ago

You will always have these issues. If people want to cheat the system they can create fake accounts, administrate profiles for family members and so on. Isolated surveys, no matter if qualitative or quantitative questions, are quite limited in their implications as long as there are incentives to give dishonest responses and no other ways to evaluate the data in the context of practice exist.

calinet6• 13 years ago

Developer at SocialSci here. You're absolutely correct, the incentive for gaming the system is always there. The difference is that we also incentivize consistency and are very proactive about tracking down fake accounts through various means. You're right that it's a difficult problem, but while Amazon turns a blind eye to such abuse, our system is designed from the start to both disincentivize and prevent it, and in the meantime encourage good consistent high-quality responses over time.

We don't claim to entirely prevent fraud, but we're very confident that we raise the barrier and reduce the rate significantly.

lnanek2• 13 years ago

Not particularly a new finding. Heck we've had posts where people had the turks post pictures and details of themselves too, and many times turks have to create a profile on site x as part of their job...

noagendamarket• 13 years ago

Amazon should use bitcoin for mechanical turk.

Mechanical Turk Workers Are Not Anonymous

Related Stories