My Half Workday as a Turker

http://www.cs.cmu.edu/~jbigham/posts/2014/half-workday-as-turker.html

by alceufc • 12 years ago

245 94 12 years ago

cs.cmu.edu

ggchappell• 12 years ago

Interesting article. One hears about MT, but not so much from this perspective.

It's also interesting that the linked page has two scroll bars (tried both Chrome & FF).

stopbits• 12 years ago

This page is just a container for a google doc in an iframe. Interesting CMS strategy.

dugmartin• 12 years ago

He should set overflow: hidden on the body

krilnon• 12 years ago

It seems to work reasonably transparently most of the time. I was in one of the author's classes in ~2010 and I think the class schedule was put online in the same way.

geronimox• 12 years ago

"At the time, I was hoping Dmitry would reject my HIT so I could go on a tirade about how unfair it was, and try to get him banned from MTurk or whatever."

I found this quite amusing. You know why this HIT was approved? Because requesters are also punished for rejecting too many HITs! He also would have been banned had Amazon discovered the scheme he was pulling.

compare• 12 years ago

This is a serious problem that requesters face. Often the workers who spam everyone with fake work will report anyone who rejects their work as retribution.

alexchamberlain• 12 years ago

Very interesting. Has anyone used Mechanical Turk to get work done?

jimmar2• 12 years ago

I've collected experimental data using Mechanical Turker. In my experience, you have to do a lot of validation to make sure the data you are getting is reasonable. You have to kick people out who aren't paying any attention at all and just want to click through your survey to get paid. (Similar techniques also have to be employed for undergraduate participants who just want to get done with your survey so they can earn their 10 points of extra credit).

If nothing else, Turkers are good at making sure your data collection instruments are working the way that they are supposed to.

pavel_lishin• 12 years ago

In my experience, you have to do a lot of validation to make sure the data you are getting is reasonable.

Do you ask any 'trick' questions that are the cognitive equivalent of a captcha?

viggity• 12 years ago

i've used it a couple times in a lame attempt to find leads. IIRC, I paid $.40 to find a name/phone number/website of construction equipment rental companies in 50 different cities. They did exactly what I wanted them to do, although I never did anything with the results. One guy seemed to pick up most of the jobs, so I tipped him an extra $10.

ctdonath• 12 years ago

Know a guy who's iPhone app uses MT to do pseudo-automated video image/motion recognition & processing. He had to train & retain his Turks, who seem quite content doing the work as an ongoing project. (Not sure if he wants further details revealed.) Between this and the article, seems the untold story may be that blind/random "turking" may be frustrating & unprofitable, but finding & developing a relationship on a project can work well for all involved.

runj__2• 12 years ago

I've used MTurk for a very subjective task: "Value the object in the image" for which I paid $0.10 per valuation (a maximum of 30 seconds was spend on the task).

The valuations were fair but the workers of Mechanical Turk may have a different view of how much something "home made" or "retro" may be worth (typically far lower than the object would be worth for someone in the market for said object).

_bpo• 12 years ago

I worked for years on systems built on top of MTurk (building trust systems, scaling tasks, etc)

The hardest thing is always task design, as the author discusses. It's quite challenging to create a task that will work in all browsers, across cultural and language boundaries, and to pay enough to incentivize the worker (while simultaneously little enough to make all the effort worthwhile).

The final effect is a bit dehumanizing on both ends. The workers feel alienated and mistrusted by the people requesting the tasks. The people requesting the tasks feel that the workers are lazy / "scammers" trying to cheat them.

We don't fully even know how to think about this kind of work yet - ideas like "hourly wage" don't seem to fit for someone working completely independently, at their own pace, possibly distracted or impaired.

Still, despite the trouble, it seems like one of the most promising fields for interaction design. There's the possibility to really improve peoples' lives (or make them worse).

abakker• 12 years ago

I've used MTurk to have interviews transcribed several times, and almost always had flawless results. (sometimes jargon or names trips them up.) I've always been incredibly impressed at the speed they get done, which is substantially better than I've ever achieved doing them myself. I usually aim to pay $10 for a 10 minute segment, which should amount to more than $10/hour for a good typist.

danielweber• 12 years ago

Last time I looked at MTurk, it was offering money if I would complete some simple task, and, oh, by the way, download this crapware onto your computer and run it.

crazypyro2• 12 years ago

To be fair, those types of HITs are forbidden by the terms (often removed quickly) and, in my experience, tend to be rare.

tectec• 12 years ago

That sounds a lot like someones research project: http://threatpost.com/research-project-pays-people-to-downlo...

bitlord_219• 12 years ago

What a dehumanizing experience.

runj__2• 12 years ago

Yes, working: how awful.

I hope you're trying to be on the workers side but either way this is a terrible thing to say. The way you look down upon people trying their very best to make a living is not helping them at all.

andreasvc• 12 years ago

It's dehumanizing to work and get paid little or nothing at all. You seem to be reading much more in the comment you're replying to than was actually said.

incision• 12 years ago

I dabble on MTurk every now and then.

It might sound strange, but I find certain kinds of survey type simple tasks incredibly relaxing and some of the HITs clearly connect to what seems to be interesting research.

Pretty much everything the author says here is spot-on and has been for a long time.

MTurk is positively flooded with HITs for generating fake reviews for products and fake content for websites. The best paying HITs by far are translation tasks - transcribing Arabic and Farsi seems to pay the best ($0.30 or more per minute of transcription).

I aim for academic projects with preference for things which require qualification (simple math / reading comprehension tests usually) as they generally pay quite a bit more.

I've earned an average of $0.27 per HIT and have never had one rejected.

Some of the more interesting / unusual tasks I've seen:

* Pinpoint various joints (shoulder, elbow, knee) in each frame of a clip showing a baseball player swinging a bat.

* Drop a pin on an estimate map location based on scenery shown in a short video clip (This was long before MapCrunch/GeoGuessr).

* Manipulate the camera in a scene of flat-shaded objects to bring them into 'correct' perspective.

* Choose 'preferred' structure designs (little houses) which appear to have been created by some sort of genetic algorithm.

VLM3• 12 years ago

"Some of the more interesting tasks I've seen"

I get distracted by categorization jobs, like look at this blueprint and if there is a blueprint number in the title box, enter it. Well, I end up admiring the print. Oh look a giant gear for some kind of mining thing. Why, what an interesting looking architectural drawing.

Similar thing happens with categorize document numbers. Apparently some municipality decided to turn millions of paper scanned pages of contract bids into piles sorted by contract, so I'd end up sitting there reading all about bids to put in some drainage culvert. Who would ever have guessed theres so many steps and processes and backfill and compaction are critical and theres so many inspectors... I mean you can half A a drainage culvert in like one paragraph but if you want it done right its 50 pages of PDF.

These jobs are probably impossible without an excellent short term memory or dual (or more) monitors.

The ones that annoyed me the most were "leave blank if no number" so I'd scour the document for what seemed like forever just to make sure. Inevitably leading me to becoming fascinated by drainage culvert design.

Its very D+D like. Whats behind this dungeon door? Oh, I see, a blueprint of a giant gear from some mining equipment. Fascinating. Like archeology but I'm getting paid for it. Well, not much pay.

zackmorris• 12 years ago

I think this is a perfect example of how money today is worth more or less than it actually is. Another way of saying that is that money is nonlinear or even non-constant.

For example, say you are broke and wandering the streets of a big city and really want to buy a bus ticket to another city for $20 or whatever. No matter how much your net worth is, you can't get that $20 easily. You could try asking various businesses to do some work for cash, you could try a song and dance routine, etc, but your choices all amount to various forms of begging.

Meanwhile you can be sitting in an office somewhere and make that money in less than 3 hours even at minimum wage, even if you do little or no work. You can even sell something on craigslist if you’re home. I suppose in desperation you could sell blood plasma, but that’s one of very few lifelines.

Nobody thinks $20 is worth much, but when you don’t have it, it’s very expensive indeed.

To me, the root of the problem is whether you are resource rich or resource poor. So the web, by virtue of being intangible, is almost quintessentially resourceless. The trend seems to be lower and lower wages for increasingly onerous labor. In other words, if you have money, you get a real world return greater than the value of your money. But if you don’t have money, then getting it requires an expenditure of resources and effort larger than the value of the money itself.

At some point in the near future, acquiring money will be so expensive from a labor standpoint that it will be cheaper to simply do things yourself and live outside of mainstream society by bartering goods and services. This really bothers me, because that shouldn’t be the goal of progress. The paradox is that even though every new fiverr and mturk create more jobs, they lower the value of work. Right now this is affecting developing countries by creating a race-to-the-bottom economy, but the futurist in me looks all the way to the end and sees how so many jobs today (especially the non-production ones like administrative/clerical work) will eventually be automated by technology and thereby make the purchase of capital through labor even more expensive.

Does anyone see a way out of this? Did I miss something fundamental?

presidentender3• 12 years ago

In the aftermath of the Bubonic plague in Europe, the shortage of labor led to higher wages for serfs, eventually ending any semblance of what we'd call 'feudalism.'

I'm not advocating a mass die-off of human beings, but any other means of limiting the supply of low-quality labor would have the same effect. If some sort of back-to-the-land subsistence thing became popular among the youth, those who didn't partake might find that a McJob paid more by virtue of there being fewer people willing to do it. Sort of a "shrug" by the lower-class Atlas instead of the elite.

VLM3• 12 years ago

I would imagine enforcement of minimum wage laws on mturk might take care of that.

The turk is so rarely updated there is probably some kind of disruptive startup opening for someone to set up a middleman service paying an hourly wage to carefully tracked humans.

This might put a stop to the horrible turk habit of claiming a 20 minute survey only takes 2 minutes or writing some kids term paper only takes 30 minutes, which is technically true if you spend no time thinking about it and can type at 250 WPM, but for almost everyone else its a bit unrealistic.

Some communities have sprung up on reddit to highlite great HITs but they're very realtime and seems like as soon as a great HIT is posted to the reddit group, in seconds its gone, used up. A better marketplace dynamic could be set up. Clearly HITs and turkers are not really commodites after all in practice, so commodity market tools do fail. Tools that better match the market would be better for all participants, and the market maker skimming off the top, so its mystifying why they put no effort into it.

scythe• 12 years ago

If some sort of back-to-the-land subsistence thing became popular among the youth

This won't become popular because it means accepting a drastic decline in living standards: you lose Internet access, public transportation, access to a variety of goods, the rural area you move to might have a corrupt government or an underfunded police department -- why do you think gun ownership is so popular in small towns? -- you experience a lot of social isolation, etc.

This idea that people should move "back to the land" can only originate from the deepest misanthropy and ignorance of the plight of the lower classes -- or perhaps the most starry-eyed technocratic utopianism, if you think that society can somehow provide poor people in disparate areas with a modern lifestyle -- and I'm not holding my breath. Of course, we could just kick poor people to the curb like this; it seems to be our policy already, city councils won't approve new housing for them, they won't approve the construction of a Wal-Mart, expansion of transit services is always opposed by a chorus of veiled racists, etc.

pjc502• 12 years ago

Subsistence farming is something that almost the entire world tries to get away from as soon as possible, because apart from the food it's near-total poverty.

Also, what land? There isn't a vast reserve of untapped productive land lying around. It's all owned (capital) by someone who won't want you farming on it.

The only humane approach to too many people is a low birthrate movement, which requires universal free contraception. Something which a lot of Americans don't want to see happen.

Loughla2• 12 years ago

This is called idolizing the pastoral. The Pastoral was/is a popular literary genre/device.

Essentially, when we get disconnected from 'the simple life', we glorify and idolize it.

Having come from back-woods, subsistence farming to a white-collar, higher education job, I can tell you that (a) very few people understand the amount of stress and work required to survive in subsistence farming, (b) seriously, stress; you can starve, or (c) how much it sucks to do without modern conveniences like computers, internet and electricity.

Ever wonder why current day subsistence farmers aren't well known for their poetry or art? Because they spend all of their time finding and growing food.

In the future, I think we'll see a combination of McJob and responsibility for one's own future, much as we've seen for every generation before. The more things change, the more they stay the same.

z2• 12 years ago

Andrew McAfee has a number of interesting articles and talks on this. E.g. http://andrewmcafee.org/2014/06/mcafee-autor-edsall-jobs-ski...

The way I see it, automated jobs will still add real value to the economy. Perhaps everyone can receive a share of that surplus through some form of universal income. At that point, assuming most of society's economic value creation is automated, monetary compensation could be completely decoupled from labor, yet everyone can be better off since everyone would be resource rich.

vidarh• 12 years ago

This question has a 170+ year long history, yet somehow it has come to the forefront against in recent years with people often entirely ignoring its history.

The foundation of the birth of modern socialism in the 1840's was based on the basic premise that technology would drastically increased efficiency and grow the economy to a point where redistribution could eradicate poverty and reduce the amount of labour necessary.

To your specific suggestion, variations over this was the viewpoint of some of the earliest socialist ideologists, who believed it possible to transform society through example and appeals to decency and charity.

Marx was one of the earliest to criticise that view strongly.

As early as the mid 1840's he made the argument that one of the main sources of eventual downfall for capitalism would not be some sudden enlightenment of the elites, but that capitalism would be too successful (the Communist Manifesto starts with a number of paragraphs gushing over the advances that capitalism and the bourgeoise have brought - Marx saw the development of capitalism as absolutely essential for progress) and eventually lead to over-production and under-employment at the same time, causing massive social upheaval and eventually leading to revolution if (though Marx also argument strongly that it was a when, not if) the ruling classes refuse to redistribute voluntarily.

bjelkeman-again• 12 years ago

I have said this before. But it may be worth repeating:

"The problem is that the worker who is going to be out of a job doesn't own the company that buys the robots to replace him/her. Ever heard of a lights out factory that has all the former assembly line workers at home with full salary? Didn't think so."

The way capitalist societies are structured doesn't really allow for income distribution that way, the way I understand it.

givan• 12 years ago

The internet helps globalize the digital labor market, competition from people that live in countries with low costs of living will drive the wages down for jobs that can be done from a computer and bureaucrats will be replaced by efficient databases, humans and paper are not very efficient.

This is natural evolution, IT wages bubble will burst because anyone with the proper education in any place in the world with lower expenses will be able to do it for less.

I hope the same happens with education, internet will change humanity in ways we can't imagine right now.

galfarragem• 12 years ago

Resuming:

1-Globalization: technology and governments based in short term economics allow work to be made where is cheaper -> wages will be similar everywhere in the world in the near future.

2-Supply and demand: technology decreases the demand of human labor (automated work) and increases the supply (mturk, fiverr, freelancer, etc) -> wages decrease.

leorocky• 12 years ago

I think the way out of this is a basic income guarantee. When a basic income is provided without requiring labor and this basic income meets the needs of an individual the cost of human labor will increase (the cost of everything will increase probably). This system assumes there is enough wealth around for the government to distribute enough to everyone.

This is Paul Krugman's take on it:

http://www.nytimes.com/2013/06/14/opinion/krugman-sympathy-f...

Kiro2• 12 years ago

the cost of human labor will increase (the cost of everything will increase probably)

How does that solve anything? Sounds like a pretty bad side-effect or is that really an objective with basic income?

hypertexthero• 12 years ago

One solution is a [Basic Income](http://binews.org/) — "an income unconditionally granted to all on an individual basis, without means test or work requirement."

A Basic Income would help solve the problem of poverty and climate change, and should be a human right.

taejo2• 12 years ago

How does it solve climate change?

roymurdock• 12 years ago

Long comment but please bear with me:

To me, it doesn’t make sense to say that money is non-linear or non-constant. Money is not a function of time, therefore it does not make sense to describe it with time-dependent terms such as “linear” or "constant”. Inflation, the change in the purchasing power of money over time, yes. Money, no.

Money is simply a token that we use to represent the intangible concept of value. I think your comment seeks to address a change in what individuals, and by extension and on aggregate, societies, fundamentally value over time.

I’ll return to that in a bit. First let me address the problem with your example: It ignores the compounding returns to performing similar actions over time. Economically: Human capital accumulation. Colloquially: Experience.

When you get your first entry-level salaried job, you are actually a liability to the company. You come in with a low level of human capital - the company is basically paying to train you, all the while putting up with any potential mistakes due to your lack of experience. Why would a company do this?

By hiring you, the company is making an investment in your stock of human capital. It takes on some risk up front in return for some amount of reward in the future when you are wiser. As you gain experience over time, (ideally) you perform increasingly valuable tasks with a higher efficiency. You have accumulated the one thing youth (or old age for that matter) cannot buy for itself: time.

Getting paid for 3 hours for "little to no work" is a drop in the bucket compared to 10 years of solid, salaried employment.

By comparison, one-off jobs which might net you $20 in a couple of hours are a completely different situation where a low level of human capital is not purchased by means of a salaried contract. Rather, it is rented with a much shorter time horizon in mind, generally for menial tasks that require little to no experience.

To connect this idea back to the larger debate of “lowering the value of work”: I think you need to specify what kind of work you are talking about. I agree that the value of human menial labor is constantly decreasing. Why?

The supply of physical, menial tasks (washing dishes or mopping floors) is determined by the inflows of the human birthrate (more humans=more eating=more dishes to wash) and the outflows of automation (dishwashing machines), among other factors. Automation adds value by adding hours to our lives that would otherwise be spent washing dishes. For menial tasks it is generally more cost efficient. Thus the outflow outweighs the inflow (decreasing birthrate) leading to a lower supply of menial labor.

I don’t think we will revert to a hunter-gatherer bartering society in the future. Markets and specialization are much more efficient forms of raising standards of living, which I define as the ultimate goal of “human progress”. I think we will eventually need to reexamine our definition of “standards of living” as we move away from the realm of the physical to that of the virtual. Perhaps one (sad) day food will be unnecessary, as we redesign our bodies to subsist on a different form of energy, or do away with physical bodies altogether. This standard of living does not take into account food, shelter, transportation, etc. – the fundamentals that we strive to improve today through progress.

Anyways, enough with the science fiction and philosophizing. The fundamental lesson here is that the market gets what the market wants. There are a lot of human beings, so menial labor, both physical and mental, is in high supply driving down its relative value/price. Technology is eating jobs from the bottom up. Increase your human capital so that your brain is more valuable than a charged hunk of steel. Be the person who knows how to operate the machine and you will never have to beg for a $20 bus ticket.

pc86• 12 years ago

For those of you like me getting a white screen with no content, the direct link is https://docs.google.com/document/d/1jzipmLrMVIH_FfxK4Hlkadxv...

robertfw• 12 years ago

FYI, Privacy badger is what caused my browser to whitescreen, as it was blocking the pass off to google docs

aceperry• 12 years ago

Kind of reminds me of the saying, "pay peanuts, get monkeys."

crazypyro2• 12 years ago

I used MTurk off and on through my teens[0] (15-18) and found that the majority of my earnings came in through the 50 word or less summaries. You get to a point where you've read enough PR department press releases that you can write the entire summary (at least to the required standards, which in my experience, is "barely comprehensible English") by looking at the headline. I was able to finish 50 word summaries in under 30 seconds. They values range from .25-.50 and they seem to come from a lot of submitters.

I've had a theory that out in the world somewhere is someone selling a guide to using MTurk to create article summaries and the guide suggests the price range of $.25-.50 because for the entire time that I've been on MTurk (many years now), every time I come back the price range is the same with small fluctuation.

Past the relatively low amount of summaries that are posted daily, money can be made by watching the right forums where people post high paying HITs (the name of the most popular forum eludes me at the moment) that are typically 5-15 minute surveys that pay .5-3.00 each.

By far, the biggest barrier to making a decent wage for me was the number of available HITs. The ones that pay over pennies each are few while the ones that are horrible time invests are everywhere. If I had an unlimited number of summary HITs, I could have made well over minimum wage.

[0] Made about $450 over the time, but this was sporadic, random working periods.

ski• 12 years ago

I'm curious, have you found the ability to write 50 word summaries in under 30 seconds to be a useful skill later in life?

It sounds like good practice in concise communication.

e12e5• 12 years ago

I'd say the above comment is about average, to average+ for hn... so probably not? (Only being partially sarcastic)

[edit: I now realize that that's not enough data, we'd also need how long it takes to write the average hn comment, and how long op took writing his :-) ]

Natsu• 12 years ago

I'm surprised someone hasn't automated jobs like that, honestly.

oftenwrong• 12 years ago

Without question it is a valuable skill.

In academic research it is especially important, and made more difficult by the complexity of the subject matter. The most successful researchers can succinctly explain their work (problem, significance, methodology and findings) to a layman. That's how you get funding. This also applies to startup founders seeking investors.

spikels• 12 years ago

Jargon Alert - HITs (Human Intellegence Tasks) are simply the individual tasks (or units of work) that the worker is paid for completing.

tripzilch• 12 years ago

Thank you! I had to ctrl-F down 75% of this thread because it wasn't explained in the article either.

pessimizer• 12 years ago

An journal article I submitted a few years ago:

Amazon Mechanical Turk: Gold Mine or Coal Mine?

http://www.mitpressjournals.org/doi/abs/10.1162/COLI_a_00057

sparkzilla2• 12 years ago

I checked Mechanical Turk as an example of how to not to price for Newslines, my crowdsourced content site. It seems that most publishers use crowdsourcing as a way to save money, and while that may work for some tasks, you will not get consistent work done.

Rather than crowdsourcing being a way to find cheap writers, content business should see crowdsourcing as a way to find a diverse pool of writers who have different interests or knowledge, who can dip in and out of work, and then pay them fair money for their time.

I pay $1 per post for writers to write 50-100 word posts that follow a very specific format. We provide feedback to the writer to try to make them make money faster. The way I figure it is, if the writer makes money then I can make money. If they stick with you, you don't have to train new people. People won't leave after half a day.

We have already paid out thousands of dollars and our site is growing fast. Writer satisfaction is high. We currently have a waiting list of hundreds of writers wanting to sign up.

http://newslines.org/newslines-rewards/

e12e5• 12 years ago

Initially I found $1/[edit: item] strangely low as an example of "high pay". Having a look at what newslines is actually about, I still think it is low, but not as low as I intially thought (I'd assumed you wanted contribution from domain experts, who'd I guess you'd price at at least 100/hour).

I'd assume anyone that can write well, should be able to make at the very minimum 30/hour writing -- That leaves on average 2 minutes/newsline. I suppose the lesson is that there are many skilled poor people out there.

(Note, this isn't meant as negative criticism, just some observations)

sparkzilla2• 12 years ago

Thank you for your observations, I appreciate the tone. In our case the writers are working to a formula, which for most people is a lot easier than creative writing. So far most of our writers are actually work-at-home moms who are happy to make around $10 an hour, for a task they see as being quite enjoyable - writing news summaries about their favorite celebrities, while sitting at home. They can finish a task in a few minutes, depending on the topic, and bank $10-$20 a day extra cash (or more) fairly easily. Then they can come back the next day for more. On our side we get a 24hr/day pool of talent that wants to keep adding content, without the need for office space and other HR issues.

cm2012• 12 years ago

You are dead wrong if you think any skilled writer can make $30 per hour. Only copywriters and technical writers can do that, and those are not easy jobs to get. Journalism, as an example, is one of the lowest paying professions. Writing jobs on Craigslist are almost always unpaid internships.

ChuckMcM• 12 years ago

This was a great read. I found it particularly interesting that the author came into each encounter 'cold', which is to say without much in the way of additional tooling. They did install a word counter extension but much of their tools were ad-hoc or non-existent.

That suggested that there may be an interesting 'toolware' market for Turkers. For example, document editing; Lets say we build a simple document editing platform (copy editing not typesetting) which includes components that are dropbox like (shared storage), Google docs like etc. And design a workflow around that toolset. So pick a HIT, get a document id, it shows up in your shared storage, use the included tool to edit it, when you're done, click 'done', and have the whole thing resubmit back for evaluation. That might allow you to focus on editing and not get hung up on a bunch of getting it done issues.

Not sure how practical that is, but if there are common Turk workflows it might be useful to build some common tooling for them.

_delirium• 12 years ago

That suggested that there may be an interesting 'toolware' market for Turkers.

In the early days of Mechanical Turk (2005-06), when almost all the HITs were seeded by Amazon themselves and were of a few common types that kept being replenished, there were browser extensions to greatly improve the UI for the common tasks (the default UI is very bad). One of the more profitable tasks was fixing the location of addresses on the new-defunct A9 BlockView (a product similar to what Google would later introduce as Street View). They'd present you about 10 photos of a streetfront that were supposed to be near an address, and you were supposed to pick the one centered around the address (e.g. the store or house entrance). Or else indicate none of the above, if the alignment was seriously off so that the correct address was out of the frame. If you used the regular website UI, you could make maybe $3-4/hr, but most of this was tedious clicking: accept HIT, click a tiny radio button, scroll down, submit HIT, request next HIT, repeat. The browser extension implemented the obvious UI improvement: use the images instead of tiny radio buttons as click targets, and then auto-submit and request the next one on click. I think in one version you could also just hit a number 0-9 on the numpad instead of clicking. With that extension I was making $20-30/hr for a little bit.

mkhpalm• 12 years ago

I wonder what pays more? Panhandling on the street corner or working on mechanical turk?

lilsunnybee5• 12 years ago

It's been over a year and a half since i've spent any serious time on Mechanical Turk trying to supplement low income, so some of my experience might be dated. With a bit of a learning curve it's plenty doable to find decently paying jobs in transcription, writing, etc. on MT, especially if you have a strong grasp of the English language. But even with higher-skilled, higher-paying jobs on that platform, the overall experience of working on MT can often end up being plenty demoralizing and unfair in many many ways, such as qualifying for skilled work and also being compensated for work performed.

The most frustrating jobs i've ever performed on the platform were usually approx 5-10 minute tasks that seemed to be decently paying and straightforward, asking you to categorize items based on some arbitrary but seemingly simple system, but later all 5 or more submissions you might have made could be blanket denied / payment refused. In that case not only do you not get payed, but having HIT submissions rejected really seriously hurts your reputation and what jobs you're allowed to apply for on MT; often many more high paying jobs on the system use high acceptance rate and flawless work history as the lowest bar for entry, and so MT workers can be screwed over incredibly by having work rejected through no fault of their own.

To not get seriously screwed newcomers are somehow supposed to know which types of HIT's to definitely avoid that could kill their rating, but without enough information to actually make that judgement. A task might seem easy and give the illusion that it's okay to send 5 or 10 submissions in a row, only to later have them all rejected and serious reputation damage done.

I think it's interesting but also really sad how this mirrors a lot of the unfairness and power imbalance in the larger working world as well. Reputation is everything, but if you just happen to get unlucky to start out your career working for clueless bosses / clients with unrealistic expectations, there's a good chance future job and career prospects are going to be seriously hindered, if not completely derailed.

Transcription work was usually very well paying i found, but doing one long transcription task for a $30 payout is a huge risk the way things were set up. Never mind that you worked for 4 or 5 hours on it; whoever assigned the job can reject it for whatever random reason they want and you get nothing. As nice as it can be to earn a nice wage for more high-skilled tasks, the possibility of being denied any payment for a days work with little recourse is really frustrating and demoralizing; the way Mechanical Turk is structured if anyone's taking a negative financial hit for work done, it's almost without fail going to be the worker.

Though this CMU professor obviously put some work into this write-up, i don't see how somebody can possibly get an accurate picture of what it's like trying to earn an income on one of these platforms by just devoting 4 hours and extrapolating based on that really limited experience. It is really easy to miss all but the most obvious and glaring problems that way, just from not sticking around long enough to even run across them.

There are plenty of MT workers who have spent much more substantial amounts of time on the platform, that are much better able to communicate benefits and pitfalls for the everyday worker. They might not be professors at prestigious universities like CMU, but having a decent amount of hands-on experience should really be the low bar for discussing pros/cons of the platform seriously; just as one example assumptions about longer-term take-home pay in the write-up were pretty naive: many better paying tasks (especially web browsing tasks and surveys paying more than a pittance) are often in limited supply and cannot be repeated either at all, or only a few times per day / week, etc. Even the best employers providing the most decent fairly-compensated work come and go, making a steady healthy income sometimes very difficult to achieve even in the best of circumstances.

Mechanical Turk has experienced a lot of success from the start largely because they created a service that makes it so easy for virtually anybody with internet access and a basic computer, to work from home and perform simple tasks / other types of work when and wherever they want, and actually get paid for it.

By any decent standards of developed-world countries the pay is shit though, and worker protections and benefits are nonexistent; workers can be denied payment and even have their reputation ruined for no good reason, with little in the way of an appeals process. But even now services like MT are still a pretty new thing. Maybe in 10 or even 20 more years most of the bugs will get worked out, and workers of every skill level can be a on more equal footing when negotiating with employers, settling disputes etc.

pjc502• 12 years ago

If there's no real appeal process, what reason is there for any MT employer to make any payouts at all? Honesty and not realising this about the system?

mpthrapp• 12 years ago

There's an unofficial ratings system for those who post the jobs[1] that most people who use MT with any sort of regularity (including myself back in the day) use. If a requester consistently fails to pay out, they'll stop getting as many workers who are willing to work on their tasks.

[1]turkopticon.ucsd.edu

lilsunnybee5• 12 years ago

Maybe i read it wrong, but from the article it sounds like the Amazon MT team is at least trying to do more in the way of preventing unnecessary rejections and bad faith task-giving. Not sure if any sort of appeals process was ever put in place, but i really doubt it, since that would create a new non-trivial administrative burden on Amazon that is likely much larger than they are willing to handle.

Seemingly arbitrary rejections always seemed to be a much bigger problem by my estimation, especially for workers already somewhat invested in the system, and needing to maintain approval ratings to even qualify for higher-paying types of work.

If Amazon hasn't implemented anything like a way of expunging old rejection records, it would be a huge benefit to their workers to have some sort of reasonable, semi-automated way of doing so, at least after enough time has passed, or enough more recent positive approvals have occurred.

I'm really out of the loop though, and don't know much about the current state of things. It would just be really nice to have some more in-depth write-ups on the MT worker experience. The article was great but from first-hand experience i know there is a lot more ground to cover.

Senior project / undergrad thesis anyone? Researching / investigating MT from a user interface, user interaction design side of things, giving recommendations for improvements to the worker experience, for HIT providers, etc. would be very interesting and potentially very beneficial for the platform.

MicroBerto• 12 years ago

I'd be interested to see how you all did if using either end of Pay4Bugs ( https://www.pay4bugs.com )

I bet you'd learn some great stuff and see some crazy software

lilsunnybee5• 12 years ago

Neat! I haven't heard of it before but will definitely check it out. Glad to see more and more options popping up for this sort of flexible, easy payment type work. This kinda thing can do a lot to help people already below, or hovering right above the poverty line, since unforeseen expenses, and not being able to get new sources of income started in time is one of the biggest most daunting problems to have to deal with.

adultSwim• 12 years ago

This is an excellent comment.

Looking at your posting history, I can see that you often take the time to write thoughtful comments. Please keep it up.

lilsunnybee5• 12 years ago

Thanks. :-)

pgl• 12 years ago

Can anyone suggest any alternatives to Mechanical Turk that offer a better experience for the worker?

garretthunyadi• 12 years ago

Check out CrowdFlower. The workers are mturk workers, but I believe that there's more attention placed on the worker.

ccb_• 12 years ago

Like Jeff, I'm a professor who works on crowdsourcing. For one of my current projects, I am working on a tool to make the Amazon Mechanical Turk marketplace fairer for workers. I have developed a Chrome plugin that tracks the length of time it takes to complete a task, and a web site that aggregates the information across many workers. Crowd-Workers.com allows workers to discovery higher paying work by sorting tasks by estimated hourly rate.

Try out our Chrome plugin and our web site: http://crowd-workers.com/landing

My Half Workday as a Turker

Related Stories