Clockwork Raven - MTurk framework from Twitter

http://twitter.github.com/clockworkraven/

by polyfractal • 14 years ago

15 8 14 years ago

twitter.github.com

polyfractal• 14 years ago

Whoops, an announcement that went with the project (probably should have linked to that instead):

http://engineering.twitter.com/2012/08/crowdsourced-data-ana...

aantix2• 14 years ago

Are there any built in controls for spam/bot responses? This is something I have been contemplating adding to Turkee ( https://github.com/aantix/turkee ).

anandkulkarni2• 14 years ago

This adds to a long list of projects like Turkit (http://groups.csail.mit.edu/uid/turkit/) that try to patch over the ease of use / spam / quality holes in Mechanical Turk.

Better platforms than Turk exist today that handle this functionality out of the box. Building more solutions like this isn't a great use of engineering time!

aantix2• 14 years ago

What are the platforms that you're referring to?

bravura• 14 years ago

Turkit, Javascript code for achieving consensus among AMT workers, especially with multiple stages. See the Find-Fix-Verify pattern: http://www.behind-the-enemy-lines.com/2011/04/want-to-improv... By doing multiple stages of crowdsourcing, you can do complicated tasks with high quality, like: Improve the grammar of these reviews. Zappos did this and saw a lift in conversion.

Crowdflower, which is good for achieving high-quality one-shot annotation, but not so much for pipelines of annotations.

MobileWorks, which is good for achieving high-quality one-shot annotation, and says in the docs that it has pipelines. I haven't figured this feature yet out.

CrowdControl, which supposedly solves every problem, but is priced as an enterprise solution.

If you want to build something cool, implement pipelines of work. i.e. build a crowd-programming layer that has subroutines. Look at what CrowdControl says they are doing.

[edit: If you also want to build something cool, implement a reputation system. Don't just assign workers a single number. Figure out what kind of tasks the workers are good as, and do a per-task reputation system. For bonus points, solve this correctly, by dynamically gauging the skillset and difficulty for each task, rather than simply grouping tasks into N clusters, where N is low.]

Email me if you want to discuss. I've been thinking about this for a while.

anandkulkarni2• 14 years ago

For a fresh-platform approach, MobileWorks: www.mobileworks.com

For something that runs as a layer on Turk, Houdini: www.houdiniapi.com

stfu• 14 years ago

Probably the one he lists in his profile ...

Clockwork Raven - MTurk framework from Twitter

Related Stories