Being A Turker

‘Turking’, i.e. crowdsourced work done using Amazon Mechanical Turk (AMT) is attracting a lot of attention. In many ways it is a ‘black box. Amazon is not transparent about how the marketplace functions, what rules govern it, and who the requesters and Turkers – who post and carry out the human intelligence tasks (HITs) – are.

Research has looked to prise open the black box, understand how it operates and use it to get the best results. It is generally considered a great opportunity for getting micro-task work completed at very cheap rates, quickly. There are concerns about AMT as a grey market; some requesters and Turkers are unscrupulous. The question for requesters has been how to design and control the crowd to get genuine work done.

Research on the Turkers themselves has been rather scant, with notable exceptions where people have contacted Turkers, often through AMT itself, done interviews, questionnaires and HITs to express their thoughts and feelings. Who they are and what they think is still unclear. What is myth or truth? We tried to better understand these invisible workers by joining their forum, Turker Nation, and looking in detail at what they discussed amongst themselves.

This is what we found:

  1. Members see Turking as work and are primarily motivated by earning.
  2. Earnings vary but Turking is low wage work: high earners on Turker Nation make ~$15-16k/yr
  3. Workers aspire to earn at least $7-10/hr, but (newbies especially) do lower paid HITs to increase their reputation and HIT count.
  4. Many Turkers choose AMT because they cannot find a good ‘regular’ job or need other income. Some are housebound, others are in circumstances where Turking is one of the few options to earn.
  5. Turker Nation provides information and support on tools, techniques, tricks of the trade, earning, and learning. They mostly share information about good and bad HITs and requesters.
  6. Relationships are key: Turkers like anonymity and flexibility but want decent working relationships with courteous communication. They want fair pay for fair work (decent wages, fairness in judging work, timely payment…) and respect works both ways: good requesters are prized.
  7. Members mostly behave ethically. Ripping requesters off is not endorsed and is justified only against dubious requesters. There is a moral duty to their fellow members.
  8. Members feel that by sharing information and acting cooperatively they can have a stronger effect on regulating the market. Many are skeptical about government intervention.

For more, see our full paper, Being A Turker, which will appear in CSCW 2014

David Martin, Ben Hanrahan, Xerox Research Centre Europe, Jacki O’Neill, Microsoft Research India, Neha Gupta, Nottingham University

About the author

dbmartin

David Martin is a research scientist working for Xerox Research Centre Europe (XRCE) in Grenoble, France. He works for the Work Practice Technology group. His work involves employing ethnographic techniques to study work and technology use from an ethnomethodological perspective. These studies are used to understand the impacts of technology in real world settings and to feed into the design of innovative new applications. His work has covered a diverse set of topics ranging from ambulance control, to banking, healthcare, and colour printing and now is focused on crowdsourcing.

View all posts

12 Comments

  • Kevin, thanks for your comment. I work with people specialising in natural language processing research who had already pointed us to the article you wrote with Karen Fort. It is an interesting article with important arguments and was something that resonated with our research, and you are referenced in our paper. Part of our ‘mission’ is to try and get academics to realise that they have an ethical duty to pay a decent amount to people carrying out HITs. This is real work and not a leisure activity to most of them, and people are currently taking advantage.

  • “Ripping requesters off is not endorsed and is justified only against dubious requesters.”

    I just want to make clear that ripping off requesters is NEVER justified on Turker Nation. Ever. The second you stoop to their level, you’re just as big of a jerk as they are.

  • Thank you for responding to our post Spamgirl. We hope that you found the rest of the blog to be a fair representation of Turker Nation and Turker Nation members as far as that is possible, since Turker Nation has a clear set of policies, but Turker Nation members represent a greater diversity of opinions. So to clarify. I want to say that it is very clear that Turker Nation takes a clear position and strict stance on ‘ripping off requesters’. It is not endorsed, period. What we were getting at, which I do believe is reflected in the views of some members, is that dubious requesters – e.g. who try to rip off Turkers (do not pay them, issue unfair blocks etc.), who post dodgy or ‘illegal’ HITs sometimes get what is coming to them, when they become ‘victims’ of scamming etc. I want to make clear this difference. In no way were we saying that TN suggests its members go out to attack dodgy requesters. It was difficult to make this clear within the confines of the blog post.

  • I hope you understand that these are specifics for TURKERNATION, there are other foums such as mturkgrind, mturkforum, mturkwiki, reddit, and more that all do not have problems with sharing answers and collective cheating. It is sad that these places exist and make turking hard for us all. When a requester is not paying enough, they justify cheating that requester because the wage is so low. On turkernation, workers just avoid the low paying requesters

  • For the most part, requesters hold power over turkers. The experienced members on Turkernation know that if you attack a requester, they can block you and report you to Amazon. Even the “shady” requesters can cause you to have your worker account suspended or removed because they want to be vindictive about comments you make to them via email. It is best not to engage with any requesters who you do not trust, it is also not a good idea to be anything less than completely professional in any dealings with any requesters.

  • Thanks for responding Mike. Yes, I agree with you. Turker Nation is different. Other sites are a lot more ‘cavalier’ about ethics. I think that wanting to get back at ‘shady’ requesters is a completely understandable reaction, and everybody at least would like to see bad requesters held to account – Please Amazon! But as you say, particularly in your second post, quite apart from the ethical side to this, there are good practical reasons to avoid trying to hit back at bad requesters. They often have more power to hurt you anyway, and attacking them just ends up tainting Turkers and AMT in general. It’s all about trying to get things to work better in a constructive manner. Easier said than done, but naming, shaming and avoiding is a good first step.

  • Spreading fibs around is not a good look. Mturk Grind does NOT scam out requesters. We have a fully fleshed out mod team, including myself, to keep up with everything. Survey content and the like are always moderated. The only thing the grind is good at is making sure you make money fast on mturk, within the guidelines of Amazon’s and the requester’s policies.

  • Why was my response deleted? Stop spreading fibs around. Mturk Grind does not scam requesters, end of story. We have a fuller moderator team in order to prevent things like that. Our goal is to help turkers succeed and make the most out of mturk while keeping it safe.

  • Zingy – thanks for your post. Sorry, your post was not deleted. I have just been away from the computer and I need to approve all posts. It is good that you have come on to have your right of reply. I personally do not have much knowledge of Mturk Grind so I apologise if I was seen to be endorsing a position against you. This was not the case. I cannot speak for Mturk Grind and I would rather you did! I think your position – as I said above – is the most constructive and useful. Thanks for commenting.

  • Sorry about getting defensive, it’s just that a lot of hard work and time has been put into our community and the crowdsourcing community in general. Seeing it being blasted for no other reason than just forum wars is silly. If you do have any more questions, don’t hesitate to contact us at the forum. Thanks.

  • Thanks for responding Zingy. I understand where you are coming from since of course this is what Turker Nation members have tried to do when defending their forum. But it was wrong to lump “all other forums” together. Thanks for the offer to help with questions. I might well take you up on that since our research into Turkers and Turking is set to continue and we’d like to broaden our understanding.