Methodological Debate: How much to pay Indian and US citizens on MTurk?

This is a broadcast search request (hopefully of interest to many readers of the blog), not the presentation of research results.

When conducting research on Amazon Mechnical Turk (MTurk) you always face the question how much to pay workers. You want to be fair, to incentivize diligent work, to expedite recruiting, to sample a somehow representative cross-section of Turkers etc. For the US, I generally aim at $7.50 per hour, slightly more than the minimum wage in the US (although that is non-binding) and presumably slightly higher than the average wage on MTurk. Now I aim for a cross-cultural study comparing survey responses and experiment behavior of Turkers registered as residing in India with US workers. How much to pay in the US, how much in India? For the US it is easy: $7.50 * (expected duration of the HIT in minutes / 60). And India?

The two obvious alternatives are

  1. Pay the same for Indian workers as US workers: $7.50 per hour. MTurk is a global market place in which workers from many nations compete. It’s only fair to pay the same rate for the same work.
  2. Adjust the wage to national price level: ~ $2.50 per hour. A dollar is worth more in the US than in India. Paying the same rate leads to higher incentives for Indian workers and might bias sampling, effort, and results. According to The World Bank, the purchasing power parity conversion factor to market exchange ratio for India compared to the US is 0.3 (http://data.worldbank.org/indicator/PA.NUS.PPPC.RF). $7.50 in the US would make $2.25 in India. Based on The Economist’s BigMac index one could argue for $2.49 in India (raw index) to $4.5 (adjusted index; http://www.economist.com/content/big-mac-index). According to (Ashenfelter 2012, http://www.nber.org/papers/w18006) wages in McDonald’s restaurants in India are 6% of the wage at a McDonald’s restaurant in the US, which could translate to paying $0.45 per hour on MTurk. Given the wide range of estimates, $2.50 might be a reasonable value.

What should be the criteria to decide and which of these two is better?

I appreciate any comments and suggestions and hope that these will be valuable to me and to other readers of Follow the Crowd.

6 thoughts on “Methodological Debate: How much to pay Indian and US citizens on MTurk?

  1. I agree that adjusting for Indian workers is a good idea, but what kinds of tasks would you have only Indian workers work on?
    If you are looking for specific demographic or academic data from Indian workers, I would pay the same as in the US, because more than likely it is only going to be one hit available.
    If you are referring to batches of hits like business card transcription or receipt transcription, I would aim for $2 an hour at first, find the very best Indian workers and then qualify only them to work for you. Then I would raise it to between 2.50-3.00 an hour so they keep working for you and are able to provide for their families.

    • The task I’m considering is responding to an online survey for academic research. The survey asks questions on work conditions on MTurk and on other work opportunities Turkers might have. One element of the study is to check for differences between US Turkers and Indian Turkers. I assume there are differences, e.g. because prior studies reported that Indian Turkers tend to be more dependent on the income from MTurk than US Turkers.

      Thus, I do not only need Indian workers, I need both US and Indian workers. To control sample size, we will post different HITs and restrict access by country of residence.
      In the end it comes down to the question how to sample representative Turkers for a survey. There have been one or two papers discussing that the fact that a Turker signs up for a survey probably already implies that he is not representative but decided to specialize on surveys (by Panos Ipeirotis an co-authors, I think). That’s an issue probably not to be overcome — at least not via MTurks mechanism of posting HITs. If the sample is biased anyways, I would at least like to bias it equally in the US and India to make the cross-country comparison as good as possible.

      The three design parameters I have to target specific subsets of Turkers are the required qualifiactions (allowing me to select), the HIT description (affecting self selection), and the payment (affecting self selection). I have a strong feeling that qualifications and HIT description should be identical in both countries. With the payment I’m not so sure.
      I more and more tend to equal nominal pay in both countries.

    • It’s an interesting mechanism to raise the payment for qualified workers. I think it does not help me on this specific research project (see my other comment above) but I will keep that strategy in mind. It might be helpful elsewhere.

  2. Thanks for posting this Henner – I agree that it’s a really interesting question, and I’m excited to see what everyone thinks!

    My personal opinion on it is that you shouldn’t adjust pay for citizens in a different country in this type of study, for a few reasons:

    1. On the technical/validity side: If most of the tasks that are posted are posted with rates geared towards US Citizens (as I’ve seen in many research projects, with people aiming to pay $6/hour an hour and up to be somewhat in line with minimum wage), you’re biasing the sample of people who’ll be doing your task, as it will probably go to people who are new Turkers and less familiar with searching the platform for higher-paying tasks, or people who don’t yet meet the qualifications for other tasks or being Masters. So you risk getting a sample that’s not entirely representative.

    2. On the personal side: it just feels kind of weird to value their work differently than the same work done by US Citizens, just on the basis of their country of origin. I get that the conversion rate is huge, but pricing the tasks differently this explicitly feels weird. Plus, I think a lot of previous work has shown that money is (one of the) biggest motivators for MTurk use in India (eg. http://aaronshaw.org/Antin_Shaw-2012-Social_desirability_bias-CHI2012.pdf), so changing the amounts that you pay is something that shouldn’t be taken lightly.

    Like I said, really interested to hear other people’s opinions and find out what you decide to do eventually!

    -e

    • I fully agree on the personal side: Same work, same payment is fair.

      On the validity you have an interesting point: I did not yet link the payment to a Turker’s experience in searching for HITs. So far I only thought of your argument with regards to qualifications.

      Bottom line: One further argument for non-discriminatory payments. Thanks, Erin.

  3. Via e-mail I received the following comment: “If you pay Indian workers U.S. wages, you’ll get a lot of them and they’ll be grateful for the generous pay. If you pay the Indian workers Indian wages they’ll get mad at you (I’ve seen this happen). So in the worst case I’d pay them half way between the Indian and U.S. wages.”

Comments are closed.