The Crowd at HICSS 2013 Series – #1

Motivation and data quality in a citizen science game: A design science evaluation

in Proceedings of the Hawai’i International Conference on System Science 2013

Kevin Crowston & Nathan R. Prestopnik
School of Information Studies
Syracuse University

Citizen science is a form of social computation where members of the public are recruited to contribute to scientific investigations. Finding ways to attract participants (i.e., motivation) and to ensure the accuracy of the data they produce (i.e., data quality) are key issues in making such systems successful. In this paper we describe the design and preliminary evaluation of a simple game that addresses these two concerns for the task of species identification.

In the game, called Happy Match, players are presented with a set of photographs of some organism (e.g., moths, sharks, rays). The players categorize each specimen on a set of characters, e.g., Shape at Rest, Forewing Main Colour, Forewing Distinctive Colour and Forewing Pattern for moths. For each character, there is a set of possible states, e.g., Arrow, Tent, Parallel, etc. for Shape at Rest. Each round of the game is seeded with one or two already-classified photographs, from which a score for the round can be calculated.

To evaluate the game on data quality primarily and motivation secondarily, we paid 200 workers from Amazon Mechanical Turk US$0.50 each to play. To motivate performance, we offered a bonus of US$0.50 for achieving a good score on the game. After playing, the workers filled out a survey about their impressions of the game. For this evaluation, we used photographs of moths for which we had known classifications to be able to compute data quality.

The main finding is that data quality was at an acceptable level for 3 out of 4 characters (all except Forewing Pattern). The pattern of errors gave us some ideas to improve the remaining character. Since we paid the AMT workers to play, it is difficult to determine the intrinsic motivation of the game. However, we did find that about 1/3 of workers played more games than required to be paid or to earn the bonus, suggesting that the game was motivating for at least some people.