What Will Others Choose? How a Majority Vote Reward Scheme Can Improve Human Computation in a Spatial Location Identification Task

SpLIT

A spatial location identification task (SpLIT)

We created a spatial location identification task (SpLIT), in which humans examine a 3D camera view of an environment to infer its spatial location on a 2D schematic map (e.g. a floor plan). However, the SpLIT are often difficult to automate because the identification of salient cues often requires semantic features that are challenging to recover from the image and the map, especially when the cues are ambiguous or do not provide sufficient information to pinpoint the exact location.

One natural question to ask is how one can leverage human computation to perform a typical SpLIT. In order to find the answer, we designed a study using Amazon Mechanical Turk to investigate how turkers can perform the task in two kinds of reward schemes: ground truth and majority vote, even when they were not familiar with the environment.

We carefully chose five pictures represent different levels of ambiguity, in terms of the extent to which workers could use the pre-defined markers (cues) to infer the location of the camera view.

The results are listed below:

  1. The accuracies of each task under both schemes drop as the level of ambiguity increases. This suggests that the lack of spatial cues did make the task harder. And it is obvious from the figure that performance in majority vote scheme was better than that in the ground truth reward scheme.

    correctness_of_points

    Accuracies of SpLIT in each of the tasks for the two reward schemes.

  2. We also calculated the accuracies of individual participants, it showed that there were more turkers that at least chose one correct location in the majority vote scheme than in the ground truth one. In addition, the percentage with high ac- curacies (0.8 and 1.0) in the majority vote scheme is also higher than that in the ground truth one.

    correctness_of_people

    Cumulative histogram of the accuracies of indi- vidual workers.

  3. We did cluster analysis over the points for each picture under both schemes then calculated the average distances and the percentages of number of answers within each cluster. The result showed that the answers in the majority vote scheme were significantly more precise than those in the ground truth scheme, and in the majority vote scheme, the clusters with the largest percentage were more likely the correct ones.

The size of the circle represents the mean inner distance of the cluster and the stroke width of the circles represents the percentage of points in that cluster.

 

To summarize the results, we found that the majority vote reward scheme in general led to a higher level of precision and reliability in the answers provided by the workers.

Though there were not enough data to fully understand this result, the current study is clearly a first step towards understanding how human computation can be incorporated into applications that support spatial tasks

For more, see our full paper, What Will Others Choose? How a Majority Vote Reward Scheme Can Improve Human Computation in a Spatial Location Identification Task.

Huaming Rao, Nanjing University of Science & Technology and University of Illinois at Urbana-Champaign
Shih-Wen Huang, University of Illinois at Urbana-Champaign
Wai-Tat Fu, University of Illinois at Urbana-Champaign

This entry was posted in -, HCOMP 2013 by Huaming Rao. Bookmark the permalink.

About Huaming Rao

I’m a PH.D. candidate from Nanjing University of Science & Technology (NUST), Nanjing, P.R China, supervised by Prof. Chuancai Liu. major on artificial intelligence & pattern recognition, and now I'm studying in the University of Illinois at Urbana-Champaign (UIUC) as a visiting student working in the CASCAD Lab , supervised by Prof. Wai-Tat Fu. My research interests broadly lie in the fields of machine learning, crowdsourcing and social computing. I'm also an experienced web developer as a fan of Ruby on Rails, AngularJs, etc. BTW, I'm enthusiastic about TDD.