CrowdCamp Report: Reconstructing Memories with the Crowd

From cave paintings to diaries to digital videos, people have always created memory aids that allow them to recall information and share it with others. How can the collective memories of many people be combined to improve collective recovery. For example, the layout of a community gathering place with sentimental or historical value could be recovered, or accidents and crimes may be explained using information that appeared trivial at first but actually has great importance.

Our CrowdCamp team set out to determine what some of the challenges and potential methods were for reconstructing places or things from the partial memories of many people.

Case Studies

We began by attempting to reconstruct common memories such as the layout of a Monopoly board. Figure 2 below shows our individual and collective attempts at this task. We found some facts that one group member recalled helped resurface related memories in other members. However, working together also introduced ‘groupthink’, where a false memory from one person corrupted the group’s final model. This is a known problem, and it is one reason why police prefer to interview witnesses separately.

Figure 1. Our reconstruction of a Monopoly board (left), compared to the true version (right).

The Effect of Meaningful Content on Memory

Next, we tried to see how information type changes the process. It’s well documented that people’s minds summarize information for better recollection. We tried 3 cases:

  • No meaning: Memorize a Sudoku puzzle (table of ordered numbers)
  • Some meaning: Memorize a set of about 30 random objects
  • Meaningful scene: Memorize a living room scene

For each, we first tried to memorize parts of the scene without coordination, then with predefined roles, e.g., different members were told to remember disjoint aspects or parts. In both cases we first wrote down what we remembered, then merged our results. Coordinated roles increased both recall and precision. Recall increased because the set of items we remembered individually was more distinct, meaning we did not redundantly memorize the same things. Precision increased because the more narrow task additional focused our attention by removing extra distractors.

Opportunities and Challenges

In some settings, prior domain knowledge allows people to organize for increased collective memory. One theme is that diversity aids in reconstruction. For example, one person may remember colors well while another may be color-blind but have a good spatial memory. Even outsiders who have no connection with the memory may be able to help.  For example, in Figure 2 below, a paid oDesk worker helps us remember our stressful first-day presentation at CrowdCamp by creating an illustration based on notes and images we provided.

An image depicting 4 presenters crying and one girl sitting at a desk in the background.

Figure 2. An oDesk worker’s rendition (left) of our stressful CrowdCamp presentation based on our notes and sketch we provided (above).

We identified three main challenges to reconstructing memories:

  • Groups, especially those containing members with strong personalities, are subject to groupthink, which can introduce errors.
  • Because some aspects of a scene are more salient, people’s memories often overlap significantly.
  • In unbounded settings, people’s accuracy decreases, likely due to an overwhelming amount of information

One consistent property was that we tended to remember nearly all of the information we could recall in total in the first few seconds or minutes, depending on the size of the task. After that, significant gains were only seen when one person’s idea jogged the memory of another.

Future Directions

We believe this work has great potential to introduce a more structured way to recreate memories using groups of people of all sizes, while avoiding problems encountered with naïve solutions. For example, approaches that mix individual recollection early on with later collaboration, while using parallel subsets of workers to minimize groupthink, could improve the way we recover knowledge in settings ranging from historical documentation to crime scenes.

What other ideas or references for recovering ideas can you think of? Anything we missed? We’d love to hear about it!

Adam Kalai, Microsoft Research

Walter S. Lasecki, University of Rochester / Carnegie Mellon University
Greg Little, digital monk
Kyle I. Murray, MIT CSAIL

CrowdCamp Report: HelloCrowd, The “Hello World!” of human computation

The first program a new computer programmer writes in any new programming language is the “Hello world!” program – a single line of code that prints “Hello world!” to the screen.

We ask, by analogy, what should be the first “program” a new user of crowdsourcing or human computation writes?  “HelloCrowd!” is our answer.

Hello World task

The simplest possible “human computation program”

Crowdsourcing and human computation are becoming ever more popular tools for answering questions, collecting data, and providing human judgment.  At the same time, there is a disconnect between interest and ability, where potential new users of these powerful tools don’t know how to get started.  Not everyone wants to take a graduate course in crowdsourcing just to get their first results. To fix this, we set out to build an interactive tutorial that could teach the fundamentals of crowdsourcing.

After creating an account, HelloCrowd tutorial users will get their feet wet by posting three simple tasks to the crowd platform of their choice. In addition to the “Hello, World” task above, we chose two common crowdsourcing tasks: image labeling and information retrieval from the web.  In the first task, workers provide a label for an image of a fruit, and in the second, workers must find the phone number for a restaurant. These tasks can be reused and posted to any crowd platform you like; we provide simple instructions for some common platforms.  The interactive tutorial will auto-generate the task urls for each tutorial user and for each platform.

Mmm, crowdsourcing is delicious

Mmm, crowdsourcing is delicious

More than just another tutorial on “how to post tasks to MTurk”, our goal with Hello Crowd is to teach fundamental concepts.  After posting tasks, new crowdsourcers will learn how to interpret their results (and get even better results next time).  For example: what concepts might the new crowdsourcer learn from the results for the “hello world” task or for the business phone number task?  Phone numbers are simple, right?  What about “867-5309” vs “555.867.5309” vs “+1 (555) 867 5309”?  Our goal is to get new users of these tools up to speed about  how to get good results: form validation (or not), redundancy, task instructions, etc.

In addition to teaching new crowdsourcers how to crowdsource, our tutorial system will be collecting a longitudinal, cross-platform dataset of crowd responses.  Each person who completes the tutorial will have “their” set of worker responses to the standard tasks, and these are all added together into a public dataset that will be available for future research on timing, speed, accuracy and cost.

We’re very proud of HelloCrowd, and hope you’ll consider giving our tutorial a try.

Christian M. Adriano, Donald Bren School, University of California, Irvine
Anand Kulkarni, MobileWorks
Andy Schriner, University of Cincinnati
Paul Zachary, Department of Political Science, University of California, San Diego

Y U No Do My Tasks?

Speeding up the completion and reducing the cost of crowdsourced work are two common goals of those using crowdsource labour. Making task more appealing is essential to meeting these goals. But, the number of people completing a task in a given time tells only half the story.

Y U No Do My Tasks?

Y U No Do My Tasks?

Understanding why crowd workers pick one task over another is essential to making maximum use of the available workforce in carrying out your task. In the same way that sunscreen lotion flies off the shelves in the summer, but is much less popular in winter, external factors may influence the behaviour and availability of the worker population at any given time.

To really understand what makes a task appealing, any metric must include a measure of the population. To factor in the number of people who might have chosen to complete a task, we propose to measure the conversion rate of tasks.

In our paper, we describe three conversion rate metrics all based on a simple ratio: the number of workers who complete a task, divided by the number who previewed the task.

For the number of workers completing a task to increase, only the number of available workers must increase (e.g. at the weekend). By including a measure of the available workforce, conversion rate metrics vary only when worker intent changes: when workers are primed to complete the task.

As pre-task interactions, including previews, are not made available to requesters using Mechanical Turk we developed a tool, turkmill, and offer this to aid others in gathering this information.

Graph showing conversion rate (at time 1.0), conversion rate over time, and nominal conversion rate (dashed line) for participants located outside of the US in our presentation and structure experiment.

Graph showing conversion rate (at time 1.0), conversion rate over time, and nominal conversion rate (dashed line) for participants located outside of the US in our presentation and structure experiment.

To demonstrate the usefulness of conversion rate metrics in understanding worker behaviour we carried out two experiments:

  • How does different presentation and structure of the task description affect the conversion rate for the task?
  • What impact does intrinsic motivation play on the conversion rate for tasks?
    • Do workers prefer non-profit organisations or commercial institutions as requesters?

As expected, we find that failing to disclose the length and difficulty of a task hinders worker uptake of the task. Unexpectedly, we also find that workers prefer tasks that are not branded by universities or non-profit requesters.

In summary, measuring the conversion rate for tasks can inform requesters about which features make a task attractive. It also improves the reproducibility of experiments by providing a measure of self-selection biases among workers.

For more, see our full paper, Crowdsourcing a HIT: Measuring Workers’ Pre-task Interaction on Microtask Markets at HCOMP 2013.
Jason T. Jacques, University of St Andrews
Per Ola Kristensson, University of St Andrews

What Will Others Choose? How a Majority Vote Reward Scheme Can Improve Human Computation in a Spatial Location Identification Task


A spatial location identification task (SpLIT)

We created a spatial location identification task (SpLIT), in which humans examine a 3D camera view of an environment to infer its spatial location on a 2D schematic map (e.g. a floor plan). However, the SpLIT are often difficult to automate because the identification of salient cues often requires semantic features that are challenging to recover from the image and the map, especially when the cues are ambiguous or do not provide sufficient information to pinpoint the exact location.

One natural question to ask is how one can leverage human computation to perform a typical SpLIT. In order to find the answer, we designed a study using Amazon Mechanical Turk to investigate how turkers can perform the task in two kinds of reward schemes: ground truth and majority vote, even when they were not familiar with the environment.

We carefully chose five pictures represent different levels of ambiguity, in terms of the extent to which workers could use the pre-defined markers (cues) to infer the location of the camera view.

The results are listed below:

  1. The accuracies of each task under both schemes drop as the level of ambiguity increases. This suggests that the lack of spatial cues did make the task harder. And it is obvious from the figure that performance in majority vote scheme was better than that in the ground truth reward scheme.


    Accuracies of SpLIT in each of the tasks for the two reward schemes.

  2. We also calculated the accuracies of individual participants, it showed that there were more turkers that at least chose one correct location in the majority vote scheme than in the ground truth one. In addition, the percentage with high ac- curacies (0.8 and 1.0) in the majority vote scheme is also higher than that in the ground truth one.


    Cumulative histogram of the accuracies of indi- vidual workers.

  3. We did cluster analysis over the points for each picture under both schemes then calculated the average distances and the percentages of number of answers within each cluster. The result showed that the answers in the majority vote scheme were significantly more precise than those in the ground truth scheme, and in the majority vote scheme, the clusters with the largest percentage were more likely the correct ones.

The size of the circle represents the mean inner distance of the cluster and the stroke width of the circles represents the percentage of points in that cluster.


To summarize the results, we found that the majority vote reward scheme in general led to a higher level of precision and reliability in the answers provided by the workers.

Though there were not enough data to fully understand this result, the current study is clearly a first step towards understanding how human computation can be incorporated into applications that support spatial tasks

For more, see our full paper, What Will Others Choose? How a Majority Vote Reward Scheme Can Improve Human Computation in a Spatial Location Identification Task.

Huaming Rao, Nanjing University of Science & Technology and University of Illinois at Urbana-Champaign
Shih-Wen Huang, University of Illinois at Urbana-Champaign
Wai-Tat Fu, University of Illinois at Urbana-Champaign

CrowdScale 2013: Call for Position Papers and Shared Task Challenge ($1500 in prizes)

CrowdScale 2013: Crowdsourcing at Scale
A workshop at HCOMP 2013: The 1st Conference on Human Computation & Crowdsourcing
November 9, 2013

Crowdsourcing and human computation at scale raises a variety of open challenges vs. crowdsourcing with smaller workloads and labor pools. We believe focusing on such issues of scale will be key to taking crowdsourcing to the next level – from its uptake by early adopters today, to its future as how the world’s work gets done. To advance crowdsourcing at scale, CrowdScale will pursue two thrusts:

Track 1: Position Papers. We invite submission of 2-page position papers which identify and motivate focused, key problems or approaches for crowdsourcing at scale.

Track 2: Shared Task Challenge. We invite submissions to a shared task challenge on computing consensus from crowds: how to generate the best possible answer for each question, based on the judgments of five or more raters per question.  Participants will submit 4-page papers describing their systems and preliminary results, with $1500 in prize money awarded to top performers.

One may participate in either or both tracks. Submitted papers will not be peer-reviewed or archived, so work shared in these papers can be later submitted to peer-reviewed venues. All papers will be posted on the workshop website to promote discussion within and beyond workshop participants. Workshop organizers will review all submissions to ensure quality, with high acceptance expected.

Position Papers
We invite submission of short (2-page) position papers which identify and motivate key problems or potential approaches for crowdsourcing at scale.  We encourage submissions identifying and clearly articulating problems, even if there aren’t satisfactory solutions proposed.  Submissions focusing on problems should clearly describe a problem of scale, why it matters, why it is hard, existing approaches, and desired properties of effective solutions.  We welcome early work, and particularly encourage submission of visionary position papers that are forward looking.

Each submitted paper should focus on one problem. We encourage multiple submissions per author for articulating distinct problem statements or methods.

Position papers are welcome to argue the merits of an approach or problem already published in earlier work by the author (or anyone else). In this case, the approach should be clearly attributed to the prior work, and the contribution of the position paper would be its argument of why the approach is promising for crowdsourcing at scale.

During the workshop, authors will self-organize into break-out groups, with each group further elaborating upon a particular critical area meriting further work and study. Each group will summarize and report its findings at the workshop’s close. In continuing discussion beyond the workshop, organizers and participants will co-author a summary paper articulating a road map of important challenges and approaches for our community to pursue.

Position paper ideas include (but are not limited to):

Shared Task Challenge
To help advance research on crowdsourcing at scale, CrowdFlower and Google are sharing two new, large challenge datasets for multi-class classification. Both datasets are available for immediate download. To make it easy to participate, we have provided multiple formats of the data, and pointers to open source software online that is available to get started.

All participants are expected to submit a paper (up to 4 pages) describing one’s method and preliminary results on shared task metrics, and to present a poster at the workshop. Final results will be announced at the workshop, with prize money awarded to the best performer(s), as well as recognition during the workshop and in our workshop report.

Shared task participants are also invited to participate in workshop discussion throughout the day.

Important Dates
October  14: Position papers due
October 20: Shared task runs due
October 27: Shared task papers due
November 9: Workshop

Please see workshop website for additional information on schedule.

Questions: Email the organizers at:

Workshop Organizers
Tatiana Josephy, CrowdFlower
Matthew Lease, University of Texas at Austin
Praveen Paritosh, Google

Advisory Committee
Omar Alonso, Microsoft
Ed Chi, Google
Lydia Chilton, University of Washington
Matt Cooper, oDesk
Peng Dai, Google
Benjamin Goldenberg, Yelp
David Huynh, Google
Panos Ipeirotis, Google/NYU
Chris Lintott, Zooniverse/GalaxyZoo
Greg Little, oDesk
Stuart Lynn, Zooniverse/GalaxyZoo
Stefano Mazzocchi, Google
Rajesh Patel, Microsoft
Mike Shwe, Google
Rion Snow, Twitter
Maria Stone, Microsoft
Alexander Sorokin, CrowdFlower
Jamie Taylor, Google
Tamsyn Waterhouse, Google
Patrick Philips, LinkedIn
Sanga Reddy Peerreddy, SetuServ

Call for Papers: Disco 2013 Human Computation Games Workshop At HComp 2013

Extending and building upon the focus of past workshops on games and human computation, the workshop Disco aims at exploring the intersection of entertainment, learning and human computation. Disco is held at HComp2013 in Palm Springs, November 9, 2013.



Both long (6 pages) and short/position papers (2 pages) can be submitted
In AAAI format:
Via easychair:

Important Dates:

  • Paper submission

    September 20, 2013

  • Notification of acceptance

    October 10, 2013

  • Camera ready submission (tentative)

    October 15, 2013

  • Workshop

    November 9, 2013

With the Internet being used worldwide, the way we think about communication, computation, artificial intelligence and research is changing. Human computation has emerged as a powerful approach to solving problems that would not be tractable without humans in the loop. Within human computation, games called games with a purpose or serious games are a successful approach to incite people to collaborate in human computation. Games are also for human means to learn.

Digital games are interaction machines and, how implicit it might be, always contain a learning component. Whether one is stacking blocks, exploring dungeons, or building cities games provide a variety of human machine interactions that range from simple puzzles to complex problem spaces. The challenges that emerge through these mechanics are precisely what foster human learning during the course of a game. Recently, human computation systems have tried to leverage the insights people have for solving these problems by observing, and automatically learning from, the interactions between players and choices made by players.

The workshop Disco is devoted to exploring the relationships between entertainment, learning, and human computation. The workshop has several goals. First, the workshop will investigate games as powerful incentives for human both to learn and to engage in human computation. Second, the workshop will pay attention at how learning can be seamlessly integrated into human computation tasks so as to improve both a player’s experience and a human computation system. Third, the workshop will explore how learning relates to entertainment and games. Finally, to close the loop, the workshop will investigate how human computation can improve the content, design and playability of games.

We are looking forward seeing you in Palm Springs:
François Bry, (Ludwig-Maximilians-University)
Markus Krause, (Leibniz University)


HCOMP 2013: Call for Papers

We invite you to join us for the first AAAI Conference on Human Computation and Crowdsourcing (HCOMP-2013) on November 7-9, 2013 in Palm Springs, California, USA

Submission deadline for Papers: May 10, 2013
Paper Format: AAAI, up to 8 pages

Submission deadline for Workshops and Tutorials: May 10, 2013
Submission deadline for Posters & Demonstrations: July 25, 2013
Format for Proposals, Posters and Demos: AAAI, up to 2 pages

HCOMP is aimed at promoting the scientific exchange of advances in human computation and crowdsourcing among researchers, engineers, and practitioners across a spectrum of disciplines. The conference was created to serve as a key focal point and scholarly venue for the review and presentation of the highest quality work on principles, studies, and applications of human computation. The meeting seeks and embraces work on human computation and crowdsourcing in multiple fields, including human-computer interaction, cognitive psychology, economics, information retrieval, economics, databases, systems, optimization, and multiple sub-disciplines of artificial intelligence, such as vision, speech, robotics, machine learning, and planning.

Submissions are invited on efforts and developments on principles, experiments, and implementations of systems that rely on programmatic access to human intellect to perform some aspect of computation, or where human perception, knowledge, reasoning, or physical activity and coordination contributes to the operation of larger computational systems, applications, and services. Submissions will be reviewed by a program committee of leading researchers from multiple areas – the complete committee list is available online.

The conference will include presentations of new research, poster and demo sessions, and invited talks. A day of workshops and tutorials, as well as a two-day CrowdCamp, will follow the main conference.

We hope you’ll submit your best work to HCOMP and look forward to seeing you in Palm Springs.

Eric Horvitz (Microsoft Research) and Bjoern Hartmann (UC Berkeley)
Conference Chairs

Announcing HCOMP 2013 – Conference on Human Computation and Crowdsourcing

Bjoern Hartmann, UC-Berkeley 
Eric Horvitz, Microsoft Research

Announcing HCOMP 2013, the Conference on Human Computation and Crowdsourcing,  Palm Springs, November 7-9, 2013.  Paper submission deadline is May 1, 2013.  Thanks to the HCOMP community for bringing HCOMP to life as a full conference, following on the successful workshop series.

HCOMP 2013 at Palm Springs

The First AAAI Conference on Human Computation and Crowdsourcing (HCOMP 2013) will be held November 7-9, 2013 in Palm Springs, California, USA. The conference was created by researchers from diverse fields to serve as a key focal point and scholarly venue for the review and presentation of the highest quality work on principles, studies, and applications of human computation. The conference is aimed at promoting the scientific exchange of advances in human computation and crowdsourcing among researchers, engineers, and practitioners across a spectrum of disciplines.  Papers submissions are due May 1, 2013 with author notification on July 16, 2013.  Workshop and tutorial proposals are due May 10, 2013.  Posters & demonstrations submissions are due July 25, 2013.

For more information, see the HCOMP 2013 website.