Reactive and Multi-platform Crowdsourcing

An essential aspect for building effective crowdsourcing computations is the ability of controlling the crowd. This means dynamically adapting the behaviour of the crowdsourcing systems in response to:

  • the quantity and timing of completed tasks
  • the quality of responses and task results
  • the profile, availability and reliability of performers.

At the purpose of controlling the crowd, we bring together various ingredients:

  • crowdsourcing: we define an abstract model of crowdsourcing activities in terms of elementary task types (such as: labelling, liking, commenting, sorting, grouping) performed upon a data set, and then we define a crowdsourcing task as an arbitrary composition of these task types, according to the model below.
Model of a multi-platform crowdsourcing campaign

Model of a multi-platform crowdsourcing campaign.

  • social networking: We show how social platforms, such as Facebook or Twitter, can be used for crowdsourcing search-related tasks, side by side with traditional crowdsourcing platforms.
  • multi-platform integration: we allow deployment of abstract crowdsourcing tasks to several invitation and execution platforms, including Twitter, Facebook, Amazon Mechanical Turk, and we collect and aggregate results from every platform. Performance of crowds depends on the execution platform (e.g., Facebook and Twitter immediately collect a lot of responses but then more professional platforms like Doodle or Linkedin outperform them), on the task type (simpler tasks are responded more frequently), on the posting time, the topic, and the language of tasks.
the selection of the task execution platform might influence the time required to get answers from the crowd (Facebook features less latency, but Doodle brings more answers)

The selection of the task execution platform might influence the time required to get answers from the crowd (Facebook features less latency, but Doodle brings in more answers in the long term).

  • expertise finding: we analyze how performer profiling can be enriched with the social activity of the performer himself and of his friends or social connections. Experiments show how different profiling options can impact on the quality and efficiency of crowdsourcing campaigns.
  • reactive rules: reactive control is obtained through rules which are formally defined and whose properties (e.g., termination) can be easily proved. Rules are defined on top of data structures which are derived from the model of the application. Rules are written in reactive style, according to the ECA (Event-Condition-Action) paradigm and allow making decisions about the production of results, the classification of performers (e.g., identification of spammers), the early termination and re-planning of tasks based on some performance measures, the dynamic definition of micro-tasks, and so on.
Simple changes in the declarative reactive rules can significantly impact on the task quality and cost. Each curve represents a rule set for controlling majority agreement, respectively: simple majority after 7 evaluations (black line); strong majority after 3 evaluations or simple majority after 7 evaluations (red line); and an additional simple majority evaluation step after 5 evaluations (blue line).

Simple changes in the declarative reactive rules can significantly impact on the task quality and cost. Each curve represents a rule set for controlling majority agreement, respectively: simple majority after 7 evaluations (black line); strong majority after 3 evaluations or simple majority after 7 evaluations (red line); and an additional simple majority evaluation step after 5 evaluations (blue line).

Reactive rules allow significant savings in terms of execution time and number of executions, as well as improvements in precision of results. 

Our system is implemented as a cloud service, where crowdsourcing campaigns are configured through a Web user interface (see image below) or through API.

Configuration user interface of our multi-platform and reactive crowdsourcing platform, CrowdSearcher

Configuration user interface of our multi-platform and reactive crowdsourcing platform, CrowdSearcher (in this step the developer adds task type, title and textual description).

A short demo video of the system is available on YouTube:

For more, see our full paper, REACTIVE CROWDSOURCING (WWW 2013). [Slides]
Alessandro Bozzon, TU Delft
Marco Brambilla, Politecnico di Milano
Stefano Ceri, Politecnico di Milano
Andrea Mauri, Politecnico di Milano

Other relevant papers:
Choosing the right crowd: expert finding in social networks (EDBT 2013).
Answering search queries with CrowdSearcher (WWW 2012).