A central challenge in crowd computing is the workflow design problem: how can we divide a complex job — for instance, editing a paper or writing a computer program — into a sequence of microtasks that can be solved by a pool of crowd workers on the web? Effective workflow design is a difficult process, requiring careful task design, extensive software development, and iterated testing with a live crowd. The complexity of workflow design limits participation in crowdsourcing marketplaces to experts willing to invest substantial time and effort, and limits the kinds of tasks that can be crowdsourced today.
What if we could use the crowd to attack the workflow design problem itself? We present Turkomatic, a tool that allows requesters to collaboratively design and execute workflows in conjunction with the crowd.
Turkomatic accepts a requester’s specification of a task in natural language, then uses workers on Amazon’s Mechanical Turk to determine how to structure workflows to achieve the objective. While workers decompose the task and solve subtasks, the requester is able to monitor and edit the resulting workflows as they are produced. The resulting workflows are executed directly by the crowd and the results returned to the requester.
We induce the crowd to design and execute workflows on our behalf via a meta-workflow called Price-Divide-Solve (PDS). This crowd algorithm asks workers to recursively divide complex tasks into simpler ones until they are appropriately short for the price offered, then to solve them. Other workers are asked to verify the solutions and combine the results into a coherent answer to the original request. Turkomatic obviates the need for requesters to implement software or design tasks because it uses pre-structured task templates to interface with the crowd. PDS is potentially capable of generating workflows for a wide variety of tasks.
To allow the requester to give input during the workflow design process, Turkomatic provides an interface for visualizing and editing workflows in real time. Requesters can edit subtask descriptions and solutions, create new subtasks, and delete unwanted subtasks. As an alternate mode of operation, it is possible to seed the system with an initial workflow that can be refined by the crowd, enabling a collaborative design process to take place.
To explore how effectively crowds can be used to support the execution of complex work, we performed two evaluations. To provide a baseline for comparison, we examined how crowds performed in producing and solving workflows without the involvement of the requester – a “fire-and-forget” model of Turkomatic. Once this was established, we looked at how requester collaboration improved the crowd’s performance in task design. We examined a variety of complex tasks, including creating and populating a blog, planning a vacation, writing simple Java programs, web research, and essay writing.
As expected, in most cases, an unsupervised crowd produced unsuitable workflows or unsuccessful results. The most common mode of failure was derailment, a phenomenon that occurred when the PDS algorithm produced unnecessarily complex decompositions and failed to terminate. This occurred when workers in different parts of a workflow failed to understand the initial meaning of a task or to effectively understand the relationship of their step to others. For instance, the itinerary planning task resulted in several visits to the same location, and a task requiring editing. However, even the unsupervised crowd produced high-quality results for tasks that could be answered without substantial decomposition, successfully solving a task to write Java code and to compose an essay.
By comparison, the results from collaboration between requesters and the crowd were substantially better. When requesters used Turkomatic’s workflow editing tools to monitor and guide the crowd’s efforts, tasks of all categories we tested completed successfully by the crowd, and the workflows were usable. This, too, is unsurprising — intervention enabled requesters to provide feedback on their initial intentions and to iterate on unsuccessful tasks, and prevented crowds from operating in a vacuum.
The price-divide-solve approach represents an effort to produce a generic algorithm for crowdsourcing arbitrary work. This strategy has value in quickly evaluating the ability of crowds to solve particular kinds of work, and it can reduce the complexity of accessing crowd platforms for casual use. However, this one-size-fits-all strategy trades off ease of use for runtime supervision: while workflows can be generated without exhaustive planning, they require requester monitoring at runtime to guarantee quality of results. In future work, we plan to investigate to what extent this supervisory function can again be assigned to crowd workers. In any case, effective workflow design is among the most common problems facing crowdsourcing researchers today. Why not collaborate with workers in solving it?
For more, see our full paper, Collaboratively Crowdsourcing Workflows with Turkomatic.