The International Workshop on Crowdwork and Human Computation CrowdWork’13 is a workshop held at the 3rd International Conference on Social Computing (SCA). We invite scientists and practitioners to submit their papers. The paper deadline is June 1st.
CROWDSOURCING CRITICAL THINKING
|
One of the challenges in using social media technologies, such as Twitter, for disaster response is that information that can help save lives is buried under the sea of other information and misinformation. This was the case in the aftermath of the 2011 Great East Japan Earthquake. For example, information on Twitter helped the rescuing of children and teachers who were stranded at a school building. However, finding this information was extremely hard because a lot of unverified tweets spread during disaster response, even after people pointed out that the unverified tweets were false rumors in their criticism tweets.
 Motivated by these observations, Tanaka, Sakamoto, and Matsuka examined if the critical thinking of crowds could help reduce the spread of misinformation. Using false tweets and criticism tweets related to the Great East Japan Earthquake, they conducted an experiment, in which half of the students in Japanese universities saw criticism tweets before seeing the false tweets, and the other half did not. They found that exposing subjects to criticism tweets increased the decision not to share the false tweets about 1.5 times, from 32% to 49%. When subjects decided to share the false tweets even after seeing the criticism tweets, they perceived the false tweets as more accurate, more important, and more anxiety-provoking than when they decided not to share the false tweets after seeing the criticism tweets. Their work, which won the best paper award in the Collaboration Systems and Technologies track, demonstrated that exposing people to criticism tweets could change their perceptions of and significantly reduce the decision to spread the associated false tweets.
 Given these findings, the group is examining how to promote the credibility evaluation by crowds to reduce the spread of misinformation and extract useful information on social media during disasters, and if it is possible to change how crowds perceive and feel about disaster-related information on social media to direct their sharing decision. Changing the perspective of crowds was the focus of another HICSS 2013 paper, which received a best paper nomination. By following this link you can find more about their research on improving social media for disaster response.
|
Information Exchange in Prediction Markets: How Social Networks Promote Forecast Efficiency
in Proceedings of the Hawai’i International Conference on System Science 2013
Liangfei Qiu - Department of Economics - University of Texas at Austin
Huaxia Rui - Simon School of Business - University of Rochester
Andrew B. Whinston - Department of Information, Risk and Operations Management - University of Texas at Austin
This paper studies the effects of information transmission on wisdom of the crowd. We provide a game-theoretic framework to resolve the question: Do social networks promote the forecast efficiency in prediction markets?
Our study shows that a social network is not a panacea in terms of improving forecast accuracy. The use of social networks could be detrimental to the forecast performance when the cost of information acquisition is high. We also study the effects of social networks on information acquisition in prediction markets. In the symmetric Bayes-Nash equilibrium, all participants use a threshold strategy, and the equilibrium information acquisition is decreasing in the number of participant’s friends and increasing in the network density. The aforementioned results are robust to two commonly used mechanisms of prediction markets: a forecast-report mechanism and a security-trading mechanism.
In the paper, we compare the performance of non-networked prediction markets (NNPM) with the performance of social-network-embedded prediction markets (SEPM). In the simulation, we use two measures of prediction market performance: the forecast accuracy and the mean squared errors (MSE) of the prediction market.
Figure #1 A & B – A Comparison between the Performances of the SEPM and the NNPM
Figure #1(a) – Forecast Accuracy
Figure 1(a) shows that when the cost of information acquisition is low, the SEPM outperforms the NNPM in terms of forecast accuracy, and when the cost is high, the NNPM outperforms the SEPM.
Figure #1(b) – MSE
In Figure 1(b), this result is robust to a different measure of prediction market performance: MSE. represents the MSE computed in the NNPM, and represents the MSE in the SEPM. When is small, , which means that the SEPM outperforms the NNPM. As increases, decreases, and when is large enough, the NNPM performs better than the SEPM.
There are two implications of this result. First, when the cost of information acquisition is low, a social network can enhance forecast accuracy in prediction markets. Second, a social network also has a negative effect on the forecast accuracy of a prediction market when the cost of information acquisition is high.
The paper at SSRN: http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2047904
Proponents of remix culture often frame remixing in terms of rich ecosystems where creative works are novel and highly generative. However, examples like this can be difficult to find. Although there is a steady stream of media being shared freely on the web, only a tiny fraction of these projects are remixed even once. On top of this, many remixes are not very different from the works they are built upon. Why is some content more attractive to remixers? Why are some projects remixed in deeper and more transformative ways?
We investigate these questions using data from Scratch — a large online remixing community where young people build, share, and collaborate on interactive animations and video games. The community was built to support users of the Scratch programming environment, a desktop application similar to Flash created by the Lifelong Kindergarten Group at the MIT Media Lab.
In our analysis, we found support for several popular theories about what makes projects remixable or generative: (1) Remixed projects are neither overly complex (i.e., too intimidating) nor too simplistic (i.e., vague and undefined); (2) Projects by prominent creators are more generative; (3) Remixes are more likely to attract remixers than de novo projects.
We also studied the originality of remixes and ask when remixing is more or less transformative. For example, a highly generative projects producing near-identical copies of previous projects may be viewed as less transformative or original. For a series of reasons — including the fact that increased generativity might come by attracting less interested, skilled, or motivated individuals — we suggest that each of the factors associated with generativity will also be associated with less original forms of remixing. We call this trade-off the remixing dilemma.
We find strong evidence of a trade-off:
Two plots of estimated values for prototypical projects. Panel 1 (left) display predicted probabilities of being remixed. Panel 2 (right) display predicted edit distances. Both panels show predicted values for both remixes and de novo projects from 0 to 1,204 blocks (99th percentile).
These results raise difficult, but important challenges, especially for designers of social media systems. For example, many social media sites track and display user prominence with leaderboards or lists of aggregate views. This technique may increase generativity by emphasizing and highlighting creator prominence while possibly decreasing the originality of the remixes elicited. Our results suggest that supporting increased complexity, at least for most projects, may have fewer drawbacks.
For more, see our full paper, “The remixing dilemma: The trade-off between generativity and originality.” Published in American Behavioral Scientist. 57-5, Pp. 643—663. (Official Link, Pay-Walled ).
Benjamin Mako Hill, Massachusetts Institute of Technology
Andrés Monroy-Hernández, Microsoft Research
Summary (TL;DR):
To deal with the massive influx of new editors between 2004 and 2007, Wikipedians built automated quality control tools and solidified their rules of governance. In our paper, we observe that these reasonable and effective strategies for maintaining the quality of the encyclopedia have come at the cost of decreased retention of desirable new editors.
The number of active editors (>=5 edits/month) is plotted over time for the English language Wikipedia.
The Story:
In 2006, the English Wikipedia faced an amazing opportunity; the open encyclopedia was growing exponentially both in new content and new contributors. With this success and growth, however, came a problem — anonymous vandalism.
In Wikipedia, content is contributed openly by Internet users, often anonymously. As the English Wikipedia gained in popularity, the potential for malicious activity grew, as well. Many feared that the vandals could overwhelm the good-faith editors tasked with keeping them at bay.
In response, Wikipedians constructed a complex immune system to fight vandalism, incorporating several strategies, including:
Our Results:
In early 2007, the English Wikipedia’s exponential growth in active editors changed directions and entered a steady decline. In this paper, we show that this decline was primarily due to a substantial drop in the retention of new, good-faith editors. Since 2007, desirable newcomers are more likely to have their work rejected, often through semi-autonomous vandal fighting tools (like Huggle). Furthermore, new users are being pushed out of policy articulation. During Wikipedia’s exponential growth period, Wikipedia’s policies and guidelines of behavior were effectively locked down against changes by new editors, and newcomers today struggle to find out where to ask for help.
For more, see our full paper, The Rise and Decline of an Open Collaboration System: How Wikipedia’s reaction to sudden popularity is causing its decline.
Aaron Halfaker, University of Minnesota
Stuart Geiger, University of California, Berkeley
Jonathan Morgan, University of Washington
John Riedl, University of Minnesota
The Theory of Crowd Capital
in Proceedings of the Hawai’i International Conference on System Science 2013
John Prpić & Prashant Shukla
Beedie School of Business
Simon Fraser University
We are seeing more and more organizations undertaking activities to engage dispersed populations through IT. Using the knowledge-based view of the organization, this work conceptualizes the theory of Crowd Capital to explain this phenomenon. A diagram of our model is shown immediately below.
Diagram of Crowd Capital Creation 
Crowd Capital is a heterogeneous knowledge resource generated by an organization, through its employ of Crowd Capability. An organization’s Crowd Capability engages the Dispersed Knowledge (Hayek 1945) of individuals –the Crowd.
Crowd Capability includes three dimensions by which an organization engages Dispersed Knowledge: a structure (some form of IT), content (the knowledge that the organization desires), and a process (internal work which sorts, filters, synthesizes, the incoming information).
Crowd Capital is always IT-mediated. In other words, forms of IT (web pages, mobile apps, sensors, software etc.) are always employed by organizations to engage the antecedent condition of Dispersed Knowledge.
Organizations exist in an environment of Dispersed Knowledge, hence, Dispersed Knowledge is not only external to the organization, but also can be engaged internally, externally or both simultaneously.
Crowd Capital can be generated through episodic or continuous forms of IT. Here we distinguish between forms of IT that necessitate community and collaboration to function, and those that do not. For example, we reason that Google’s ReCaptcha and Citizen Science applications like Foldit, do not require community and collaboration to generate Crowd Capital, whereas Innovation Communities (von Hippel 2005) and Peer Production (Benkler & Nissenbaum 2006) do.
If you’re interested, you can find a preprint copy of Prpić & Shukla (2013) here:
http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2193115
We very much look forward to your comments!
REAL-TIME TRANSCRIPTION is a vital accommodation for deaf and heard of hearing people in their daily lives. Captioning is typically expensive due to the years of training that is required.
LEGION:SCRIBE introduced a method that used multiple non-experts to caption audio with high quality and low latency, at far lower costs. We recently developed TimeWarp to help make the task easier for individual workers without hurting collective performance. TimeWarp makes each captionist’s job easier by selectively slowing down and speeding up the playback speed of the audio.
An example of time warping using 4 workers. The workers each individually hear their segment (matched by color) in slow motion. However, the crowd can collectively still keep up in real-time.
OFFLINE CAPTIONISTS OFTEN SLOW DOWN AUDIO to make it easier to caption. However, this necessarily puts the worker behind real-time. That’s fine for offline captioning, but means it can’t be used by one person and still keep up with real-time speech.
TimeWarp relies on:
For the parts of the audio workers as asked to type, the audio is played slower. In order to catch up with real-time, the audio is played slightly faster during parts in between where the worker listens for context.
WARPING TIME IMPROVES ACCURACY, COVERAGE, AND EVEN LATENCY. Our experiments showed:
The surprising improvement in latency is due to workers being able to keep up with each word as it was said, instead of memorizing it and then typing it later.
For more, see our full paper, Warping Time for More Effective Real-Time Crowdsourcing.
Walter S. Lasecki, University of Rochester
Christopher D. Miller, University of Rochester
Jeffrey P. Bigham, University of Rochester
Although crowdsourcing is a useful social computing technique, its unreliability has greatly undermined its utility. In this study, we found that carefully manipulating the social transparency and various peer-dependent reward schemes can successfully motivate crowds to generate high-quality work.
Previous research has shown that social transparency can make people more accountable for their own actions in online collaborative work. Nevertheless, it is not easy to utilize social transparency in crowdsourcing since crowd workers usually work individually. Our previous study on peer consistency evaluation demonstrated that simply making the rewards of crowd workers depend on each other can create social effects between workers, motivating them to perform significantly better. However, not all peer-dependent reward schemes create positive social effects for collaborative work. The possible social effects of peer-dependent reward schemes from previous literature are summarized below:
We conducted a 3X2 experiment by setting two levels of social transparency (low: anonymous, high: demographic information revealed) and three different peer-dependent reward schemes (individual, teamwork, competition). The main findings of our experiment are as follows:
1. Social transparency successfully motivated crowds to generate high-quality outcomes when their rewards were codependent.
When the workers worked individually, there was no significant difference between the workers who were anonymous and those who shared their demographic information. However, when making the rewards of the workers codependent, the difference between the performances of the two groups became significant. This shows that connecting the crowds is the key for us to utilize social transparency to enhance the reliability of crowdsourcing outcomes.
2. Social loafing harms the performance of crowds when only the collective outcomes are evaluated.
In team environments, the rewards of the workers were decided by the average of their performance and their teammates. We found that when a crowd worker was paired with a teammate that had good performance, they performed significantly worse. This result shows that social loafing really creates a negative effect on crowd work when the personal effort is difficult to evaluate.
3. Social facilitation was effective only when there was social transparency between the crowds.
In competitions, the rewards of crowds were decided by the positive difference between their performances and their opponent’s. We found that, when workers shared their demographic information, the workers were motivated to outperform their opponents. However, this effect did not exist when the participants worked anonymously, which indicates that social transparency makes social facilitation more effective.
For more, see our full paper, Don’t Hide in the Crowd! Increasing Social Transparency Between Peer Workers Improves Crowdsourcing Outcomes
Shih-Wen Huang, University of Illinois at Urbana-Champaign
Wai-Tat Fu, University of Illinois at Urbana-Champaign
Motivation and data quality in a citizen science game: A design science evaluation
in Proceedings of the Hawai’i International Conference on System Science 2013
Kevin Crowston & Nathan R. Prestopnik
School of Information Studies
Syracuse University
Citizen science is a form of social computation where members of the public are recruited to contribute to scientific investigations. Finding ways to attract participants (i.e., motivation) and to ensure the accuracy of the data they produce (i.e., data quality) are key issues in making such systems successful. In this paper we describe the design and preliminary evaluation of a simple game that addresses these two concerns for the task of species identification.
In the game, called Happy Match, players are presented with a set of photographs of some organism (e.g., moths, sharks, rays). The players categorize each specimen on a set of characters, e.g., Shape at Rest, Forewing Main Colour, Forewing Distinctive Colour and Forewing Pattern for moths. For each character, there is a set of possible states, e.g., Arrow, Tent, Parallel, etc. for Shape at Rest. Each round of the game is seeded with one or two already-classified photographs, from which a score for the round can be calculated.
To evaluate the game on data quality primarily and motivation secondarily, we paid 200 workers from Amazon Mechanical Turk US$0.50 each to play. To motivate performance, we offered a bonus of US$0.50 for achieving a good score on the game. After playing, the workers filled out a survey about their impressions of the game. For this evaluation, we used photographs of moths for which we had known classifications to be able to compute data quality.
The main finding is that data quality was at an acceptable level for 3 out of 4 characters (all except Forewing Pattern). The pattern of errors gave us some ideas to improve the remaining character. Since we paid the AMT workers to play, it is difficult to determine the intrinsic motivation of the game. However, we did find that about 1/3 of workers played more games than required to be paid or to earn the bonus, suggesting that the game was motivating for at least some people.
We introduce CommunityCompare, a visual analytic system to enable online community leaders to make sense of their community’s activity with comparisons.
Emma, an online community leader, wants to learn how she might increase member visits (views) by looking at example communities that are doing well on this metric. Each line in the main chart represents a community; Emma’s community is highlighted in orange. She wants to find example communities that have similar size and age compared to her community, and at the same time have more views than hers. To achieve this, she creates two visual filters on the community size and community age axes around her community, and creates another filter on the views axis above her community. She now sees 8 communities shown in the main chart and listed in the Select Communities widget. She wants to know what has been posted in these communities, so she clicks on the name of each community in the list to view their Most Valuable Posts below. She sees several examples of posts that give her new ideas for discussion topics in her community. She then saves the list of communities and decides to contact the leaders to learn more about their activities.
We motivate and inform the system design with formative interviews of community leaders. From additional interviews, a field deployment, and surveys of leaders, we show how the system enabled leaders to assess community performance in the context of other comparable communities, learn about community dynamics through data exploration, and identify examples of top performing communities from which to learn.
For more, see our full paper, CommunityCompare: Visually Comparing Communities for Online Community Leaders in the Enterprise.
Anbang Xu, University of Illinois at Urbana-Champaign
Jilin Chen, IBM Research – Almaden
Tara Matthews, IBM Research – Almaden
Michael Muller, IBM Research – Cambridge
Hernan Badenes , IBM Research – Almaden
Social Widgets powered by AB-WebLog.com.