People love looking at visualizations of data, prompting the creation of online systems like IBM’s Many Eyes and Data360 where groups of web users can gather to create, analyze, and discuss graphs. Organizations and collective knowledge at large benefit from the insights generated by groups as they collaboratively analyze socially-relevant data. Yet what seems a win-win situation falls short of effective when negative effects of group-think, like social influence, diminish the quality of the collective signal. Asch  first showed this in a well known experiment where subjects were asked to match the length of a given line to that of one of three lines lines of different length that were also shown. When confederates answered before the subject, and each picked the wrong line, the subject more often than not chose the wrong line as well.
In Asch’s work, the subject’s reliance on the erroneous group response is thought to be based on his desire to fit in with the group. It is also possible, though, that a person will rely on social information in even a simple task out of a desire to be accurate him or herself. We applied the concept of social proof, when a person looks to others’ behavior in deciding whether to engage in the activity him or herself, to a graph perception task in which subjects were motivated to be correct. We set up a series of proportion judgment and linear association estimation tasks on Amazon’s Mechanical Turk, and asked subjects to supply their best judgments with and without access to social information on other workers’ responses. We were interested in knowing how the final group response would be affected by the presence of a social signal. Prior responses for a given graph task were shown in a histogram representation with a peak value set to the actual group response when no social information was shown. Yet in order to see how the quality of the social information signal affected the new group response in our social conditions, we also created a set of histograms centered on a peak value one standard deviation from the actual non-social group response. Subjects in our social condition saw a mix of both types of social histograms: some with a peak at the “faithful” or near accurate response, versus others with a peak centered on the “offset”, biased response.
Our results showed that the lowest mean errors for a given graph task resulted when the Turkers saw the faithful social histogram that closely approximated the true value for the task. The mean errors for the group who saw no social information were only slightly higher. Yet those who saw the more biased offset social histogram with a peak value one standard deviation from the non-social group response made significantly more errors over both other conditions. In other words, the quality of the crowd’s response for the task depends on how accurately the social signal they are shown captures the truth.
This prompted a second question – how much does the amount of prior social information shown affect whether a new user relies the signal? We re-ran both social conditions for the same graphs, but this time, we systematically varied the number of responses shown in the graph. One might assume that less prior responses makes for a less “trustworthy” social signal, hence new users will rely less on the social information for their own judgments. Yet analysis showed that it didn’t matter how many responses are shown. In other words, a social signal based on as few as 1 or 5 prior responses was as good in the eyes of our subjects as one based on 50 scores. The implication is that a dynamic like an information cascade take hold when social signals are in place, with initial responses propagating across a community as new users weight the social answer over and above their private judgment.
In combination, our findings raise some challenging questions for the design of crowdsourcing systems for visual analytics. If social information can lead to less accurate group decisions, should the info be shown at all? Given our observation that the “faithful” social histogram centered on the non-social group response led to slightly lower errors than seeing no social information at all, it may be possible for social information to improve the group response under certain conditions. This possibility may well extend to other online systems where social information is displayed. Should social information be withheld by such systems instead, until a large enough number of responses have been gathered? What happens when systematic biases, or shared human tendencies to be biased in the same direction (such as to over or underestimate visualized quantities in a visual analytics context) , cause the group response to be inaccurate regardless? Are there ways that system designers can intelligently combine responses to get accurate collective signals, combining what we know about the social dynamics that can occur with knowledge of systematic biases affecting how people interpret visual and other information? These are just a few of the questions we are now considering.
You can read the full details of this research in our CHI 2011 publication:
Hullman, J., Adar, E., and Shah, P. 2011. The impact of social information on visual judgments. In Proc of CHI ’11. ACM, New York, NY, USA, 1461-1470.
You can download the paper here.
 Asch, S.E. Effects of group pressure upon the modification and distortion of judgment. In Groups, leadership and men
. Edited by H. Guetzkow. Pittsburgh, PA., 1951
About the author: Jessica Hullman (firstname.lastname@example.org, http://jhullman.people.si.umich.edu) is a PhD student at the University of Michigan. Her research looks at how the challenges presented to the quality of collective insight when non-expert users gather to analyze visualizations, and how those challenges might be overcome by design interventions at the graph or system level.