Leaderboards are not only used competitively

Point scoring and leaderboards are one of many techniques used to encourage engagement in crowdsourcing activities. But do they have a motivational effect? How do people actually relate to them? ctd3

We studied the behavior of volunteers collecting data for an environmental organization, Close The Doors. They registered whether shops left their doors open or kept them closed during winter, using a mobile app while going about their everyday lives over a two week period. We compared the performance and attitudes of volunteers who scored points displayed on a leaderboard with those who used a control version of the mobile app – still collecting data, but no performance feedback.

We found that:

  • The top scorers in the points group substantially outperformed the top scorers in the control group.
  • But the lower scorers in the points group performed less well than the lower scorers in the control group.
  • Unless additional payment was used alongside points, there was no statistically significant difference in the data collection performance between those awarded points and the control group.

We conducted interviews with top, medium and low scorers in each group to understand what was happening.

  • The top scorers were motivated by the leaderboard, competing with those close to them and spurring each other on, This resulted in increased performance. So they performed better than the top scorers in the control group.
  • Low scorers were demotivated by the leaderboard, feeling they couldnt catch up and so gave up as the experiment progressed

Our CSCW2014 paper focuses on the attitude of those in the middle. Three of the four mid-scoring interviewees who were interviewed (unlike all but one of the top and low scoring interviewees) did not express competitive attitudes to the leaderboard. Rather, they viewed it as a means of understanding what other volunteers were doing, with the aim of making a typical contribution.

  • They were positively motivated to make a contribution on a par with others. One explicitly said they wanted to be in the middle of the leaderboard.
  • However the score required to be in the middle is determined by the performances of those below, not by those above.
  • So despite the positive motivation, the actual contribution of those in the middle was lower than those in the control group.

So some are motivated pr demotivated by competition, while others are motivated more by playing their part. Corwdsourcing systems could support the latter motivation by using normification in addition to gamification. This is to provide information about the behaviour of others in a way which encourages non-competitive comparison. Perhaps crowdsourcing systems could use adaptive, personalised interfaces to tailor the motivational information they provide based on the psychology of the individual.

For more, see our full paper, Competing or Aiming to be Average? Normification as a means of engaging digital volunteers.

Chris Preist – University of Bristol
Elaine Massung – University of Bristol
David Coyle – University of Bristol

This entry was posted in -, CSCW 2014 by ChrisPreist. Bookmark the permalink.

About ChrisPreist

Dr Chris Preist is Reader in Sustainability and Computer Systems at the University of Bristol Department of Computer Science. His research interests include (i) the application of digital technology (particularly social computing and mobile technology) to engage communities around environmental and social sustainability. (ii) The role of gamification techniques and social norm theories in such engagement (and, more broadly, in crowdsourcing and citizen science), and the analysis of the interaction between competitive and contributional motivations of participants. (iii) Environmental impact of digital technology, and more broadly consumer electronic goods. Particularly the understanding of user behaviour and how this affects such impacts. The role of interface design on influencing user behaviour with regard to environmental impacts ...

4 thoughts on “Leaderboards are not only used competitively

  1. I’m so glad you are studying this question. I personally get very demotivated my leaderboards, I’m glad I’m not the only one. I also think it is interesting to study the people in the middle. You said that, for people in the middle were motivated by the fact that “the score required to be in the middle is determined by the performances of those below, not by those above.” Does this imply in perhaps a cynical sense that the motivating factor for middlers was not to be the best, but to at least beat somebody? Otherwise, I’m not sure what to make of this.

    Great work! Very interesting problem! Thanks!

    • Thanks very much for your comment, Lydia.
      Just to be clear, the people in the middle were not motivated by the fact that “the score required to be in the middle is determined by the performances of those below, not by those above.” Rather, they are motivated to be a ‘typical’ contributer on a par with others, and they see that ‘being in the middle of the leaderboard’ is a measure of this. In our interviews, they were not coming out as wanting to ‘beat’ someone, but rather to ‘do their part’ and contribute alongside others.

      It is a consequence of the nature of the information a leaderboard presents and how it presents it that the score required to be ‘in the middle’ is set by those below, not above.
      We hypothesise in the paper that if the same information was presented in a different way – for example a scatterplot of contribution – then people would interpret ‘being in the middle’ differently. Effectively, a leaderboard interpreted in a ‘normalising’ way probably encourages median performance. Other means of displaying the same information could encourage mean or mode performance instead….

  2. Really interesting stuff! We also played with leaderboards and partial information (in a less controlled setting though – a field trial of a crowdsourced mapping tool – http://www.collabmap.org) as well and found that how you reveal information and how you couple monetary incentives with leaderboards can really change behaviour. This would go against one of your result on ‘low performers’ – leaderboards only have an impact if you reveal how hard it will be to beat the next one above (i.e., the points). Did you see hints of ‘the effect of ‘points’ visibility’?

    • Thanks for your comment, Gopal – I will check out collabmap….

      I dont quite understand what you are saying wrt your findings going against one of our results, and perhaps you can clarify what you found – but I will respond now by expanding on what we found: our leaderboards do display points for everyone. The people who were low on the leaderboard – who ended up in a poor position early on due either to lack of effort early on or due to circumstances such as being out of Bristol for a couple of days early in the trial – found the leaderboard demotivating as they felt there was no way they could ‘catch up’ with the leading group, and so they tended to ‘give up’. We didnt explore a setup with a leaderboard showing only position but not score. Did you?
      With regard to financial incentives – we also trialled a group who had finance linked to point scoring on the leaderboard. We observed that collectively they scored substantially more, but a z-score analysis showed that the distribution pattern was similar to the non-financial leaderboard group, rather than the control. In other words, the lower members experienced the same ‘demotivation’ from a relative perspective (and the qualitative data bears this out), though from an absolute perspective they did score more points than equivalent members in the control group.

Comments are closed.