What do users’ comments tell us about designing better tutorial systems?

If you’ve spent any time using complex software, you’ve probably had help from web-based tutorials. Though they are community-created content, web tutorials serve as the de-facto source of help for many users of these applications.

We analyzed user comments posted on popular tutorials for Word, Photoshop, and Excel to understand:

  • How web tutorials are currently being used,
  • the social practices in their comment sections,

and,

  • what these findings imply for creating new and better tutorial systems.

Check out the graphic below for a high-level sketch of our findings. Afterward I’ll talk about two results that we found particularly interesting.

Our analysis revealed insights into how people use tutorials, and the social practices in their comment areas.

Our analysis revealed insights into how people use tutorials, and the social practices in their comment areas.

First, we found an unexpected use of web tutorials that we’ve termed “expert shadowing”. In this scenario the user attempts to recreate a complex end result by mimicking the actions of an expert. The user’s primary goal in this use appears to be recreational; the tutorial is allowing the user to experience using the software at a level of ability they couldn’t otherwise, and this is both challenging and rewarding. This is qualitatively different from reading a tutorial to learn a new skill, or applying its instructions to a current problem, and it suggests an opportunity for tutorial systems that are designed to explicitly create this type of rich experience for users.

Second, we identified social practices that produce valuable information that could help other users, or improve the tutorial content. For example, when users run into trouble following a tutorial, they sometimes post “help-me” stack traces describing where they got stuck and their actions leading up to the problem. This information could help subsequent users, but in current tutorial designs it’s typically stuck in a long list of comments at the bottom of the page, where it’s unlikely to be seen when it’s needed most. In the paper, we outline some ideas on how tutorial interfaces could embrace and support valuable social practices to enable tutorials that improve over time as users use and respond to them.

For full details see our ICWSM 2013 paper, Understanding the Roles and Uses of Web Tutorials.
Ben Lafreniere, University of Waterloo
Andrea Bunt, University of Manitoba
Matthew Lount, University of Manitoba
Michael Terry, University of Waterloo

4 thoughts on “What do users’ comments tell us about designing better tutorial systems?

  1. Thank you for the post!

    It’s interesting to see various types of conversation in online tutorial systems. I like the idea that the conversation can be used further to improve the tutorial systems.

    One question that I had was if the different types of comments often appeared together on the same tutorial page or usually on different pages.

    Thanks!
    Souneil

  2. Thanks for the comment!

    We found that the different social practices were mixed together in the comment sections of tutorials, and some of the practices were much more common than others. For example, comments thanking the author (Reader-Author Communication) were much more common than the other practices.

    This is another reason why we feel there’s an opportunity for redesigning tutorial interfaces. In current designs, comments that might be useful to other users of the tutorial can get lost in a long list of Reader-Author Communication comments.

  3. If you were to combine this with some of the insights of Adam’s work on query log analysis for feature popularity, and then make a complete guess: how commonly would you say that people use tutorials vs. other means of learning or experimentation in software these days?

  4. My intuition is that consulting tutorial content is extremely common, and may be the most common method of learning to use software right now. I have some evidence to back this up from lab studies where I’ve observed people performing tasks using GIMP. In this case, seeking help on the web is much, much more common than going to the built-in help system.

    My sense is that this would extend to other applications as well, but there are other learning contexts of use that I’m not sure about, and other learning methods that are not available during lab studies. For instance, I’m not sure how use of tutorials compares to asking for help from peers. My guess is that tutorials would still win, because Google is always a click away, whereas knowledgeable friends are not. But it’s hard to say definitively.

    I also suspect that some users are more comfortable going to the web for help than others. I get the sense (entirely unverified) that more technically adept users are more willing to immediately consult Google when they face a problem.

    It would be really interesting to try and answer some of these questions.

Comments are closed.