Voyant: Generating Structured Feedback on Visual Designs Using a Crowd of Non-Experts

Crowdsourcing offers an emerging opportunity for users to receive rapid feedback on their designs. A critical challenge for generating feedback via crowdsourcing is to identify what type of feedback is desirable to the user, yet can be generated by non-experts. We created Voyant, a system that leverages a non-expert crowd to generate perception-oriented feedback from a selected audience as part of the design workflow.

The system generates five types of feedback: (i) Elements are the individual elements that can be seen in a design. (ii) First Notice refers to the visual order in which elements are first noticed in the design. (iii) Impressions are the perceptions formed in one’s mind upon first viewing the design. (iv) Goals refer to how well the design is perceived to meet its communicative goals. (v) Guidelines refer to how well the design is perceived to meet known guidelines in the domain.

Voyant decomposes feedback generation into a description and interpretation phase, inspired by how a critique is taught in design education. In each phase, the tasks focus a worker’s attention on specific aspects of a design rather than soliciting holistic evaluations to improve outcomes. The system submits these tasks to an online labor market (Amazon Mechanical Turk). Each type of feedback typically requires a few hours to generate and costs a few US dollars.

Our evaluation shows that users were able to leverage the feedback generated by Voyant to develop insight, and discover previously unknown problems with their designs. For example, the Impressions feedback generated by Voyant on a user’s poster (see the video above). The user intended it to be perceived as Shakespeare, but was surprised to learn of an unintended interpretation (see “dog” in word cloud).

To use Voyant, the user imports a design image and configures the crowd demographics. Once generated, the feedback can be utilized to help iterate toward an effective solution.

Try it: http://www.crowdfeedback.me


For more, see our full paper, Voyant: Generating Structured Feedback on Visual Designs Using a Crowd of Non-Experts.
Anbang Xu, University of Illinois at Urbana-Champaign
Shih-Wen Huang, University of Illinois at Urbana-Champaign
Brian P. Bailey, University of Illinois at Urbana-Champaign

About the author

Anbang Xu

Anbang Xu is a Ph.D. candidate in Computer Science at the University of Illinois at Urbana-Champaign, specializing in Human-Computer Interaction. His research in HCI focuses on the design, implementation, and evaluation of crowd-powered systems for supporting creative activities. His research interests include crowdsourcing, creativity support tools, and visualization. He is also the main founder of www.crowdfeedback.me, which is based on his dissertation on Voyant.

View all posts


  • Very neat! Seems like it would also work nicely for large teams in situations where in-person crits are not practical. In particular, this might be a fun in-class tool: both for peer-feedback and for facilitating crits of third-party designs.

  • Hi Krzysztof, thank you for your insightful comments! Our system currently recruits workers from an online labor market, but the system could be applied to a large design course and students in the course could be used as a crowd.

  • Pretty nice post. I just stumbled upon your blog and wanted to say that I’ve
    truly enjoyed browsing your blog posts. In any case I
    will be subscribing to your feed and I hope you write again very soon!

  • I will immediately take hold of your rss as I can’t find your e-mail subscription hyperlink or newsletter service. Do you’ve any? Kindly allow me recognize in order that I could subscribe. Thanks.
    CADEN http://www.net-ict.be/