Exploring Real-Time Collaboration in Crowd-Powered Systems Through a UI Design Tool

This blog post is a summary of our CSCW 2018 paper “Exploring Real-Time Collaboration in Crowd-Powered Systems Through a UI Design Tool” [1] by Sang Won Lee, Rebecca Krosnick, Sun Young Park, Brandon Keelean, Sach Vaidya, Stephanie D. O’Keefe, and Walter S. Lasecki from the University of Michigan.

Imagine while at lunch with colleagues the perfect idea pops into your head — a promising interface design for your team’s new app. You sketch it on a napkin to communicate the idea to your colleagues, and say you’ll create a higher fidelity mockup on your computer later in the day, after your slew of previously scheduled meetings. But what if you could create that higher fidelity mockup while still sitting at the lunch table? What if that higher fidelity mockup could be created immediately based on your sketch and verbal description?

Prior work from our team and collaborators enables end-users to do just this: sketch and verbally describe their desired UI and have a higher fidelity mockup created in real-time, by crowd workers from Mechanical Turk using visual drag-and-drop tools (Apparition [2] and SketchExpress [3]).

An overview of the continuous real-time crowdsourcing workflow in Apparition: a requester speaks and sketches their UI requirements to crowd workers who create a corresponding higher-fidelity UI in the shared canvas. Workers acknowledge requests and communicate questions to the requester via a chat box, and the requester can see progress on the canvas in real-time.

Feasibility studies [2, 3] showed that Apparition and SketchExpress could successfully coordinate crowd workers to create UI prototypes specified by a natural language description. However, those prior studies did not explore patterns in how different end-user requesters interact with such continuous real-time crowdsourcing systems and the challenges they face.

In our new paper at CSCW 2018 [1], we specifically explore how end-users communicate their UI requirements to workers, what differences there may be based on end-users’ design background, and what challenges they encounter. We recruited 10 participants, 5 with a background in design and 5 without, to act as individual requesters of the Apparition crowd-powered system. We provided them a paper sketch of a “to-do” app and asked them to communicate the requirements of the UI via sketch and verbal description. In one condition (T1) participants were told to communicate as if they were working with a colleague who later would receive their sketch and audio recording. In the second condition (T2) the same participants were told to communicate and work in real-time with crowd workers available on the Apparition platform. We summarize our results below:

• Requesters appreciate real-time collaboration with crowd workers: A major potential benefit of real-time collaboration (T2) over offline requests (T1) is the ability for requesters to immediately correct workers’ mistakes and answer workers’ clarifying questions; if working offline, it may take multiple exchanges to accomplish the desired prototype, especially if the requester’s initial instructions are ambiguous. During the post-study interviews, most participants commented on these benefits.
• Non-designers describe and reference visual features; designers use domain-specific language: Requesters who were designers often used more domain-specific language (e.g., “mobile device screen”) than requesters who were not designers, who instead used more basic descriptive language (e.g., “interface like a rectangular box, its height is bigger than the width”) along with more verbal (e.g., “this”), click, and gesture references to the canvas.

Average number of instances (per requester) of a particular description or reference technique when communicating with crowd workers.

• Crowd workers have trouble understanding design jargon: Crowd workers, who generally had minimal design experience, sometimes found designers’ domain-specific language challenging to understand. For example, when a designer requested a “hamburger menu”, one worker actually wrote the word “hamburger” on the canvas, clearly misunderstanding the requester’s intent:

The image on the left is the requester’s in-progress sketch of the desired UI, where they verbally describe the 3 horizontal lines at the top-right as a “hamburger menu”. The image on the right shows the workers starting to replace the sketch with higher-fidelity UI elements, and illustrates one worker misunderstanding the “hamburger menu” verbal description, adding a textbox with the text “Hamburger”.

• Having asymmetric communication modalities is confusing: In the Apparition system, requesters have the ability to verbally speak as well as type in a chat box to workers, but workers only have the ability to type in the chat box. During our study [1], we saw that this asymmetry in modalities caused confusion in a few ways:
  a) requesters did not always monitor the chat box,
  b) workers did not always acknowledge requests in the chat box,
  c) chat box messages reviewed in retrospect might not make sense without the corresponding verbal and canvas context (see figure below),
  d) some requesters felt uncomfortable that workers were not also able to speak.

Chat history over the span of 9 minutes of one session. It is difficult to understand worker questions and “ok” acknowledgments without context of the corresponding canvas state.

• Having a shared visual canvas used for both requests and work can be confusing and cluttered: Requesters drew their rough sketch on the same canvas that workers would later create their higher fidelity version. This resulted in some challenges:
  a) workers were sometimes hesitant to remove the requester’s sketched elements,
  b) when both sketched and higher fidelity element versions were present, the canvas appeared cluttered,
  c) when workers eventually did remove the requester’s sketched elements, the requester’s initial specifications were now gone and could no longer be referenced.
  Workers organically addressed some of these challenges by having a “sandbox” space for creating elements before replacing the sketched version, or by using a split canvas, keeping the original sketch intact on one side and creating the higher fidelity version on the other side.

A cluttered canvas, with checkboxes and text originally sketched by the requester, and higher-fidelity elements later overlaid by workers.

Based on our findings, we present recommendations for designing future continuous real-time crowdsourcing systems:

1. Turn requester’s speech into a structured task list to make requests easier for workers to revisit.
2. Reconsider requester and worker communication modalities to improve speed and effectiveness.
3. Preserve the requester’s original visual specification, even if in a shared workspace.
4. Improve requester and workers’ awareness of each others’ current activity.
5. Train workers not only to use visual tools, but also how to effectively collaborate.

Our vision is that direct collaboration with crowd workers will enable effective crowdsourcing of open-ended, creative tasks (e.g., content and UI creation, photo editing, data analysis, programming), and we believe there is much exciting work ahead in designing and studying such continuous real-time crowdsourcing systems. Please read our paper [1] for more details, and come see Sang Won Lee’s talk at CSCW!

References

[1] Sang Won Lee, Rebecca Krosnick, Sun Young Park, Brandon Keelean, Sach Vaidya, Stephanie D. O’Keefe, and Walter S. Lasecki. Exploring Real-Time Collaboration in Crowd-Powered Systems Through a UI Design Tool. In Proceedings of the ACM Conference on Computer-Supported Cooperative Work and Social Computing (CSCW 2018). New York, New York.

[2] Walter S. Lasecki, Juho Kim, Nicholas Rafter, Onkur Sen, Jeffrey P. Bigham, Michael S. Bernstein Apparition: Crowdsourced User Interfaces That Come To Life As You Sketch Them. In Proceedings of the International ACM Conference on Human Factors in Computing Systems (CHI 2015). Seoul, Korea.

[3] Sang Won Lee, Yujin Zhang, Isabelle Wong, Yiwei Yang, Stephanie D. O’Keefe, Walter S. Lasecki. SketchExpress: Remixing Animations For More Effective Crowd-Powered Prototyping Of Interactive Interfaces. In Proceedings of the ACM Symposium on User Interface Software and Technology (UIST 2017). Quebec City, Canada.