A Workflow for Co-Creating Augmented Reality Experiences
Research for Design(ing)
01. The Challenge
As augmented reality (AR) continues to move away from the fringes and into the mainstream, interaction designers are increasingly confronted with a new medium that challenges our existing design workflows. Those who are tasked with designing and prototyping experiences situated in AR will quickly find that the process is both time and resource intensive. Take for example a common workflow:
In this scenario, designers move from low fidelity 2D sketches and then must translate these sketches into three dimensions using code heavy prototyping tools like that of Unity. It is only at this point when you can get these prototypes in the hands of users before continuing on to iterate and refine. While this workflow may prove fruitful for some, it may be worthwhile to ask how you can make the earlier phase of this process more robust by gathering insights from people(users) before jumping into code.
Despite the novelty of the medium and the associated technical challenges, there remains an opportunity to integrate, adapt, and expand upon existing design and co-creation methodologies to ensure that AR experiences are grounded in human needs and understandings.
02. Existing Methodologies
Particularly in the context of generative design research, co-creation workshops and toolkits are not without precedent. Take for example the work of Liz Sanders, whose generative “tools” allow designers and researchers to gain a better understanding of people’s dreams and aspirations by allowing participants to express themselves through the act of making.
“The projective quality of the toolkit components is deliberate. They are simple and ambiguous so that the participant can project his or her own aspirations onto the artifacts that they make…”
By providing participants with an intentionally simple palette of make “tools,” i.e., anything from legos to puppets to pipe cleaners, and a simple prompt/rule, they are given license to create freely. The resulting artifacts they produce are not design solutions but rather a reflection of their needs, understandings, and aspirations.[E. Sanders, Generative Tools for Co-Designing] Perhaps by gaining this more nuanced understanding of people through co-creating exercises, designers will be better equipped to address the complexities that arise from merging the physical and digital worlds.
In addition to looking at co-design methodologies more broadly, it is also important to acknowledge other efforts being made to enable the creation of low fidelity prototypes for Augmented Reality specifically. One of the more prominent examples of this comes from Apple during their 2018 World Wide Developers Conference. During the session titled, “Prototyping for AR,” designer Praveen Sharma describes using a combination of paper UI elements, rulers, toothpicks and other objects coupled with an iPhone and Keynote to simulate an Augmented Reality experience.
Sharma suggests that designers film the UI elements situated within the physical environment, and use keynote to overlay animations and other screen locked components. The resulting prototypes are effective tools to simulate an experience without having to invest time and resources involved in developing a functional product. However, the workflow outlined by Sharma is predicated upon the fact that there is already a pre-existing design to be tested, complete with schematic UI elements and content. Calling into question what a toolkit that was focused on creating, rather than simulating or evaluating, would look like.
03. Vignette, a Toolkit for Co-Creating AR Experiences
Building upon the existing landscape co-creating methodologies and low fidelity prototyping tools, the resulting creation focused toolkit consists of two components aimed at designers who are looking to introduce elements of co-design into their AR prototyping workflow. The first being a “scaffold” which leads designers through a series of prompts to help them craft an exercise that is aligned with their goals. The document (pictured above) asks designers to consider the following:
Scaffold Question 01. What is the question you are trying to answer?
Scaffold Question 02. What is a simple activity that might give users context to aid in answering the question?
Scaffold Question 03. How might a prototyping kit be used to gain a better understanding of the problem at hand?
The second component is a simple prototyping kit consisting of a series of acrylic panes which serve as a canvas for drawing in 3D space. Within the kit, designers will find a clear panel for sketching larger visualizations, while small panels of varying sizes and opacities allow users to layer information over an object they are looking to augment. While intended to be used in conjunction with the scaffold, the prototyping kit itself is intentionally left as a blank slate, giving designers the flexibility to use it for whatever prototyping task they see fit.
The resulting artifacts are intended to exist as a vignette, or snapshot of the user’s sketch situated within the physical environment which can, in turn, be used to inform the design of an Augmented Reality experience.
04. Putting it to the Test
In order to evaluate the effectiveness of the toolkit, I used the scaffold to design a series of workshop exercises that explore one the questions that I am actively looking at as part of my MDes thesis at Carnegie Mellon. These “questions” are intended to be focused and limited in scope and perhaps only a subset of the larger research question at hand. For instance, while my thesis looks to understand how one might leverage the learning and collaborative potential of AR to build a platform for the makers, the question explored as part of this exercise is one that has arisen through other research methods:
Scaffold Question 01: What is the question you are trying to Answer? How might A.R. be used to reveal unseen information about how things work?
Scaffold Question 02: What is a simple activity that might give users context to aid in answering the question?
Give participants a series of make tools, in this case LittleBits, and have them build a simple assembly based on a prompt.
With the research question established, I used to the scaffold to devise an activity that could be used to help give participants some context to aid in the prototyping exercise that would come later. Considering that the overarching theme was making, participants were given a limited palette of tools, i.e., LittleBits, and were asked to investigate them, become familiar with how they work, and ultimately go on to build an assembly. To facilitate this, they were first given a worksheet (shown above) that prompted them to indicate any questions they had about the tools in front of them. These worksheets served a dual role, giving participants a reference for later exercises but also giving me as the designer a detailed look at what people do and do not understand about the tools, which may later inform design explorations.
To continue to build a foundation for the prototyping exercise, participants were then asked to create a simple assembly using the Little Bits, “with a motor that changes direction when you more closer/further away, and turns on and off when you clap.” Although challenging for some, the activity allowed participants to gain a better understanding of how these tools worked through hands-on exploration.
Scaffold Question 03: How might a prototyping kit be used to generate potential solutions?
Based on your understanding of these tools, visualize the unseen process at work using the prototyping kit.
With a base level of understanding established and an assembly that could be “augmented,” participants were then introduced to the prototyping kit. With the kit in hand, they were tasked with visualizing the unseen processes at work within their assembly, paying particular attention to the questions they had raised during the earlier exercises. Much like in the co-creation exercises of Liz Sanders, participants were given the freedom to respond to the prompt in any way they like as long as it made sense to them.
The resulting artifacts were vignettes that reflected the participant’s understandings of the unseen processes at work in their assemblies, and of perhaps equal importance, the visualizations start to reveal the mental models they rely upon to convey concepts like sound, distance, and connection that are often difficult to make concrete.
Across each prototype, some commonalities emerged, for instance, the use of radiating waves and arrows to reflect the input from sensors and the use of dimension lines to indicate distance. While these representations were largely aligned with my expectations, I did not anticipate the participant’s tendency to anthropomorphize and personify the behaviors of their assemblies. In some cases manifesting itself simply as correlating sound sensors to depictions of ears but in others, the annotations were framed from the perspective of the assembly itself, i.e., “When you clap I turn on and off.” Additionally, each vignette attempted to address the processes at two scales, meaning they often visualized the behaviors of each bit using the smaller panels while using the large pane to speak to systems level connections and flows.
As noted earlier, the artifacts that resulted from the exercise were not solutions by design but rather a reflection of the understandings and mental models of people who will go on to be users. By introducing elements of co-design into the workflow for creating AR experiences, the resulting products may, in fact, be grounded in a more nuanced understanding of needs, questions, and aspirations of people.
05. Reflection
The resources for prototyping AR experiences in low fidelity are scarce and for a good reason. The very notion of merging physical and digital worlds brings about questions of how to bridge the gap between two and three dimensions as you make the leap from sketch to prototype. Despite this, however, there is an opportunity to adapt and expand upon existing methodologies by resituating them in this new context. As I work towards my MDes thesis, and in design practice more generally, time and resources are stretched thin furthering the need to fold people (users) into design workflows as early as possible, allowing them to do more than solely evaluate designs but giving them a platform to help create.
The toolkit outlined here is very much a work in progress but suggests an alternative means of approaching design problems situated in AR. The resulting workflow will continue to be deployed in workshop settings throughout the coming months in an effort to inform a series of making explorations as part of my thesis work.
06. Try it yourself
You can can access the exercise design scaffold and prototyping kit cut files via the Dropbox links below.
This project was created as of Carnegie Mellon’s Fall 2018 Master of Design Seminar III: Advanced Interaction & Service Design Concepts
Instructors: Dr Dan Lockton and Ahmed Ansari
