Expanding the Design Process for Mixed Reality
How to help your team ideate, communicate, and avoid paradigms when designing mixed reality experiences
As Microsoft launched the HoloLens to an audience of eager developers in 2016, the team had already partnered with studios inside and outside of Microsoft to build the device’s launch experiences. These teams learned by doing, finding both opportunities and challenges in the new field of mixed reality design.
To help new teams and partners innovate more effectively, we turned their methods and insights into a curriculum of design and development lessons that we teach to developers in our Mixed Reality Academy (including week-long design workshops we offer to our enterprise partners).
Below is a snapshot of these learnings, part of a larger process we use to help our enterprise partners prepare their teams for mixed reality development. While many of these methods were originally targeted for HoloLens development, ideating and communicating in 3D are critical to the full spectrum of mixed reality experiences.
Thinking spatially during the design process
Any design process or design framework is meant to iterate thinking: To approach a problem broadly, to share ideas with others effectively, to evaluate those ideas and reach a solution. Today, we have well-established design and development methods for building experiences on desktops, phones, and tablets. Teams have clear expectations of what is necessary to iterate an idea and turn it into a product for users.
Often teams are a mix of development, design, research, and management, but all roles can participate in the design process. The barrier to entry to contribute an idea for a mobile app can be as simple as drawing a rectangle for the device’s screen. While sketching boxes and lines for the UI elements, with arrows to indicate motion or interactions, can be enough to establish technical requirements or define potential user behavior.
With mixed reality, the traditional 2D design process begins to break down: sketching in 3D is difficult for most people and using 2D tools, like pen and paper or whiteboards, can often limit ideas to those dimensions. Meanwhile 3D tools, built for gaming or engineering, require a high degree of proficiency to quickly flesh out ideas. The lack of lightweight tools is compounded by the technical uncertainty inherent with new devices, where foundational interaction methods are still being established. These challenges can potentially limit the design contributions of your team to only those with 3D development backgrounds — drastically reducing the team’s ability to iteration.
When we work with external partners, we hear stories of teams ‘waiting for the developer to finish the prototype’ before they can continue with their design process. This often means the rest of the team is blocked in making meaningful progress on the product, overloading the developer to solve both the technical implementation as well as major components of the user experience (as they attempt to put a rough idea into code).
Techniques for expanding the design process
Our teams at Microsoft have a set of techniques to more effectively include their team and quickly iterate through complex design problems. While not a formal process, these techniques are intended to supplement rather than replace your workflow. These methods allow those without specialized 3D skills to offer ideas before diving into the prototyping phase. Different roles and disciplines (not just 3D designers and developers) can be part of the design process, uncovering opportunities and identifying possible challenges that might occur later in development.
Generating ideas with bodystorming
Getting your team to think about events occurring in the real world, beyond the traditional world of 2D devices, is key to developing innovative mixed reality experiences. At Microsoft, we have found the best way to do this is to encourage interaction with physical props in a real-world space. Using simple, cheap crafting materials we build physical props to represent digital objects, user interfaces, and animations in a proposed experience. This technique is called bodystorming and has been a staple of product ideation within industrial design for decades.
Simple, physical props level the playing field for participants, allowing individuals with different skill sets and backgrounds to contribute ideas and uncover opportunities inherent to mixed reality experiences instead of being locked into the paradigm of 2D thinking. While technical prototyping or high-fidelity storyboarding requires a skilled 3D developer or artist, a few Styrofoam balls and cardboard can be enough to showcase how an interface might unfold in physical space. These techniques apply to both mixed reality development with HoloLens and the immersive headsets. For example, a set of plastic connectors might roughly illustrate the size of holograms that appear in a HoloLens experience or as props to act out interactable elements or motion designs in a virtual world.
The purpose of bodystorming is to ideate quickly and efficiently by encouraging participants to think spatially.
Bodystorming is a technique to quickly generate ideas and evaluate ideas that are too nebulous to prototype. At Microsoft, bodystorming is most commonly used to quickly vet an idea, although it can be helpful to host a more in-depth session if you involve outside stakeholders who are not familiar with mixed reality development or need to distill very broad scenarios. Remember that the purpose of bodystorming is to ideate quickly and efficiently by encouraging participants to think spatially. Detailed artwork or precise measurements are not important at this stage. Physical props need only meet the minimum requirements to explore or communicate an idea. Ideas presented through bodystorming are not expected to be fully vetted, but the process can help narrow down possibilities to test later during the in-device prototyping phase. As such, bodystorming is not intended to replace technical prototyping, but rather offset the burden of solving both technical and design challenges during the prototyping phase.
Acting and expert feedback
Following the bodystorming process of ideating with physical objects in the real world, the next step is to walk through an experience with these objects. We call this phase of the process acting and it often involves staging how a user would move through the experience or a specific interaction.
Acting with physical props allows participants to not only experience the thinking through the perspective of the user but allow outside observers to see how events play out. This presents an ideal time to include a wider audience of team members or stakeholders who are able to provide specific ‘expert’ feedback. For example, if you are exploring a mixed reality experience designed for hospitals, acting out your thinking to a medical professional can provide invaluable feedback. Another example is when you might have a specific challenge you are trying to understand, like spatial audio or budgeting for asset quality vs. performance. In these cases, acting gives experts a quick, rough concept of how the experience might unfold, without the need for a device-based prototype.
This sort of acting is not formalized, with no need to bring in professional talent, but in situations where we want to share the thinking with others who are not present we will video record a ‘scene’ of interaction or include a storyboard artist to observe and sketch key moments. Acting can also be a very lightweight activity, often happening in-situ during the bodystorming phase. Deciding which to use depends on the audience (and fidelity needed to elicit the necessary type of feedback) but ultimately comes down to whatever will most effectively capture your team’s thinking.
Capturing ideas with storyboards
The best method for conveying the ideas and concepts of your proposed experience depends on your intended audience, as well as the type of feedback your next iteration requires. When presenting new ideas to team members, low fidelity re-enactments of bodystorming can be enough to bring someone up to speed. When introducing your experience concepts to new stakeholders or potential users, the best method is often storyboarding. Storyboarding is a technique common to the entertainment industry, usually found behind the scenes in movies and video game development, and helps convey both the overall flow of an experience (at low fidelities) as well as the aesthetic look and feel (at high fidelities). Just as with prototyping, understanding the fidelity needs of your storyboard is key to gathering the right kind of feedback and avoiding counter-productive discussions.
Low-fidelity storyboards are the right fidelity for quick discussions, especially when conveying high-level ideas. These can be as simple as stick-figure drawings and primitive shapes to denote virtual elements in a scene or the proximity of interactive components (both physical and virtual). While these are useful given the relative ease and low skill barrier to execute, remember the lesson from bodystorming: Not everyone can see 2D depictions of an experience and understand the 3D implications.
High-fidelity storyboards are a powerful tool when bringing in new stakeholders or when combining insights from a bodystorming session with the proposed aesthetic direction of your experience. Storyboards can build off mood boards to illustrate the final appearance of a virtual experience, as well as capture key moments that may be pivotal to the final product. Keep in mind that high-fidelity storyboards often require an artist, especially one embedded within the team, who can capture difficult-to-describe ideas. At Microsoft we have added storyboard artists, often with backgrounds in game development, to our mixed reality teams who attend meetings prepared to quickly capture sketches that will later be expanded into higher fidelity mockups. These individuals work closely with technical artists, helping to convey the artistic direction of assets used in the final experience.
Expanding your team’s skills
In addition to the techniques described, altering your process to better accommodate mixed reality experiences depends on more closely connecting technical and design roles. Working with partners at Microsoft, we have witnessed several teams successfully transition into 3D development, and find the biggest advantage comes from team members stepping out of their comfort zone and developing skills that give them more involvement throughout the development process (not just limiting their skills to design or development).
Altering your process to better accommodate mixed reality experiences depends on more closely connecting technical and design roles.
Feedback and iteration is key to any successful design, and with mixed reality experiences this sort of feedback often skews to more technical concepts, especially given the relative nascence of mixed reality’s technology and tools. Building your design team’s skills in technical areas, including a proficiency with tools like Unity or Unreal, helps those individuals better understand the approach of developers and provide more effective feedback during prototyping. Similarly, developers who understand the fundamental UX concepts of mixed reality experiences (and potential design pitfalls) can help them offer implementation insights during the early phases of planning.
We usually get questions from partners working in mixed reality about how best to grow their team and skill sets with 3D tools. This is a difficult question to answer given the state of the industry, as teams are faced with building tomorrow’s experiences with yesterday’s tools. Many teams are gathering individuals with backgrounds in gaming and entertainment, where 3D development has been a staple for decades, or encouraging their existing teams to pick up tools like Unity and 3D modeling software. While these are both essential needs for meeting the baseline of mixed reality experiences today, there will very likely be a great number of specialized tools and skills to help build tomorrow’s ground-breaking applications.
The field of mixed reality design will change dramatically in the near future, with more focus on data and cloud services in holographic experiences. This next generation of experiences will take advantage of advanced natural language processing and real-time computer vision, as well as more inherently social scenarios within immersive experiences. For example, experiences in virtual reality allows designers to build imaginative environments and spaces, requiring a skill set more akin to architecture or psychology rather than traditional interface design. Or consider experiences in the real world with HoloLens, where the use cases often involve highly specialized fields such as medicine, engineering, or manufacturing, where specific environments and real-time data are essential elements of the experience. Working across roles, leveraging both design and specialized knowledge, and having a willingness to learn new tools are invaluable skill for teams working in mixed reality.
Helping your team explore mixed reality quickly and effectively
In the early days of HoloLens, these techniques were borne out of necessity as the device’s early prototype hardware proved to be a less than ideal format for quickly iterating through design ideas. Interaction methods, including fundamental UI in the operating system were explored with bodystorming, while acting helped our usability team understand fundamental user behaviors for scenarios in the workplace. In the end, the team was able to establish a strong baseline for the core UX of HoloLens, communicating concepts across team roles, allowing OS development to move quickly in parallel with hardware development.
Whether it is bodystorming, acting, reviewing with experts, or storyboarding, these techniques are intended to save you time and effort. While we are still very much in the early days of HoloLens and virtual reality development, expanding your design process for mixed reality will help your team spend their energy exploring new and challenging design problems rather than overcoming difficulties communicating ideas.
Read more about the areas of opportunity Microsoft’s partners have been exploring, as well as ways to start designing with mixed reality on the Microsoft Dev Center.
To stay in-the-know with Microsoft Design, follow us on Dribbble, Twitter and Facebook, or join our Windows Insider program. And if you are interested in joining our team, head over to aka.ms/DesignCareers.
