9 Methods for HoloLens & Mixed Reality UX Design

Brian Hui
Brian Hui
May 12, 2017 · 10 min read

Image for post
Image for post
Left: Jing sketching. Middle: Alex with the HoloLens. Right: ArtDroid-82 user flow

We all have a process when we design, our flows usually follow a semi-flexible framework that lets us as designers be flexible depending on the client’s requirements. With UX design for mixed reality, the rules and guidelines are constantly changing as developers create content for this fledgling digital medium. It really is an exciting time!

Working as part of a group, HelloLens, our team was tasked by Dana Karwas, our professor at New York University Tandon School of Engineering’s , Joshua Walton, Principal Interaction Designer of the Microsoft HoloLens, and later on, Nicholas Kamuda, Creative Director for Microsoft HoloLens and Windows Holographic, with .

Our final product, the HoloStudio, included a digital assistant, ArtDroid-82 (aka, “Arty”), to help guide users and disseminate information in a mixed reality experience where users could see art work, resize, reposition, and rotate the artwork, as well as see the process of how the art was created.

Image for post
Image for post
Concept visual of Arty with the user at an interactive installation demonstration inside the HoloStudio.

Before we began, we all knew it was going to be a challenge and the biggest question that hung over our heads on the first few days was just how exactly we should go about designing for mixed reality. No one had an answer, so we had to use the resources of other companies and people to derive our own tools and design process for mixed reality. In order to develop a mixed reality experience, our team utilized these tools to complete our UX design project for mixed reality and the HoloLens.

These methods are divided into three areas:

  • Project Goals & Research
  • Brainstorming & Prototyping
  • Communicating Ideas

Project Goals & Research

1. Defining Project Goals

Since the HoloLens was relatively new technology, our team set out to create clear and defined goals to the project. These goals were vague enough that they gave us some wiggle room while also being narrow enough to focus our attention to solving specific problems. Defining the goals and vision of the project is critical to keep scope creep to a minimum as well as helping you refocus your attention should you get too excited (it happens, we know!).

2. Reading Up on Mixed Reality & Sense Ratios

Since there is no real standard, everyone is making it up as they go. During the beginning phases of the project, we read a lot about what other people were doing. The biggest articles that helped us the most include Joshua Walton’s talk at Build 2016 on , Juan’s article on , by Lucas Rizzotto, watching videos on the , reading articles from , and reading about the works of the Seattle-based company, .

We also sought to familiarize ourselves with . McLuhan’s idea of sensory ratios claims that our perception of media by our individual sensorium has the potential to change our mental interaction processes. With mixed reality, we have the potential to create experiences that will allow users to use natural affordances that will have no change to the sensory ratio. Our team believes that how content is displayed is going to play a big part in allowing for a minimal or negligent change to our natural sensory ratios so that we won’t feel bombarded or overwhelmed by the content that is displayed.

3. Understanding the Technology: HoloLens Demo

To understand the HoloLens better, our team went to the Microsoft Store on Fifth Avenue in New York City to take part in demos that showed off the capabilities of the HoloLens. These sessions were one hour in length and demonstrated to us three use cases for the HoloLens — as a mixed reality showroom, pinning holograms into a room for playful and fun interactions, and a mixed reality game.

Image for post
Image for post

The one hour demo session that we went to was key to helping our team understand the vision for Microsoft has for the HoloLens. It also introduced us to concepts that are important when developing a mixed reality experience. 3D sound, gestures, voice commands, and the limited field of view all had to be accounted for when we started to think about just how this experience would take shape.

Brainstorming & Prototyping

4. Table of Elements: Basic Framework for Mixed Reality Design

If you’re not familiar with the Table of Elements, it is a design document that was used by 8ninths and then modified by Juan to include Mixed Reality Design Methods. Our team made no changes to that iteration of the Table of Elements as it helped us envision what elements we wanted to include in our experience including what design methods we wanted to utilize.

Image for post
Image for post
Our Table of Elements for the HoloStudio.

However, it is not a bad idea to include almost everything you think you want to do so that you remember what options you have. By the time our project was completed, we would have a more concise table as elements were added or removed depending on if we used it in the project or not.

5. The Interactions Concept Map: Defining Potential Interactions

After understanding what the HoloLens could do, our team opted to dive right into thinking about what this could mean in terms of this design challenge. We sat down and brainstormed potential interactions that our team could integrate into the app that would meet the goals we set at the beginning of the project.

Image for post
Image for post
Interactions Concept Map for the HoloStudio

We wanted the experience to have as few obtrusive interface elements as possible as the user would be processing information about the holograms in relation to the room or space that they would be in. Therefore, this brainstorming session was heavily focused on using air tap gestures as well as voice commands to interact with the artwork while having only a few buttons to tap on and interact with. We had to take a lot of things into account when we were brainstorming including. One of the hardest things our team had to do in this session was to stop thinking of design in a two dimensional space and instead try to think of it in three dimensional space. We had to think of interactions that would leverage the freedom and mobility that the real world environment provides to our users

The interactions concept map helped us define potential interactions for all the art pieces we needed to include. Here, we basically included any ideas our team had on how users could interact with specific types of art pieces. We even had branches for Interface and Extras for potential interface options and extra features we could include. This document was helpful to our team as it allowed us to think of what was potentially possible. In later stages, we could narrow our options down.

6. Sketching & Storyboarding: Defining Ideas

To help us gather feedback and visualize our initial concepts, our team decided it would be better to sketch out some of our ideas. We used different colors to represent different elements of the mixed reality experience:

  • Physical room elements would be in black or grey
  • Mixed reality holograms would be in blue
  • Interactions and gestures would be in orange
  • Audio cues and dialogue would be represented in purple

This color coding scheme was borrowed from Juan’s article. Color coding these elements allowed us to quickly identify and differentiate the elements and allowed us to convey our ideas in a quick and efficient manner to other people.

Image for post
Image for post
An audio line is represented in purple in this sketch.
Image for post
Image for post
Blue represents holograms that would populate the room.
Image for post
Image for post
The user interactions and gestures are represented in orange.

Due to time constraints, we chose to sketch only a few of our ideas so our team could collectively decide on what direction we wanted to take with this project. Regardless, we think that sketching and storyboarding can clearly help you communicate your ideas, identify weak points of your experience, and help you think more about end-to-end interactions in a more cohesive manner.

7. Prototyping: Representing Holograms in Physical Space

One of the first problems we faced when entering the prototyping phase was the representation of holograms in physical space. Should we be using digital software to prototype? What would be the best way to prototype so that users can adequately test our ideas?

Image for post
Image for post
Jing and Will constructing a painting hologram

To solve some of these problems, we decided to use resources that were quickly available to us: a roll of brown paper, markers, printer paper, PVC pipes, as well as miscellaneous objects to represent holograms. Constructing these placeholder holograms allowed us to use bodystorming techniques in tandem with these placeholders to adequately communicate what our experience looked like. It would prove useful when we tested our prototype experiences with people as it allowed people to understand where and when holograms would appear, as well as how the holograms should appear when they interacted with it.

Image for post
Image for post
Will and Ankit bodystorming our prototype
Image for post
Image for post
Bodystorming with constructed holograms (L to R: Jing, Will, Tyler, Ankit)

This method let us gather a good amount of user feedback. In particular, people loved that they could move around the room and “interact” with holograms while the team basically acted the experience out. To us, constructing the prototypes and then bodystorming them was a blast. It allowed us to be more imaginative and engaged while we continued to craft our ideas. This method is especially ideal if you aren’t a great 3D modeller.

Communicating Ideas

8. Visual Flows: Animated & Static User Flows

To supplement traditional user flow diagrams, our team decided to create animated visual flows. Our team found that communicating user flows through an animated visual flow helped people better understand how our project experience should play out.

Image for post
Image for post
User flow with iconography.

Traditional user flows played a part in our process as well, We had to come up with a way to communicate the complexity of a mixed reality user flow in a static diagram. We chose to use icons to represent different input and output to prevent a wordy and drawn out user flow. When we received feedback for this document, people were able to quickly discern what the user was supposed to do and what would happen as a result of user input.

9. Concept Visuals: Bringing Ideas to Life

To help other people see what we wanted our ideas to become, we created concept visuals to better visualize our ideas. We compiled assets from online sources to create visualizations that would translate our ideas across a visual medium. This method can potentially strengthen your ideas in addition to having strong sketches, storyboards, and visual or static user flows to better communicate your ideas.

We wanted to better communicate how users could interact with artwork. These concept visuals allowed us to better articulate our ideas and gather support and feedback on our potential interactions. Since the HoloLens is limited to one user viewing at a time (unless you’re streaming or in spectator view), our team thought it was a fantastic tool to help bring our ideas to life for a wider audience.

Image for post
Image for post
Image for post
Image for post
Image for post
Image for post

By taking cues from architectural renders and video game concept art, creating these visualizations is helpful to illustrate an ideal version of your experience. These can then be used for many purposes, including communicating with stakeholders, bringing your ideas to life for wider audiences, and acquiring additional support.


The tools you could use to help design in mixed reality is vast. Since the guidelines and rules for mixed reality UX design seem to be ever-changing, we had to create new methods or modify existing methods to help us in the whole process. We chose to utilize these tools as we already had a defined goal, but not a defined process.

In summary, the tools that we used included:

Project Goals & Research:

  • Setting goals to help define vision and keep your team on track
  • Reading articles about how others have approached design for mixed reality
  • Experiencing the HoloLens to understand its capabilities and limitations

Brainstorming & Prototyping:

  • Table of Elements to help build a framework for your mixed reality experience
  • Interactions Concept Map for brainstorming potential interactions
  • Sketching and storyboarding
  • Prototyping by bodystorming and constructing holograms

Communicating Ideas:

  • Creating both static and animated user flows to better communicate the user experience
  • Creating concept art to communicate your ideas to stakeholders

We thought that these tools helped us refine our ideas and prototypes into something more useful and engaging. We hope that you’ll find some of these tools to be helpful in your projects as well!

Want to read more about how my team designed for mixed reality? .

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store