What I learned designing for mixed reality with NASA
Lessons from a graduate student project
NASA has approached mixed reality in innovative ways — making it possible for scientists to examine the Martian landscape from their desks, for engineers to collaborate on full-scale spacecraft designs, and for astronauts to repair parts of the International Space Station.
As part of my graduate student project, I had the chance to work on a HoloLens applications called ProtoSpace in collaboration with the Ops Lab at NASA Jet Propulsion Laboratory. I designed interactions for a handheld controller — as an alternative to voice and gesture controls — which helped engineers interact with 3D objects. An alternative interface becomes necessary when voice and gesture controls are restricted by context. For example, in collaborative group settings, voice commands disrupt the flow of conversation and hand gestures are fatiguing and similarly disruptive.
What is mixed reality?
Mixed reality is best explained in contrast to other immersive technologies:
Virtual reality: VR replaces your current reality with an entirely digital one and disconnects you from your immediate surroundings.
Augmented reality: AR overlays digital information atop your physical world, but with limited spatial awareness.
Mixed reality: With MR, digital objects are integrated into your physical environment. They interact with and lock into your physical world. Spatial awareness is high because your environment is constantly scanned.
With MR, the blend of physical and digital presents a unique set of design challenges, and best practices are yet to be discovered. For that reason, I wanted to share the lessons I learned designing for mixed reality with NASA:
1. Consider the benefits of multimodal interaction
There’s room for creativity when it comes to combining different modes of interaction. Multimodal interaction means users can input information from a variety of sources (e.g., voice controls, hand gestures and gaze) and receive outputs in a variety of formats (e.g., speech synthesis, graphics). Researchers even experimented with opportunistic controls, transforming objects within a person’s environment into buttons and other UI components.
Input modality was an interesting challenge for my team while working on ProtoSpace. We discovered that existing input methods had limitations in group settings. For example, voice commands disrupted the flow of conversation and hand gestures were fatiguing and similarly disruptive. We found that a controller input allowed NASA’s engineers to interact with the system in a more unobtrusive way during group discussions.
An additional modality offered more flexibility in switching between input methods based on preference and context.
2. Use paper prototypes to rapidly test assumptions
With few best practices to rely on, many of our design hypotheses were derived from analogous research and intuition. Prototyping with paper (and other cheap materials) allowed us to test our designs quickly and inexpensively. We tested these assumptions using a Wizard-of-Oz technique, which worked quite well given there was consistent and immediate feedback when people interacted with our system.
By inviting our target users to interact with the prototype, we could gather preliminary feedback on the logic of our interactions.
3. Use storyboards to help you think deeply about the world
Designing for mixed reality can get quite complex. For my team, it involved understanding user psychology, the use of surrounding space, information access, social dynamics, and how these factors all interact.
Storyboarding is a great way to think through these complex scenarios. Lillian Warner shares a valuable resource for storyboarding with mixed reality: a first-person user flow and a third-person user flow.
Working through each scenario from a first- and third-person perspective is a great way to guide your thought process and to communicate you designs to others.
4. Never stop asking “Why?”
Mixed reality is an exciting technology with tons of possibilities, but designers have to be critical of when the technology should (and shouldn’t) be used. How does mixed reality support product and business goals? Is it the best medium to meet those needs? It’s often tempting to build something because the technology allows for it, but it’s important to keep questioning these assumptions and asking “Why?”
Final Thoughts
Mixed reality poses many interesting design challenges, and there’s opportunity for young designers to help shape the field. I hope the lessons I’ve shared here can help those who are taking their first look into the field.
Thanks for reading!
This story is published in The Startup, Medium’s largest entrepreneurship publication followed by +388,268 people.
Subscribe to receive our top stories here.
