Designing for Mixed Reality
How to think about 3D space in HoloLens
Holographic computing is a powerful thing. The amount of data we can absorb in three dimensions is astounding. Experiencing things in an immersive context speeds up our understanding, helps us make decisions faster, and hard-codes our memories in spatial awareness.
The design implications that stem from such a rich environment are equally astounding. Think about your favorite map app. On a 2D screen, we spend a lot of time panning, zooming, and switching views to understand things like elevation, depth, and proximity. Now imagine that same map in HoloLens 3D. Suddenly, everything that our minds previously had to stitch together — different views and angles arranged like a Lego map — is now instantly understood. It’s alive, it’s complete, it’s right there in front of you. Experiencing something digital in the same way we understand the physical world is pretty magical.
As designers, this real-world understanding is a constant and relatively new challenge. How do we translate our knowledge of pixels and resolutions to the physicality of 3D realms? How can UI designers confidently iterate across different “worlds”? How does design thinking evolve from here?
Get into the space
Designing for an immersive mixed reality platform means imagining and exploring the literal space around the user. Think about the walls, floor, ceiling, windows, and furniture that may exist. Anticipate everything about the user’s environment. Walk a mile in their shoes, as it were. Contextualize.
Now consider how building on the existing elements in the room might enhance the experience and uncover opportunities to create solutions that feel more grounded and natural. From the opposite perspective, consider the negative impacts of existing furniture and such. Uncover ways to provide guidance to the user in setting up their space, to curb potential distractions and disarray.
Keep your design explorations as physical as possible. Use real objects as stand-ins for holograms. You won’t get a sense of the physical space if you don’t create it, stand in it, move around it. Prototyping for holographic applications is a little goofy — embrace it. Get your team together and craft the experience using anything you’ve got handy. Blocks, foam, exercise balls, stuffed bears, each other, whatever. The early, rudimentary prototypes will inform your design direction and interaction model.
The distance challenge: legibility and accuracy
Depth perception is a huge challenge in 3D design. It affects every piece of information and object the user interacts with. The mind perceives things as smaller at a distance, which means details at a distance are more difficult to decipher, especially text. The amount of muscle control required for the eye to focus on objects at a distance is much greater than focusing on elements closer to the user. The slightest movement of the head affects perception. Reading small text from far away requires the muscles in the
neck to hold the head as still as possible. The eyes need more time to focus, so they must remain motionless while blinking intervals are increased. Basically — human eyesight is a cool but complex thing. All these processes are happening automatically, but prolonged reading at a distance can cause fatigue. This is why the humble road sign is so great. It presents the exact right information at various distances, quickly and effectively. Referencing real-world patterns like this creates powerful virtual world solutions.
In line with legibility, virtual spaces require precision. We’re imperfect — we misjudge distances, we move awkwardly, we can’t focus on too much stuff at once. Being precise in your design considerations means overcoming the natural movement of the body, especially at a distance. Imagine a tiny button ten meters in front of you. You walk forward, maybe you start talking to someone next to you, you miss the button. You have to recalibrate for a moment, try again.
As a designer, you can change that interaction. In the virtual space, you’re free to dictate the size of fonts, buttons, and any other bits of information that need to be observed. Create designs based on how far the content will be from the user. Keep in mind that to maximize effectiveness, the bulk of primary content should stay between one and two meters from the user. Elements closer than one meter can cause nausea, especially if they’re connected to the user’s gaze.
The user feels confident when they can easily absorb the information required to make choices. Swiftness and accuracy replace fatigue and frustration. The goal is to be responsive and always provide the right amount of feedback in reply to the user’s input. An experience that has a flexible workflow, makes the user feel confident, understands intent, and responds with the desired result is rewarding and engaging.
Think like a game developer
Which is not to say, “develop a game for HoloLens.” The point here is to design the scenes all around you, and to keep in mind that all that immersion and scenery will undoubtedly affect performance. Game development practices are commonly measured by frame rate, and it’s the same with HoloLens. To produce quality graphics and more immersive experiences, we’re always balancing what we do visually against a GPU budget. For designers, these are the major performance consumers: polygons, object count, texture size, transparency, complex shaders, dynamic lighting and shadows.
As you design the space around the user, certainly don’t shy away from playing with all the design elements available to you. The trick is to balance left brain and right brain, developer and designer. Be both, be conscientious of everything going on in the space. The magic of HoloLens lies in that 360°-ness. That cinematic, fluid experience. What’s behind the user, below the user, in their periphery? Where’s the action? What should they hear and see, and where should they hear and see it? Immerse the user. Be playful. Remember that with the HoloLens we’re no longer restricted to designing within a physical frame. The world is our canvas.
Start building
Thinking about 3D space only scratches the surface. Once you’ve oriented yourself, done the prototyping, the explorations, the iterations; you’ll need a solid toolbox to start building. HoloLens has a ton of guidance for developers and designers alike. Start exploring these resources, and have fun!
Adobe Illustrator Texture Template
As designers, we work with physical objects or on a whiteboard to understand the size and relationship of the UI system. Using this template, it’s easy to plot the UI system out in Illustrator to finalize the design and aesthetic. We can even compare how it fits within the user’s field of view.
All about typography in HoloLens
My colleague Dong Yoon Park wrote this fantastic piece about text in 3D space. There’s also a nice overview of a recent Meetup talk from Yoon on the same topic.
HoloToolkit-Unity
After installing Unity, this collection of scripts and components is the quickest way to get something interactive running on the HoloLens. There are plenty of examples included with the toolkit and a community of active contributors.
On the developer side, here is a list of all the tools you may need to develop applications for HoloLens.
Holographic Academy
If you have some C# scripting abilities, here’s a set of tutorials that cover the fundamentals of holographic experiences and input methods like Gaze, Gesture, and Voice. The Documentation section has more in-depth study of designing for holograms.
Microsoft HoloLens Dev Center
The central hub for all things HoloLens has links to everything listed above, plus things like the Galaxy Explorer source code, video gallery and so much more.
Microsoft Store
To see what other developers are building on HoloLens, I suggest games like RoboRaid for its spatial understanding, and Young Conker for their use of the environment including things like sofas and tables. Fragments is another must see, it’s probably the best example of the visual quality bar a HoloLens experience can achieve, it’s so immersive and amazingly performant. There are so many great apps on the Store and new ones being added all the time. There’s always something new to check out.
If there’s anything I missed, let me know in the comments below. I can’t wait to see what you build!
To stay in-the-know with Microsoft Design, follow us on Dribbble, Twitter and Facebook, or join our Windows Insider program. And if you are interested in joining our team, head over to aka.ms/DesignCareers.
