Best Practices For Mixed Reality Design in 2017
Working with the biggest strengths and weaknesses of this revolutionary medium
Last updated on January 16th, 2016.
Mixed Reality is here, and it’s bound to be one of the biggest new technologies of this decade. For the past few months, I’ve had the opportunity to develop several experiences for the Microsoft Hololens, today’s leading MR device, and learned several things in the process. The best practices below are all about harnessing the medium’s strengths — and hopefully you’ll find something useful to take into our own projects.
If you are a developer with wisdom to share, please mention it in the comments and I’ll be glad to add it to the article!
1) Start fresh — this is an entirely new way of designing!
MR is an entirely new medium, and its best practices are far from being fully defined. Because of this, there’s no greater asset for a designer today than an open mind, so experiment as much as you can and always be open to new forms of interaction, no matter how strange they may seem at first. The road to consumer MR will be paved by unintuitive design choices.
2) Design for the human instinct
Millions of years of evolution have trained us to be very perceptible to motion and contrast — successful MR apps will take full advantage of such involuntary human reactions to guide users through experiences.
These include spotlights, object permanence, subtle changes in lighting, differences in size and color as well as any discernible motion. When used sparingly, these tools can be extremely powerful when smoothly guiding user behavior.
3) Respect the user’s personal space
To design comfortable experiences, it’s important that new elements are always introduced from a reasonable distance. Objects that find themselves too close to the user and obstruct their field of view can quickly remove any sense of control and cause discomfort.
While invading personal space can be a powerful tool in storytelling, most applications won’t share these same intentions. A distance of 1 meter was found to be best in research, and it’s the Microsoft standard distance for introducing in new windows and objects.
4) Avoid 2D assets when possible
Because MR is all about blending in, 2D assets tend to negatively stand out. While there are instances where 2D assets can be intelligently used for performance bumps, particles effects and other illusions, an over-reliance on 2D means that most of the objects in your scene are not realistically responding to the environment, casting shadows or bouncing light on one another. The end result is an inevitably lesser experience that doesn’t respect the rules of the world it’s in.
5) Design for context
You’ve probably heard of some dystopian visions of the future where user’s visions are covered by pop-ups, useless information and intrusive advertising. Mixed Reality, however, is about the extreme opposite.
At its heart, MR is all about contextual computing — well designed experiences will only show the information that’s needed, where it’s needed, when its needed. Make sure that’s true through the entirety of your application. If the world is your canvas, you must above all respect it.
6) Turn voice commands into conversations
MR’s focus on voice commands gives designers a great opportunity to create interactions that feel intuitive and avoid an over-reliance on traditional UI elements (such as windows and buttons). But shouting arbitrary voice commands at the emptiness of space only goes so far. Instead of isolated voice commands, try to design conversations between the user and your app that feel more natural:
- Ensure that your app can handle variations of the same command: e.g. Both “Enlarge picture” and “Make the picture bigger” should have the same outcome, befitting various speaking habits.
- Give users a clear visual point of reference for them to talk to. This can be a character or a simple animation that triggers on speech.
- Have the app play a sound or even talk back at the user whenever it acknowledges a command. You can even go as far as to give your app a voice and personality of its own.
- Design commands so they can be detected in larger word constructs. “Can you enlarge the picture?” may come out more naturally to certain users than “Enlarge picture” — but both commands share the same central keywords that can be detected just as easily. Design for flexibility.
7) Blend into the user’s physical space
Regardless of whether you’re creating a simple application or an otherworldly experience, it is vital that your project seamlessly blends into the user’s physical space. This includes obvious things, like making sure no objects are going through walls when they shouldn’t, but also finer visual detail, like the casting of shadows of digital objects on real world surfaces..
These details are harder to pull off from a technical standpoint, but are worthy of mention as they add multitudes to the believability of MR experiences.
8) You have the user’s spatial mesh — play with it!
One of the biggest features of MR is its ability to understand our physical environment. This not only means that we can realistically superimpose digital information on top of the user’s space, but also that we can modify it in any way we wish.
A solid example of this is Microsoft’s RoboRaid, where robots literally break through the user’s walls in realistic fashion. But much more can be done — creators can set up virtual shelves, expand spaces by pushing out walls, create windows and portals — any change is possible, and the question becomes which one is the best for the app at hand.
9) Avoid using transparency
Mixed Reality holograms already have a natural opacity to them that can vary in intensity depending on the user’s settings. This means that saved some exceptions, you should largely avoid to use transparent shaders, as there’s no way of telling it how it will look in various hardware configurations. It also causes a noticeable performance bump, so there’s that.
10) Take into account the user’s height
Self-explanatory — when designing interfaces, make sure that your app adapts to the height of a multitude of users.
11) Mixed Reality is inherently social. Always develop with multiplayer at the back of your mind.
Anyone who has experienced a multi-user Hololens app knows that everything changes when there’s a second person present — and that’s also when the medium shines. And even if you don’t have multiple MR devices to develop for, the simple act of asking “ What changes when a second user is a part of the experience?” can yield into design breakthroughs and a more human-centric experience that can be expanded in the future.
In the end, humans are social animals and MR adds a whole new layer of possibility into human interaction — applications that play with this well will be a force to be reckoned with.
12) If you can’t blend in, less is more.
Experiences with too many elements that don’t blend in can easily be overwhelming and disorienting to the user. Objects that don’t blend drag the user’s attention away, causing you to lose control over the desired user experience. In cases like these, working with less can be more, as a smaller and more contained experience with fewer elements can be much easier to orchestrate from a UX standpoint.
13) Performance is everything: use optimized shaders and geometry
When developing applications for Mixed Reality, use shaders and geometry created specifically for the hardware. The Microsoft Holotoolkit has a wide array of useful scripts and shaders designed for performance that are useful in any setting.
14) Sometimes, colors look differently in MR than what you would expect. Be flexible!
Because of the way that the HoloLens renders colors (2 green channels, 1 red and one green), some colors don’t look as striking than their 2D counterparts.
15) Be visual — and magical whenever you can.
Apps that make full use of the visual capabilities of Mixed Reality can create breathtaking moments that are incomparable to anything today in any other medium. Communicate whatever you can visually and creatively, combining all of the tips above to surprise the user in beautiful (and useful!) ways.
It doesn’t matter if your app is designed for productivity or entertainment — if there’s a way you can cause user delight, you should go for it. Mixed Reality is, after all, the science of magic. And if we want this new reality to be beautiful, then perhaps this might be the most important item of this list.
Lucas Rizzotto is an Immersive Experience Designer, Artist and Developer. You can follow him on Instagram, Twitter, Facebook or contact him through his website.
Also, sign up to my mailing list so you can see more of my stuff (:
