Designing Windows Mixed Reality

Guidance for immersive and holographic head-mounted displays

We are entering a tremendously exciting new era of spatial computing. Breakthroughs in displays, spatial sound, sensors, environmental awareness, input, and 3D graphics are leading us, and challenging us, to define new types of experiences. Together we’re exploring a new frontier that is dramatically more personal, intuitive, immersive, and contextual. Oh, and potentially profitable too… some analysts estimate mixed reality will be a $110 billion market by 2021.

To help Windows Mixed Reality developers and designers create and innovate, we offer actionable design guidance, with related code on GitHub. Authored by Microsoft designers, developers, program managers, and researchers, and in some cases by you, the broader community, the guidance focuses on user-experience for holographic devices (like HoloLens) and immersive devices (like the Acer and HP Windows Mixed Reality headsets). So, consider it a set of topics about how-to design for Windows Mixed Reality head-mounted displays. As the guidance evolves, we will always have the latest overview in the About this design guidance article on Windows Dev Center.

Because we are learning right along with you, we won’t always be able to offer “canonical” or final guidance. Some of what we share will be in the spirit of ‘lessons we’ve learned’ and ‘you might want to avoid going down that path’. All of this is in the spirit of inspiring and empowering you with design thinking, best practices, and open source controls and sample apps that you can use directly in your own work. We’ll do our best to listen, and to share our insights even if they are exploratory and early. And we look forward to hearing from you, learning from you, and working closely with you.

Overview

Here’s a quick overview of how the design guidance is organized. Design-related articles are on Windows Dev Center, and related open-source code is on the Mixed Reality Design Labs and HoloToolkit GitHub repos:

Design guidance on Windows Dev Center

The Dev Center design guidance is organized in these sections, each with multiple articles:

• Get started with design — Understand high-level thoughts and principles.
• Interaction design — Learn about input, commanding, navigation, and other interaction basics for designing your apps.
• Style — Make your app delightful by using color, typography, and motion.
• App patterns — Learn capabilities that span immersive and holographic apps.
• Controls — Use these as building blocks to create your own app experience.
• Sample apps — Build great experiences from samples designed and created with open-source controls.
• Design tools and resources — Jump-start your project with design templates and tools.

In the months ahead, we’ll expand this taxonomy to include a broader set of design topics. Whenever possible, we’ll give you a heads-up about what is coming next, so please keep checking our mixed reality design homepage on Dev Center.

Open-source code on GitHub

Unity code for our designs can be found in several places on GitHub:

• The HoloToolkit repo offers fundamentals you already care about… a collection of scripts and components that help accelerate the development of Windows Mixed Reality apps. HoloToolkit provides the building blocks for our Mixed Reality Academy, the training program we offer to teach enterprise partners and agencies about developing for Windows Mixed Reality.
• The Mixed Reality Design Labs repo shares our early design prototypes, explorations, and emerging controls and patterns. It includes the latest HoloToolkit as a module. Design Labs is not “official” design guidance or code, so think about it as a way for creators like you to get closer to the design conversations we have at Microsoft, and explore design-centric extensions of HoloToolkit. As the Labs work matures and gains wider adoption, we will evolve it capabilities like HoloToolkit and developer tools.

Objectives

Here’s a quick look at some high-level objectives that guide this work.

The right balance of global and local design

We offer two levels of design guidance: global and local. Our global design guidance is embodied in the Fluent Design System, which details how we think about fundamentals like light, depth, motion, material, and scale across all Microsoft design — our devices, products, tools, and services. The local design guidance in Dev Center covers topics unique to HMDs, for example 3D environments and objects; shared environments; the use of sensors, eye tracking and spatial mapping; and the opportunities of spatial audio. Throughout our guidance you will see us refer to both these global and the local aspects, and hopefully this will help you ground your work in a larger foundation of design while taking advantage of the design differences of specific devices.

Help solve customer challenges

We wrestle with many of the same issues that you do, and we understand how challenging your work is. It’s exciting to explore and define a new frontier… and it can also be daunting. Old paradigms and practices need to be re-thought; customers need new experiences; and there is so much potential for innovation. Given that, we want this work to be as comprehensive as possible, moving well beyond a style guide. We aim to deliver a comprehensive set of design guidance that covers mixed reality interaction, commanding, navigation, input, and style — all grounded in human behavior and customer scenarios.

Point the way towards a new, more human way of computing

In addition to addressing specific customer needs, we’ll push for innovations that are only possible with this new medium. New ways of human behavior; new ways of people relating to themselves, their activities, and their environments; new ways of seeing our world… we want our guidance to reflect all these aspirations too.

Meet creators where they are

You are a diverse and multi-talented community, and we want our guidance to reflect that. You have different skill-sets (beginning, intermediate, advanced), use different tools (Unity, DirectX, C#, other), are familiar with various platforms (Windows, iOS, Android), come from different backgrounds (mobile, enterprise, gaming), and are working on different size teams (solo, small, medium, large). With this diversity in mind, we’ll deliver guidance that is relevant to different perspectives.

Share as much as possible, from experimental to explicit

One of the challenges of offering design guidance in this new medium is that we don’t always have definitive guidance to offer. Just like you, we are learning, experimenting, prototyping, problem-solving, and adjusting as we hit obstacles. Rather than wait for some mythical future moment when we have it all figured out, we aim to share our thinking with you real-time, even if it is not conclusive. Of course, our goal is to be definitive wherever we can, providing clear, flexible design guidance tied to open-source code, and actionable in Microsoft dev and design tools. But getting to that point takes many rounds of iteration and learning, so we will be openly sharing as we go, even design-thinking that is experimental.

Have a discussion

Perhaps most importantly, we want to engage with you, the community of holographic and immersive designers and developers, to define this exciting new era of design. We know we don’t have all the answers, and we believe many exciting solutions and innovations will come from you. We aim to be open and available to hear about them, and discuss with you online and at events, and add value wherever we can. We are excited to be a part of this amazing creative community, embarking on an adventure together.

What’s next

Looking ahead, we see lots of opportunity, and will focus on:

• More controls, prototypes, and sample apps on GitHub
• More design-guidance on Dev Center
• Deeper application of the Fluent design system to head-mounted UX
• Deeper info-architecture (beyond the seven categories mentioned above)

Together we are entering a tremendously exciting new era of computing. I hope our design guidance is helpful to you, and this article provides some meaningful context as you explore it. We welcome your feedback as we move forward. Please dive in, check out our Fluent design toolkits, and let us know your thoughts by commenting here, on GitHub, or LinkedIn.
Let’s co-create the future together!

Thanks for taking the time to read this post!
For more about mixed reality design, please follow me on Twitter and Medium.

To stay in-the-know with Microsoft Design, follow us on Dribbble, Twitter and Facebook, or join our Windows Insider program. And if you are interested in joining our team, head over to aka.ms/DesignCareers.