The Pursuit of More Personal Computing
How Microsoft’s bet on mixed reality creates new opportunities for designers and developers
Tomorrow’s opportunities are uncovered by building products today. The solutions these products provide reveal what’s necessary to advance the future. With mixed reality this is especially true: Meaningful insight comes from getting hands-on with real work — real devices, real customers, real problems.
At Microsoft, I’m part of the design team helping enterprise partners build experiences for their business using Windows Mixed Reality. Over the past year, our team has focused on HoloLens and understanding how Microsoft’s flagship holographic device can deliver value to customers today. Working closely with designers and developers from these companies, our team focuses on uncovering solutions that would be technically unfeasible, financially impractical, or otherwise impossible without HoloLens.
Building these solutions helps Microsoft’s internal teams prepare for the next generation of computing. Learning how individuals and businesses interact with core technologies like mixed reality, voice, and AI, helps Microsoft build better devices, platforms, and tools for developers. If you are a designer or a developer exploring this space, understanding what our teams are learning from partners today is critical to preparing for tomorrow’s mixed reality opportunities.
Microsoft’s ambition for mixed reality
We live our lives among two worlds: the physical and the digital. Both have fundamental strengths that we leverage to augment and extend our abilities. We can talk in-person with a friend, using our physical senses to understand things like body language, or we can augment our ability to talk by video-chatting with a friend from miles away. Until now these two worlds, and their respective strengths, have been fundamentally separated.
The physical world is one of atoms and physics. We use our senses to make decisions, leveraging years of learned behavior to interact with objects and people in our environments. Despite the ease of these interactions, we are limited by our physical abilities and the laws of nature.
The digital world is one of bits and logic. Computations are made instantly while information can be distributed effortlessly. Despite the speed and flow of information, interactions are too often limited to small screens, abstract inputs, and noisy feeds.
What if we could combine the advantages of the physical and digital worlds? This is the cornerstone of experiences across the spectrum of mixed reality: a medium where the physical and digital co-exist and seamlessly interact. Combining these worlds builds a new foundation for interacting more naturally with technology — an evolution in personal computing.
The spectrum of mixed reality has exploded as developers have begun exploring the opportunities of immersion and presence. Bringing users into the digital world with immersive (virtual reality) experiences and creating digital objects in the physical world with holographic (augmented reality) experiences. But what are the benefits of mapping the physical world to the digital world? What happens when we give computers eyes to see?
The fundamental camera vision technology behind holograms acts like a pair of eyes for the computer to see the environment around you: Objects in the world, people around you, changes as they happen. A digital understanding of your context in the physical world. This leads to an enormous amount of information, the implications of which we are only beginning to understand.
Culminating core technologies
Computing is too often a tangible thing. Grabbing our devices to tell them who we are and what we want. Contorting our thinking and aligning what we say to match what we believe the computer needs to hear.
The promise of mixed reality, especially in the real world with holographic experiences, is to lessen the burden of interacting with technology. Reducing cognitive load as users navigate the layers of abstraction inherent to computing today. How can we design experiences that not only take advantage of contextual understanding, but make it easier to draw insight and take action? Two contributing technologies are also attempting to solve this problem:
- Voice, in terms of speech and conversation, is enabling users to communicate with computers through more natural means — answering bots through text or issuing commands to conversational hardware.
- Artificial intelligence is powering experiences that distill insights from increasingly complex datasets. While AI is an enormous topic, recent progress has provided the groundwork for devices that rely on computer vision, more natural digital assistants, and recommending actions to users.
Mixed reality provides a medium to combine these technologies into a single user experience. Voice becomes a powerful, natural method for input when wearing a holographic headset. AI acts as a critical cipher to contextualize the enormous amounts of information connecting the physical and digital worlds. This is why Sataya Nadella refers to the HoloLens as ‘the ultimate computer’, it’s a culminating device for three core technologies. A platform to empower humans to more easily interact with the growing complexity of devices and services.
Less interface in your face
A culminating device that connects the physical world to the digital world allows us to design experiences that fit more naturally, without cumbersome abstractions. Consider the experiences you’ve created: When the barriers of abstraction are removed, how much of your interface is left? Which parts of your app flow change when you know the user and their context? How many menus and buttons remain?
For example, think about shared experiences in mixed reality like the OnSight tool NASA’s Jet Propulsion Laboratory built for scientists. Instead of building a system to look at Martian data (abstracting the data onto screens or displays), they brought scientists inside the data, effectively putting them on the surface of Mars as they collaborated.
Instead of finding the button to draw attention to some Martian geology, scientists can point to it directly. No menus, no pens, no learning curve to using the tool effectively. By leveraging our known abilities from the physical world, more natural interactions in mixed reality can circumvent deep technical literacy in even the most advanced industry tools.
Likewise, voice and AI can extend natural interaction in experience like this. When digital assistants can ‘see’ into the world around us, conversations will feel less cumbersome. A bot in NASA’s tool might extract context from the scientists conferring over Martian geology, acting as a ready (and conversational) source of information when scientists gesture and ask about ‘this’ or ‘that’. A computer that knows your context is ready to jump in with information appropriate to you, through a method of interaction most appropriate to your context.
Building on a foundation
In this world of contextual experiences, catering to the mobility of the user is key. Moving fluidly between devices and services, with highly personalized contexts, users will travel about the physical world seamlessly — creating a massive platform challenge. When the physical world becomes a digital end-point, interoperability reigns supreme.
“Bringing together the digital universe and the physical world will unlock human potential… enabling every person and organization on the planet to achieve more.”
— Satya Nadella
Windows Mixed Reality is an effort to create a platform for an ecosystem of devices, enabling developers to create immersive, affordable, and compatible experiences for the largest possible audience. The future will not be limited to a single manufacturer, let alone a single device. Headsets, mobile devices, PCs, accessories… all these physical things must interoperate (as well as digital things like graphs and services) through an underlying platform to successfully deliver on the promise of mixed reality.
Designing for tomorrow’s experiences today
Each one of the core technologies behind this new class of experiences are enabling designers and developers to create compelling and successful experiences today. By reducing abstraction, we can interact more directly with the digital world, allowing us to design in ways that augment and amplify human abilities. Voice technology (through bots and digital assistants like Cortana) is allowing users to carry out increasingly complex conversations and scenarios, while AI technology (through tools like Microsoft Cognitive Services) is causing companies to rethink how users will interact with everything from social media to supply chain management.
As a culminating medium, it’s hard to say what the far future of mixed reality will look like for consumers. We know these experiences will rely on both a new class of design tools as well fundamental support from the platform. The tools and supporting services will be refined by solving real, tangible problems and getting hands-on with real hardware. Experimenting today with the intersection between mixed reality, voice, and AI is key. Microsoft is just getting started with this new class of computing and learning from the success of enterprise experiences today will help create a better platform for the consumer experiences of tomorrow.
Read more about enterprise projects and the areas of opportunity Microsoft’s partners have been exploring, as well as ways to start designing with Windows Mixed Reality on the Windows Dev Center.
To stay in-the-know with Microsoft Design, follow us on Dribbble, Twitter and Facebook, or join our Windows Insider program. And if you are interested in joining our team, head over to aka.ms/DesignCareers.
