Open-Source Building Blocks for Windows Mixed Reality Experiences

Mixed Reality Toolkit-Unity is an open-source project to share foundational components, building blocks for common interactions and UI controls for building Mixed Reality experiences in Unity.

Windows Mixed Reality devices — Microsoft HoloLens and immersive HMD by Samsung

From Mixed Reality Toolkit, I picked up some of the most frequently used building blocks and interaction patterns. For each building blocks, I’ll show where you can find the example scene in the repository, and how you can use it in your projects.

Whether you are creating apps for HoloLens or immersive headset, you need to set up your scene with proper input module, camera, cursor or motion controllers to interact with your content. And Mixed Reality Toolkit makes it easy.

Example Scenes

When you first start creating apps for mixed reality, these two example scenes are the perfect place to start with.

HoloToolkit-Examples/Input/Scenes/InputManagerTest.unity
HoloToolkit-Examples/Input/Scenes/MotionControllerTest.unity
Find technical details on the README File
InputManagerTest and MotionControllerTest scene

Through these scenes, you can learn how to use MRTK’s InputManager and access data handle events from the motion controller buttons.

How to use: automatic scene setup

When you import MRTK release Unity packages or clone the project from the GitHub repository, you are going to find a new menu ‘Mixed Reality Toolkit’ in Unity.

Under ‘Configure’ menu, you will see the menu ‘Apply Mixed Reality Scene Settings’.

When you click it, it removes the default camera and adds foundational components — InputManager, MixedRealityCameraParent, and DefaultCursor.

You can also manually add these from the project panel. You can find these components as prefabs. When you search MixedRealityCamera, you will be able to see two different camera prefabs.

The difference is, MixedRealityCamera is the camera only prefab whereas, MixedRealityCameraParent includes additional components for the immersive headsets such as Teleportation, Motion Controller and, Boundary.

MixedRealtyCamera supports both HoloLens and immersive headset. It detects the device type and optimizes the properties such as clear flags and Skybox.

Below you can find some of the useful properties you can customize such as custom Cursor, Motion Controller models, and Floor.

Default Input setup for HoloLens & Immersive headset — Gaze, Gesture, Motion Controllers, Floor
Interactable Object Example scene in MRTK

Interactable Object examples show how to make any object interactable and make it respond to standard input states such as observation, targeted and pressed. In mixed reality, a button could be anything from a coffee cup to a balloon, it doesn’t have to be a flat rectangular shape that we have been using in 2D screens. It is important to provide clear visual feedback so that people can understand which object is interactable.

Example Scene

You can find the examples of Interactable Objects in this scene

HoloToolkit-Examples/UX/Scenes/InteractableObjectExample.unity
Find technical details on the README File
InteractableObjectExample scene

How to use

To make an object interactable, you can use series of the scripts called CompoundButton. They are modular scripts for various interaction elements such as mesh, text, animation, icon, sound, and speech.

Compound Button script series in MRTK

You can mix & match these scripts for your purpose.

Interaction states

  • Disabled: Object is inactive
  • Observation: Cursor not on the object, Hand not raised
  • Observation Targeted: Cursor on object, Hand not raised
  • Interactive: Cursor not on the object, Hand ready gesture
  • Targeted: Cursor on object, Hand ready gesture
  • Pressed: Cursor on object, Hand air-tapped gesture

InteractableObjectExample.unity scene includes HolographicButton prefab and ToolBar prefab made with HolographicButton + ObjectCollection.

HolographicButton and ToolBar prefab in MRTK

Interaction Receiver: Handle events in a single place

Interaction Receiver is a convenient building block that allows you handle the events from multiple objects in a single place.

As you can see in this diagram, Interaction Receiver can store list of the objects and when it receives events, it can do something based on the triggered object.

How to use

Design & Development Guidelines

Bounding Box and App Bar example scene in MRTK

Bounding box is the standard interface for manipulating objects in Windows Mixed Reality. You can find this in the HoloLens shell as well as the cliff house in the immersive headset. With this single UI, you can move, rotate and scale objects. App bar is used to display contextual buttons on the objects. In default, it shows Hide, Adjust, and Remove button. Adjust button shows the gizmo affordance for manipulation and remove button deletes the object. You can also add additional custom buttons in the AppBar prefab.

Example Scene

You can find the examples of Bounding Box & App Bar in this scene

HoloToolkit-Examples/UX/Scenes/BoundingBoxGizmoExample.unity
Find technical details on the README File
BoundingBoxGizmoExample scene

How to use

Simply assign BoundingBoxRig script to any object. BoundingBoxBasic prefab is required to get blue wire visualization.

TwoHandManipulatable script is optional. This allows you move, rotate, and scale using two hands. In default, Bounding Box does not move the object.

You can also add your custom buttons to the AppBar. Search AppBar prefab in the project panel and find Button property in the Inspector panel. You will be able to adjust the number of Buttons and define your custom buttons.

Adding custom button to the App Bar

Design & Development Guidelines

Two Hand Manipulation Example scene in MRTK

Two Hand Manipulation allows you manipulate objects using both hands. You can scale, rotate and move by tap and hold and moving your both hands. This is the same interaction behavior introduced in April 2018 Update (also known as RS4) HoloLens as well as in the cliff house for the immersive headset. It works with motion controllers too. It is an intuitive way to manipulate the objects in space.

Example Scene

You can find the examples of Two Hand Manipulation in this scene

HoloToolkit-Examples/Input/Scenes/TwoHandManipulationTest.unity
Find technical details on the README File
TwoHandManipulationTest scene

How to use

Simply assign TwoHandManipulatable script to any object.

Object Collection Example scene in MRTK

With Object Collection, you can easily layout an array of objects in 3D space. You can specify the surface type, row, radius, cell width/height and facing options. With these various options, you can quickly construct three-dimensional grid system.

Example Scene

You can find the examples of the Object Collection in this scene

HoloToolkit-Examples/UX/Scenes/ObjectCollectionExample.unity
Find technical details on the README File

How to use

Example of making a grid layout UI with HolographicButton prefab and ObjectCollection.

Holographic Button + Object Collection

Design & Development Guidelines

Example of Tag-Along + Constant View Size + Surface Magnetism on Start menu

Solver system allows you combine various object positioning behaviors and run them in order. The Start menu is a good example that shows multiple behaviors such as Tag-along, Surface Magnetism, and Constant View Size. It follows you in a certain range and if it collides with other object surfaces, it follows the surface and automatically scales down or up to maintain the size.

Modular Solver script series provide following object positioning behaviors:

  • Tag-along
  • Billboarding
  • Radial view
  • Body lock
  • Surface magnetism
  • Constant view size

In MRTK, there is a simple Tag-along script but Solvers could be better when you want to combine multiple positioning mechanisms such as Tag-along and Surface magnetism.

Example Scene

You can find the examples of the Object Collection in this scene

HoloToolkit-Examples/Utilities/Scenes/SolverExamples.unity
Find technical details on the README File

How to use

Solver Radial View example
Solver Body Lock example

Solver Surface Magnetism + Spatial Processing

Solver Surface Magnetism + Spatial Processing

This example shows how to make objects smoothly align with the physical surface. Find the example scene and README on Spatial Processing + Solver Surface Magnetism example

Design & Development Guidelines

Voice input is a natural way to interact with an object in mixed reality. It can help users to reduce the time and minimize effort. Windows Mixed Reality shell provides system level commands such as “Select”, “Close”, “Move This” and “Face Me”.

With MRTK, you can easily add voice commands to your experiences.

Example Scene

You can find the examples of the voice input in this scene

HoloToolkit-Examples/Input/Scenes/SpeechInputSource.unity

How to use

Design & Development Guidelines

Spatial Understanding Example scene in MRTK

Spatial Understanding is one of the most exciting capabilities in HoloLens. With Spatial Understanding, you can analyze your environment and identify the types of the physical surfaces — such as Floor, Ceiling, Wall, and Platforms. When you leverage these physical surfaces, you can create realistic holograms that interact with the real-life environment.

Example Scenes

In MRTK, there are three different examples. Spatial Mapping, Spatial Processing, and Spatial Understanding.

Examples/SpatialMapping/Scenes/SpatialMappingExample.unity
Examples/SpatialMapping/Scenes/SpatialProcessing.unity
Examples/SpatialUnderstanding/Scenes/SpatialUnderstandingExample
Find technical details on the README File — Spatial Mapping & Processing
Find technical details on the README File — Spatial Understanding

Spatial Mapping

You can simply drag and drop SpatialMapping prefab and you can get this kind of meshes of your environment. It is a little bit rough but it is quick and easy way to make your app interact with the environment.

Spatial Processing

If you want to get more refined clean surfaces, you can use Spatial Processing. Spatial Processing converts the meshes into planes by merging vertices. Through this, you can get clean planes for the walls, tables, and floors.

Spatial Understanding

This is the most sophisticated one. The example scene shows the room scanning experience and gives you identified surfaces. Using this technique, you can place your content on specific surface type.

Spatial Processing Example scene in MRTK

Design & Development Guidelines

If you build an experience for the immersive headset with motion controllers, you can use AttachToController to attach any objects and UI to the controllers.

Example Scene

You can find great examples in the Mixed Reality Academy Tutorial — Motion Controllers.

https://docs.microsoft.com/en-us/windows/mixed-reality/mixed-reality-213
HoloToolkit/Input/Scripts/Utilities/AttachToController.cs
Mixed Reality Academy — Motion Controller Tutorial
Mixed Reality Academy — Motion Controller Tutorial

How to use

Design & Development Guidelines

Mixed Reality Toolkit Standard Shader example scene

MRTK/StandardShader is a new shader which provides various kinds of effects such as glowing border, hover highlighting, and animated properties. With this shader, you can easily achieve Fluent Design System’s elements such as light, depth, and material. It is optimized for the performance in HoloLens to run at 60fps.

Example Scene

You can find great examples of the materials based on MRTK/Standard Shader.

HoloToolkit-Examples/StandardShader/Scenes/MaterialGallery.unity
You can easily achieve near plane fade with MRTK/Standard Shader

Next Step

I introduced some of the useful building blocks in MRTK. I hope this can help your mixed reality journey! Join the community by opening an issue, reporting a bug, and contributing your code!

Mixed Reality Toolkit (GitHub)

Mixed Reality Academy Tutorials

Guidelines on Windows Mixed Reality Dev Center

Japanese version

My other stories on Medium