MRTK 101 / FAQ: How to use Mixed Reality Toolkit Unity for Basic Interactions (HoloLens 2, HoloLens, Windows Mixed Reality, Open VR)

Yoon Park (박동윤)
Aug 22 · 6 min read
  • How to simulate input interactions in Unity editor?
  • How to grab and move an object?
  • How to resize an object?
  • How to move or rotate an object with precision?
  • How to make an object respond to input events?
  • How to add visual feedback?
  • How to add audio feedback?
  • How to use HoloLens 2 style button prefabs?
  • How to make an object follow you?
  • How to make an object face you?

How to simulate input interactions in Unity editor?

MRTK supports in-editor input simulation. Simply run your scene by clicking Unity’s play button. Use these keys to simulate input.

  • Press W, A, S, D keys to move the camera.
  • Hold the right mouse button and move the mouse to look around.
  • To bring up the simulated hands, press Space bar(Right hand) or left Shift key(Left hand)
  • To keep simulated hands in the view, press T or Y key
  • To rotate simulated hands, press Q or E(horizontal) / R or F(vertical)

Learn more about Input Simulation in the MRTK documentation.

How to grab and move an object?

To make an object grabbable, assign these two scripts: ManipulationHandler.cs and NearInteractionGrabbable.cs(for direct grab with articulated hand tracking input) ManipulationHandler supports both near and far interactions. You can grab and move an object with HoloLens 2’s articulated hand tracking input(near), hand ray(far), motion controller’s beam(far), HoloLens gaze cursor & air-tap(far).

NearInteractionGrabbable and ManipulationHandler.cs assigned to an object
Grab and move with input simulation — far interaction with hand ray / near interaction with articulated hand

How to resize an object?

ManipulationHandler.cs supports two-handed scale/rotation. This works with various input types such as HoloLens 2’s articulated hand input, HoloLens 1’s gaze + gesture input, and Windows Mixed Reality immersive headset’s motion controller input.

Resizing object with two-handed manipulation

Learn more about Manipulation Handler in the MRTK documentation.

How to move or rotate an object with precision?

Assign BoundingBox.cs to an object to use Bounding Box which is the interface for scaling and rotating an object. By default, it shows HoloLens 1 style blue handles and wires. To use HoloLens 2 style proximity-based animated handles, you need to assign prefabs and materials. Please refer to Bounding Box documentation and the BoundingBoxExamples.unity scene for the configuration details.

BoundingBox.cs assigned to an object
Default Bounding Box and HoloLens 2 style in the BoundingBoxExamples.unity scene

Learn more about Bounding Box in the MRTK documentation.

How to make an object respond to input events?

Assign PointerHandler.cs to an object. In the inspector, you will be able to use events OnPointerDown(), OnPointerUp(), OnPointerClicked(), OnPointerDragged()

To use these events in a script, implement IMixedRealityPointerHandler.

PointerHandler.cs assigned to an object

Learn more about Input System in the MRTK documentation.

How to add visual feedback?

Assign Interactable.cs to an object. In the inspector, create a new theme. Using Interactable’s theme profiles, you can easily add visual feedback to all available input interaction states.

Interactable.cs assigned to an object. Created a new color theme ‘CubeTheme’ for the Target object ‘Cube’

Interactable provides various types of themes including the shader theme which allows you to control properties of the shader per interaction state.

Learn more about Interactable in the MRTK documentation.

Another important building block for visual feedback is the MRTK Standard Shader. With MRTK Standard Shader, you can easily add visual feedback effects such as hover light and proximity light. Since MRTK Standard shader performs significantly less computation than the Unity Standard shader, you can create a performant experience.

Create a new material and select the Shader ‘Mixed Reality Toolkit > Standard’. Or you can pick one of the existing materials that use MRTK Standard Shader.

Assigned material with MRTK Standard Shader’s proximity light and border light options

Learn more about MRTK Standard Shader in the MRTK documentation.

How to add audio feedback?

Add AudioSource to an object. Then, in the scripts that exposes input events(e.g. Interactable.cs or PointerHandler.cs), assign the object to the event and select AudioSource.PlayOneShot(). You can use your audio clips or choose one from MRTK’s audio assets.

Audio Source assigned to an object. AudioSource.PlayOneShot configured in the Interactable’s OnPress() and OnRelease() events.

How to use HoloLens 2 style button prefabs?

MRTK provides various types of HoloLens 2’s shell(OS) style buttons. It provides sophisticated visual feedbacks such as proximity light, compressing box, and a ripple effect on the button surface.

Simply drag and drop one of the HoloLens 2 style pressable button prefab into your scene. The prefab uses Interactable.cs which is introduced above. You can use exposed events such as OnClick() in the Interactable to trigger actions.

Use events under Interactable to trigger actions

Learn more about Button prefabs in the MRTK documentation.

How to make an object follow you?

Assign RadialView.cs script to an object. It is part of the Solver script series that allows you to achieve various types of object positioning in 3D space. SolverHandler.cs will be automatically added.

Below is an example of RadialView configuration to achieve ‘lazy follow’ tag-along behavior just like the Start menu in the HoloLens shell. You can specify the minimum/maximum distance and minimum/maximum view degrees. The example below shows positioning the object between 0.4m and 0.8m range within 15°. Adjust Lerp Time values to make the positional update faster or slower.

Radial View Solver options
Example of using Radial View Solver on the Toggle Features Panel in the HandInteractionExamples scene

Learn more about Solvers in the MRTK documentation.

How to make an object face you?

Assign Billboard.cs script to an object. It will always face you, regardless of your position. You can specify the pivot axis option.

Billboard.cs script assigned to an object with Pivot Axis option Y
The left object always faces the camera with Billboard script, with Y-axis pivot option
Example of using Billboard script in the Tooltip prefab to make the labels always face the camera

Ready to create amazing experiences for mixed reality? Visit the pages below and learn more about MRTK and mixed reality.

Resources

You can find some of the foundational building blocks on MRTK GitHub’s landing page.

Microsoft Build 2019 session video: Building Apps for HoloLens 2 using Unity and Mixed Reality Toolkit

Microsoft HoloLens & Mixed Reality Meetup: HoloLens 2 Design & Development Overview

MRTK’s UX Building Blocks (Japanese): at Tokyo HoloLens Meetup, July 2019

Read my other stories

Yoon Park (박동윤)

Written by

Designer & Creative Technologist. UX Designer @Microsoft HoloLens. Creator of Typography Insight App. Parsons MFADT alum. Opinions are my own.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade