Image for post
Image for post

MRTK 101 / FAQ: How to use Mixed Reality Toolkit Unity for Basic Interactions (HoloLens 2, HoloLens, Windows Mixed Reality, Open VR)

Learn about how to use MRTK to achieve some of the most widely used common interaction patterns in mixed reality

Image for post
Image for post
  • How to simulate input interactions in Unity editor?
  • How to grab and move an object?
  • How to resize an object?
  • How to move or rotate an object with precision?
  • How to make an object respond to input events?
  • How to add visual feedback?
  • How to add audio feedback?
  • How to use HoloLens 2 style button prefabs?
  • How to make an object follow you?
  • How to make an object face you?

How to simulate input interactions in Unity editor?

  • Press W, A, S, D keys to move the camera.
  • Hold the right mouse button and move the mouse to look around.
  • To bring up the simulated hands, press Space bar(Right hand) or left Shift key(Left hand)
  • To keep simulated hands in the view, press T or Y key
  • To rotate simulated hands, press Q or E(horizontal) / R or F(vertical)

Learn more about Input Simulation in the MRTK documentation.

How to grab and move an object?

Image for post
Image for post
NearInteractionGrabbable and ManipulationHandler.cs assigned to an object
Grab and move with input simulation — far interaction with hand ray / near interaction with articulated hand

How to resize an object?

Resizing object with two-handed manipulation

Learn more about Manipulation Handler in the MRTK documentation.

How to move or rotate an object with precision?

Image for post
Image for post
BoundingBox.cs assigned to an object
Default Bounding Box and HoloLens 2 style in the BoundingBoxExamples.unity scene

Learn more about Bounding Box in the MRTK documentation.

How to make an object respond to input events?

To use these events in a script, implement IMixedRealityPointerHandler.

Image for post
Image for post
PointerHandler.cs assigned to an object

Learn more about Input System in the MRTK documentation.

How to add visual feedback?

Image for post
Image for post
Interactable.cs assigned to an object. Created a new color theme ‘CubeTheme’ for the Target object ‘Cube’

Interactable provides various types of themes including the shader theme which allows you to control properties of the shader per interaction state.

Learn more about Interactable in the MRTK documentation.

Another important building block for visual feedback is the MRTK Standard Shader. With MRTK Standard Shader, you can easily add visual feedback effects such as hover light and proximity light. Since MRTK Standard shader performs significantly less computation than the Unity Standard shader, you can create a performant experience.

Create a new material and select the Shader ‘Mixed Reality Toolkit > Standard’. Or you can pick one of the existing materials that use MRTK Standard Shader.

Image for post
Image for post
Image for post
Image for post
Assigned material with MRTK Standard Shader’s proximity light and border light options

Learn more about MRTK Standard Shader in the MRTK documentation.

How to add audio feedback?

Image for post
Image for post
Audio Source assigned to an object. AudioSource.PlayOneShot configured in the Interactable’s OnPress() and OnRelease() events.

How to use HoloLens 2 style button prefabs?

Simply drag and drop one of the HoloLens 2 style pressable button prefab into your scene. The prefab uses Interactable.cs which is introduced above. You can use exposed events such as OnClick() in the Interactable to trigger actions.

Image for post
Image for post
Use events under Interactable to trigger actions

Learn more about Button prefabs in the MRTK documentation.

How to make an object follow you?

Below is an example of RadialView configuration to achieve ‘lazy follow’ tag-along behavior just like the Start menu in the HoloLens shell. You can specify the minimum/maximum distance and minimum/maximum view degrees. The example below shows positioning the object between 0.4m and 0.8m range within 15°. Adjust Lerp Time values to make the positional update faster or slower.

Image for post
Image for post
Radial View Solver options
Example of using Radial View Solver on the Toggle Features Panel in the HandInteractionExamples scene

Learn more about Solvers in the MRTK documentation.

How to make an object face you?

Image for post
Image for post
Billboard.cs script assigned to an object with Pivot Axis option Y
The left object always faces the camera with Billboard script, with Y-axis pivot option
Example of using Billboard script in the Tooltip prefab to make the labels always face the camera

Ready to create amazing experiences for mixed reality? Visit the pages below and learn more about MRTK and mixed reality.

Resources

You can find some of the foundational building blocks on MRTK GitHub’s landing page.

Microsoft Build 2019 session video: Building Apps for HoloLens 2 using Unity and Mixed Reality Toolkit

Microsoft HoloLens & Mixed Reality Meetup: HoloLens 2 Design & Development Overview

MRTK’s UX Building Blocks (Japanese): at Tokyo HoloLens Meetup, July 2019

Read my other stories

Image for post
Image for post
Designing Type In Space for HoloLens 2

Written by

Designer & Creative Technologist. UX Designer @Microsoft HoloLens. Creator of http://typeinsight.org Opinions are my own. http://dongyoonpark.com

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store