Building an XR Application in Unity with MRTK [Part 2] — Building the ball

The most important game component in this application is the player’s ball. This ball will follow the player’s hand, which ever one is first available. Using their hand, the player will shoot this “ball” into the environment to hit targets and score points. The player should also be able to retrieve the ball back to their person. This section will describe the process of creating this functionality for the application.

Building the prefab

1. Create a new empty GameObject in root scene, rename this GameObject to Ball and resize scale to <0.1, 0.1, 0.1>
2. Create a new Prefabs folder, if does not exist, under your Assets and drag this GameObject into this folder to create an original prefab. Open the prefab by select the Open button with the ball selected in Inspector window.
3. Ensure all GameObjects in the prefab are on the Ball layer which we configured in our project settings in Part 1.

Open prefab view after selecting ball in scene view

1. MRTK ships with some texture and model assets out of box that can be re-used. Search in the Project Window for Model_Bucky and drag the FBX file (NOT the prefab) under the Ball GameObject. Rename this bucky model to Mesh and change scale to <0.5, 0.5, 0.5>.

1. Select the Ball GameObject and add a Sphere Collider component, a RigidBody component (enable Is Kinematic property), and an AudioSource component.
2. Further, add an Overlap script component (from MRTK solvers). NOTE: after adding this component, the SolverHandler component will be auto-added.
3. Update the SolverHandler and Overlap properties to match the following. The most important property to set is the Tracked Target Type to Hand Joint. The Overlap script, while enabled, will force the ball to follow a specific position. This property indicates what Transform to follow/track against in which case with Hand Joint, it will follow either of the available hands, and in particular the palm hand joint.

1. Finally, create a new script called OrbManager and add it to the Ball GameObject.

Creating the OrbManager script

In the newly created OrbManager class, first step is to wrap the component in our namespace XR.Break and to include necessary namespaces we will use (see below).

1. On the OrbManager class, add the IMixedRealityInputHandler interface. This will allow the GameObject to respond to input events populated by MRTK.
2. Add a RigidBody property, a MeshRenderer property, SolverHandler property, and UnityEvents for when we fire the ball and when we retrieve it. This will be useful for coordinating actions in editor. Marking these private but with SerializedField attribute ensure they will be visible in Editor but not editable at runtime by other classes.

Defining States

Next, step we are going to define some state parameters for the manager to track against. There are three valid states for the ball.

1. Idle = the ball is being controlled by nothing, hide the ball.
2. SourceTracked = the ball is being tracked by one of the hand input sources (i.e the Overlap solver will be controlling it’s position to follow a hand)
3. PhysicsTracked = the ball is being controlled by physics in Unity

This OrbState enum is stored in our CurrentState property which defines what state we are currently in.

There is one other dimension of state to store for the ball. The player needs an ability to control how much power will be delivered to the ball when fired. The amount of force used in launching the ball will be determined by timer that calculates how long the user has been holding the select gesture, bounded by a max value. Thus, the manager can be in a state of “powering up” to fire the ball or not which is tracked by the IsPoweringUp property.

Responding to State Changes

Both of our state properties, IsPoweringUp and CurrentState, have update methods that execute when state changes, PowerUpUpdate() and OrbStateUpdate() respectively.

For the time being, PowerUpUpdate will just reset our timer when we change states (i.e the user is getting ready to launch the ball or it has fired).

The OrbStateUpdate is important as it drives the unity editor properties defined earlier at the top of the class. Effectively it controls whether each component is enabled or not based on our state. If we transition to any new OrbState, we reset our IsPoweringUp state since a user can no longer be powering up at any orb state change.

1. If Idle, disable everything include the mesh renderer
2. If SourceTracked, only enable the Overlap solver via the solverHandler.UpdateSolvers property. When true, Overlap solver will execute it’s update property and thus control the ball’s position via tracking the hands.
3. If PhysicsTracked, enable the RigidBody so the ball is manipulated by the physics engine in Unity.

Registering for Global Input

As referred earlier, this component needs to respond to input from the user for when to fire and when to “power up” the ball for launch.

When the user selects via closing their hand, this will result in a input down event fired by MRTK. When the user opens their hand, this will result in a input up event fired by MRTK. The interface added earlier, IMixedRealityInputHandler, provides interface functions (OnInputUp and OnInputDown) for the OrbManager to respond to these input actions.

Generally, MRTK only calls this interface if this GameObject is in focus when an action/input occurs. For our OrbManager, we want to listen to all input events and filter/respond to only particular input sources (i.e the hand currently controlling the ball).

To make OrbManager subscribe globally to all input events, the script needs to register itself with the Mixed Reality input system via RegisterHandler<IMixedRealityInputHandler>(). The general practice is to register ourselves with the input system OnEnable() and then un-register ourselves from events OnDisable().

Furthermore, we need to define the logic in our script for when an InputUp or InputDown event occurs. This is a bit nuanced for the purposes of this article but the OrbManager script only wants to respond to events that are fired from our currently tracked hand and that are of input action type “Select”. (read more about MRTK Input actions here). This is handled by our if statement in each interface call. In the following section, it will be illustrated how our local trackedController reference is captured by OrbManager. Every input source in MRTK such as a hand or controller is assigned an input source id which unique identifies it in the system.

If the user selects via closing their hand (i.e OnInputDown fires), then we want to start powering up our ball if we are in the SourceTracked state.

If the user releases by opening their hand (i.e OnInputUp fires), then we have two paths to execute. If the user is holding the ball (i.e SourceTracked), then we want to fire it into the world. Else, we want to retrieve the ball from the scene to the player’s hand and begin tracking the ball via our hand again.

The Update Loop

This is the final piece of the puzzle for our OrbManager script. The update loop will perform the necessary work that needs to be done every frame based on our current states.

First thing needed to be done is to determine if our hand tracking has changed from the last frame. HoloLens can lose hand tracking if the user lowers one or both of their hands out of view from the device. For example, if the user lowers their hands to their side while looking up or ahead. The device’s cameras cannot see the hands and thus not track them.

Fortunately, it is easy for OrbManager to determine if a tracked hand is available because the Overlap/SolverHandler components record this very information. Remember, the SolverHandler was constructed in editor to track the palm hand joint of any hand available. The SolverHandler’s TransformTarget property if not null, returns the transform of the palm of our hand. If it is null, then there is no hand available.

Thus, if our tracking state changes, we want to get the IMixedRealityController instance of our currently tracked hand and the LinePointer instance on that hand. These are retrieved via helper methods GetTrackedController() and GetLinePointer(). This is performed at the beginning our Update loop function.

What is a LinePointer? Out of box, MRTK creates various pointers when an input source is detected such as a controller or hand. The LinePointer is a type of pointer that shoots a ray from controllers/hands into the world and allows for easy far interaction selection. The LinePointer will be useful as it provides an effectively calculated ray from the hand out into the world which will be useful to fire the ball.

Next step in the Update loop is to react to the currently set states based on whether we are tracking or not.

If we are not tracking, then set our current state to Idle. There is no hand to track against so we are going to hide the ball.

If we are tracking against a hand, then either enter the SourceTracked state or increase our powerUpTimer because the user is holding down the select gesture(i.e their hand is remaining closed).

Ready, aim, fire!

Finalizing Connections in Editor

With the code pulled together, the final piece to getting this component working is to hook up the necessary references in editor.

IMPORTANT: The Animator and Power Up fields will be added in a later section, styling the ball. At this point in time, ensure all other properties match.

For the audio source files, MRTK ships out-of-box with some default audio clips. These can be found by searching in the Unity Project Window for each clip or use one of your own!

Review

With this prefab built, go back to the root scene and enter play mode.

Remember that holding the space bar will bring up the right simulated hand which can be controlled by the mouse. Hold down the left mouse button for a period of time and then release. This should shoot the ball into the scene. Select the left mouse button again quickly and this should return the ball back to the hand. Hide the hand by releasing the space bar and all components should disappear.

NOTE: The graphic below represents the completed, polished ball. The additional styling components are explained in the appendix.

Previous section: [Part 1] — Install the tools

Next section: [Part 3] — Scanning the environment (AR)

Table of Contents

References

• Code Source GitHub

microsoft/MixedRealityToolkit-Unity

Pointers

Solvers

Input actions

Full OrbManager source: