Building an XR Application in Unity with MRTK [Part 6] —Creating the score board

Thus far, we we have created a Ball object that responds to user input, targets that spawn and react to our Ball and a dynamic environment for the scene. To bring it full circle, we will now create some UI to inform the user of their progress by tracking and displaying a score. Every time the user hits a bulls-eye or shoots their ball through a ring, they will accumulate points that will be displayed in the game. The user will also have the ability to use voice commands to toggle hiding/showing the scoreboard. Furthermore, instead of placing the score at a particular point in a dynamic, unknown scene, we will have the content follow the user’s head and remain visible in their FOV.

Simple Scoreboard for XR game

Setting Up Voice Commands

In order to react to voice commands from the user, we need to register our keywords with the MRTK system. This can be done in the MRTK profile configuration. Furthermore, we also have to enable the UWP capability for when the app is deployed. This will prompt the user requesting that our application have access to the microphone so we can process voice commands.

Enabling UWP Microphone Capability

1. Open the Player Settings window
2. Expand the Publishing Settings section
3. Scroll down to Capabilities
4. Enable the Microphone capability checkbox

Adding Keywords to MRTK

1. Select the MixedRealityToolkit GameObject in the root scene and view it in the Inspector window.
2. Select the Input tab on the left-hand side and then expand the Speech section. The Speech section configures the voice commands registered within the system for MRTK.
3. If not already done so, clone the default speech commands profile by clicking the clone button and provide a new name (i.e AppMixedRealitySpeechCommandsProfile)
4. Click the + Add a New Speech Command Button twice
5. For first keyword entry, set Keyword property to “Show Scoreboard” and KeyCode property to Comma (or other desired shortcut key)
6. For second keyword entry, set Keyword property to “Hide Scoreboard” and KeyCode property to Period (or other desired shortcut key)

Creating the ScoreManager script

Most of the functionality of the ScoreBoard can be fulfilled by using Unity and MRTK components. However, we need a custom script to keep track of the current score and toggle visibility of the Scoreboard visuals.

1. Create a new script called ScoreManager.
2. Like TargetManager, ScoreManager will be a Singleton since there will be only one instance needed in the scene.
3. Next create a reference property for the TextMeshPro label that will display the score named scoreLabel
4. Finally, we need to store the actual score. We will utilize a uint variable since we do not want the score to go negative and we will create getter/setter property that will update our TextMeshPro label whenever the score changes. This ensures we only perform the work to update our UI when our state actually changes.

Next we need to create public functions that allow other components to modify the score property. Thus we have an AddScore(uint) and a ResetScore() available and again note we only take in a uint so we cannot reduce the score and make it negative. IMPORTANT: Calls should be made to the Score property, not the score variable directly. This way the UI updates accordingly as state changes.

Finally, we need to define the logic for showing/hiding the visuals. We simply need to create some public functions that are accessible to other components. We also want to initialize our score and UI to 0 when the game starts on Awake().

Building the Scoreboard prefab

1. Create a new empty GameObject named Scoreboard in the root scene
2. Drag this GameObject into your Prefabs folder to create an original prefab and open the prefab view
3. Create a new TextMeshPro label named Label as a child of the Scoreboard GameObject. Scale, size, and stylize to desired configuration. It should remain roughly in the middle of it’s Scoreboard parent so it remains in view. It is also recommended to center align the text.
4. Add an Overlap component to the Scoreboard GameObject. On the SolverHandler component that will be auto-added, modify the Additional Offset property to <0,0,1>. Ensure theTrackedTargetType property is set to Head.
5. Add the ScoreManager component to the Scoreboard GameObject
6. Add a SpeechInputHandler script component to the Scoreboard GameObject.
7. Uncheck the Is Focus Required property and click the + button near the bottom right to add a new keyboard.
8. Expand the new keyword entry, for the Keyword property select from the drop-down the “Show Scoreboard” item. Click the + button for adding a UnityEvent entry into the Response() field. Assign the Scoreboard GameObject as the assigned object and then select the ScoreManager.Show() function.
9. Click the + button again to add a new keyboard entry.
10. Expand the new keyword entry, for the Keyword property select from the drop-down the “Hide Scoreboard” item. Click the + button for adding a UnityEvent entry into the Response() field. Assign the Scoreboard GameObject as the assigned object and then select the ScoreManager.Hide() function.

The Overlap solver component will keep the Scoreboard GameObject tied to the player’s head and the Additional Offset property will keep it in front of the user’s view.

The ScoreManager will keep track of the current score and update the TextMeshPro label with the latest value. This component will also show and hide the label visual as desired.

The SpeechInputHandler component will listen for voice command input events from the MRTK system and if the keyword matches one of our entries listed, then the component will execute that function. Furthermore, disabling Is Focus Required ensures we receive all global speech input events.

Scoring against the Targets

Now that we have a simple but functioning score board we need to register when points are awarded to the Player. We can do this simply in the Target classes we created in Part 4. When a bulls-eye target has collided with the Ball, we add a call to the ScoreManager Singleton to increase the score by some amount.

Likewise, we can modify the logic for RingTargets by increasing the score by double since rings require the ball to go through instead of just making contact.

UI/Text on HoloLens2

One final important note exists between the UI/Text and how these are rendered on HoloLens 2.

The Windows Mixed Reality platform provides advanced logic to stabilize holograms based on head movement for any application. There are multiple techniques to accomplish this but the default one, if enabled, on HoloLens 2 is Depth Buffer Late-Stage Re-projection. The actual technique could require a whole in-depth article of it’s own but for the purposes of XR developers, the key fact to know is that the platform requires a valid depth buffer. Effectively, the platform will use the depth buffer data to determine how far and where certain holograms exist and thus as the head moves every so slightly, can transform the image to make it appear more stable to the user.

More details on hologram stability for Windows Mixed Reality can be found here.

To review, a depth buffer is a texture that instead of storing color values (RGB) per pixel of the scene, it stores depth values which equate to how far an object is at every pixel rendered of the scene.

One can easily view their scenes depth buffer in Editor using MRTK.

1. Select the MixedRealityToolkit GameObject and view it in Editor
2. Select the Editor tab on the left-hand side.
3. Enable the Render Depth Buffer property and open the Game View window.

So, now you may ask, why does any of this matter for UI/Text? The answer lies in the fact that: transparent/UI GameObjects in Unity, by default, do NOT render to the depth buffer!

To illustrate this, consider the gif below. When toggling the Render Depth Buffer property in MRTK, notice how the opaque primitive all render to depth but the text data does not!

In editor, there is no noticeable difference while in play mode. However, when the application is deployed to a HoloLens device, one will notice that transparent GameObjects and UI/Text will shake and “stutter” in view, sort of like dramatic anti-aliasing. This is obviously not a wonderful experience for the user. Thus, how can we overcome this issue?

The easiest way to fix this is to have data written to the depth buffer, either by:

1. Modifying all materials to always write to the depth buffer
2. Placing an opaque background behind the text like a quad panel that the UI sits on

For the purposes of our Scoreboard, we can modify the TextMeshPro label as follows.

1. Open the Scoreboard prefab and select the TextMeshPro label GameObject. Scroll to the material information.
2. Change the shader to Mixed Reality Toolkit/TextMeshPro
3. Click the Fix Now button under the Depth Write Mode section.

Now, our Scoreboard label will write to the depth buffer which is evident in viewer:

Review

With this final piece, we effectively have a simple, but complete XR game! We will have one more article to discuss building and deploying the application to device.

Every article thus far has had ample opportunity for extensions and improvements and it is recommended for the reader as an exercise to enhance this component as well. Great example points to consider:

• Design UI to instruct user how to hide/show keyboard.
• Extend UI to give additional stats such as num of bulls-eye vs Ring Targets hit
• Design UI to provide instructions and/or tutorials for how the user should play the game
• Create main menu to restart game or add a timer to the game to spice up the game design

Previous section: [Part 5] — Placing targets on the environment

Next section: [Part 7] — Building the app

Table of Contents

References

• Code Source GitHub

Hologram stabilization

Hologram stability - Mixed Reality

Speech

Camera in Unity - Mixed Reality