Prototyping an AR interface on the Hololens
Not gonna lie, Unity had a steep learning curve, especially due to the limited resources available for UI and interaction design. Most of the tutorials centered around game development — Unity’s bread and butter. Microsoft’s Mixed Reality Academy , Unity’s UI tutorials and the wonderful world of internet forums were essentially the only (reliable) resources we had at our disposal.
As we didn’t have too much time to build the prototype, we focused our efforts on giving our audience a simple walkthrough of Dash’s main menu and key resources, such as a 3D models and schematics.
Input
At first, we hoped to incorporate gaze and gestural input for the prototype but due to the limited time to build it, we decided to use key input using the keyboard. There were two reasons for this:
- It was easier to build. We didn’t need to fully build and deploy the app onto the Hololens.
- As we used the Unity Remoting to live-stream onto the Hololens, the key input allowed the end-result on the Hololens to be a lot smoother. When attempting to use gaze and gesture, the prototype displayed was very slow to respond to input.
Animation
One tricky aspect of building on Unity was the animation of buttons. In order to use custom animations to show when buttons was highlighted or pressed, we created spritesheets that illustrated each keyframe in the animation sequence (see image above). Following this, each sprite was placed as keyframes on Unity’s animator.
Annotations On 3D Model
To show how interactive 3D models could enhance learning both in the classroom and out in the workshop, we wanted to illustrate an animated engine piston with annotations pointing to its parts. As the model rotated, however, it was important for the labels to always face the user. Credit here goes to Pierre Amelot from CMU’s HCI department for sharing how to do it and his code with us.
