Vive Tracker Overlay Project.
In March 2017 I was lucky enough to get my hands on one of the first Vive Trackers.
Using the Tracker with Leap Motion, OpenVR_Overlays and a wireless microphone I built a tracked avatar.
The problem is that when people are in VR they can’t see you, can’t hear you and can’t interact with you. The goal was to build a solution using the Vive Tracker to give demonstrators presence.
The first challenge was getting something to persist across any SteamVR experience. I knew this was definitely possible from resources like OVRDrop - a pretty cool app that mirrors your desktop within SteamVR.
The OpenVR Overlay interface provides access to draw 2D images over the 3D scene no matter which application is running, this was used to build the first attempt:
This demo was created with the following steps
1. Setup non-VR unity scene.
2. Use SteamVR scripts to track Tracker.
3. Build a cube using 6 2D planes around the Tracker
4. Directional audio from the location of the Tracker
5. Use OpenVR to overlay the planes
This was pretty cool, although it was still 2D which seemed to be a limitation of the OpenVR Overlay API. Still, it was time to get creative…
I wanted a 3D overlay, and I knew that it was possible, I had seen Triangular Pixel’s Tracker Cat video. I just didn’t understand how.
After spending some time failing to use or understand the Compositor, I thought of a w̶o̶r̶k̶a̶r̶o̶u̶n̶d solution. Instead of rendering a plane to world space, render it to screen space. Essentially cover the entire screen with a plane that renders a camera feed from another unity scene. Think of it like wearing an AR headset… in VR. Here’s a demo using the Leap Motion:
This demo was created with the following steps
1. Setup non-VR unity scene
2. Use SteamVR script to track HMD
3. Attach camera to HMD
4. Incorporate Leap Motion asset, as a child of the HMD
5. Set the HMD camera’s target texture to a render texture
6. Use OpenVR to overlay the render texture
HMD* (head-mounted-display)
This irony was that the second attempt was actually much simpler than the original, it was also much better. It felt really good.
Now that I understood how to do get a ‘3D overlay’ it was time to reincorporate the Tracker. I stuck the Tracker and the Leap Motion onto a headset, and built in some directional audio:
This demo was created with the following steps
1. Setup non-VR unity scene
2. Use SteamVR script to track HMD attached to a camera
3. Use SteamVR script to track Tracker attached to gameobject with hat model
4. Directional audio from the location of the Tracker
5. Incorporate Leap Motion asset, as a child of the Tracker
6. Set the HMD camera’s target texture to a render texture
7. Use OpenVR to overlay the render texture
This setup has vastly improved giving demos, as it’s much easier to be heard and seen. The setup also helps when introducing people to VR by mitigating “VR-shyness” - where people are reluctant to move, look around or engage with the environment at all.
I’ve put the project on Github. Check it out here.
So there we have it. A solution that can integrate into any VR experience, creating an avatar with a tracked head, tracked hands and directional audio.
Hopefully people will stop bumping into me now.
