Oh Joy: On An Alternate HoloPlayer Control Scheme
If you’ve gotten a chance to use a HoloPlayer One, you know that the primary mode of interaction is 3D finger tracking via the integrated RealSense SR300 depth camera.
Your finger is a great, straightforward and intuitive tool to “touch” the hologram, but there are times when we want more precise control over our content than afforded by the meaty finger. Our questions were — “How do we trigger a command or action when all we’re tracking is a point in space?”, “ How does the user tell the system that they want to manipulate a holobject?” and “How does the system feed back to the user that this has happened?”
I was brought on board at Looking Glass to work towards solving that problem, and for the past few weeks I’ve been working on finding an alternate control device with which we might poke and prod our favorite floating pixels.
I’ve also spent the past few weeks helplessly watching hundreds of hours — and dollars—vanish over the event horizon of my own personal temporal-financial black hole: the Nintendo Switch.
The Switch “Joy-Con” controllers have several features that make it attractive to a potential “poke-r” of holograms. They have buttons, an accelerometer and gyroscope array for positional tracking, and a linear resonant actuator for vibration feedback — Nintendo’s much-touted ‘HD Rumble’.
Used as a holo-controller, one can:
- Manipulate onscreen objects with button and joystick inputs rather than using the keyboard.
- Send orientation data to the HoloPlayer, enabling the user to point with not only a position, but also a direction. Great for effects like directional brushes or lighting.
- Send vibration feedback — give a little “bump” to let the user know that the position tracked by the depth camera overlaps with a floating holographic object. Or let the user feel a texture when working with a virtual material in the hologram.
We don’t even need a Switch to do it!
The Joy-Con itself uses standard Bluetooth communications and can be connected to any host computer. On top of the HoloPlay SDK used to make new apps in HoloPlayers, I’ve created an open-source Unity package that exposes interfaces to motion data and button and joystick input, as well as making it easy for developers to dynamically send vibration data to the Joy-Con. I’m currently in the process of using this library to retrofit Joy-Con controls into existing HoloPlayer demos for more immediate immersive and tactile experiences — if you happen to have a HoloPlayer and a Joy-Con around, stay tuned for new content real soon.
Remaining problem: how do we get the depth camera to pick up the Joy-Con? It’s a gray blobby little ovoid with no pointy bits. I now present the product of countless minutes of painstaking research: The Joy-Con Wand, meticulously handcrafted from a Joy-Con, a rock candy stick, and some spray paint.
Want to make your own Joy-Con wand? Buy the following:
Then follow these handy Ikea-style instructions:
There you have it — your very own Joy-Con wand attachment.
Welcome to the future!
If you’d like to jump into the hologram revolution, we’ll be launching these systems in late November — you can sign yourself up to be one of the first to prototype on a HoloPlayer One on our website.
Follow Looking Glass on Facebook, Instagram and Twitter. Get involved in the Hologram Hacker Forum here!
