The mouse was for the computer. The …….. is for Augmented Reality.

Nat Martin
Apr 15, 2019 · 5 min read

The original HoloLens was the most magical piece of technology I had ever experienced. As someone who has a love-hate relationship with their smartphone, experiencing HoloLens was exciting because I glimpsed a future where technology was no longer an obstacle to the real world, but instead a tool to enhance it.

Still, my 2016 HoloLens experience was far from flawless: The hand tracking was not great, and the mode of interaction was counter-intuitive. You use the HoloLens(v1) by raising your hands in the view of the headset and performing a special pinching gesture — known as an air tap — to click. These kinds of gestures were hard to learn and unnatural to execute. And often simply not reliably recognised by the device. So I decided to explore what the alternatives were.

I’ll give a rundown here:

Hand tracking

Image for post
Image for post
Using a Hololens V1 (Credit: Jason Urgo)

Over the past few years, progress has been made to increase the resolution of hand tracking. Now Leap Motion provides excellent tracking and the fully-articulated hand tracking on the new HoloLens appears to be greatly improved.

However, even if these resolution problems are entirely resolved, I don’t think camera-based hand tracking is the silver bullet for XR input. For hand tracking to work your hands need to be in the view of the camera. Even wider field of view promised by new hardware, (e.g Leap Motion) there’s no escape from the tiresome act of holding up your hands for long periods of time (aka Gorilla Arm Syndrome).

This Minority Report–style input takes up space and looks stupid. I’m aware social norms around technology change quickly — I mean look at airpods — but I really struggle to imagine a future where everyone is waving their hands in front of a headset on the bus to work.

So, what other options are there for the future of XR?


Eye tracking can play a role in XR input too.

Handheld Controllers

Yes — as Palmer Lucky pointed out — the controller’s electromagnetic tracking is far from perfect (and even further from perfect when near anything large that’s made of metal) … but it does make the headset much more usable than the Hololens v1.

Yet, the biggest problem with the Magic Leap controller is that it undermines the core experience of AR: it stops you interacting directly with the real world. Imagine using it on a construction site, while driving a car, or during your breakfast…

Image for post
Image for post
Using a Magic Leap during breakfast (Credit: Magic Leap)

EMG Controllers

Image for post
Image for post
CTRL Labs Demo (Credit: CTRL Labs/ VentureBeat)

These EMG technologies have the potential to be a good compromise between handheld controllers and camera-based hand tracking. However, there are still some unresolved signal processing challenges before EMG is ready to be a reliable input device. (Having said that, I am yet to try Ctrl Labs latest demos.)

All of the above methods will likely play a role in the future of XR interaction, however none of them provide a complete, viable and intuitive input solution for XR creators today.

So, we made LITHO…

LITHO is a wearable controller, complemented by our tracking software and UX toolkit. The controller itself is worn between your first and second finger. It has a capacitive trackpad on the underside, an array of motion sensors, and provides haptic feedback. LITHO’s design means you can wear it while doing everyday activities (e.g typing, driving, drilling).

Image for post
Image for post
Image for post
Image for post
LITHO let’s you precisely manipulate content in the real world.

LITHO’s tracking software outputs the approximate position and rotation of the controller relative to the phone’s/headset’s AR camera — allowing developers to create apps that let you interact with objects in the real world, simply by pointing.This is achieved by taking data from the camera and the IMU to estimate the controller’s position in space. In real life this equates to an intuitive experience which feels somewhere between a typical ‘3DOF’ VR controller and a fully tracked controller like the VIVE. We have created demo use cases which include AR 3D creation apps that allow you to design buildings in context and to scale.

What’s more, LITHO can also be used without a visual output. For example, with your phone in your back pocket you can create apps that directly interact with connected objects such as lights. And these interactions can be enhanced further through LITHO’s haptic feedback.

Image for post
Image for post
LITHO lets you directly interact with connected devices simply by pointing.

LITHO is unique in that it’s cross-platform and output agnostic. This means that developers creating Unity apps for niche AR headsets, like HoloLens, can build the app for iOS and Android too — with one click. This enables billions more people to try these complex AR experiences for $199, as opposed to thousands.

Litho is shipping now. If you’re interested in developing with LITHO, please order your device here.

Hope you enjoyed reading our first medium article. We plan to write articles on the below topics in the coming months:

  • Why input is the barrier to handheld AR.
  • How spatial interfaces can change how we process and understand data.
  • The problems with XR terminology, and the solutions.
  • Rapid prototyping spatial interfaces.

Leave a comment if any of these are of particular interest, or you have any questions or feedback.


An interface for the real world.

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store