Project North Star by Leap Motion
A ton of attention in the mixed reality space has been on Magic Leap and the Magic Leap One announced recently, with videos of NBA Star Shaquille O’neal trying out the device to promote a media partnership. The device is highly anticipated, and Magic Leap has not been afraid to pull every PR stunt to promote the yet unavailable products, like releasing statements that developers must agree to keep devices ‘under lock and key’ as a part of the agreement to get one in early stages.
This weeks announcement by Leap Motion that it has also been playing with mixed reality HMDs. Leap Motion released a gesture interface device several years ago. I got a couple on a Kickstarter campaign in 2014. I remember not being very impressed with the interface. To be fair I was using the device to control a Windows environment, which was not designed for gesture control. The small device would sit in front of your PC and allow you to wave your hands in front of your screen to do ‘mouse actions’ like scroll, click, swipe, etc.
In 2014 at TEDxKyoto, our interactive media team and member Sagar Patel used this device to create a sound and motion experience, where music and a user’s motion were mixed to create a visual interactive station at the event.
Things got more exciting when VR developers started strapping a Leap Motion device on the front of VR headsets. This allowed the use of hand gestures in virtual environments, and many popular VR titles built in support . At the same TEDxKyoto event mentioned before, one of our partners, Fuji Xerox debuted a VR card game experience using the Leap Motion and the HTC Vive.
The recent announcement by Leap Motion brings this hand gesture to mixed reality and looks more like a Microsoft Hololens or a Meta 2. The main input and tracking device is a modified Leap Motion, so the set-up is mainly geared toward tracking hand gestures. It doesn’t do the robust environment tracking the other devices do, and it meant to be used for virtual wearable devices. It places a Leap Motion just above your eyes to track hand positioning. Then displayed an overlay, mostly on or near hands to allow an interface that seems both intuitive and natural. Microsoft and the Hololens, while is fantastic at tracking the environment and putting static virtual content in indoor spaces, sometimes feels unresponsive to gesture control. They designed a pinch and release system to move virtual objects. The Leap Motion seems much better at gesture control, but probably won’t allow for static items outside of view, we shall see.
I am most impressed with how well these demos seem to deal with the issue of occlusion. One of the biggest issues, my own AR in learning application ARienation included, is that without proper depth sensing, all overlays appear in front of all other real objects. These demos seems to show very good occlusion of hands with the virtual objects, this makes interacting with them seem much more natural.
Theses devices will not be for sale, but perhaps even better, Leap Motion is releasing the plans and the SDK to build software for the device to the public in open source. This means we can expect multiple companies to jump in and try to create an HMD using the technology. And in the meantime, we are free to try and make our own. As a researcher in AR/VR/MR in learning contexts, I find this very exciting. In my opinion, this may be more useful in classroom and vocational training than Magic Leap or Hololens, helping to guide surgery or mechanical maintenance much better as the level of control is much higher.
I will be following this story as it progresses. Let’s we can build something good with this technology Leap Motion is giving us.
