LiDAR and ARKit Meet Pantomime for Reach-In Augmented Reality
(Updated to reflect developments with RealityKit and LiDAR in 2021.)
Apple’s new ARKit is bringing motion tracking for augmented reality to billions of devices. But as this video shows, while that’s great for viewing mixed reality scenes from a varying point of view, ARKit’s illusions break down when users get close to objects and surfaces, or try to physically interact. Pantomime’s unique approach supports close-up interaction and reaching in, while ARKit was the first step in taking Pantomime from small tabletop experiences into the effortless room-scale navigation and integration with real spaces it enjoys today.
Background
Pantomime’s founders — alumni of the teams that invented virtual reality and created The Sims — have always seen immersive worlds a bit differently from the “VR headsets will soon be everywhere!” crowd. Long before Pokémon GO, we figured for immersive worlds to be social and scalable — a common lifestyle rather than an isolating novelty — head-mounted displays must be optional.
Pantomime’s strengths have been:
- viewing and reaching into virtual worlds with handheld devices
- 3D models of devices for fast, accurate, low-latency interaction
- realistic physics with momentum, collisions and vivid sounds
- runs on common hardware — nothing new required
- networked experiences across tablets, phones, PCs and headsets
Pantomime has been betting that software-based 3D motion tracking without markers would soon be a commonplace commodity.
Now that bet is paying off: Apple’s ARKit and Google’s ARCore will provide the 3D tracking and surface recognition to make immersive augmented reality worlds work anywhere — while Pantomime makes them deeply interactive.
[Actually, thanks to recent developments with LiDAR, parts of this article first written in 2017 that refer to ARKit tracking are fast becoming a kind of period piece. The introduction of accurate depth tracking with LiDAR, built into devices like iPhone Pro and iPad Pro models, have changed the landscape and pushed things like markers into ancient history. A LiDAR sensor captures an accurate, growing depth model of the shape of surfaces in the space up to 16 feet in front of the rear camera as it moves. As presented briefly at the end of the article, since 2020 LiDAR has liberated Pantomime far beyond its tabletop experiences or anything ARKit could deliver.
To include LiDAR in our discussion, we’re bending our “common hardware — nothing new required” proviso above — just slightly. With Apple’s Pro devices, you never buy a LiDAR sensor — just a top quality mobile device with better performance and cameras than a cheaper one, for roughly $100 more. The overwhelming majority of iPhone Pro and iPad Pro don’t even know or care what LiDAR is, and didn’t get the device for Augmented Reality. In its October 2021 iPhone 13 Pro launch, LiDAR was an afterthought and augmented reality was never mentioned.
Compare that to whether someone might not know they’ve bought a VR headset. In that sense, though Apple’s LiDAR-enabled Pro iPhones and iPads aren’t currently the most widespread models, they’re quite common — millions have been bought, often subsidized by phone carriers — and thus they meet our “common device” criteria. By the time the first software Killer App for LiDAR-enabled mobile changes the landscape the way Pokemon GO did in 2016, there may already be 100 million magical AR devices lurking in the marketplace. And in that sense, LiDAR continues to play to Pantomime’s strengths.]
ARKit Brings Interactive Pantomime to Real Tables and Floors
Before ARKit, Pantomime provided portable 3D worlds where devices interact on a virtual surface, like a synthetic table or ice rink, directly under the real device. Balance the device on a real surface like a table, and the flat virtual surface lines up perfectly underneath it. Or play in mid air, on a bench or a bus, and still see the virtual surface you’re playing on. In this kind of portable virtual world, carrying the device moves the virtual world with you. To move a device through a world, you stroke the screen, or walk and tumble the device on its edges and corners.
Until now, Pantomime was best suited for tabletop experiences — play on a surface, whether real or virtual. By knowing the shape and dimensions of the mobile device it’s running on, and with a model of how it’s supported, Pantomime’s patented method provides local motion tracking using the device’s 3D accelerometers and gyroscopes. Since no video image processing is required and the motion sensors sample 100 times a second — much faster than a video frame — tracking is faster, stabler and lower latency than video based tracking— and very accurate as long as the device is in contact with the table. So interactive tracking works even in low light, if the camera is blocked, or you’re so close or moving so fast that the image is blurred.
ARKit Limitations
ARKit and ARCore are known to fail if a device is moved too quickly, if its camera is flat against a table or too close to objects, or in poor lighting. It has no model of how the device you’re holding is shaped or supported, never mind how it might collide or interact with augmented reality objects. The closer your device gets to an object, the bigger the problems. The video above shows several of the the issues and Pantomime’s patented solutions.
Pantomime Makes ARKit Reach-In Interactive
Pantomime’s approach, designed for rich close-up interaction, lets users accurately push, paddle, touch, throw, scoop, pour, and squash virtual objects in the scene!
2021 Update:
Revolutionary Realism with LiDAR on iPad and iPhone Pros
Pantomime is now fully integrated with Apple’s recent deployment of LiDAR depth sensing in all recent iPad Pro and iPhone Pro models. It eliminates all the space distortion and other issues of ARKit and tabletop limitations on Pantomime’s prior technology. Users can easily hold and throw virtual objects and effortlessly reach into physically realistic scenes and touch or scoop virtual objects with their devices and tools attached them.
Our flagship app Reality Faucet runs on every LiDAR device; its built-in Reality Store offers a growing library of free and paid In-App Purchases that include physically realistic objects with a wide variety of native physical behaviors.
This video provides a glimpse of Reality Faucet and its library in its August 2021 release:
The Reality Faucet app is available at the App Store here:
