First steps into an AR Cloud: Relocalization

Back in June of 2017, on the tails of the WWDC 2017 ARKit announcement, when the world’s imagination began to run wild with the sudden availability of smartphone AR features on tens of millions of iOS devices, Ori Inbar of Super Ventures wrote a seminal post about the “AR Cloud”. In his post, Ori explains what is needed to transition from the novelty AR apps we see today toward the AR future of which we’ve been dreaming — a revolutionary new ubiquitous computing interface providing natural interaction with virtual objects and interfaces. @keiichiban’s Augmented City paints a vivid picture:

Keiichi Matsuda’s “Augmented City” (2010) (The 3D Red/Blue anaglyph can be disabled on the Youtube page.)

If you missed Ori’s article, check it out, or watch his supplementary video here:

Ori Inbar’s original “AR Cloud” explanation

Since then, many more “AR Cloud” posts have appeared on Medium, largely supporting Ori’s position, with some finer points made about the relationship between AR Cloud and ephemeral shared state solutions like Google Anchors.

As of today, just days after the Augmented World Expo 2018, the most topical and hotly contested AR Cloud technology is relocalization. (Relocalization enables your device to accurately learn its position and orientation in real world space, which in turn enables multi-user and persistent AR experiences.) Until fairly recently, AR positioning has required markers, such as those used by Vuforia:

Vuforia marker tracking

Markers have been a staple in AR tracking for years, but markerless versions of AR tracking tech are now rapidly emerging. By my count there are now over half a dozen markerless relocalization plays in motion. Here they are, in no particular order:

Google’s Visual Positioning Service (VPS)

Once Apple launched ARKit, Google made the decision to drop the depth camera requirement, and pursue RGB-only based SLAM techniques. In doing so, area definition files were dropped from the newly repackaged ARCore, and Android-based relocalization has been off the table ever since. Almost a year later, Google’s “Visual Positioning System” is resurfacing, which promises relocalization without an infrared depth camera.

Google VPS teaser

As of the writing of this post, Google VPS is not yet available to the public.

Vertical’s Placenote SDK

On it’s own, this free app is not an all in-one AR Cloud solution, but instead offers people the opportunity to experiment with relocalization today. For app developers, Vertical has launched their Placenote SDK so that more specific use cases can be addressed. Unlike many of the other SDKs outlined in this post, the Placenote SDK is openly available now.

6D.AI

Additionally, 6D has recently released video demonstrating real-time mesh construction, which has interesting use cases for AR applications that benefit from topological information about the immediate environment.

real-time mesh creation

As of the writing of this article, the 6D.AI SDK is in beta via invitation only. Applications can be submitted here.

Fantasmo’s Camera Positioning Standard (CPS)

  1. At first glance, the size of the tracked space appears to be larger than some of the other demo videos.
  2. They call out autonomous robot navigation as a use case for a “machine readable” 3D map.
  3. Their pending Camera Positioning Standard (CPS) proposal promises future interoperability (though the standard itself appears to be unpublished at the time of the writing of this article). Details here are scant, but these are the first traces of shared open infrastructure in the service of creating AR common spaces. More on this later.

Fantasmo’s approach is an ambitious one. To learn more, visit camerapositioning.io.

Selerio

Selerio is also accepting applications to access their SDK.

Scape

Unlike the indoor propositions, Scape’s relocalization is global, meaning that once your device is localized, you know your precise position in world coordinates, not just local, relative coordinates. (More on this in an upcoming post.)

visualization of Scape AR markers

Scape is currently accepting applications for their private beta which offers relocalization in London, England.

Sturfee

I can’t be sure, but I believe Sturfee may also provide global coordinates.

Sturfee is accepting signups for its SDK now.

Vuforia’s Object Tracking

In previous releases, Vuforia has showcased “Extended Tracking” which like ARKit and ARCore, uses sensor data to track device pose even when markers have left the camera field of view. (It is unclear to me if this work is compatible with ARkit or ARCore.)

Jidomaps

Jidomaps demonstrates saving and loading an AR frame of reference.

Apple’s Arkit 2.0

Apple ARKit 2 WWDC Demo

Now, just a few days later, the first multi-user ARKit 2.0 test applications are appearing:

In Summary

First, no attempt has been made to test the veracity of any of the claims made by these companies; nor has any testing been done to measure the relative performance of the techniques employed under the myriad of possible environments and circumstances. There is a huge amount of work to be done by the community at large to sift through and sort out how best to use these technologies. In other, more mature industries, I have seen head to head “shoot-out” tests in which third party labs test for accuracy and speed — this style of testing is a much needed next step for those of us collectively trying to build in Augmented Reality.

Second, and yet more important to my own thesis, relocalization is just the most immediate of many obstacles between the novelty pop-up-book AR and the AR future of our dreams (or nightmares). My hope is that the discussion around “AR Cloud” will come to include some of the other necessary services toward internet-scale AR. Challenges surrounding privacy, security, application fragmentation, open standards development, content management, identity management, community management — each in their own right difficult in a world without augmented reality, yet exacerbated in an augmented world.

Hopefully this post will help set the stage for that discussion. Stay tuned for follow up posts and please feel free to reach out with comments or questions.

XR & Spatial Technology

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store