Successful Relocalization with 6D
How beta developers can get the most out of multiplayer and persistence testing on the 6D platform
Last month, 6D.ai's Rik Basu showed us how to make Augmented Reality applications with persistence using the 6D Reality Platform. This time, we will focus on a crucial piece of user experience (UX) that makes persistence, along with multiplayer, possible at all: relocalization.
This post is to help explain to beta devs what is going on in the beta system right now. We are continually improving our APIs and we will make it simpler over time for both devs and consumers.
Before we start, let's imagine a real world situation: you just woke up and you feel disoriented within those first few seconds. This is not your bed. Where are you? You're not home. How did you get there? What's on the other side of that door? Pretty quickly, the furniture, the pictures on the wall, some details help you remember: this is your parents' guest room, you're staying with them for the holidays. With that sudden realization comes a tremendous amount of context: you arrived late last night, they have been living here for about 10 years now, you will be catching up over breakfast downstairs, but not before you shower in the small bathroom they just remodeled down the corridor, etc.
Most of the time, you know where you are because you already knew where you were the second before, and you know how much you’ve moved in the interval. This is also how, in AR, tracking works. In the cases when you do not know the location well, you need to compare the clues around you with your memory of the places you have seen before — a place with clothes, tags, a register is a store, a place with a stage and lots of seats is a theater, etc. This is how relocalization works.
Devices using the 6D Reality Platform collectively contribute three-dimensional geometry to the AR Cloud. This builds up a machine readable form of the world’s ground-level three-dimensional data in the cloud, adding new locations that have been meshed and thus can be remembered. This aggregation of 3D data is a unique capability of the 6D Reality Platform, and ensures that this memory grows over time and the UX for consumers continually gets better & better.
In order to leverage the AR Cloud, an app using 6D calls the SixDegreesSDK_LoadFromARCloud() API function. This function triggers a background thread that downloads a local piece of that collective memory. Once loaded into what we call a map, the platform compares it with what it sees from the camera's live stream. Quickly, matching elements are found and precise coordinates and orientation (together called pose) are returned. From there, relocalization is considered successful, so location-specific data can be loaded, and tracking takes over, which is less resource-intensive.
Just like you need to scrutinize your surroundings to understand where you are, you need to point your device at details that have been seen before to relocalize. One difference however, humans understand depth through stereoscopy thanks to our two eyes, but the 6D Reality Platform understands depth through motion, as it has only one camera at its disposal. For this reason, during relocalization (when the SixDegreesSDK_GetLoadStatus() API function reports SixDegreesLoadStateRelocalizing), the device has to be in motion. A step to the left, a step to the right, while pointing at those familiar details, will lead to a quick relocalization success. If it does not come, point at something else and try again. Sometimes the best results are obtained by moving around a room, going from object to object. As you move around the room, the local (on device) map gets bigger & more likely to find a match. Then later when you save to the cloud, the Global map (in the cloud) grows from the local contribution from the device, saving more viewpoints in that physical area & increasing the likelihood of a successful relocalization next time.
Additional tips:
- It does not help to scan before loading from the AR Cloud. Relocalization is attempted frame by frame in real time.
- It does not help to hold the device steady and point at an object, as it prevents depth detection, which relocalization cannot do without. You don’t need to move very much at all, but you need some motion, similar to initializing ARKit.
- Lateral movement is a more effective than rotation to detect depth. Try a step to the left, or a step to the right, instead of a panorama picture.
- What makes relocalization more successful also makes mesh-building more successful; try applying those tips the next time you scan an area and use the SixDegreesSDK_SaveToARCloud() API function.
You can join the beta here if you are interested in building on 6D or want to reach out to our team with any questions.
