Saving ARKit Planes and meshes across multiple sessions using PlacenoteSDK

Users are tired of scanning surfaces for planes every time.

Planes and persistent features!

The ephemeral nature of ARKit sessions means the user has to constantly look for planes when starting a new app, even if the environment is the same. A lot of the recent spate of AR apps have this problem.

Constant ask to setup ruins what is otherwise a great experience! (Playground AR)

Using Placenote’s scene recognition technology this can become a thing of the past! I’ll show you here how to do this for ARKit using Unity’s ARKit plugin! However, a similar solution is easily possible using Scenekit.

Surfaces are important

If you are looking to augment reality, you must bring in an aspect of reality into the digital world. Planes detected in the users environment are the most basic building block for you to do so.

ARKit’s method of detecting planes is one of the hallmarks of its capabilities and with Apple recently adding the much-demanded Vertical plane detection, this feature is quickly becoming a must-have for truly immersive apps. Apple went one step further and now allows for custom non-rectangular meshes to be detected as well! Once these planes and meshes are detected, you can turn them into real game surfaces or use them to occlude your content!

Game surfaces! (h/t @nk)

If you have never seen Placenote work before, I suggest you try the sample apps, moreover, there are guides available that walk you through building a simple example in Unity and Scenekit.

Saving scenes in Map Meta-data

If you know how to save a map, load a map and get a unique ID using Placenote in Unity you will have noticed that there is a meta-data field attached to all maps. This is a generic JSON-capable field that is attached to all maps and then available in the cloud. In our sample apps so far we take generic GPS data and save it with each map as floats.

if (useLocation) {     
   metadata["location"] = new JObject ();     
   metadata["location"]["latitude"] = locationInfo.latitude;              
   metadata["location"]["longitude"] = locationInfo.longitude;
   metadata["location"]["altitude"] = locationInfo.altitude;
}    
LibPlacenote.Instance.SetMetadata (mapId, metadata);

However, using a JSON serializer like JSON.NET for C#, you can serialize and then store any serializable data structures. Unity connects with ARKits service to find planes and meshes and uses Unity specific GameObjects to render them. So, we can define a custom serializable object that captures everything Unity would need render a mesh or a plane (if iOS 11.2 and older):

[System.Serializable]
public class ARPlaneMesh
{
   public Matrix4x4 transform;
   public Vector3[] vertices;
   public Vector3[] boundaryVertices;
   public Vector2[] texture;
   public Vector3 extent;
   public Vector3 center;
   public int[] trIndices;
   public string id;
 }

To save a list of meshes we can create a object as an array of ARPlaneMeshes.

[System.Serializable]
public class PlaneMeshList
{
   public ARPlaneMesh[] meshList;
}

Unity has a special AnchorManager class that manages Planes and meshes it finds in the environment. Once you want to save a map we iterate through all of the Unity plane details to be found in the manager and transfer it to our custom variable type, ARPlaneMesh and PlaneMeshList. The code here is a bit involved but it captures all the details you’d want to remember between Sessions:

We can then serialize and save this list,

metadata["planes"] = JObject.FromObject (saveList)
LibPlacenote.Instance.SetMetadata (mapId, metadata);

Maps saved with the planes in the meta-data!

This list of planes is now available to all your users! To retrieve later on, or on a different phone, extract the meta-data from the map with the relevant map id. When retrieving the map, lets also retrieve the meta-data:

LibPlacenote.Instance.ListMaps ((mapList) => {
    foreach (LibPlacenote.MapInfo mapId in mapList) {
       if (mapId.userData != null) {
           if (mapId.placeId == mCurrMapId) {
             //Correct map ID. 
             mCurrMapInfo = mapId;
             break;
           }
       } else {}
     }
     //Other functions to load map
}

Once the map is loaded and localized (as denoted by a status change), we can use Unity’s utility functions to render the planes (or turn them into colliders or occlusions!):

if (currStatus == Status.RUNNING && prevStatus == Status.LOST) {
   mLabelText.text = "Localized";
   var mapMetaData = mCurrMapInfo.userData["planes"];
   PlaneMeshList planeList = mapMetadata.ToObject<PlaneMeshList> ();        
   foreach (var plane in planeList.meshList) {
      GameObject go = PlaneUtility.CreatePlaneInScene (plane);
   }
}

Localized and all planes loaded!

Try it or incorporate it into your own AR app. This code is all open-source! I have organized this into a Unity project with the related UI’s and adequate factoring. The project is available on Github:

https://github.com/Placenote/PlacenoteSDK-SavePlanes

Don’t hesitate to reach out to us on our slack channel or find me @pj_mukh