Markerless Persistent Rendering of SCNode’s using ARKit and Placenote SDK

A software tutorial for building persistent content using ARKit

Note: A consistently updated step-by-step guide for integrating Placenote in SceneKit or Unity is available here https://placenote.com/documentation

I really like Ray Wenderlich’s iOS team. For anyone new to SceneKit (and by extension ARKit), they have a great set of resources to get started. One of their tutorials walks you through a very simple SceneKit app that is similar to Fruit Ninja but with simple shapes instead of fruit. This is predicated on the idea that it is extremely straightforward to create simple shapes and put them in front of you in ARKit. Once you have setup a simple ARKit scene (I trust this is easy given the myriad of other resources, I like Apple’s), if you want to put a sphere one meter in front of you in swift it’s just:

func renderSphere() {  
  let geometry:SCNGeometry = SCNSphere(radius: 0.1) //units, meters
  let geometryNode = SCNNode(geometry: geometry)      
  geometryNode.position = SCNVector3(x:0.0, y:0.0, z:-0.3)
  scnScene.rootNode.addChildNode(geometryNode) 
}

To control the workflow a bit, I’m going connect this renderSphere function to a UIButton action and also jazz up my sphere by adding some texture!

Now with some creative placing of where you start your app you get:

Look! it’s a virtual planet next to a real globe. Hah! SceneKit has all these primitives to easily render basic shapes (SCNSphere, SCNBox, SCNPyramid etc.).

However, what if you start in a slightly different spot than before?

Oh no! The planet is no longer next to the globe. This is because the SCNVector3 point (0,0,1) is in reference to where the latest ARKit session was initiated. It was easy to create the sphere, but it is not persistent.

We can use the Placenote SDK here to make the sphere (or any SCNode) appear in the same place all the time.

Integrating the SDK

Download and integrate the Placenote library into your ARKit XCode Project. There are detailed instructions for this on the Github (under ‘To integrate this into your own app’). Next, make your main ViewController a delegate of PNDelegate by adding it to your class definition:

class ViewController: UIViewController, ARSCNViewDelegate, ARSessionDelegate, PNDelegate {

And, add your main ViewController to be the multiDelegate of the PlacenoteSDK as the ViewController initializes (in viewDidLoad()or viewDidAppear())

LibPlacenote.instance.multiDelegate += self

As you add ViewController to be a multiDelegate, you will also need to add the following delegate functions to receive information from Placenote SDK. We’ll make use of these later.

//Receive a pose update when a new pose is calculated   
func onPose(_ outputPose: matrix_float4x4, _ arkitPose: matrix_float4x4) -> Void 
//Receive a status update when the status changes   
func onStatusChange(_ prevStatus: LibPlacenote.MappingStatus, _ currStatus: LibPlacenote.MappingStatus) -> Void

Finally, the Placenote SDK has a utility called CameraManager that allows the PlacenoteSDK to manage the position of the ARKit Camera between ARKit Sessions. You activate it by simply initiating the object and passing in the ARKit Scene and Camera Object,

if let camera: SCNNode = sceneView?.pointOfView {  
  camManager = CameraManager(scene: sceneView.scene, cam: camera)  
}

Mapping and Localizing

Now, you are ready to render your planet and use the PlacenoteSDK to remember where it was! To do this is a simple 3 step process.

1. Start a Mapping Session, place all the content in place
2. Save the map and retrieve a relevant Map ID
3. Load the map using the Map ID and start a Localization Session.

Once the map is localized, after step 3, the Placenote SDK retrieves the old ARKit session and moves the Camera position so that all subsequent ARKit sessions are the same as the first one!

So! To start the mapping session, simply call startSession and place the sphere where you want it. I also use the small label (sessionInfoLabel) to see feedback on the main screen

LibPlacenote.instance.startSession() 
shapesRendering = true 
renderSphere()  
sessionInfoLabel.text = "Mapping.."

Additionally, ARKit sends you an image update in a session callback, send it to Placenote like so:

func session(_ session: ARSession, didUpdate: ARFrame) {  
  let image: CVPixelBuffer = didUpdate.capturedImage  
  let pose: matrix_float4x4 = didUpdate.camera.transform  
  if (shapesRendering) {    
    LibPlacenote.instance.setFrame(image: image, pose: pose)  
  } 
}

Next! Step 2, you’ll want to save the map so we can recall it later. You do this by calling saveMap. I’ll do this here under the same UIButton and just co-ordinate the actions, so my final actions under the UIButton action is,

if (!shapesRendering) {  
  shapesRendering = true   
  renderSphere()   
  LibPlacenote.instance.startSession()   
  sessionInfoView.isHidden = false   
  sessionInfoLabel.text = "Mapping.."   
  button.setTitle("Save Map", for: .normal) } 
else {   
  LibPlacenote.instance.saveMap(savedCb: { (mapID: String?) -> Void in    
      print ("MapId: " + mapID!)    
      self.sessionInfoLabel.isHidden = false 
      self.sessionInfoLabel.text = "Map Saved.."
      LibPlacenote.instance.stopSession()  },
 
   uploadProgressCb: {(completed: Bool, faulted: Bool, percentage: Float) -> Void in        
    //Nothing to do here  
  }) 
}

Note, how the saveMap function returns a unique Map ID. I just output it to the console here, but you could also save it programmatically or retrieve it later using the swift API. Putting it all together and running the app, you get:

The map is saved and uploaded to the cloud for access on any phone running with your API Key (more on this in a future blog post!).

Now, Step 3, to retrieve the map everytime you turn on the app, lets comment out everything under the UIButton action and make a call to loadMap. Once the loadMap function returns successfully, you can again, start a session. This time, as you call on startSession after loading a map, the Placenote SDK will try to localize the old ARKit session for you,

if (!shapesRendering) {  
  shapesRendering = true   
  LibPlacenote.instance.loadMap(mapId: "<Map ID Here>",
    downloadProgressCb: {(completed: Bool, faulted: Bool, percentage: Float) -> Void in     
      if (completed) {       
        LibPlacenote.instance.startSession()  
      }  
    }) 
}

For now, I just copy paste the Map ID as it was output to the console when we saved the map. However, as mentioned before, the Map ID can be saved programmatically, associated with a user, a GPS point, whatever you fancy! Note that the code to send image frames to the Placenote SDK stays the same!

func session(_ session: ARSession, didUpdate: ARFrame) {  
  let image: CVPixelBuffer = didUpdate.capturedImage  
  let pose: matrix_float4x4 = didUpdate.camera.transform  
  if (shapesRendering) {  
    LibPlacenote.instance.setFrame(image: image, pose: pose)  
  } 
}

Now, remember when you set up a delegate with a function called onStatusChange? The moment PlacenoteSDK finds the map you created before, the status will change from ‘lost’ to ‘running’. When this happens, you can render the sphere exactly at (0,0,1) like so,

func onStatusChange(_ prevStatus: LibPlacenote.MappingStatus, _ currStatus: LibPlacenote.MappingStatus) {
  if (prevStatus == LibPlacenote.MappingStatus.lost && currStatus ==LibPlacenote.MappingStatus.running) {  
    renderSphere() 
    sessionInfoLabel.text = "Map Found!"  
  } 
}

Lets run this app in place now! No matter where I start, when I come back to my globe, I find my sphere right next to it in the same spot!

Try it yourself..

If you want to see this exact code in action, its available on my Github on a branch (remember to update PlacenoteSDK-Bridging-Header.h with your own API-key!).

However, here at Vertical we’ve done a couple of more things to furnish the idea of rendering basic shapes. We let the user add random shapes by tapping on their screens using UITapGestureRecognizer and saving the shapes as a JSON file using swift. By naming this JSON file after the MapID and showing the list of these maps using UITableView, users can easily create and retrieve random shapes they put around them! If you want to try the full app it can be found on Github! Just follow the instructions and start experimenting with Persistence in ARKit :).