AR technology for Android — Part 4: AR Cloud Anchors

Imagine the next scenario: you are with your friends…you are an Android user…your friends are IOS users…using your phone you add virtual objects to an AR scene…your friends can view and interact with these objects in real-time…Whoaa! What kind of witchcraft is that? Let me explain!

As I mentioned in the previous part of this article, the great advantage of ARCore is given by Cloud Anchors. I want to share with you my experience with ArCore Cloud Anchors. Step-by-step we will create an Android app using this innovative feature.

Anchors, anchors, anchors…What is an anchor in ARCore?

An anchor describes a fixed position in the real world. The value of an anchor’s pose (pose = position, and orientation) is automatically adjusted by ARCore as its motion tracking improves over time. An anchor can be created at a Trackable Position and Orientation in ARCore. Cloud Anchors are anchors that are hosted in the cloud and can be resolved by multiple users to establish a common frame of reference across users and their devices.

Let’s “throw the anchor” together!

I think the best way for a programmer to learn is to write code. Using Kotlin programming language, we will step-by-step create an Android app that will place in the real space the same virtual wolf from the previous part of the article. We will see how we can host and resolve the anchor created by posting the wolf in our app and, among the steps, we will learn together the main concepts about AR Cloud Anchors.

Requirements:

• A supported ARCore device, connected via a USB cable to your development machine (and also connected to the Internet via WiFi).
• ARCore 1.8 or later.
• A development machine with Android Studio (v3.0 or later).

To resolve a Cloud Anchor from the app, the device will need a Cloud ID. The Cloud ID is usually very long and this is the reason why we should store it as a short code in the Firebase Realtime Database. The Firebase Realtime Database is a cloud-hosted database. Data is stored as JSON and synchronized in real-time to every connected client. When you build cross-platform apps with iOS, Android and JavaScript SDKs, all of your clients share one Realtime Database instance and automatically receive updates with the newest data.

If you read the third part of this article, you are familiar with ARCore and you can integrate it into your own app easier. Follow the next steps to share the AR experience with others using Cloud Anchors.

Step 1. Open Android Studio IDE and create a project.

Step 2. Open AndroidManifest.xml and add camera and internet permissions and ARCore metadata.

<uses-permission android:name="android.permission.CAMERA"/>
<uses-permission android:name=“android.permission.INTERNET"/>
<application
…
<meta-data
android:name="com.google.ar.core"
android:value=“optional"/>
</application>

Step 3. Go to Google Cloud Console, create a new project and generate an API KEY by tapping the “Create credentials” button. You will find more details about API Keys here. Add the code bellow inside AndroidManifest and paste your API KEY to the metadata value.

<meta-data
android:name="com.google.android.ar.API_KEY"
android:value="Paste your API Key here" />

Step 4. It’s time to connect your app with Firebase. I suggest you create a connection using Firebase Assistant. Go to Tools, click on Firebase and select Realtime Database. Follow the steps there and you will connect your app to Firebase very easy.

When the connection is done, go to Firebase Console, choose your recently created project, select Database from the left side of the screen, select Rules tab and configure your write and read rules for public access.

Step 5. Configure app only for each module that uses Java 8 language feature (either in its source code or through dependencies) and add ARCore and Sceneform dependency in the build.gradle file (Module.app).

android {
…
compileOptions {
sourceCompatibility JavaVersion.VERSION_1_8
targetCompatibility JavaVersion.VERSION_1_8
}
}
dependencies {
implementation 'com.google.ar:core:1.9.0'
implementation "com.google.ar.sceneform.ux:sceneform-ux:1.9.0"
}

Step 6. Open activity_main.xml and replace the TextView with a Button. When this button is tapped, the camera permission is requested. If permission will be enabled the activity responsible for loading the AR Experience will be started.

Note: We will request camera permission using the easypermissions library.

companion object {
const val CAMERA_REQUEST = 201
}
private fun onButtonClicked() {
val perms = arrayOf(Manifest.permission.CAMERA)
if (!EasyPermissions.hasPermissions(this, *perms)) {
EasyPermissions.requestPermissions(
this,
getString(R.string.camera_permission_rationale),
CAMERA_REQUEST,
*perms
)
return
}
startSimpleArActivity()
}
@AfterPermissionGranted(CAMERA_REQUEST)
private fun startSimpleArActivity() {
startActivity(Intent(this, ArCoreActivity::class.java))   
 }
override fun onRequestPermissionsResult(requestCode: Int, permissions: Array<String>, grantResults: IntArray) {
EasyPermissions.onRequestPermissionsResult(requestCode, permissions, grantResults, this)
}

Step 7. Create a new activity ArCoreActivity responsible for rendering the virtual content over the real world images captured with the device camera. Do not forget to declare the new activity in AndroidManifest.xml.

Step 8. Before building and adding the model to the scene, we need to add the asset into our project. Go to https://free3d.com/3d-model/wolf-rigged-and-game-ready-42808.html and download the free wolf asset. Unzip the archive and copy the .obj and .mtl files from the OBJ folder. Paste the files inside the sampledata->wolf. If you don’t have a sample data directory, create it: right-click on app directory -> New -> Sample Data Directory.

Step 9. Import the asset: right click on .obj, choose the first option Import Sceneform Asset and click finish.

Note: If you don’t see this option, install Google Sceneform Tools(Beta) plugin:

• Windows: File-> Settings-> Plugins->Browse Repositories
• Mac OS : Android Studio -> Preferences -> Plugins

A Wolf_One_obj.sfb file is generated inside assets directory and your module build.gradle file contains a sceneform.asset() line of code.

• data/wolf/Wolf_One_obj.obj is the source asset path specified during import;
• default is model specified during import;
• sampledata/wolf/Wolf_One_obj.sfa is the .sfa output path specified during import;
• src/main/assets/Wolf_One_obj is the .sfb output path specified during import.

sceneform.asset('sampledata/wolf/Wolf_One_obj.obj',
        'default',
        'sampledata/wolf/Wolf_One_obj.sfa',
        'src/main/assets/Wolf_One_obj')

Step 10. Create a class ArCoreFragment.kt that extends ArFragment. Override getSessionConfiguration() method and add the following lines of code in order to create a configuration for ArFragment.

• Set the instruction view of planeDiscoveryController to null as a step to disable the initial hand gesture.

planeDiscoveryController.setInstructionView(null)

• Create an ARCore configuration

val config = super.getSessionConfiguration(session)

• Enable Cloud Anchor mode and return ArFragment configuration

config.cloudAnchorMode = Config.CloudAnchorMode.ENABLED
return config

Step 11. Add the custom ArFragment that you recently created inside activity_main.layout.

<fragment
android:layout_width="match_parent"
android:layout_height="match_parent"
app:layout_constraintStart_toStartOf="parent"
app:layout_constraintEnd_toEndOf="parent"
app:layout_constraintTop_toTopOf="parent"
app:layout_constraintBottom_toBottomOf="parent"
android:id="@+id/sceneform_fragment"
android:name=“com.example.cloudanchorsardemo.ArCoreFragment"/>

Step 12. Inside the same layout, add two buttons, one is to clear the anchor and another is to resolve a hosted anchor by id. The result should look like this:

Step 13. Build the model using ModelRenderable class which works asynchronously. First of all, we need to create the ModelRenderable by setting the source that will load the model. thenAccept() method is called if the model is successfully loaded and returns the built model. As its name says, the exceptionally() method is called if the model cannot be created.

private fun placeWolf(fragment: ArFragment, anchor: Anchor?, model: Uri) {
ModelRenderable.builder()
.setSource(fragment.context!!, model)
.build()
.thenAccept { renderable -> addNodeToScene(fragment, anchor, renderable) }
.exceptionally { throwable ->
val builder = android.app.AlertDialog.Builder(this)
builder.setMessage(throwable.message)
.setTitle("Error!")
val dialog = builder.create()
dialog.show()
null
}
}

Step 14. Create a function responsible for adding the node to the scene. It creates an AnchorNode on the Anchor and a TransformableNode with the parent as AnchorNode. All these concepts are explained in the previous part (step 11).

private fun addNodeToScene(fragment: ArFragment, anchor: Anchor?, renderable: Renderable) {
val anchorNode = AnchorNode(anchor)
val node = TransformableNode(fragment.transformationSystem)
node.renderable = renderable
node.setParent(anchorNode)
fragment.arSceneView.scene.addChild(anchorNode)
node.select()
}

Step 15. Create a global enum AppAnchorState that contains the status of the cloudAnchor.

private enum AppAnchorState {
NONE,
HOSTING,
HOSTED,
RESOLVING,
RESOLVED
}

Step 16. Declare two variables, an ArCoreFragment, and an Anchor, and create a method responsible for setting the Cloud Anchor.

private var arCoreFragment: ArCoreFragment? = null
private var cloudAnchor: Anchor? = null
private fun setCloudAnchor(newAnchor: Anchor?) {
if (cloudAnchor != null) {
cloudAnchor?.detach()
     }
cloudAnchor = newAnchor
appAnchorState = AppAnchorState.NONE
}

Step 17. Create ResolveDialog class with corresponding layout resolve_dialog.xml and add the code below. This class will display a dialog where the user will type in the anchor id. See code on Github.

Step 18. Add the following methods inside ArCoreActivity.kt class.

private fun initListeners() {
clear_button.setOnClickListener { setCloudAnchor(null)}
     resolve_button.setOnClickListener(View.OnClickListener {
ResolveDialog(
this,
object : ResolveDialog.PositiveButtonListener {
override fun onPositiveButtonClicked(dialogValue: String) {
resolveAnchor(dialogValue)
}
}).show()        
    })
arCoreFragment?.setOnTapArPlaneListener { hitResult: HitResult, plane: Plane, _: MotionEvent ->
if (plane.type != Plane.Type.HORIZONTAL_UPWARD_FACING ||     appAnchorState != AppAnchorState.NONE) {
}
val newAnchor =   arCoreFragment?.arSceneView?.session?.hostCloudAnchor(hitResult.createAnchor())
setCloudAnchor(newAnchor)
appAnchorState = AppAnchorState.HOSTING
Toast.makeText(this, "Now hosting anchor...", Toast.LENGTH_LONG).show()
arCoreFragment?.let { placeWolf(it, cloudAnchor,  Uri.parse("Wolf_One_obj.sfb")) }
}
}

This method is responsible for settings listeners on clear and resolve buttons. It also sets the TapArPlaneListener on arCoreFragment and hides the plane discovery controller to disable the hand gesture (optional). When the clear button is tapped, we set the cloudAnchor to null. When an upward facing plane is tapped, we create an Anchor and pass it to setCloudAnchor function. Then we call placeObject() on the cloudAnchor, placing our 3D wolf at the tapped point.

fun showMessage(message: String) {
Toast.makeText(this, message, Toast.LENGTH_LONG).show()
}

As the name says, showMessage() method displays some messages to the users using Toast.

private fun setCloudAnchor(newAnchor: Anchor?) {
if (cloudAnchor != null) {
cloudAnchor?.detach()
}
cloudAnchor = newAnchor
appAnchorState = AppAnchorState.NONE
}

This method ensures there is only one cloud anchor at any point in time.

@Synchronized
private fun updateAnchorIfNecessary() {
if (appAnchorState != AppAnchorState.HOSTING && appAnchorState != AppAnchorState.RESOLVING) {
return
}
val cloudState = cloudAnchor?.cloudAnchorState
cloudState?.let {it->
if (appAnchorState == AppAnchorState.HOSTING) {
if (it.isError) {
Toast.makeText(this, "Error hosting anchor.. $it", Toast.LENGTH_LONG).show()
appAnchorState = AppAnchorState.NONE
} else if (it == Anchor.CloudAnchorState.SUCCESS) {
firebaseDatabaseManager?.nextShortCode(object :    FirebaseDatabaseManager.ShortCodeListener {
override fun onShortCodeAvailable(shortCode: Int?) {
if (shortCode == null) {
showMessage("Could not get shortCode")
return
}
cloudAnchor?.let {
firebaseDatabaseManager?.storeUsingShortCode(
shortCode,
it.cloudAnchorId)
}
showMessage("Anchor hosted! Cloud Short Code: $shortCode")
}
})
appAnchorState = AppAnchorState.HOSTED
}
} else if (appAnchorState == AppAnchorState.RESOLVING) {
if (it.isError) {
Toast.makeText(this, "Error hosting anchor.. $it", Toast.LENGTH_LONG).show()
appAnchorState = AppAnchorState.NONE
} else if (it == Anchor.CloudAnchorState.SUCCESS) {
Toast.makeText(this, "Anchor resolved successfully", Toast.LENGTH_LONG).show()
appAnchorState = AppAnchorState.RESOLVED
}
}
}
}

This method checks the anchor state and updates it if necessary.

Step 19. Paste below code inside onCreate() method.

override fun onCreate(savedInstanceState: Bundle?) {
super.onCreate(savedInstanceState)
setContentView(R.layout.activity_arcore)
arCoreFragment =  supportFragmentManager.findFragmentById(R.id.sceneform_fragment) as ArCoreFragment?
arCoreFragment?.planeDiscoveryController?.hide()
arCoreFragment?.arSceneView?.scene?.addOnUpdateListener {
Scene.OnUpdateListener { p0 ->  updateAnchorIfNecessary() }
}
arCoreFragment?.arSceneView?.scene?.addOnUpdateListener { p0 ->    updateAnchorIfNecessary() }
firebaseDatabaseManager = FirebaseDatabaseManager(this)
    initListeners()
}

Step 20. Build your project and run it on your hardware connected device.

The full code is available on Github.

That’s all! Do not hesitate to let a comment below with your feedback. Enjoy coding!

https://www.zipperstudios.co

Zipper Studios is a group of passionate engineers helping startups and well-established companies build their mobile products. Our clients are leaders in the fields of health and fitness, AI, and Machine Learning. We love to talk to likeminded people who want to innovate in the world of mobile so drop us a line here.