Develop your HelloAR app in Android studio using ARCore and Sceneform

Codemaker
Codemaker
May 16 · 8 min read
Image credits to proandroiddev.com

What is Augmented Reality

Augmented Reality is “a technology that superimposes a computer-generated image on a user’s view of the real world, thus providing a composite view”. Essentially, AR is a technology which enables us to render computer generated 3D object models into the real world and have it interact with its surrounding as if it were physically present at the same location.

What is ARCore

ARCore is a platform for building Android AR experiences. It enables your phone to sense its environment, understand the world and interact with the information.

ARCore works on 3 principles:

  • Motion Tracking: It allows the phone to understand its current position relative to the real world.
  • Understanding the Environment: It allows the phone to detect the size and location of all type of surfaces: vertical, horizontal and angled.
  • Light Estimation: It allows the phone to sense the environment’s lighting condition.

Sceneform

ARCore in itself isn’t an SDK, rather it is an engine that helps SDKs to render the objects. Hence, in order to make use of this functionality, Google released Sceneform SDK to enable developers to build Android AR apps without having to learn OpenGL.

Sceneform comes with many nifty features such as:

  • An automatic compatibility check for ARCore enabled phones.
  • Checking for camera permissions.
  • A scene graph API to abstract all the complexities.
  • A plugin for manipulating 3D assets.

Getting Started

you first need to enable ARCore in your project. This is simple as we will be using Android Studio and Sceneform SDK. There are two major operations Sceneform performs automatically:

  • Checking for availability of ARCore
  • Asking for camera permission

Create a new Android Studio project and select an empty activity.

Adding Dependencies

Add the following dependency to your project level build.gradle file:

dependencies { classpath 'com.google.ar.sceneform:plugin:1.10.0' }

Add the latest ARCore library as a dependency in your app’s build.gradle file:

dependencies {
...
implementation 'com.google.ar:core:1.10.0'
implementation "com.google.ar.sceneform.ux:sceneform-ux:1.10.0"
}

Sceneform SDK requires minSdkVersion greater than or equal to 24. So, make sure that you set the minSdkVersion>= 24. Also, make sure that you have included the Maven repository in your project level build.gradle.

Updating Manifest

Add the following lines in your AndroidManifest.xml file:

<uses-permission android:name="android.permission.CAMERA" />
<uses-permission android:name="android.permission.INTERNET"/>
<uses-feature android:name="android.hardware.camera.ar" android:required="true"/>

Using ARCore requires camera permission and a camera enabled phone. Also, add a meta data to your application tag:

<meta-data
android:name="com.google.ar.core"
android:value="required" />

If your app strictly requires the device to be ARCore enabled, then set required = true or if AR is not a primary feature or you have handled compatibility for non-compatible devices you can set required = false.

Then sync Your project with Gradle files and wait for the build to finish. This will install the Sceneform SDK to the project and Sceneform plugin to AndroidStudio. It will help you to view the .sfb files. These files are the 3D models which are rendered in your camera. It also helps you in importing, viewing, and building 3D assets.

Presently with our Android Studio sync is complete and Sceneform SDK Installed, we can begin with creating our absolute first ARCore application.

In the first place, we have to add the Sceneform piece to our design record. This will be where we place all our 3D models. It deals with the camera instatement and authorization taking care of.

Head over to your fundamental design document. For my situation it is activity_main.xml and include the Sceneform part:

<?xml version="1.0" encoding="utf-8"?>
<LinearLayout xmlns:android="http://schemas.android.com/apk/res/android"
xmlns:tools="http://schemas.android.com/tools"
android:layout_width="match_parent"
android:layout_height="match_parent"
android:orientation="vertical"
tools:context=".MainActivity">
<fragment
android:id="@+id/ux_fragment"
android:name="com.google.ar.sceneform.ux.ArFragment"
android:layout_width="match_parent"
android:layout_height="0dp"
android:layout_weight="9" />
<LinearLayout
android:id="@+id/gallery_layout"
android:layout_width="match_parent"
android:layout_height="0dp"
android:layout_weight="1"
android:orientation="horizontal" />
</LinearLayout>

This is all that you need to do in the layout file.

Checking compatibility at runtime

We will check if the device:

  1. Is running Android API version >= 24.
  2. Can support OpenGL version 3.0.

The above conditions are mandatory for a device to support AR applications using ARCore and Sceneform SDK.

We intend to finish the activity if these conditions aren’t satisfied. However, you can still continue to support other features. Add the method below in your class That will help with the Compactibility Check:

private boolean checkIsSupportedDeviceOrFinish(final Activity activity) {
if (Build.VERSION.SDK_INT < Build.VERSION_CODES.N) {
Log.e(TAG, "Sceneform requires Android N or later");
Toast.makeText(activity, "Sceneform requires Android N or later", Toast.LENGTH_LONG).show();
activity.finish();
return false;
}
String openGlVersionString =
((ActivityManager) activity.getSystemService(Context.ACTIVITY_SERVICE))
.getDeviceConfigurationInfo()
.getGlEsVersion();
if (Double.parseDouble(openGlVersionString) < MIN_OPENGL_VERSION) {
Log.e(TAG, "Sceneform requires OpenGL ES 3.0 later");
Toast.makeText(activity, "Sceneform requires OpenGL ES 3.0 or later", Toast.LENGTH_LONG)
.show();
activity.finish();
return false;
}
return true;
}

Adding 3D models to our application

It is now time to download and import the 3D models to be rendered into our application. In our case, we will be rendering a 3D cube in a corner of our room and moving it around.

You can download 3D models from anywhere, but Google has provided an excellent repository POLY to download 3D models for your application. You can download the models in .fbx, .obj or .gltf format. We will be downloading the .fbx file.

Open the project view in your android studio project and expand the app folder. You will notice a folder named “sampledata”. If not, go ahead to create one.

After your model finishes downloading, you will need to extract the downloaded zip file into this sample data folder.

You will find a .fbx file and a png image of the model. We’ll import the .fbx file in our application using the sceneform plugin.

Importing the model using Sceneform plugin

You need to set the ar plugin in your app’s Gradle file as well.
Add the following below the dependencies:

apply plugin: 'com.google.ar.sceneform.plugin'

Add the following lines at the end of the app’s build.gradle:

sceneform.asset('sampledata/cube.fbx',         
'default',
'sampledata/cube.sfa',
'src/main/assets/cube')

The build.gradle of the app looks like this finally:

apply plugin: 'com.android.application'
apply plugin: 'com.google.ar.sceneform.plugin'

android {
compileSdkVersion 28
defaultConfig {
applicationId "com.example.arcorefirst"
minSdkVersion 24
targetSdkVersion 28
versionCode 1
versionName "1.0"
testInstrumentationRunner "android.support.test.runner.AndroidJUnitRunner"
}
buildTypes {
release {
minifyEnabled false
proguardFiles getDefaultProguardFile(’proguard-android.txt’), 'proguard-rules.pro'
}
}
compileOptions {
sourceCompatibility = '1.8'
targetCompatibility = '1.8'
}
}

dependencies {
implementation fileTree(dir: 'libs’, include: [’*.jar’])
implementation 'com.android.support:appcompat-v7:28.0.0'
implementation 'com.android.support.constraint:constraint-layout:1.1.3'
testImplementation 'junit:junit:4.12'
androidTestImplementation 'com.android.support.test:runner:1.0.2'
androidTestImplementation 'com.android.support.test.espresso:espresso-core:3.0.2'
implementation 'com.google.ar.sceneform.ux:sceneform-ux:1.10.0'
}

sceneform.asset(’sampledata/cube.fbx’,
'default’,
'sampledata/cube.sfa’,
'src/main/res/raw/cube’)

Android ARCore Sceneform requires Java 8 or higher version.

The sja and sjb are the Sceneform Asset Description and Sceneform Binary files. The sjb file is visible in the 3D viewer. It is shipped with the APK. and sja file is used to set properties for the sjb file.

Integrating the Model

Add the following code in your java file:

@Override
@SuppressWarnings({"AndroidApiChecker", "FutureReturnValueIgnored"})
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);

if (!checkIsSupportedDeviceOrFinish(this))
return;

setContentView(R.layout.activity_main);

arFragment = (ArFragment) getSupportFragmentManager().findFragmentById(R.id.ux_fragment);
arFragment.setOnTapArPlaneListener(
(HitResult hitresult, Plane plane, MotionEvent motionevent) -> {
if (plane.getType() != Plane.Type.HORIZONTAL_UPWARD_FACING)
return;

Anchor anchor = hitresult.createAnchor();
placeObject(arFragment, anchor, R.raw.cube);
}
);
}

Adding the Model to the AR Scene

Our AR fragment is the container of the scene and hence we need to add a model to the fragment whenever it is clicked. Hence, we’ll add an onTapListener to our fragment.

private void placeObject(ArFragment arFragment, Anchor anchor, int uri) {
ModelRenderable.builder()
.setSource(arFragment.getContext(), uri)
.build()
.thenAccept(modelRenderable -> addNodeToScene(arFragment, anchor, modelRenderable))
.exceptionally(throwable -> {
Toast.makeText(arFragment.getContext(), "Error:" + throwable.getMessage(), Toast.LENGTH_LONG).show();
return null;
}
);
}

private void addNodeToScene(ArFragment arFragment, Anchor anchor, Renderable renderable) {
AnchorNode anchorNode = new AnchorNode(anchor);
TransformableNode node = new TransformableNode(arFragment.getTransformationSystem());
node.setRenderable(renderable);
node.setParent(anchorNode);
arFragment.getArSceneView().getScene().addChild(anchorNode);
node.select();
}

Using the hitResult, we can get the location tapped and create an anchor node which is the root node of our scene (image an augmented reality scene as an inverted tree).

Next, we create a TransformableNode which will be our chair and set it to the anchornode. A transformable node can react to location changes and size changes when the user drags the object or uses pinch to zoom.

Let’s have a look at some terminologies here:

  • Scene: It’s the place where our 3D world will be rendered.
  • HitResult: It is an imaginary ray of light coming from infinity and it’s first point of intersection with the real world is the point of tap.
  • Anchor: A fixed location in the real world. Used to transform local coordinates (according to user’s display) to the real-world coordinates.
  • TransformableNode: A node that can react to user’s interactions such as rotation, zoom and drag.

Here’s how your final java file would look like:

package com.example.arcorefirst;

import android.app.Activity;
import android.app.ActivityManager;
import android.content.Context;
import android.net.Uri;
import android.os.Build;
import android.os.Bundle;
import android.support.v7.app.AppCompatActivity;
import android.util.Log;
import android.view.MotionEvent;
import android.widget.Toast;

import com.google.ar.core.Anchor;
import com.google.ar.core.HitResult;
import com.google.ar.core.Plane;
import com.google.ar.sceneform.AnchorNode;
import com.google.ar.sceneform.rendering.ModelRenderable;
import com.google.ar.sceneform.rendering.Renderable;
import com.google.ar.sceneform.ux.ArFragment;
import com.google.ar.sceneform.ux.TransformableNode;

public class MainActivity extends AppCompatActivity {
private static final String TAG = MainActivity.class.getSimpleName();
private static final double MIN_OPENGL_VERSION = 3.0;

ArFragment arFragment;
ModelRenderable lampPostRenderable;
private Uri selectedObject;

@Override
@SuppressWarnings({"AndroidApiChecker", "FutureReturnValueIgnored"})
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);

if (!checkIsSupportedDeviceOrFinish(this))
return;

setContentView(R.layout.activity_main);

arFragment = (ArFragment) getSupportFragmentManager().findFragmentById(R.id.ux_fragment);
arFragment.setOnTapArPlaneListener(
(HitResult hitresult, Plane plane, MotionEvent motionevent) -> {
if (plane.getType() != Plane.Type.HORIZONTAL_UPWARD_FACING)
return;

Anchor anchor = hitresult.createAnchor();
placeObject(arFragment, anchor, R.raw.cube);
}
);
}

private void placeObject(ArFragment arFragment, Anchor anchor, int uri) {
ModelRenderable.builder()
.setSource(arFragment.getContext(), uri)
.build()
.thenAccept(modelRenderable -> addNodeToScene(arFragment, anchor, modelRenderable))
.exceptionally(throwable -> {
Toast.makeText(arFragment.getContext(), "Error:" + throwable.getMessage(), Toast.LENGTH_LONG).show();
return null;
}
);
}

private void addNodeToScene(ArFragment arFragment, Anchor anchor, Renderable renderable) {
AnchorNode anchorNode = new AnchorNode(anchor);
TransformableNode node = new TransformableNode(arFragment.getTransformationSystem());
node.setRenderable(renderable);
node.setParent(anchorNode);
arFragment.getArSceneView().getScene().addChild(anchorNode);
node.select();
}

private boolean checkIsSupportedDeviceOrFinish(final Activity activity) {
if (Build.VERSION.SDK_INT < Build.VERSION_CODES.N) {
Log.e(TAG, "Sceneform requires Android N or later");
Toast.makeText(activity, "Sceneform requires Android N or later", Toast.LENGTH_LONG).show();
activity.finish();
return false;
}
String openGlVersionString =
((ActivityManager) activity.getSystemService(Context.ACTIVITY_SERVICE))
.getDeviceConfigurationInfo()
.getGlEsVersion();
if (Double.parseDouble(openGlVersionString) < MIN_OPENGL_VERSION) {
Log.e(TAG, "Sceneform requires OpenGL ES 3.0 later");
Toast.makeText(activity, "Sceneform requires OpenGL ES 3.0 or later", Toast.LENGTH_LONG)
.show();
activity.finish();
return false;
}
return true;
}
}

Let’s see what’s happening here.

  1. First, we get the fragment we added in our layout file with the help of supportFragmentManager and the fragment id.
  2. Then we need to load the model into the scene. For this, we use the ModelRenderable class provided by the Sceneform SDK. With the help of ModelRenderable’s setSource() method, we can load our model by passing in the name of the generated .sfb file.
  3. Model is being built on a background thread, so after the model is loaded, it’s presented to the main thread which then renders it to the scene.
  4. We receive the model inside the thenAccept method. If there’s any error in building the model, an exception is thrown.

There you have it! Your own AR App in Android studio :)

I hope you had fun reading and/or following along. In the next story, we will look into how to build more features and interactions into this App. Stay tuned!

If you are interested in further exploring, here are some resources I found helpful along the way:

Geek Culture

Proud to geek out. Follow to join our +500K monthly readers.

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store