Sceneform — Introduction to AR

Recently, I tried ARCore using Unity and really loved it. But being an Android developer, it seemed restrictive. I wanted to add it more UI components to interact with ARCore and make it more wholesome. Enter Sceneform.

ARCore with Unity — Beginner on Android

Sceneform allows you to access ARCore using Java and by extension use Android component in conjunction with it. For example, show a model, tap on a button to play a sound.

So let us being. To learn, we will implement an application along the way. This would allow us to learn it properly and hopefully answer queries. Also note that we need to set the minimum API version as 24.

First, we need to add Sceneform to gradle. As of writing this post, the recent available version was 1.5.1.

implementation 'com.google.ar.sceneform.ux:sceneform-ux:1.5.1'

We will also add compile options for Java 8 support.

compileOptions {
    sourceCompatibility JavaVersion.VERSION_1_8
    targetCompatibility JavaVersion.VERSION_1_8
}

Next we need to add the permissions for Camera and also specify that for this application the device needs to support AR functionality. So we make the following changes in AndroidManifest.xml

<uses-permission android:name="android.permission.CAMERA" />
<uses-feature android:name="android.hardware.camera.ar" android:required="true" />

Inside application definition we need to add the meta data information in manifest too.

<meta-data android:name="com.google.ar.core" android:value="required" />

Now we need some 3D models which we can render. Now bear in mind that the models are required to be light (low polygon count) for faster rendering. Once such website from wherein you can get 3D models is:

Poly

From here you can download OBJ or GLTX models which we will use in our application development.

For the purpose of development, we can use the model of Pluto.

Pluto

Click on the download button and it should download a zip file which contains the model (OBJ), material (MTL)and texture (JPG). It is not mandatory to know what are all these, but knowing a bit about them will be interesting. The OBJ can be though of as a wireframe model. It contains all the polygon data which creates the model. The MTL contains the material definitions like colors for example which will be applied on the model. It also contains information about textures to be used. Finally, textures are the potion of images which will be defined in the material file and applied to the model. You can think MTL as the map file between them.

Next, we need a tool which would create a structural information from these models and other files. For this, we will use a plugin called Google Sceneform Tools. This plugin is currently in Beta phase, but stable for development purpose. Download and install the plugin from Android Studio.

Lets start by creating an Android project. Nothing special here, just the same steps that we follow. Before we begin writing any code, one very important change that we need to do is, instead of extending from Fragment, we need to extend from ArFragment.

public class ArPuppyFragment extends ArFragment

This simple line does all the heavy lifting work for us. It provides us with the view on which we can continue with our development. In the layout file, we can simply use the fragment as

<fragment
    android:id="@+id/sceneform_fragment"
    android:name="com.soumya.arcore.ArPuppyFragment"
    android:layout_width="match_parent"
    android:layout_height="match_parent"
    android:layout_alignParentTop="true"
    />

This looks good till here. Let us run the application and see what happens. We will observe that we are required to provide access to Camera and hence the permission dialog is displayed. Once approved, the camera is launched. You will be displayed a tutorial screen on how to detect planes.

Hand tutorial screen (can be hidden)

Try moving the left and right on horizontal surface and see what happens? White dots will be coming up. Congratulations!!! You have detected a plane.

Dev Note:
If you want to remove the tutorial screen then you can provide the following lines of codes.
arFragment.getPlaneDiscoveryController().hide();
arFragment.getPlaneDiscoveryController().setInstructionView(null);

So before we continue, we need to understand what is plane. In real world, if we need to place an object, say for example your phone, you need to put it on a plane, for example the table. Similarly to put a virtual model into the real world, we need a plane. Once a plane is detected, we can add the models to that particular plane.

Models added to the plane will be anchored to it. Anchors are similar to ship’s anchor. They are used to define the position compared to the real world where the virtual object is being displayed.

Now with the knowledge of the basic understanding, first, we need to add the 3D model. For this, create a folder named sampledata inside the app level folder.

Once the folder is created, you can create a subfolder inside it and call it pluto. Inside this folder, we need to copy the OBJ, MTL and texture files.

Now, we need to generate the sceneform assets. For this we will be taking the help of the plugin we had downloaded earlier. For this, right-click on the OBJ file and select Import Sceneform Asset. It would create two new files with the extension sfa and sfb. These two files will be used to render the model.

If you click on the SFB file inside the assets folder, you will be shown a preview of the model which has been imported.

Till will take some time to generate the model files once you click finish.

Perfect! Now we need to drop the model once a plane is detected. Hence, the first step is plane detection. From the fragment we need access to Scene. This object contains all the nodes and its information. Node contain the renderable (3D model) and positional data. It is a graph.

puppyFragment = (ArPuppyFragment) getSupportFragmentManager()
.findFragmentById(R.id.sceneform_fragment);
puppyFragment.getArSceneView().getScene()
.addOnUpdateListener(updateListener);

Here is the listener implementation:

private Scene.OnUpdateListener updateListener = frameTime -> {

    Frame frame = puppyFragment.getArSceneView().getArFrame();
    if(frame == null) {
        return;
    }

    for(Plane plane : frame.getUpdatedTrackables(Plane.class)) {
        addObjectModel(Uri.parse("pluto.sfb"));
        break;
    }
};

onUpdate will be called called whenever the content of the scene is being updated. In such an instance, get can get all the detected planes corresponding to the frame. A frame can have multiple planes. We can iterate through all the planes and based on the plane which we need, we can work on it. For our case, let us consider that we add the model as soon as we detect a plane.

private void addObjectModel(Uri object) {
    Frame frame = arFragment.getArSceneView().getArFrame();
    Point center = getScreenCenter(arFragment);

    puppyFragment.getArSceneView().getScene()
            .removeOnUpdateListener(updateListener);

    if(frame != null) {
        List<HitResult> result = frame.hitTest(center.x, center.y);
        for(HitResult hit : result) {
            Trackable trackable = hit.getTrackable();
            if (trackable instanceof Plane && ((Plane) trackable).isPoseInPolygon(hit.getHitPose())) {
                placeObject(arFragment, hit.createAnchor(), object);
                break;
            }
        }
    }
}

The way to get the center of the screen is:

private Point getScreenCenter() {

    if(puppyFragment == null || puppyFragment.getView() == null) {
        return new android.graphics.Point(0,0);
    }

    int w = puppyFragment.getView().getWidth()/2;
    int h = puppyFragment.getView().getHeight()/2;
    return new android.graphics.Point(w, h);
}

The above method is going to add the model to the plane. To this, we need to pass the ArFragment and the Uri of the model, for example:

Uri.parse("Pluto.sfb")

Before placing the object, we need to perform a hit test. Think of this process as sending laser lights to the plane on a coordinate with respect to the device screen and figuring out as to whether a viable plane is present on which the model can be rendered. This would give us a proper HitResult which can be used to create an anchor.

private void placeObject(Anchor anchor, Uri object) {
    ModelRenderable.builder()
        .setSource(puppyFragment.getContext(), object)
        .build()
        .thenAccept(modelRenderable -> addNodeToScene(anchor, modelRenderable, object))
        .exceptionally(throwable -> null);
}

This method will load the 3D model and a renderable form which will be placed in the defined anchor. Anchors in ARCore does that same thing it does to a ship. It fixes the object from drifting away. Hence in AR too, once the object has been placed into the real world, it will stay in that location until and unless it has been moved by user or programatically.

ModelRenderable provides an async implementation for the load process. Once the loading has been done we can proceed to add the node to the scene.

private void addNodeToScene(Anchor createAnchor, ModelRenderable renderable, Uri object) {
    AnchorNode anchorNode = new AnchorNode(createAnchor);
    TransformableNode transformableNode = new TransformableNode(puppyFragment.getTransformationSystem());
    transformableNode.setName(object.toString());
    transformableNode.setRenderable(renderable);
    transformableNode.setParent(anchorNode);

    puppyFragment.getArSceneView().getScene().addChild(anchorNode);

    transformableNode.setOnTapListener((hitTestResult, motionEvent) -> {
        //Perform callback action, like bark
    });
    transformableNode.select();
}

Finally we come to the node wherein the model which has been loaded will be added to the scene. Like we have mentioned earlier that we need nodes to connect and form the graph, we will create an anchor node form the anchor. Next we create another node which would contain the renderable information. In this instance we are using TransformableNode. This transformable node would contain the renderable and the anchor node. Once complete, we just need to add it to the scene.

Pluto on my desk

There we have it. Pluto’s 3D model has been added to the real world.

So now that we have Pluto being displayed, let us go ahead, add the button and when you tap on the button make him bark. his purpose of this task is to show the easy integration with other Android components.

This is how the layout would look like.

<RelativeLayout
    xmlns:android="http://schemas.android.com/apk/res/android"
    xmlns:app="http://schemas.android.com/apk/res-auto"
    xmlns:tools="http://schemas.android.com/tools"
    android:layout_width="match_parent"
    android:layout_height="match_parent"
    tools:context=".MainActivity">

    <fragment
        android:id="@+id/sceneform_fragment"
        android:name="com.soumya.arcore.ArPuppyFragment"
        android:layout_width="match_parent"
        android:layout_height="match_parent"
        android:layout_alignParentTop="true"
        />

    <Button
        android:layout_width="match_parent"
        android:layout_height="wrap_content"
        android:id="@+id/bark_pluto"
        android:text="Bark"
        android:textSize="16sp"
        android:layout_alignParentBottom="true"
        android:layout_margin="8dp"
        android:visibility="gone"
        />

</RelativeLayout>

Now, inside onUpdate, once the plane is detected, we can set the visibility of the button to true and then on click of the button we can perform the barking or any other operation that we want to perform.

Dev Note:
puppyFragment.getArSceneView().getScene()
            .removeOnUpdateListener(updateListener);
The listener is being removed inside the addModel method as because if not done then it would add multiple models and burden the system with each scene update. Hence using this we want to make only one model to be drawn for our application.

This is just a small portion of the implementation using ARCore with Sceneform. The libraries itself are evolving with each release. For example, skeletal animation for the models are not yet supported, but there is an open request for it being implemented.

google-ar/sceneform-android-sdk

Hope this post has been able to provide (even if a small) head start into the development using Sceneform. Do let me know your thoughts and feedback. Till then augment the reality to the core.