Creating an Augmented Faces React Native App with Sceneform.

In this article I’ll guide you to create an augmented faces app (like Snapchat) with React Native.

To follow this tutorial I expect you to know react native.

Project Setup

Let’s create a new project:

Voilà, we have now an empty react native project.

I strongly recommend to open your project with Android Studio after installing a library that contains native code.

Remember to set minSdkVersion to 24 and to update your AndroidManifest.xml as mentioned in the library docs.

Now let’s run it with:

And voilà, we well see a screen like this one below:

Now comes the fun.

Creating an AugmentedFacesView

Let’s clear the App.js file and set this basic view:

As said in the library’s docs it does not handle permissions so we will do it using the PermissionsAndroid component.

We will:

  • Import the PermissionsAndroid component from React Native.
  • Create a field in the state to handle permissions, let’s call it cameraPermissionGranted.
  • Create (and bind) an async method to check permissions and request them if needed, let’s call it checkPermissions.
  • Call the checkPermissions method in the componentDidMount lifecycle method.
  • Add an AugmentedFacesView node (with styles filling the screen) to the render method conditioned by the cameraPermissionGranted state value.

Our code might be looking like this at this point:

And we will see the front-facing camera in our screen!

No more screenshots for now, I did not do my hair tonight.

Adding an Augmented Face

According to the docs, the AugmentedFacesView component exposes a method and a prop, being addAugmentedFace and setAugmentedFace.

Let’s add a new field to our state that will be passed to setAugmentedFace, the initial value will be -1 as mentioned in the docs, we will call it augmentedFaceIndex.

In order to use methods, we must create a ref first, so let’s initialise it in the constructor and assign it in the node, let’s call it augmentedFacesView.

The package itselt has 2 sets of model+texture that can be used in this example, we will create a constant outside the class with them as shown below:

Having it in place, now we have to tell our app to load them assets, we will do it using the ref previously created and the addAugmentedFace method.

As said in the docs, the addAugmentedFace method is a promise that returns an index when it fulfills correctly, so we need a new field in the state where successfully loaded indexes will be saved, let’s add it as augmentedFaces being an empty array.

To load the assets we will create a new method, I will create it as loadAssets in this example and we will call it after the permissions are granted.

The method will iterate our ExampleSet and for each iteration it will call the addAugmentedFace method, when it resolves we will add the item to the state with an additional property: index.

Now, we will render the loaded assets in a FlatList. We will create a View with absolute position over our AugmentedFacesView node and at the bottom we will add the FlatList, rendering the previous structure.

The FlatList’s renderItem must return a TouchableOpacity (import it fron react native) that will we use to change the augmentedFaceIndex.

Our code now allows us to pick an augmented face and use it and looks like this:

And we are done!

… Ey! I want to add my own Augmented Face assets! How do I do that?

It’s not that complex, to create a model or texture you can follow the Arcore docs.

Said that, I will only teach you how to load them into your app.

Importing my own assets

Having your assets ready, you have two options:

  • Host them and load remotely.
  • Bundle them in your app.

The way they are loaded on runtime in the app is the same, calling the addAugmentedFace method, the difference is that if it’s a remote asset you will use an URI, otherwise, you’ll use a path (pointing to android/src/main/assets/ but skipping that part like in the example above)

As a rule, your AugmentedFaceModel requires a model property, so if you’re using only a texture you can set model to ‘models/face.glb’ (and I’m taking note of this for a future update haha).

You can find example code in here.

As a final note, I want to mention that this tutorial was possible thanks to the SceneView Community.