Making an Augmented Reality App with React Native, Expo, and Google Poly

In this article, we will learn how to display 3D objects from the Google Poly library in a React Native / Expo application.

Setup Project

To start off, create a blank Expo application. I prefer to use Expo XDE, but you can also use the CLI if that’s what you like.

Install Dependencies

We’re going to use three.js to display our objects using the wonderful expo-three library. Go ahead and install the dependencies we’ll be using:

npm install -S three expo-three expo-graphics expo-asset-utils

We’ll also need to grab a few files from three.js to help us load the 3D models. We need to use OBJLoader and MTLLoader. I just grabbed the code from Github (OBJLoader.js and MTLLoader.js) and pasted it in my project in a /util folder.

MyProject/util/OBJLoader.js

Setup Camera

Now we’ll setup the camera and scene that we’ll use for our augmented reality app. Expo provides a component that makes this really easy.

Go into your main screen and import a few components:

import Expo from ‘expo’;
import ExpoTHREE, { three } from 'expo-three';
import ExpoGraphics from 'expo-graphics';

Now head down to your render method and add an ExpoGraphics.View:

render() {
   return (
     <View style={{flex:1}}>
       <ExpoGraphics.View style={{flex:1}}
         onContextCreate={this.onContextCreate}
         onRender={this.onRender}
         arEnabled={true}
       />
     </View>
   )
}

The onContextCreate function is called as soon as the graphics context is created and gives us a lot of useful variables. This is where we can setup our three.js scene, like so:

onContextCreate = async ({gl, scale, width, height, arSession}) => {
 // Initialize renderer…
 this.renderer = ExpoTHREE.createRenderer({gl});
 this.renderer = setPixelRatio(scale);
 this.renderer.setSize(width, height);
 
 // Initialize scene…
 this.scene = new THREE.Scene();
 this.scene.background =  
    ExpoTHREE.createARBackgroundTexture(arSession, this.renderer);
 
 // Initialize camera…
 this.camera = ExpoTHREE.createARCamera(arSession, width / scale,
     height / scale, 0.01, 1000);
 
 // Initialize lighting…
 var ambientLight = new THREE.AmbientLight(0xaaaaaa);
 this.scene.add(ambientLight);
}

Most of this is pretty standard for setting up a three.js scene, but some of it uses the Expo framework. The pixel ratio and size of the renderer are important for making sure everything looks good on the screen. The scene background and camera are what add the actual camera image to our scene. The lighting is important for being able to see the objects that we’ll be adding later — if there is no light to shine on them, they will just appear as black blobs!

The render method gets called multiple times and is used for updating your scene with animations or whatever you might want to do. The simplest render method would look something like this:

onRender = (delta) => {
  this.renderer.render(this.scene, this.camera);
}

Now try running your app and you should see your phone’s camera image. Remember to use an actual device instead of a simulator!

One last thing: To get rid of those pesky yellow boxes, put this somewhere at the top of your screen file:

console.disableYellowBox= true;

Note: Sometimes the yellow boxes are helpful!

Google Poly API

Now let’s write some code that queries the Google Poly API. First head to the developer page and get an API key. I usually put all of my API keys in a file called /constants/ApiKeys.js, but you can do whatever works for you.

Next, skim through the API docs to get an idea of the inputs and outputs that are available. Try out the API explorer to test some actual queries!

Let’s make a class that can help us work with the API. But before we get started, let’s think a little bit about how we want to use our class…

• We want to let users search for whatever kind of Google Poly objects they want. So we’ll provide them with a TextInput where they can type their search query.
• There could be a large number of results, so let’s only show users a few at a time unless they click “Load More” or something like that to get more results.
• Since we are using the three.js OBJLoader, we want to limit options to only those that support .obj format.

With these requirements in mind, I came up with the following class, which I called /api/GooglePoly.js.

A few notes:

• Pass in your API key to the constructor rather than hard-coding it as a constant in the class. This lets you share the code with your friends without giving them your key!
• I hard-coded a lot of parameters in my getQueryURL function to make the inputs a little simpler. You can customize this however you’d like.
• Since the keywords will come from user input, make sure to sanitize it! Otherwise, you open your application to risks if unexpected inputs are provided. You can sanitize your inputs by calling encodeURIComponent(). You could also use a query-building library to help you build the query string, but I was too lazy for that.
• getSearchResults returns a promise which gives the next page of results. But the class also aggregates ALL results in this.currentResults, so we can access all of them if we want.

Now you have a nice class for working with the Google Poly API. Write up some code to test it out! I put mine in the constructor of my main screen (but remember to delete it after you make sure it works!)

import GooglePoly from ‘./api/GooglePoly’;
import ApiKeys from ‘./constants/ApiKeys’;
...
constructor(props) {
  super(props);
 
  var googlePoly = new GooglePoly(ApiKeys.GooglePoly);
  googlePoly.setSearchParams(“duck”);
  googlePoly.getSearchResults().then(function(results) {
    console.log(“Got some results!”, results);
  });
}

Render Google Poly Objects

Now that we can successfully retrieve Google Poly objects, lets try rendering them into our scene. This actually takes quite a bit of code, most of which I got from Google’s Github repo.

We need to write a function that will take a result from the Google Poly API and give us back an object we can place in our scene. I’m going to put the function in my GooglePoly class, but if you’re not comfortable with using that class for both the API querying and the three.js building, then you can put it somewhere else.

First, import a few libraries into our GooglePoly.js file:

import ExpoTHREE from ‘expo-three’;
import AssetUtils from ‘expo-asset-utils’;
import * as THREE from ‘three’;
require(“./../util/OBJLoader”);
require(“./../util/MTLLoader”);

Then, create a function similar to this:

This function takes an object from the Google Poly API along with a success callback and a failure callback. If the three.js object is successfully created, it will call success(object). If there is an error, it will call failure(errorMessage).

In order to be able to render the object, let’s give users a way to select an object…

Create a Searchable UI

GooglePolyAsset.js

In our app, we want to give users the ability to search for any kind of object they want. So they’ll type in their search query and get a gallery of matching results. Let’s make a component for what one of those results will look like. I’m going to call mine /components/GooglePolyAsset.js.

This component will look something like this, but feel free to update the styling however you’d like.

Quack!

SearchableGooglePolyAssetList.js

Now let’s make a component that allows users to search for Google Poly assets and displays them in a list. I called my component /components/SearchableGooglePolyAssetList.js. That’s quite a mouthful! But at least it describes what it does. PS: You could also split this up into a few sub-components if you’re conscientious!

This will give you a component that can accept your instance of GooglePoly and display search results. The component will look like this when there are no results:

And like this once the user searches for something:

Ok, it’s not the most attractive UI (or code)! But it allows users to search and select items from the Google Poly library. You should make it look better!

Putting It Together

We should have all the pieces we need, so now let’s put them all together!

Put the Search Into a Modal

Let’s put the search component into a Modal because it’s a little more convenient than navigating to a new screen. In your main screen, add a modal like this:

<Modal 
  visible={this.state.searchModalVisible} 
  animationType="slide">
    <SearchableGooglePolyAssetList
      googlePoly={this.googlePoly}
      onCancelPress={this.onCancelPress}
      onAssetPress={this.onAssetPress}
    />
</Modal>

And of course, add the needed functions:

onCancelPress = () => {    
  this.setState({searchModalVisible: false});  
}   
onAssetPress = (asset) => {    
  this.setState({currentAsset: asset});
  this.setState({searchModalVisible: false});  
}

Add Some Buttons

Let’s add some buttons that allow us to bring up the search modal as well as a couple buttons that let us add/remove the selected object.

First, we’ll start by implementing the necessary functions:

onSearchModalPress = () => {    
  this.setState({searchModalVisible: true});  
}
onRemoveObjectPress = () => {    
  if (this.threeModel) {      
    this.scene.remove(this.threeModel);    
  }  
}
onAddObjectPress = () => {    
  // Remove the current object...    
  this.onRemoveObjectPress();     
  // Add the current object...      
  GooglePoly.getThreeModel(this.state.currentAsset, function(object)   
  {      
    this.threeModel = object;      
    ExpoTHREE.utils.scaleLongestSideToSize(object, 0.75);      
    object.position.z = -3;      
    this.scene.add(object);    
  }.bind(this), function(error) {      
    console.log(error);    
  });  
}

The first function just shows the modal by setting searchModalVisible to true. The second function removes the selected object by calling this.scene.remove(...). The third function adds an object to the scene by first removing the existing object (if there is one) and then calling getThreeModel() to create the three.js model and adding it to the scene.

Finally, let’s hook those functions up to some buttons! I decided to use Icon buttons, but you can use whatever you’d like.

<Icon.Button size={40} name="plus" backgroundColor="transparent" 
    onPress={this.onAddObjectPress} />            
<Icon.Button size={40} name="magnify" backgroundColor="transparent" 
    onPress={this.onSearchModalPress} />            
<Icon.Button size={40} name="minus" backgroundColor="transparent" 
    onPress={this.onRemoveObjectPress} />

That’s it! Try customizing it yourself by adding screenshot functionality or placing multiple objects in the scene with different animations!

Video Tutorial

Code

ProProgramming101/PolySnap

References

Web Quickstart | Poly | Google Developers

three.js - Javascript 3D library

expo/expo-three

googlevr/poly-sample-web