Augmented Reality in Android Apps using ARCore
Android Integration for Location based Augmented Reality using ARCore
Would you like to spot different kind of places near-by from your point of view without looking on a Google-like map using Augmented Reality?
If so, then welcome to
A comprehensive guide on how to implement an Android App that lets you scan for various venues around your GPS location and display markers on the screen pointing to their location, in real time, using Open-Source SDKs.
Before jumping in to the actual implementation, let’s add to the agenda the software and hardware components needed so this feature can work.
Software
Firstly, we need a list of venues. The most important components of this venues objects would be their geolocation. To generate such a list, we will use the Foursquare API. Sadly, this API does not provide us with an altitude for those venues so we will use a random function to generate a height for each element, this property being used in the drawing of the markers.
For the drawing part, we will use ARCore’s Sceneform to host the 2D nodes that are rendered on-screen, and the ARCorelocation as a wrapper library for handling the Sceneform’s Anchors position, orientation and scaling based on real world coordinates. Those anchors will represent our physical venues.
Hardware
On the hardware side, the most important components are:
Camera: To provide the user a live experience of the surrounding environment
GPS: This component is needed to broadcast the geolocation of the user’s device as a pair: latitude and longitude. Those values, along with the destination’s geolocation, are used for real-world location tracking
Magnetometer/Compass: To know the device’s orientation relative to the Earth’s Magnetic Nord. Using it’s output, one can calculate a bearing value based on the user device’s coordinates and the destination coordinates. Simply put, using this value, the heading from source to destination can be established, so we know where to draw the location markers on the screen
Android device with Google ARCore compatibility: The minimum SDK version required by this library is 7.0 or API 24. Also, a list of the compatible devices with ARCore can be found here
As a side note, this guide’s purpose is not to get you introduced with what concepts like ARCore or Sceneform are and it does not dive deep into geometry concepts, but it allows the reader to understand what, when and why something is used.
Diving in to the Implementation
ARCore compatibility
First thing first, we have to check whether the device is compatible or not with ARCore. We can achieve this compatibility check at runtime, using the following simplified function.
As seen in the example, we can anytime use a callback to get the ARCore status. Based on the value of this status, we can show/hide ARCore related functionality (e.g. hide a button that starts AR functionality).
Setup the LocationScene
Next thing, we need to create an xml layout that would be the container for our Sceneform. The widget used is com.google.ar.sceneform.ArSceneView.
After requesting the user’s permissions for location and camera, we should setup the ArSceneView’s session and the LocationScene which uses our ArSceneView widget. We will use the LocationScene instance to manage our AR markers that will be displayed.
By setting the above parameters to the locationScene, we force it to minimize the refreshing of the displayed markers, so that the markers are static on the screen and don’t move that much. Also, marker overlapping should be avoided if setting the offsetOverlaping parameter to true. By setting removeOverlapping, we force the framework to draw only the closest to us marker, in case of overlapping.
If the LocationScene’s setup is successful, we can now try to obtain the latitude and longitude for our device using the above declared locationScene object. But the device’s GPS does not instantly broadcasts those values, so we need to wait until it provides valid data. To do so, we will use an AsyncTask to acquire the device geolocation. Once we get valid data, we can proceed to fetch the venues that are near our geolocation.
Using Retrofit, we setup the Foursquare API to fetch those venues. This API has configurable parameters like the the category of the venue or the limit of the venues list to be returned. You can freely set those parameters based on your needs.
Once we have the venues list from the API, we can proceed to the fun part, the markers rendering. We will split the implementation of the rendering function in two main parts, for better visibility.
Setup and render the venues markers
Okay, seems like a lot of stuff happens here. Let’s get it step by step. Firstly, we will parse each venue from our venues set (set preferred so no duplicates exists). Then, for each venue, we will create an object of type CompletableFuture<ViewRenderable> by using the factory builder provided by Sceneform.
But are those classes more exactly?
Sceneform provides a high level API which allows the developer to render realistic 2D/3D scenes from the device’s camera Point Of View . In our case, we will render some 2D Anchors.
ARCore Anchor represents a virtual object from the Sceneform.
CompletableFuture is a Java 8 interface which declares the contract for an asynchronous computation. It does it’s job on a background thread and posts the result on the main thread (yes, RxJava sort of step-brother). Why we use this? All build() methods in Sceneform return a CompletableFuture, with the object being built on a separate thread and a callback function being provided.
ViewRenderable renders a 2D Android View in 3D space by attaching it to a Node.
AnchorNode is a Node that is automatically positioned in world space based on an ARCore Anchor.
Now to continue, as you can see in the 4th line of the snippet, while building the rendarable, we pass a custom view to it. This view represents the aspect of the node, so you should customize it based on your needs.
Create a LocationMarker
By using the method anyOf(completableFuture), we are able to know if the viewRenderable was built or not and handle both the cases.
Now, for each venue, when it’s corresponding view renderable has finished loading, we will create a LocationMarker object containing this view rendarable along with the geolocation of the current venue. This newly created object represents the marker that will be added to the scene.
Using the setVenueNode() method, one can obtain the corresponding view renderable for a specific venue and then create an ARCore Node. This is the place where we should setup our marker design and click listeners. For this example we will set a layout that will contain the category icon, the name and the distance from our device to the given Venue.
Once you are done with the layout setup, you need to return the Node, which will be passed to the LocationMarker created above, the last one being added to the Scene.
Then, the ARCore Location SDK will perform all the calculations needed (azimuth, distance, scaling, position on screen, etc.) so that our marker will be displayed correctly on the screen. We can also set an altitude for the marker (in this example, using a random generator) so that the marker will be raised as needed (this is done in the 2nd part of the render function).
By default, the markers are rendered at the device’s current altitude.
Add the LocationMarker to the Scene
Once we have setup the initial data for our marker, it’s time to add it to the list of markers that are rendered on the screen.
Before we add the marker to the scene, we should specify a scaling mode and a scaling modifier for it. By choosing FIXED_SIZE_ON_SCREEN scaling type, we will be able to customize our own scaling mode. If we want, we can let the SDK perform the scaling, by specifying,for example, the scaling type GRADUAL_TO_MAX_RENDER_DISTANCE.
The scale modifier represents a value used to compute the marker’s scaling every frame. Initially, we will set a default value, and then we will change it by computing a new value based on the real world distance between our device and the marker’s location.
Next, we add the marker to the locationScene’s markers list, enable the node, refresh the scene and make our marker’s layout visible.
As you can see from the last snippet, there is a function setRenderEvent(), which is called on the locationMarker object. This function is called every frame and allow us to update the Node associated with the Marker. Thus, we are able to get the distance from our device to the target location and update it’s view accordingly.
Moreover, knowing the distance, in meters, in real time, we can compute a new scale modifier for the marker. By doing that, we can, for example, make the markers that are closest to us appear bigger than those that are far away. An important thing to remember is that those computations are done in each frame for all visible markers on the screen (present in the scene).
If the distance is bigger than a set threshold, we can detach that marker from the scene. Another case when we should do this is when the marker is too close from our device, and it would obscure the screen. All we have to do is to call the following function on the desired locationMarker we want removed.
Update the venues markers
As stated before, this second part of the rendering function is actually an update listener performed by the Sceneform on our markers.
We proceed to the computation of the altitude for each marker in the scene. For this app, we are going to raise the distant markers above the closest ones, based on fully-customizable distance intervals.
The last lines of the rendering function are pretty much self explanatory. If the current frame is null or the camera is not tracking any markers, we should exit the update function. Otherwise, we should allow the locationScene to process the current frame, refreshing the markers and recalculating values like distance or height for the markers that exist in the scene.
Lastly, one important thing to do before actually calling the rendering function is to reset and clear all the variables and lists/sets used.
Results
If you’ve reached that far, our app should now render different venues as pins/markers on the screen as you are scanning the environment.
You can add more functionalities, for example implementing a venues filter, which allows only certain venues to be rendered at a time, depending on the selected category from the filter.
Taking a deep breath..
Okay, so maybe you wonder what actually happened? To wrap things up, the we made the following steps, in chronological order:
- A runtime check to check if the device running the app is compatible with ARCore
- Permissions requested from the user to allow the app to use the hardware camera and the GPS location
- Sceneform and LocationScene SDK setup
- Request device geolocation from the LocationScene SDK and wait until it is established
- Fetch venues from Foursquare API, using the device’s geolocation, obtaining a list of different venues from around us
- For each venue, create a layout and setup a LocationMarker to be rendered, applying custom algorithms for the computation of the height and the scaling of the marker, in real time
- Add the venue to the Scene
- Repeat the last two steps until all venues are parsed
- Start scanning around with your new app :)
Okay, but where is the practical example?
Please feel free to check the more detailed sample app’s source code available here.
Also, don’t hesitate to leave a comment if you have any questions :)
