Build an augmented reality walk with AR.js

Cultural organisations have long used audio tours to augment our experience of moving through a place. Now we can use Augmented Reality (AR) to do the same. For example, in 2019 Apple launched guided walks in five cities across the world where users can ‘discover’ AR artworks. This blog post will show you how to augment your user’s experience of a place by creating a mobile-friendly AR walk in under an hour

The code for this post is drawn from a recent Research Software Engineering project with Dr Michael Richardson (@mjrichardson1), Newcastle University. Accessed through user’s mobile phones, the application includes an interactive ‘treasure map’ showing their location, a route, and ‘storytelling points’. At each storytelling point, the user uses their phone cameras to look for 3-dimensional AR models.

AR pirate ship

I used Ionic 4 with Leaflet.js to build a web application deployed using Google Firebase. I’ll focus on the code most relevant to the AR functionality as you can use your preferred frameworks and deployment method for everything else.

1. Set up your web framework

I wanted to get the feel of a mobile app without having to deploy as such, so chose Ionic 4, a flexible web framework with engaging UI components such as sliders, toast messages and fab buttons. If I want to deploy for Android or iOS in the future, Ionic will mean I can do that with little extra work.

Ionic slider used here for onboarding

2. Choose your 3D model

In this example, I’ll show you how to use a 3D model consisting of .obj and .mtl files, but you can also use .gltf models, see here.

An AR pirate in my back yard (I’m writing this during self-isolation!)

Create your own 3D model or browse the internet using sites such as Turbosquid. If you want to use the same as me download the ‘Pirate Kit’ from Kenney, extract the contents of the folder and look for pirate_captain.obj and pirate_captain.mtl.

3. Create your AR experience with 3 files

We’ll be using AR.js (AR for web) and A-Frame (creates VR experiences using HTML).

I’m going to use 3 files for each AR interaction. For example, the AR at the start of the walk has 3 files:

There are two reasons why I decided to populate the A-Frame HTML using a separate JavaScript file. Neither are difficult without using a separate file, this just made it easier.

• Use multiple locations: I needed to be able to test my models when working in the office and at home, and in their intended location.
• Fetch models from local or session storage: I can deal with patchy signal along the walk route by saving the models into local storage at the start of the walk.

We’re going to start by creating an ‘anchor’ page to house the AR for the starting point of your route. If you are using Ionic, generate a new page called ‘start’ using the command ionic generate page start . Paste this code into start.page.html (alter the path accordingly):

<iframe src="../../assets/ARFiles/start.html"></iframe>

Next, we will create the HTML page that actually holds all the important AR stuff. Create a new file called ‘start.html’ and paste in this code:

This code can be divided into 2 parts:

• Scripts: we are including A-Frame, AR.js , and three.js (line 10). We are also including the JavaScript file you are yet to write.
• AR scene: we use A-Frame tags to create an a-scene. You could populate this with the AR model and location details here, but our JavaScript file will do that.

Next, create a JavaScript file called ‘start.js’, I’m keeping mine alongside the start.html file. Paste in this code:

This is quite a bit of code but it is just dealing with the location and models to populate your A-Frame. Key sections are:

• Lines 8–25: list of places you would like your model to appear
• Lines 35–61: attempting to retrieve the .obj and .mtl files from local storage (see the full repo for how these have been saved) and pulling them from the server if there’s any problem with this.
• Everything else: creating HTML elements to populate the a-scene we created in start.html. Notable lines are creating a gps-entity-place with longitude and latitude (line 79) and an obj-model containing references to the .obj and .mtl files (line 82). You can also add rotation (line 83), scale and position using a right-handed coordinate system.

4. Create your map

Your users need to know where to go to find the start of your AR model. I used Leaflet.js, with custom markers and with a ‘pirate-y’ map from Thunderforest. I’ll be brief here and direct you to Leaflet’s quick start guide for more information. You can see my complete file here.

1. Add the leaflet stylesheet and script to index.html.
2. Add <div id=”mapid” ></div> to the HTML page for the page you want your map to appear in.
3. Initialise your map and map tiles in your JavaScript or Typescript file.
4. Create and markers.
5. Add user’s current location and a route if you would like.

You have now got the basics on your first AR point and a map to add more to. With the flexibility of using a JavaScript file to populate your HTML file, you can easily add multiple locations for your AR to appear and load your AR models from local storage to deal with patchy connections en-route. I used Google Firebase as a quick and easy way to deploy this pilot, check out their hosting instructions here.

I have a little mystery I’m hoping to solve with this tutorial. Firebase deploys from the www folder Angular creates when you run ng build. Here’s the mystery: the AR models don’t appear or only appear briefly if I have built with the -prod flag. If you have any ideas why, please let me know!