Making AR Work

Idea

Before we get into the process, let’s clarify how augmented reality (AR) works. We want to overlay objects/visuals in front of the screen when the camera sees it. There are two main ways of doing that: object detection and surface detection.

Object detection: we tell the software to detect a particular object; it can be an image, a shape, or a 3D model. Once the camera detects that object, the defined overlay shows up on the screen.

Surface detection: we use the camera to detect a plane surface. Once it detects that surface, the defined overlay shows up on that surface/plane.

Software

There are two free* tools that enable the implementation of augmented reality: Unity and Vuforia.

Unity

Unity is a game development software developed by Unity Technologies. Over the years, it has become a standard for developing interactive experiences for phones and desktops. Even though the software has “game” attached to it’s description, it’s capabilities extend way beyond just games.

Unity Website

Vuforia

In its simplest form, Vuforia is an AR generator website that integrates with Unity. So, once we create an object detection license (called Target), we import that license in Unity to make AR work.

*While both these tools are free, Vuforia has a subscription after the number of detections surpass 1000.

Vuforia Developer Portal

Workflow

1. Vuforia (https://developer.vuforia.com/)

• Create an account or login.
• Under License Manager, create a Development Key. Think of it as the license we would use in Unity.
• Under Target Manager, add a Database. This would have the details of the object the camera would detect. So, add image/object targets here. For instance, if we want the camera to detect multiple images, upload each image as a new target.
• Note: for plane detection, a target is not required. Only the license key would be copied in Unity.
• We now have the “License Key” and the “Database” for using in Unity. Download the database and copy the license key.
• Check this YouTube link for detailed instructions on the setup:

1. Unity (https://unity.com/)

• Download Unity — select Free (Personal). This would download Unity Hub, which is a Unity software manager.
• Often, the latest version is not the best choice for development. The stable versions with LTS (Long Term Support) are best bets.
• During installation, there will be an option to select what we want to install. Check “Vuforia” and “Android/ iOS (based on the operating system in use)” and complete the installation.
• Detailed steps on setting up Vuforia for Unity are detailed in Vuforia’s documentation:
• Getting Started with Vuforia Engine in Unity
• For help understanding Unity and setting up the scene, refer the Unity documentation (it is well-written and helpful)
• Unity User Manual (2019.2)

Potential Challenges

• Console Log: The exporting process for Unity tends to give a lot of errors in the Console window. Often, that has to do with the extensive system and software settings required for export, and the software versions of Unity and Vuforia. When we go to File> Export> Build, the Player Settings option has a lot of configurations that must match the project.
• Model Size: When building the scene in Unity, we had to make sure that the size of the models (exhaustive meshes due to photogrammetry) did not interfere with the rendering of the scene. In some cases, that meant simplifying the meshes using external tools to accommodate the design of sculptures (details in the Photogrammetry section).
• Hardware: Working in AR and on Unity requires a decent hardware configuration for PC/Mac. We advise working with a minimum of 16GB RAM and a 512GB hard drive. That simplifies the loading time in Unity, because importing and exporting packages is a big part of the project development.

Note: The process detailed above talks about AR using Unity. We adopted this approach because the web platform, WebAR, is still under development is not ready for consumer use. When it is ready, platforms like A-Frame and Styly, which only work with WebVR, will handle most of the work. Then, activation using AR would be simplified — open a link on your phone and AR starts working from your browser.