What Do You Need To Know To Create An Application Using ARKit?

Over the past few years, the development of mobile applications is gradually moving towards augmented reality. With applications that use AR, you can simplify everyday tasks. For example, AR can help you plan home renovations. With the help of special applications, you can get the size of the room (Measure) or see how to best place your the furniture in your apartment (IKEA Place). My name is Kristina Shevtsova. Together with ARchy, I create applications using ARKit on a regular basis. Today I will share my knowledge and we will try to figure out what you need to know before creating an application using ARKit. In this article we will answer the following questions:

1. Technical requirements and limitations of ARKit. What tools does the developer of an AR application for iOS need?
2. ARKit functionality. What can ARKit do and how do its versions differ from each other?
3. Types of AR applications. What are the scenarios of using AR-functionality in an iOS application? How to track ARKit-compatible devices correctly.
4. ARKit terminology. What concepts should a beginner developer of an AR-application know?

Technical requirements and limitations of ARKit

Let’s start by comparing the requirements of two versions of the framework: ARKit and ARKit 2.

We should point out that the table above does not cover all the limitations. If you need to use face tracking in your application, it would only be possible on devices with TrueDepth camera, namely iPhone X and later iPhone versions, and iPad Pro with A12X Bionic chip.

ARKit functionality

Here is a list of the functionalities of the first version of this framework:

1. Detecting horizontal surfaces (such as a floor, table, etc.).
2. Light exposure. Used to be able to see how 3D objects cast a shadow, or to see glares of light on shiny surfaces.
3. Hit-testing. You can use this option for the following scenario: the users click anywhere on the screen of their device to place a 3D object there. ARKit converts the coordinates on the device screen to real-world coordinates. For this conversion, such indicators as the distance of the real object from the user’s device, light exposure, and many others are taken into account.

Between ARKit 1 and ARKit 2 there was an intermediate version of the framework — ARKit 1.5 (supported by iOS 11.3). In this version, Apple added recognition of vertical surfaces (such as walls) and 2D images, as well as face recognition.

Despite the fact that ARKit 2 has more technical limitations, this version of the framework also has more functionalities:

1. Saving and recovering AR map. This means that we can save the map with the objects placed in augmented reality. If necessary, we are able to display the map again. We can also transfer it to the server and display it on another device.
2. Multiplayer mode.
3. Tracking moving 2D images.
4. Tracking static 3D objects.
5. Improved face tracking. Now you can track gaze direction (left and right eye separately), and determine whether the person sticks out the tongue.

Now we understand the basic ARKit limitations and functionalities. To develop the first AR application, we need Xcode 9+ and one of the devices listed above.

Types of AR applications

Applications developed with ARKit come in several types. How we use ARKit functionality depends on the type of AR application. Let’s study the options of indicating device compatibility depending on how the application uses ARKit:

● If the main functionality of the application is based on using AR (using the main camera), then in Info.plist you must add the arkit key to the UIRequiredDeviceCapabilities. Using this key will make the application available only for ARKit-compatible devices.

● If AR is a secondary feature of the application, you must ensure that the device supports the AR configuration you want to use. To do this, check the isSupported property of the corresponding subclass of ARConfiguration.

● Using the face tracking feature requires having TrueDepth camera on the device. To determine whether the current device supports face tracking, you must check the ARFaceTrackingConfiguration.isSupported property.

It is clear that there aren’t any AR applications without the camera. You must also request the user’s permission to access the camera. To do this add the NSCameraUsageDescription key into Info.plist. In the value of this key, display the text that will explain why your application needs access to the camera. If you use the ARKit face tracking feature, your application must include a privacy policy that describes how you will use face tracking and face data.

ARKit terminology

Now let’s look at the basic concepts of ARKit:

1. ARSession — an object that coordinates the main processes required for augmented reality. These processes include collecting data about a device’s movement, controlling a device’s camera and analyzing images captured by the camera. ARSession combines all these results to set up a correspondence between the real space in which the device is located and the virtual space in which you create AR content.
2. ARConfiguration is an abstract base class for the ARSession configuration. ARConfiguration subclasses are responsible for how augmented reality looks and are required for starting the AR session.

ARKit includes the following configuration classes:

● ARWorldTrackingConfiguration. Allows you to determine planes, hit testing, set environment-based lighting and identify images and objects.

● AROrientationTrackingConfiguration. Provides basic AR features and track only a device’s orientation.

● ARImageTrackingConfiguration. Detects and tracks previously known images.

● ARFaceTrackingConfiguration. Uses the front camera and tracks the movement and expressions of the user’s face.

● ARObjectScanningConfiguration. It is designed to collect high-precision spatial data to use for scanning 3D objects for later detection. Use it for development only! High-precision spatial mapping delivers high performance and has high energy consumption. This type of configuration disables ARKit functions that are not required for scanning reference objects. For end users, use ARWorldTrackingConfiguration.

3. ARSCNView — a view to display AR content.

ARKit is a very young but very promising technology. Now you have the basic knowledge to finally work with it. After reading this article, you will not encounter any unexpected difficulties while creating your AR application. Now all you have to do is to come up with an interesting idea and start turning it into reality. Good luck!