Practical application of face tracking using ARKit

Photo by bady abbas on Unsplash

Intro to ARKit

ARKit is Apple’s augmented reality development framework, and it’s compatible with millions of iOS devices.This framework allows developers to create augmented reality apps and take advantage of the hardware and software capabilities of iOS devices. It was launched late 2017 along with their iPhone X with TrueDepth camera system.

What AR means — Through the phone camera, we can add 3D/2D elements to our world and interact with them in real time (be it a piece of furniture in the corner of our room or some cool sunglasses on our face), thereby augmenting our own reality.

Features of AR Platform

1. Face-tracking — With Apple’s TrueDepth camera, it can accurately identify many features of a face in its view:
   - Shape of face
   - Position of face
   - Rotation of face
   - Distance of face from camera
   - Position of the nose, eyes, lips
   We can use this information to place certain elements on specific regions of the user’s face and that object will “stick” to the face and behave as if it were part of the user.
2. Plane-tracking — With the use of Artificial Intelligence, the phone back camera setup can detect horizontal surfaces like floors and tabletops quite well and then place a 3D model on the detected surface creating a feel that the object is really in their room. Use cases include placing a couch or a fridge in your room or even a car in your garage.
3. Image-tracking — The camera can also track a single or a set of predefined images, when found, can trigger an AR experience. Use cases can include AR business cards or billboards.

This article will be more focussed on the face-tracking capability within ARKit and its practical application. One of the practical applications could be virtual try-on of makeup products.

Let’s say you had to model a generic face, what are the features you would include?

Eyes, nose, lips, chin, cheeks — right?

This is the same thing Apple does as well. They have generated a 3D model of a face which has these exact features.

ARFaceGeometry.obj

They have also generated a mapping of the 3D face on to a 2D image (3D artists will know the terminology — UV mapping)

Wireframe texture

This flat image wraps around the 3D geometry of the face to give us the feature-points. We can clearly see where the eyes, nose and mouth are in the image.

Wireframe texture on top of 3D face geometry

We can use this image itself to do a variety of makeup effects.
Let’s take an example of lipstick. Using Photoshop or any other image creation tool, we can use the previous image as reference and prepare an image (of same dimensions) which looks like this.

Creating a lipstick image using wireframe texture as a base

Actual lipstick texture we are going to use

As we can see, the lipstick is exactly where the lips were in the previous image. Now, if we apply this image on to the 3D face mesh and see in AR.

Output Result - Lipstick applied in AR

But this is only one face. No two persons have the exact same face structure unless they are twins. How do we make sure it fits every face?
That’s the neat part, we don’t have to. ARKit modifies this 3D mesh to match the user’s face dimensions and expressions in real-time. So, the image needs to be created only once and it will be mapped onto each and every user face when they try it on.

Similarly we can create images for other products like blush, eyeliner, eyeshadow, etc.

You can start exploring with the sample project provided by Apple here.