ARKit 2.0 support in UE4 4.20

In this post, I am going to talk at a high level about how the ARKit 2.0 features are supported in UE4 4.20. In subsequent posts, I intend to give more detail about each feature and provide ideas on using them in your apps. Until then, this will give you enough information to experiment with the features.

The major features that were added in this release of ARKit are:

  • World AR Map Persistence
  • Environment Texturing
  • Object Detection

Minor features in this release are:

  • Image Tracking
  • Eye Gaze Target, Individual Eye Transforms, and Tongue detection for FaceAR

Since UE4 hadn’t fully integrated the ARKit 1.5 changes, this means we had to add support for:

  • Vertical Plane Anchors
  • Image Detection

The following sections talk about the features/changes at a high level.

Session Configuration

  • World Alignment — which is used to specify what kind of transform tracking you want from ARKit (previously hidden)
  • There are new options in the Session Type (Image, Object Scanning)
  • Vertical Plane Detection
  • Enable Auto Focus — this works well for most apps, so is on by default. Some apps that require macro camera focus and small objects may see them swim as the focus changes. I think those are less frequent and can disable the auto focus feature as needed.
  • Max Num Simultaneous Images Tracked — tracking an image in a scene can be expensive. This limits how many images to track at once.
  • Environment Capture Probe Type — this is the new Environment Texturing feature which uses machine learning to build cubemaps from the camera data collected as the camera scans the scene. These cubemaps can be used for image based lighting and/or making more realistic reflective surfaces.
  • World Map Data — this is the binary data that was saved in a previous AR session (persistence). When provided, the AR session will use that data as the basis for the AR world, rather than building a new AR world from scratch.
  • Candidate Objects — just like image detection/tracking needs to know what to look for in a scene, its size, etc. candidate objects need a similar descriptor before the AR session can search for them.
ARSessionConfig in UE4 4.20 with the ARKit 2.0 features shown

Image Tracking

There are 2 new configuration options for image detection and tracking. The first is a new AR session mode that only detects and tracks images in a scene. It skips all of the other work that is done when building an AR world, so is great for performance if you are only interested in detecting and tracking images.

The second option tells ARKit how many instances of images you want to track in a scene. Note: this can be the same candidate image seen multiple times or different images. It’s there to control how much performance you want to give to tracking, since tracking unlimited number of instances could be too much for your framerate targets.

Environment Texturing

There are two ways to generate this data. The first is to set the Environment Capture Probe Type to Manual. When you do this, you must tell ARKit where you want an environment probe to be located and the bounding box that you want it to include. After that, an AREnvironmentCaptureProbe is created for you, which will hold the cubemap that is generated. As the camera data is applied to a subsequent updates, the capture probe will update its cubemap data. The second way to is to Environment Capture Probe Type to Automatic. In this mode, ARKit will automatically create probes within the scene and update their textures. In my testing, I found that it creates many probes (and overlapping) which has two problems: The process of updating those cubemaps is pretty expensive and can generate hitches; and, secondly, you have to manually decide how to render objects with multiple probes around it. Because of those two problems, I recommend that you stick to a Manual probe that covers the potential play area for your game.

UE4 already supports image based lighting and reflections via the SkyLight actor you place in a level. If you set that SkyLight actor to be image based, you can then use the cubemap from the AREnvironmentCaptureProbe to be the image it lights and reflects (see below).

Using the texture from a AREnvironmentCaptureProbe as a SkyLight source texture

SkyLights can also blend between two different cubemaps, which some developers use as the basis of a time of day system. You can use that feature to blend between two capture probes, so that moving objects don’t pop when moving between those probe areas. One drawback to using a SkyLight for image based lighting is that it doesn’t expect the cubemap to be changing underneath, so it doesn’t automatically know to regenerate its internal data. I am hoping to add a ARSkyLight that knows when the capture probe was updated and triggers a regeneration of the underlying SkyLight data. Longer term, I want to double buffer the cubemap textures, so that updates can be blended between rather than the pop that happens when the texture is wholesale replaced.

World Map Persistence

Object Scanning and Detection

Face AR Additions

Wrapping Up

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store