AR Environment Probes in UE4 4.20

Joe Graf
Joe Graf
Jun 29, 2018 · 5 min read

In my previous post, I talked about the support for ARKit 2.0 that will be present in the UE4 4.20 release (link below). In this short post, I am going to into more detail on how to use the environment probe feature in ARKit 2.

Project Setup

All of these posts that I do are based upon our Handheld AR Template when generating a new project. Using that template is not required, but it does automate some of the Blueprint creation for you and will give you a common frame of reference with my samples.

There are 4 steps to setting up your project to use ARKit 2.0’s environment probes. The first step is to set your project’s rendering settings to use the Metal Forward Desktop renderer. Note, this is an expensive setting as you are using the full desktop renderer with a pre-pass, more complex lighting, etc. However, it’s needed in order to get the reflection support. The image below shows the settings I used for this post.

iOS Project Rendering Settings

Your ARSessionConfig also needs to know to generate the environment probes. There are two options for when this is enabled. Automatic analyzes the scene and generates environment probes for you. I find it often generates more probes than I can reasonably use, but it is the easy way to get one created. The second method is to manually specify a bounding box and location. This is the most performant option as you have total control on how many get created and there are no updates for environment probes that you aren’t using. For this post, I have mine set to automatic as seen below.

Automatic Environment Probe Generation

In the previous post, I mentioned that I hoped to get a AR specific SkyLight implementation written. I did manage to get this into the 4.20 release. There’s nothing amazing about this code, but it does understand how the environment probes are updated over time. It notices when the probe texture has been updated and forces a regeneration of our underlying render data for lighting and reflections. This is all done on the rendering thread, so shouldn’t affect your gameplay unless you’re render thread bound.

The easiest way to get a ARSkyLight added to your level is to create a Blueprint subclass. There’s no changes that you need to do to the new class, but this makes it easy to drag into your levels. The image below shows the one I created for this post.

A simple Blueprint wrapper for ARSkyLight

The last step is to delete the standard SkyLight that our template creates and add your newly created BP_ARSkyLight to the level instead.

Using the ARSkyLight

Because there can be multiple environment probes created for a scene, you need to tell the ARSkyLight which probe is the one that you are interested in. Since I am using the automatically created probes for this post, I added a simple Blueprint level script to periodically check for the creation of an environment probe. The level Blueprint grabs the first one it finds and sets that as the source for the ARSkyLight. After the probe is set in the ARSkyLight, it takes care of the rest of the work of checking for changes in the underlying cubemap data. My Blueprint is below.

Level Blueprint that polls for a new environment probe

I am sure you’re curious what the results look like. Given that these cubemaps are generated by some AI as it samples the scene from the camera data, I am quite impressed by the results. They aren’t perfect yet, but this is a 1.0 feature, so I expect them to improve over time. The image below is from an iPad Pro running iOS 12 beta 2.

Environment probe in action (see caveats below)

Caveats

There are a couple of caveats that need to be mentioned. The first is that the Metal Forward Desktop renderer requires a pre-pass. On some devices that pre-pass results in z-fighting due to different math optimizations that the metal compiler uses. In the image above, the black splotches on the sphere are examples of this z-fighting. Metal 2.1 introduces [ position, invariant ] attributes which fixes the z-fighting. UE4 4.21 will support Metal 2.1 as a project setting. Unfortunately, there was not enough time to get that into the 4.20 release. However, the changes are available in our Dev-Rendering stream. The CL #s that I know of at this time that are needed are: 4173488 and 4175308. Remember that is an active development branch, so Caveat Emptor.

The second caveat is that I noticed that different devices sometimes generated slightly different resulting cubemaps (radar with sample images filed). Specifically, my iPhone X that I use for FaceAR testing would have the scene slightly rotated…sometimes. One benefit of the iPhone X is that the z-fighting prevalent on the iPad Pro is nonexistent. Below is an image from an iPhone X session of this project.

iPhone X running this example app

One more thing to note is that repeating pattern you see in both images happens whenever there is insufficient data for the AI to make a good texture. If you keep scanning the area, that repeating textures goes away as it fills in the data with good details.

Next Up

In the next installment, I’ll walk you through shared AR spaces using UE4’s built in networking.

Update

In 4.21, Metal 2.1 is supported so the z-fighting is gone.

4.21 AREnvProbe Sample

Joe Graf

Written by

Joe Graf

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade