In my previous post, I talked about the support for ARKit 2.0 that will be present in the UE4 4.20 release (link below). In this short post, I am going to into more detail on how to use the environment probe feature in ARKit 2.
Project Setup
All of these posts that I do are based upon our Handheld AR Template when generating a new project. Using that template is not required, but it does automate some of the Blueprint creation for you and will give you a common frame of reference with my samples.
There are 4 steps to setting up your project to use ARKit 2.0’s environment probes. The first step is to set your project’s rendering settings to use the Metal Forward Desktop renderer. Note, this is an expensive setting as you are using the full desktop renderer with a pre-pass, more complex lighting, etc. However, it’s needed in order to get the reflection support. The image below shows the settings I used for this post.
Your ARSessionConfig also needs to know to generate the environment probes. There are two options for when this is enabled. Automatic analyzes the scene and generates environment probes for you. I find it often generates more probes than I can reasonably use, but it is the easy way to get one created. The second method is to manually specify a bounding box and location. This is the most performant option as you have total control on how many get created and there are no updates for environment probes that you aren’t using. For this post, I have mine set to automatic as seen below.
In the previous post, I mentioned that I hoped to get a AR specific SkyLight implementation written. I did manage to get this into the 4.20 release. There’s nothing amazing about this code, but it does understand how the environment probes are updated over time. It notices when the probe texture has been updated and forces a regeneration of our underlying render data for lighting and reflections. This is all done on the rendering thread, so shouldn’t affect your gameplay unless you’re render thread bound.
The easiest way to get a ARSkyLight added to your level is to create a Blueprint subclass. There’s no changes that you need to do to the new class, but this makes it easy to drag into your levels. The image below shows the one I created for this post.
The last step is to delete the standard SkyLight that our template creates and add your newly created BP_ARSkyLight to the level instead.
Using the ARSkyLight
Because there can be multiple environment probes created for a scene, you need to tell the ARSkyLight which probe is the one that you are interested in. Since I am using the automatically created probes for this post, I added a simple Blueprint level script to periodically check for the creation of an environment probe. The level Blueprint grabs the first one it finds and sets that as the source for the ARSkyLight. After the probe is set in the ARSkyLight, it takes care of the rest of the work of checking for changes in the underlying cubemap data. My Blueprint is below.
I am sure you’re curious what the results look like. Given that these cubemaps are generated by some AI as it samples the scene from the camera data, I am quite impressed by the results. They aren’t perfect yet, but this is a 1.0 feature, so I expect them to improve over time. The image below is from an iPad Pro running iOS 12 beta 2.
Caveats
There are a couple of caveats that need to be mentioned. The first is that the Metal Forward Desktop renderer requires a pre-pass. On some devices that pre-pass results in z-fighting due to different math optimizations that the metal compiler uses. In the image above, the black splotches on the sphere are examples of this z-fighting. Metal 2.1 introduces [ position, invariant ] attributes which fixes the z-fighting. UE4 4.21 will support Metal 2.1 as a project setting. Unfortunately, there was not enough time to get that into the 4.20 release. However, the changes are available in our Dev-Rendering stream. The CL #s that I know of at this time that are needed are: 4173488 and 4175308. Remember that is an active development branch, so Caveat Emptor.
The second caveat is that I noticed that different devices sometimes generated slightly different resulting cubemaps (radar with sample images filed). Specifically, my iPhone X that I use for FaceAR testing would have the scene slightly rotated…sometimes. One benefit of the iPhone X is that the z-fighting prevalent on the iPad Pro is nonexistent. Below is an image from an iPhone X session of this project.
One more thing to note is that repeating pattern you see in both images happens whenever there is insufficient data for the AI to make a good texture. If you keep scanning the area, that repeating textures goes away as it fills in the data with good details.
Next Up
In the next installment, I’ll walk you through shared AR spaces using UE4’s built in networking.
Update
In 4.21, Metal 2.1 is supported so the z-fighting is gone.
