Realistic reflections and environment textures in ARKit

Reflection is an essential part of our everyday environment. Naturally, for a realistic augmented reality experience, virtual objects should contain it too. Starting from ARKit 2.0 developers can use automatically generated environment textures for realistic reflections.

“In 3D asset rendering, environment textures are the basis for image-based lighting algorithms where surfaces can realistically reflect light from their surroundings. ARKit can generate environment textures during an AR session using camera imagery, allowing SceneKit or a custom rendering engine to provide realistic image-based lighting for virtual objects in your AR experience.”

— Apple doc.

Environment texture gathers scene information in the form of a cube map; this texture later is used as a reflection probe. Because it’s impossible to have a fully complete cube map in realistic use cases(basically, to complete cube map the user has to complete 360-degree panorama), ARKit builds up missing parts of the cube map texture with machine learning.

Let’s take a look at a small demo of the reflective object.

As you can see, the sphere reflects real world objects that are placed on the table, which in my opinion is ultimately awesome 🤩

For this demo, I created a sphere and a reflective material for it.

I created a PBR material with a metalness = 1 and roughness = 0, here is a cheatsheet of how different materials look like depending on these values.

Metalness and roughness are two of the basic components for any physical based rendered material. You can check this article for more info about PBR lighting in SceneKit.

To enable environment texture map generation for an AR session all we have to do is set the environmentTexturingproperty of ARSession configuration:

From this point, ARKit will automatically create AREnvironmentProbeAnchor objects. AREnvironmentProbeAnchor provides environmental lighting information for a specific area of space.

AREnvironmentProbeAnchor is a subclass of ARAnchor, which means that they have the same lifecycle. Its extent property represents the area around the anchor’s position which can be later used when projecting its texture for parallax correction. The texture is represented by environmentTexture property.

Currently, the environment texturing supports two settings:

• For EnvironmentTexturing.automatic ARKit automatically creates, positions, and adds AREnvironmentProbeAnchor objects to the session.
• ForEnvironmentTexturing.manual you identify points in the scene for which you want light probe texture maps by creating AREnvironmentProbeAnchor objects and adding them to the session.

If configuration.environmentTexturing is set to .automatic you don’t have to work with AREnvironmentProbeAnchor directly to achieve great results.

That’s it about the environment texturing and reflections in ARKit. You can find the source code of the reflective sphere that was shown earlier here. I hope you learned something new from this small post!