RealityKit 911 — Custom Material with Metal shaders

RealityKit’s programmable materials, called CustomMaterials, are available on iOS 15 and later (in visionOS use ShaderGraphMaterial). This type of materials uses two special Metal functions to manage model’s vertex_position and pixel_data. Apple Metal is native low-level API for interacting with 3D graphics hardware. As a developer, you are able to create chunks of code written in Metal Shader Language (MSL), which is actually a cut-down version of C++, that dramatically changes the look and feel of any AR/VR material.

According to Apple definition:

Custom materials allow you to leverage RealityKit’s existing shader pipeline to render physically based or unlit materials that support custom Metal shader functions. These functions modify how RealityKit renders an entity. Custom materials support two different types of custom Metal shader functions: geometry modifiers and surface shaders.

RealityKit’s first type of Metal shader is a built-in fragment shader that fires once for every screen pixel. RealityKit’s fragment shader calls your surface shader, meaning that surface shaders are also called once for each of the entity’s fragments (pixels).

The other type of Metal shader that RealityKit uses is the vertex shader. Vertex shaders fire once for every vertex in the entity. If you supply a geometry modifier when creating a custom material, RealityKit’s vertex shader calls it. Geometry modifiers fire once for every vertex in the entity.

Well, with the theory everything is clear, let’s move on to practice. We have a default Reality Composer’s scene with notorious steel box.

Steel box in Reality Composer’s default scene

Shaders.metal

Starting with Xcode 11, Metal development support has been added to the Simulator app, so we don’t have to run our RealityKit app on the real iOS device every time a tiny change is made.

First of all, we have to create a surface shader in Metal. The surface shader, the same way as geometry modifier, must be inside function with [[visible]] attribute.

The meaning of the “basicShader” function is simple: I want the color of the texture to constantly change over time. To do this, I used the trigonometric functions sin() and cos() which both form a sinusoid, but with a shift of 90 degrees.

Sinusoidal waveforms with negative and positive half waves

Examine the code. For the Red and Green channels, I took the absolute values ​​returned by these functions (just positive values). Blue channel’s value is a result of simple math operations.

#include <metal_stdlib>
#include <RealityKit/RealityKit.h>
using namespace metal;
[[visible]]
void basicShader(realitykit::surface_parameters shader)
{
    realitykit::surface::surface_properties ssh = shader.surface();
    float time = shader.uniforms().time();
    half r = abs(cos(time * 2.5));
    half g = abs(sin(time * 5.0));
    half b = abs(r - (g * r));
    ssh.set_base_color(half3(r, g, b));
    ssh.set_metallic(half(1.0));
    ssh.set_roughness(half(0.0));
    ssh.set_clearcoat(half(1.0));
    ssh.set_clearcoat_roughness(half(0.0));
}

Look at the graph to find out how the math affected sinusoids.

Resulted graphs for Red, Green and Blue channels

You can experiment with any trigonometric functions at www.desmos.com.

Second Metal function is about positioning. The general idea behind the math here is to make vertices change their position, so we can move a part of the model or the whole model. Cool?

The code below is geometry modifier and all parameters there are self-explanatory because we utilize the same principles as in previous function.

The only thing you have to remember is that the Metal’s Normalized Device Coordinates (NDC) go from negative 1.0 to positive 1.0 (for X and Y axis), where the origin (0.0, 0.0) is a model’s center, and for Z axis, the range is 0.0 to +1.0 (so, positive Z-values go away from the camera).

Thus, the NDC’s center is (0.0, 0.0, 0.5).

Left-handed NDC coordinate system of Apple Metal

I used offset property for applying an elliptical orbit for cube model.

[[visible]]
void basicModifier(realitykit::geometry_parameters modifier)
{
    float3 pose = modifier.geometry().model_position();
    float time = modifier.uniforms().time();
    float speed = 1.5f;
    float amplitude = 0.1f;
    float offset = 0.05f;
    float cosTime = (cos(time * speed)) * amplitude;
    float sinTime = (sin(time * speed)) * (amplitude + offset);
    modifier.geometry().set_model_position_offset(
        float3(cosTime, sinTime, pose.z + 0.1)
    );
}

ViewController.swift

Swift part isn’t complicated. At first we declare an instance of MTLDevice. It can be considered as our direct connection to the GPU. And then we declare an instance of Metal library where both shader functions live. The other part of code is familiar to you because it’s a regular assignment of RealityKit’s material. By the way, I implemented perpetum rotation using physics, not animation.

import UIKit
import RealityKit
import Metal
class ViewController: UIViewController {
    @IBOutlet var arView: ARView!
    override func viewDidLoad() {
        super.viewDidLoad()
        self.arView.environment.background = .color(.darkGray)
        let boxScene = try! Experience.loadBox()
        boxScene.children[0].scale *= 5.0
        let box = boxScene.steelBox!.children[0] as! ModelEntity
        // Metal device and library
        let device = MTLCreateSystemDefaultDevice()
        guard let defaultLibrary = device!.makeDefaultLibrary()
        else { return }
        let shader = CustomMaterial.SurfaceShader(
                                           named: "basicShader",
                                              in: defaultLibrary)
        let modifier = CustomMaterial.GeometryModifier(
                                           named: "basicModifier",
                                              in: defaultLibrary)
        do {
            box.model?.materials[0] = try CustomMaterial(
                                            surfaceShader: shader,
                                         geometryModifier: modifier,
                                            lightingModel: .lit)
        } catch {
            print("Can't find Custom material")
        }
        self.arView.scene.anchors.append(boxScene)
        // Rotation about Y axis 
        box.components[PhysicsBodyComponent.self] = .init()
        box.components[PhysicsMotionComponent.self] = .init()
        box.physicsBody?.massProperties.mass = 0.0
        box.physicsMotion?.angularVelocity.y = 1.0
        box.generateCollisionShapes(recursive: true)
    }
}

Result

What we’ve got? We have got animation loop of three colors — contemporary green, mustard gold and noble purple on a metallic surface with a clear coating. And we’ve got a parallelepiped’s movement along its elliptical orbit.

That’s all for now.

If this post is useful for you, please press the Clap button and hold it.

¡Hasta la vista!