RealityKit 911 — Fake Shadows for stationary models

Raytraced shadows are one of the most computationally intensive features in computer graphics. On the other hand, even DepthMap shadows (also known as Projective shadows) are anyway a heavy burden for CPU / GPU processing and memory consumption, albeit to a lesser extent.

What if we give up rendering shadows of stationary objects? What if we use Fake shadows for those objects? That’s a crucial moment for any AR developer, because using fake shadows saves battery life and reduces processing. All we have to do is to turn shadows rendering off and then bake our Fake shadows into a texture. Let’s do that.

Can NUKE help us making shadows?

The Foundry NUKE is a professional compositing toolkit that allows us create texture with baked shadows (whether they are stationary or moving). NUKE is based on nodes, like SceneKit. The difference is, NUKE has a graphical representation of a node graph that is extremely useful. Every node has its own properties that can be adjusted separately. You have to download a Non-commercial version of The Foundry NUKE to start our journey.

Watch these video tutorials to find out how to start using NUKE.

Our Augmented Reality scene is quite simple: two drummer models, as well as a textured floor and a textured wall (shadow catcher).

Scene has no shadows

In Xcode create a RealityKit project for macOS app with window size 1920x1080. Load a Reality Composer project containing 3 .usdz files inside separate scenes (drummers, floor and wall). Hide the wall and the floor, then assign a new Unlit shader with a default white color for drummers and take a screenshot.

Wall and floor are hidden

In Xcode run this macOS app allowing you to render just drummers:

import AppKit
import RealityKit
class ViewController: NSViewController {
    @IBOutlet var arView: ARView!
    override func awakeFromNib() {
        // Black background
        arView.environment.background = .color(.black)
        
        // Scene 1 – Drummers
        let drummersScene = try! Experience.loadDrummers()
        // Scene 2 – Wall
        let wallScene = try! Experience.loadWall()
        // Scene 3 – Floor
        let floorScene = try! Experience.loadFloor()
        let whiteTexture = UnlitMaterial()
        // I merged 2 USDZs in Maya, so their hierarchy is specific 
        let modelEntity = drummersScene.twoDrummers?.children[0]
                .children[0].children[0].children[0]
                .children[0].children[0].children[0] as! ModelEntity
        modelEntity.model!.materials[0] = whiteTexture
        arView.scene.anchors.append(drummersScene)
        // arView.scene.anchors.append(wallScene)
        // arView.scene.anchors.append(floorScene)
    }
}

Screenshot (Cmd–Shift–4) should look like this.

White UnlitMaterial

Compositing in NUKE

Load a screenshot of white material into The Foundry NUKE using Read node, than invert it applying an Invert node. For calling any node use Tab hotkey (keep your mouse pointer over Node Graph while pressing Tab).

NUKE script processing our fake shadows

Feed original image into A input of a Copy node, then inverted image into B input of Copy node. Consider that B-stream in NUKE is a general channels’ stream (A-streams are optional).

In Properties pane copy Red channel data into Alpha channel data. After that premultiply RGB of a second image (B-stream) by Alpha channel of a first image (A-stream) using Premult node.

Channel reordering and premultiplication ops

Transform node helps you translate your shadow along X and Y axis.

With Grade node we control a transparency of our fake shadows

Now we can see the shadow over a wall texture. But our shadow is opaque, isn’t it?

Opaque shadows

We can easily fix it decreasing a multiply parameter in a Grade node. Semi-transparent shadows look much more pleasant. Pay particular attention to the parameter channels=rgba in Grade node.

Transparent shadows

As you can see from the comp, we composited a shadow over wall’s texture using a Merge node with Over operation.

NUKE has 30 compositing operations. Over is a default one. Over adds RGBA channels of a foreground image with RGB channels of a background image, that multiplied by inversion of a foreground’s Alpha.

RGB1 + RGB2 * (1.0 – A1)

Have you noticed that our shadow has sharp edges?

Let’s blur them using a Blur node with a corresponding parameter size=30.

Now it looks more believable.

Blur is optional. Use blur only when a model is far from a wall.

Comparison time!
Here are opaque black shadows. Look unrealistic, right?

And these are nice semi-transparent shadows.

To render the new wall’s texture on your disk use Write node. Remember that Non-commercial version of NUKE supports a rendering of images with max resolution 1920x1080. Save new texture with a name fake.jpg.

Let’s assign the new texture to a wall object (do not forget to put fake.jpg file into Xcode’s Assets.xcassets directory). Here’s a Swift code for iOS app:

import SwiftUI
import RealityKit
struct ContentView: View {
    var body: some View {
        return ARViewContainer()
            .edgesIgnoringSafeArea(.all)
    }
}
struct ARViewContainer: UIViewRepresentable {
    func makeUIView(context: Context) -> ARView {
        let arView = ARView(frame: .zero)
        let wallScene = try! Experience.loadWall()
        var fakeShadowsTexture = SimpleMaterial()
        fakeShadowsTexture.baseColor = .texture(try! .load(
                                                     named: "fake"))
        let wallModel = wallScene.wall!.children[0] as! ModelEntity
      
        wallModel.model!.materials = [fakeShadowsTexture]
        arView.scene.anchors.append(wallScene)
        return arView
    }
    func updateUIView(_ uiView: ARView, context: Context) { }
}

The same way you’re capable of creating fake shadows for furniture, trees, buildings and other stationary AR objects.

That’s all for now.

If this post is useful for you, please press the Clap button and hold it.

¡Hasta la vista!