RealityKit 911 — Fake Shadows for stationary models
Raytraced shadows are one of the most computationally intensive features in computer graphics. On the other hand, even DepthMap shadows (also known as Projective shadows) are anyway a heavy burden for CPU / GPU processing and memory consumption, albeit to a lesser extent.
What if we give up rendering shadows of stationary objects? What if we use Fake shadows for those objects? That’s a crucial moment for any AR developer, because using fake shadows saves battery life and reduces processing. All we have to do is to turn shadows rendering off and then bake our Fake shadows into a texture. Let’s do that.
Can NUKE help us making shadows?
The Foundry NUKE is a professional compositing toolkit that allows us create texture with baked shadows (whether they are stationary or moving). NUKE is based on nodes, like SceneKit. The difference is, NUKE has a graphical representation of a node graph that is extremely useful. Every node has its own properties that can be adjusted separately. You have to download a Non-commercial version of The Foundry NUKE to start our journey.
Watch these video tutorials to find out how to start using NUKE.
Our Augmented Reality scene is quite simple: two drummer models, as well as a textured floor and a textured wall (shadow catcher).
In Xcode create a RealityKit project for macOS app with window size 1920x1080. Load a Reality Composer project containing 3 .usdz files inside separate scenes (drummers, floor and wall). Hide the wall and the floor, then assign a new Unlit shader with a default white color for drummers and take a screenshot.
In Xcode run this macOS app allowing you to render just drummers:
import AppKit
import RealityKitclass ViewController: NSViewController { @IBOutlet var arView: ARView! override func awakeFromNib() { // Black background
arView.environment.background = .color(.black)
// Scene 1 – Drummers
let drummersScene = try! Experience.loadDrummers() // Scene 2 – Wall
let wallScene = try! Experience.loadWall() // Scene 3 – Floor
let floorScene = try! Experience.loadFloor() let whiteTexture = UnlitMaterial() // I merged 2 USDZs in Maya, so their hierarchy is specific
let modelEntity = drummersScene.twoDrummers?.children[0]
.children[0].children[0].children[0]
.children[0].children[0].children[0] as! ModelEntity modelEntity.model!.materials[0] = whiteTexture arView.scene.anchors.append(drummersScene) // arView.scene.anchors.append(wallScene)
// arView.scene.anchors.append(floorScene)
}
}
Screenshot (Cmd–Shift–4) should look like this.
Compositing in NUKE
Load a screenshot of white material into The Foundry NUKE using Read node, than invert it applying an Invert node. For calling any node use Tab hotkey (keep your mouse pointer over Node Graph while pressing Tab).
Feed original image into A input of a Copy node, then inverted image into B input of Copy node. Consider that B-stream in NUKE is a general channels’ stream (A-streams are optional).
In Properties pane copy Red channel data into Alpha channel data. After that premultiply RGB of a second image (B-stream) by Alpha channel of a first image (A-stream) using Premult node.
Transform node helps you translate your shadow along X and Y axis.
Now we can see the shadow over a wall texture. But our shadow is opaque, isn’t it?
We can easily fix it decreasing a multiply parameter in a Grade node. Semi-transparent shadows look much more pleasant. Pay particular attention to the parameter channels=rgba in Grade node.
As you can see from the comp, we composited a shadow over wall’s texture using a Merge node with Over operation.
NUKE has 30 compositing operations. Over is a default one. Over adds RGBA channels of a foreground image with RGB channels of a background image, that multiplied by inversion of a foreground’s Alpha.
RGB1 + RGB2 * (1.0 – A1)
Have you noticed that our shadow has sharp edges?
Let’s blur them using a Blur node with a corresponding parameter size=30.
Now it looks more believable.
Comparison time!
Here are opaque black shadows. Look unrealistic, right?
And these are nice semi-transparent shadows.
To render the new wall’s texture on your disk use Write node. Remember that Non-commercial version of NUKE supports a rendering of images with max resolution 1920x1080. Save new texture with a name fake.jpg.
Let’s assign the new texture to a wall object (do not forget to put fake.jpg file into Xcode’s Assets.xcassets directory). Here’s a Swift code for iOS app:
import SwiftUI
import RealityKitstruct ContentView: View {
var body: some View {
return ARViewContainer()
.edgesIgnoringSafeArea(.all)
}
}struct ARViewContainer: UIViewRepresentable { func makeUIView(context: Context) -> ARView { let arView = ARView(frame: .zero) let wallScene = try! Experience.loadWall() var fakeShadowsTexture = SimpleMaterial() fakeShadowsTexture.baseColor = .texture(try! .load(
named: "fake")) let wallModel = wallScene.wall!.children[0] as! ModelEntity
wallModel.model!.materials = [fakeShadowsTexture] arView.scene.anchors.append(wallScene) return arView
} func updateUIView(_ uiView: ARView, context: Context) { }
}
The same way you’re capable of creating fake shadows for furniture, trees, buildings and other stationary AR objects.
That’s all for now.
If this post is useful for you, please press the Clap button and hold it.
¡Hasta la vista!