The Top 9 Most In-Demand Features for RealityKit 3.0

Published in

Geek Culture

4 min readJun 19, 2021

RealityKit is great, nobody argues. However, the framework isn’t perfect yet. RealityKit 2.0 does not have all the features that we’re used to in SceneKit. Therefore, I decided to write about nine important features that I would like to see in the next major release of the framework. I believe that Apple’s RealityKit team members will read this story.

1. Particle system (solved in 2023)

Native particle system is an essential element of any 3D package or game engine. I expect to see an advanced particle system (not the same as in SceneKit, but rather the same as in Maya or Houdini). At least, I want to see such features as 3D particles’ emitter, lighting particles, particles’ collisions and the ability to inherit a color from texture.

2. allowsCameraContol

Many SceneKit developers highly appreciate this property. It allows you dolly, truck and/or pedestal your active camera. If you forgot to disable it in AR app it may totally destroy your AR experience. Nonetheless, it is very difficult to live without it.

sceneView.allowsCameraControl = true

Even though this feature is not implemented in RealityKit 2.0, you can try to implement it yourself. Read this post for a hint.

3. shadowOnly

In SceneKit we were accustomed to Lighting Model named shadowOnly. In other words it’s shadow catcher. It’s hard to overestimate its benefits.

let material = SCNMaterial()
material.lightingModel = .shadowOnly

I know about Reality Composer’s floor, capable of catching a shadow, but it would be better to have a shader.

4. Pivot Point

Why there’s no pivoting in RealityKit 2.0 and Reality Composer 1.5? You have to agree that it’s inconvenient, right? SceneKit has it:

let node = SCNNode()
node.simdPivot.columns.3.y = 0.25

Many of you may argue with me — just nest a model in a group (Entity), and you’ll get a way to control the pivot. But you must admit, with a large number of objects in the scene, this is very inconvenient way to control object’s transformations.

5. Physics accuracy

One of the main wishes of game developers is an engine for accurate physics simulations. The current version of RealityKit is very far from ideal in terms of dynamics compared, for example, to Unity. The number of parameters for controlling physics in RealityKit is poor, and collisions do not work out well.

6. Xcode AR Simulator (solved in 2023)

ARKit/RealityKit developers urgently need an AR Simulator similar to ARCore Emulator in Android Studio. No comments.

A positive shift in this direction has already been outlined, developers can test their apps in Swift Playgrounds on iPadOS. Also we’ve got a visionOS simulator in Xcode 15.

7. Capturing Texture during Scene Reconstruction

Even though a new API called Object Capture was announced at WWDC 2021, we still need an API to instantly capture textures using LiDAR Scanner.

8. Depth Map Visualization

It would be great to simultaneously render final composition of Depth Data for video part and for models out-of-the-box (without manual input of the Metal code).

9. Play all animations from the list

Currently, RealityKit developers can play only the first element in the collection of animations associated with the entity. It is important to resolve this situation as soon as possible.

let entity = ModelEntity()
let animation = entity.availableAnimations[2]       // error
entity.playAnimation(animation)

Active Scheme vs iOS Simulator (solved in 2022)

In Xcode 13.2 there’s no such issue anymore. But in previous versions of Xcode, all developers saw this annoying message many times.

The problem was trivial — chosen iOS Simulator scheme didn’t support session instance property. I’m glad it’s fixed at last.

Another options

Of course, the list of desired options could be longer — there could be such features as morphing, skinning, camera constraints, physics constraints, doubleSided material (although the double-sided shader can currently be activated via pythonic USDZ schemas), etc, but I decided to limit the list to the most important ones.

That’s all for now.

If this post is useful for you, please press the Clap button and hold it.

¡Hasta la vista!