At GTC2021 today, NVIDIA announced a new update to their Maxine SDKs, the technology behind the NVIDIA Broadcast app. Amongst the new capabilities of the SDKs are skeletal body tracking and background removal.
These effects are powered in real-time by NVIDIA RTX™ GPUs and their dedicated Tensor Cores that accelerate AI. Notch is the first partner to integrate body tracking, and we’ve released a new update to Notch Builder 0.9.23 today which includes these features.
“As an industry-standard software used for live events graphics, Notch understands the importance of leveraging innovative technology to simplify the workflow and was quick to integrate these new capabilities into Notch Builder. With the addition of these new AI-driven camera effects, Notch continues to push the boundaries of what’s possible.”
— Gerardo Delgado Cabrera, Senior Product Manager at NVIDIA
Let’s take a look at how these features work in Notch!
Body Tracking
Using just a regular camera input, the body tracker allows a full human body to be tracked in 3D space in real-time, without any additional special hardware, using the Tensor Cores on NVIDIA RTX GPUs.
In Notch, the skeletal data from the body tracker can be targeted to a motion capture rig to control the animation of a 3D character. Individual joints can be used to drive interaction with other nodes and effects.
In this video, you can see body tracker consistently track our product specialist Armin as he moves around (including turning a full 360!). For good measure, we’ve also added the existing face tracker to show how both features can be creatively combined.
See our manual: http://manual.notch.one/0.9.23/en/topic/nodes-video-processing-feature-tracking-nvidia-ar-body-tracker
This technology has some interesting applications. Interactive or stage projects that previously required hardware tracking systems or depth cameras may need just a regular camera, simplifying the setup considerably.
We’ve also tested this feature for virtual production setups and it has huge potential. Being able to reliably track people in 3D using just the camera that’s already there to film them is practical and economical. Some use cases to explore would be everything from tracking hands in 3D, to generating virtual shadows and reflections from talent walking on an LED floor.
We believe these additions to an XR workflow, in particular, can help make it easier than ever to create believable XR content.
Virtual Background (Background Removal)
Being able to create video processing effects that isolate the subject from the background has been a much-requested feature ever since live cameras started being used on stage. With NVIDIA’s AI-driven Virtual Background this is now possible without any additional hardware.
Virtual Background also leverages RTX GPU Tensor Cores, using AI to work out a mask for the subject in real-time. This technology has been developed primarily for webcams, video calls, and streamers, where it works superbly, running at high resolution and with considerably more accuracy than other solutions we’ve seen. It handles difficult conditions well, even coping with dark clothing on dark backgrounds.
See our manual: http://manual.notch.one/0.9.23/en/topic/nodes-post-fx-image-processing-nvidia-virtual-background
While it was designed to work for a person in front of a webcam and works best on shots of the upper body + head, we thought we’d see how it did on a range of live stage camera footage, and it still performs well — giving it great potential for use on IMAG effects!
Update today
For Notch users, these new features are incredibly exciting. For example; interactive projects can now be much easier to set up, using just a regular camera. For stage productions, 3D avatars are now possible under ideal conditions without the need for specialist hardware tracking solutions.
These new features are available now to all Notch Builder users simply by installing the NVIDIA Broadcast redistributables (make sure to pick the version that matches your GPU and be sure to install both the Video FX and the AR SDK ones) and updating Notch Builder to the latest version (click the update notification in the software or go here to download the updated installer).
We look forward to seeing what you’ll create!
