NVIDIA announced today that Notch is among the first partners to integrate their new AR SDK, which is part of the NVIDIA Broadcast Engine.
These GPU-accelerated SDKs deliver AI-powered features that Notch users can take advantage of in a soon-to-be-released software update. The first features coming to Notch are Face Tracking, Face Landmark Tracking, and the new Face Mesh following shortly behind.
Face and Landmark Tracking
This technique uses AI-based technology to detect human faces as well as determine the facial features including the contour, face shape, lips, eyes, and eyelids, using up to 126 key points.
Face Mesh
Using a 3D morphable model to create a 3D mesh representation of a human face with over 6000 polygons and a full 6 degrees of freedom head pose — all in real-time of course.
Face tracking has been a long-standing, and much-loved feature in Notch, but up to now, it ran entirely on the CPU. The overhead of which was very high, taking a whole frame to process, even when the source video feed is downsampled to 1/16 of its original pixel count. Additionally, the source video typically has to be first downloaded from the GPU to the CPU in order to be processed. The combination of these factors means that the CPU solution incurs 2–3 frames of latency.
Being able to leverage the Tensor Cores improves this significantly. Processing time is now considerably reduced even when the full resolution input video is used for tracking; and the video can be processed directly on GPU without being downloaded. This reduces the latency to be within the same frame as the source video, while also increasing the quality & stability of the tracking vs the old CPU solution.
That’s great news for live events or broadcast, where latency and responsiveness are key.
“The speed and quality of the new (and easy-to-integrate!) AR SDK face tracker is incredible. Through the use of dedicated Tensor Cores, Notch users have access to sub-frame accurate face tracking with high-resolution cameras in a live environment. We’re excited to see what this, as well as other NVIDIA Broadcast features, will bring to the live show and virtual production environment.”
— Matt Swoboda, Founder, and Director at Notch.
The new features will be available in a maintenance release of Notch Builder 0.9.23 within the next two weeks.
