In Notch release 0.9.23, we introduce an interesting new feature: AI-based upscaling of images and video. We have taken advantage of new technology created by NVIDIA to use machine learning, accelerated by the Tensor Coresin the latest NVIDIA GPUs (RTX onwards), to be able to upscale an image by a factor of 2x, 4x or 8x.
In principle, this works by pre-training an AI on a data set of images, each one rendered at different resolutions, so the AI learns how to recreate the higher resolution version from the lower resolution — filling in the gaps accordingly. This means that sharp lines and details remain sharp through the upscaling process rather than blurring out. You can see the difference in this comparison which shows a 960x540 image, scaled up using regular scaling and the NVIDIA NGX AI upscaler:
The potential of a really good upscaler is obviously enormous. Have a ton of old stock content or old show content created at a lower resolution that you now need to display on a super high-resolution LED screen, that you can’t re-render? AI upscale it instead. Trying to produce content for a 16k widescreen at super high quality (perhaps using the new path tracer in Notch?) and it’s taking too long for your deadline? Render it at 8k or even 4k and upscale it to 16k.
We’ve integrated the AI Upscaler in two separate places in Notch: there are now options to use it on video export, and therefore for video transcodes via the render queue; and there’s a new AI Upscaler video node for more general use, which also exposes the two quality options for the upscaler (“fast” and “high quality”). They require the very latest NVIDIA drivers and an RTX-range NVIDIA GPU to work. Upscaling a 1080p image with the high-quality denoiser takes a couple of seconds, so this is definitely a process for export rather than real-time.
This application of machine learning to the content creation process, like AI denoising for path tracers, while full of promise, is still in its relative infancy. The quality of results depends heavily on the way the AI was trained in the first place; if your content is sufficiently similar to the training data set, it’s likely to do better. Our tests have shown that thin lines and text fare particularly well compared to more traditional upscaling methods, remaining sharp after upscaling with the AI filling in the details. Noise does less well, coming out as slightly blurrier noise.
This technology and the features it unlocks within Notch is only going to improve with time. Larger training data sets and improvements in techniques will surely bring quality improvements, and we can only expect the next range of NVIDIA GPUs to come with even faster Tensor Cores. Right now, it’s definitely worth checking out in Notch and seeing how it can apply to your workflow.
