How to AI-Upscale Your Video Collection with Nvidia Maxine

Do you have a bunch of low-res and poorly encoded video files saved somewhere? I certainly do. Wouldn’t it be great to make use of your GPU and upscale them with AI?

Well, that’s what I thought, and I even went ahead and upscaled some of them with existing tools such as Waifu2x (through a GUI called Waifu2x-Extension-GUI) and Topaz Video AI. The problem is that the process takes absolutely ages — it took me ~5 hours to 4x upscale a ~1 hour video on my RTX 3070.

However, recently Nvidia released its Maxine SDK that includes a number of audio/video/AR processing features, and one of them is SuperResolution! (aka AI upscaling)

The main advantage is that on my machine is more than 5 times faster than Waifu2! Your mileage (and video files) may vary of course.

Here are a couple of results.

If you’re looking at the above examples on a small screen, the quality might not look very obvious. But on a TV or a monitor full-screened, the difference is night and day.

All the original compression artifacts are gone as well. The resulting images are much cleaner. Pay particular attention to the fruit and the furniture.

Disclaimer: Although they are much better than the original, the end results are not perfect. There are parts (such as skin) that Waifu2x & Topaz Video AI do better. But the speed trade-off is well worth it.

Here is how you can do it. The instructions below are for Windows, but this works on Linux too. And I assume you already have basic knowledge of using command line in Windows.

Of course, this works on Nvidia GPUs only. It supports all GPUs from the 2000-series and up. Sorry 1000-series owner, this needs Tensor-cores to work.

Before you begin, make sure you have a very recent version of Nvidia driver installed (511.65 or later).

Firstly, we need to head over to Nvidia NGC to download the Maxine SDK. We only need the Video Effects part of it to do SuperResolution. The Audio Effects SDK and Augmented Reality SDK are not needed.

Maxine Windows Video Effects SDK | NVIDIA NGC

You need an account to download it, but it’s free to register.

Note that although you can find the SDK on GitHub, but it’s better to download the full SDK package through NGC instead, because it comes with all the required libraries and AI models.

Once downloaded, extract it somewhere (e.g. D:\MAXINE-VFX-SDK), preferably on a fast storage device.

Now open up a command prompt, and change to the VideoEffects directory (e.g. D:\MAXINE-VFX-SDK\samples\VideoEffectsApp).

Next, we need to make sure the paths are pointing to the right directories so that the app knows where to find the libraries.

SET PATH=%PATH%;..\external\opencv\bin;..\..\bin;

Then simply run the below command.

VideoEffectsApp.exe --progress --effect=SuperRes --mode=0 --model_dir=..\..\bin\models --in_file=C:\vids\in.mp4 --resolution=1440 --out_file=C:\vids\out.mp4

But before you run it, there is a couple of things to change with the above command.

The “mode” parameter can be either 0 or 1. The 0 option removes artifacts and is generally a better option to use. Using 1 will apply very heavy sharpening and is only recommended for extremely poor source videos.

The “in_file” and “out_file” parameters are the input and output video files.

The “resolution” parameter must be 1.3333x, 1.5x, 2x, 3x, or 4x times the vertical resolution of the source video. For example, if the source video is 480x360 and you want to 4x upscale it, then specify 1440 here.

Then we wait.

The amount of time it takes depends on your GPU, storage, source video, and the parameters you specify. With the ~1 hour video I mentioned before, it took me around 45 mins. Much faster than Waifu2x.

Once done, make sure you have both the original and the upscaled videos open side by side and compare the difference!

One thing to note is that the video file assembled by the VideoEffectsApp.exe tool may have its frame rate set incorrectly. This can be easily resolved by ffmpeg without re-encoding.

For example, if the source video is 29.97fps, but VideoEffectsApp.exe made it 29fps, we can convert the frame rate back like so:

ffmpeg -itsscale 0.9676343009676343 -i C:\vids\out.mp4 -c copy C:\vids\out-29.97.mp4

The number “0.9676343009676343” above is the ratio we want to scale to. In this case it’s 29 divided by 29.97.

In addition, this tool doesn’t process audio. But again, ffmpeg comes to our rescue. What we want to do, is to take the audio track of the source video file, and add it to our upscaled video file. And we do it like so:

ffmpeg -i C:\vids\out-29.97.mp4 -i C:\vids\in.mp4 -c copy -map 0:v:0 -map 1:a:0 -shortest final.mp4

In this command, we take both video files as inputs using the “-i” argument. But we only use the video stream of the first input specified as “0:v:0”, and only the audio stream of the second input specified as “1:a:0”.

“-c copy” means no re-encoding.

“-shortest” means that if there is any difference in duration between the video stream and the audio stream, use the shortest one.

Side note. In the SDK, you might see another effect called “Upscale”. This is not what we want. We want “SuperResolution”.

The difference between these 2 is that “Upscale” is super-fast, as in faster than real-time, but the quality is nowhere near “SuperResolution”. “Upscale” is good for use cases such as upscaling a webcam feed or a CCTV stream.

There is also another effect called “ArtifactReduction”. We don’t need to explicitly use it as “SuperResolution” already includes it.

Hope you enjoyed it. Now go ahead and upscale all the old low-res video you have lying around!

Any questions? Let me know.

For advanced usage, see Nvidia’s official documentation of the Video Effects SDK here — https://docs.nvidia.com/deeplearning/maxine/vfx-sdk-system-guide/index.html#videoeffects-cli-kbc-intro

Hopefully soon someone will come along and make a nice GUI for it.