Video streaming is everywhere and they have become the most universal way the internet is communicating. With the advent of high-speed internet, media generation has been consistently growing year over year. Toolchain supporting video streaming/encoding/decoding has been slowly catching up with its growth. Video encoding and decoding processes are computationally expensive and to support it, NVIDIA has come up with their specialized hardware encoders and decoders in their GPU lineup.
Traditionally encoding and decoding been done on CPU which is getting outdated. Currently, there is a better way to do it using NVIDIA GPU which is super-efficient. NVIDIA has come up with NVIDIA VIDEO CODEC SDK a hardware-accelerated encoder/decoder library for video encoding, decoding.
It supports the processing of many streams compared to the current standard using CPU (x264). The SDK consists of two hardware acceleration interfaces: NVENCODE & NVDECODE. Below Fig 2 & Fig 3 shows the performance boost of using the SDK over current standards (x264) for video encoding and decoding and going forward it will not be a CPU intensive task anymore.
NVENCODE API for video encode acceleration (also referred to as NVENC)
NVIDIA GPU starting from Kepler, have specialized hardware which provides fully accelerated hardware-based video encoding and is independent of graphics performance. NVENC makes it possible to:
- Encode and stream games and applications at high quality and ultra-low latency without utilizing CPU
- Encode at very high quality for archiving, OTT streaming, web videos
- Encode with ultra-low power consumption per stream (Watts/stream)
NVDECODE API for video decode acceleration (formerly called NVCUVID API — also referred to as NVDEC)
- Faster decoding which makes it suitable to be used for transcoding applications
- Video decoding for several popular codecs like MPEG-2, VC-1, H.264 (AVCHD), H.265 (HEVC), VP8, VP9
- Supports both Windows and Linux platform
Before proceeding further, one needs to have some knowledge about video encoding/decoding tools like FFmpeg or GStreamer (we will be covering GStreamer).
GStreamer is a powerful streaming media framework that supports modular and pipeline based workflows. It has a wide variety of plugin ecosystem. Using GStreamer pipelines we can create a complex streaming workflow.
Now that we have a high-level understanding of NVENC/NVDEC and GStreamer, let’s see how do we install and setup the SDK with GStreamer. GStreamer uses a meson and ninja build system for its builds.
GStreamer has four main modules GStreamer, base, good, bad, ugly and each has its own set of dependencies. I’ve consolidated the dependencies for all of them below. Let’s install the required dependencies for GStreamer.
# Ubuntu 18.04 & Tesla T4 compatible driver and CUDA# gstreamer
sudo apt-get install automake unzip python3-pip
sudo apt-get install cmake git autoconf autopoint
sudo apt-get install gtk-doc-tools glib-2.0
sudo apt-get install bison flex libglib2.0-dev
sudo apt-get install libunwind-dev libdw-dev libgtk-3-dev
sudo apt-get install libx11-dev xorg-dev libglu1-mesa-dev freeglut3-dev libglew1.5 libglew1.5-dev libglu1-mesa libglu1-mesa-dev libgl1-mesa-glx libgl1-mesa-dev libtool libreadline-dev# gst-plugins-base
sudo apt-get install libtheora-bin libtheora-dev libtheora-doc
sudo apt-get install libvorbis-dev
sudo apt-get install libcdparanoia-dev
sudo apt-get install alsa-base alsa-tools
sudo apt-get install libasound2-dev
sudo apt-get install libopus-dev libvisual-0.4-dev libpango1.0-dev# gst-plugins-good
sudo apt-get install libwavpack-dev libspeex-dev qt libjack-sdk libjpeg-dev libdv-dev libsoup2.4-dev qtdeclarative5-dev\
sudo apt-get install libcairo-dev# gst-plugins-bad
sudo apt-get install x265 x264# gst-plugins-ugly
sudo apt-get install libx264-dev# gst-libav
sudo apt-get install yasm nasm libbz2-dev liblzma-dev
Now, we are ready to build GStreamer from the source using the command given below, it uses the master branch but it can be customized to pull specific releases.
$ git clone https://gitlab.freedesktop.org/gstreamer/gst-build.git;
$ cd gst-build;
$ source $HOME/.bashrc;
$ $HOME/.local/bin/meson build/; # meson setup — wipe build/# NVIDIA GPU above kepler is required for below command
$ $HOME/.local/bin/meson configure build/ -Dgst-plugins-bad:nvcodec=enabled;$ ninja -C build/;
Now, we can start using the SDK for video encoding and decoding. Let’s see some of the commands used for video encoding and decoding using NVENC/NVDEC with GStreamer.
Pipelines using NVENC/NVDEC
RTSP to resized RTSP stream
$ gst-launch-1.0 rtspsrc location=$RTSPURL ! rtph264depay ! h264parse ! nvh264dec ! videoconvert ! videoscale ! video/x-raw,width=640,height=480 ! nvh264enc ! h264parse ! rtph264pay name=pay0 pt=96
RTSP to flv
$ gst-launch-1.0 rtspsrc location=$RTSPURL ! rtpbin ! rtph264depay ! h264parse ! nvh264dec ! video/x-raw ! nvh264enc ! h264parse ! flvmux ! filesink location=xyz.flv
GStreamer supports a variety of pipelines for streaming both video and audio media. To build more pipelines using GStreamer check out the references section. The above setup can be easily dockerized using nvidia-docker and running the docker container with the pipeline string as runtime argument will open up doors for cool projects. There are not many resources online which provide installation steps for NVIDIA Video codec SDK and GStreamer. I hope this will help a lot of people.
I’ll also be writing about Deepstream SDK from NVIDIA, about how to use their examples and apply some cool deep learning algorithms on videos.
I’m sure there might be some mistakes in the post, please feel free to shout out.