Using GStreamer plugin to extract detected object from NVIDIA DeepStream

In our recent computer vision project, we are using Nvidia Deepstream

• To detect the object
• Moreover, we want to save the detected object image in particular folder for further analytics.

After so many hours of research, we are able to extract detected object from NVIDIA DeepStream. we are happy to share our experience with AI community, so it would be helpful to others.

What is GStreamer ?

GStreamer is a framework for creating streaming media applications. The GStreamer framework is designed to make it easy to write applications that handle audio or video or both. It isn’t restricted to audio and video, and can process any kind of data flow.The framework is based on plugins that will provide the various codec and other functionality. The plugins can be linked and arranged in a pipeline. This pipeline defines the flow of the data.The GStreamer core function is to provide a framework for plugins, data flow and media type handling/negotiation. It also provides an API to write applications using the various plugins.

https://gstreamer.freedesktop.org/documentation/application-development/introduction/gstreamer.html

Obtained from https://gstreamer.freedesktop.org

What is Deepstream ?

NVIDIA’s DeepStream SDK delivers a complete streaming analytics toolkit for AI-based multi-sensor processing, video and image understanding. Deepstream SDK heavily uses the open source multimedia handling library Gstreamer .The DeepStream SDK can be used to build end-to-end AI-powered applications to analyze video and sensor data. Some popular use cases are: retail analytics, parking management, managing logistics, robotics, optical inspection and managing operations.

Obtained from https://developer.nvidia.com/deepstream-sdk

For Deepstream Installation Details:

https://docs.nvidia.com/metropolis/deepstream/4.0.2/dev-guide/index.html

Deepstream Custom GStreamer Plugin:

In order to save the detected object, We need to create custom gstreamer plugin (gst-plugin), Deepstream SDK provide sample gst-plugin. The sources for the plugin are in sources/gst-plugins/gst-dsexample directory in the SDK. This plugin was written for GStreamer 1.14.1 but is compatible with newer versions of GStreamer. This plugin derives from the GstBaseTransform class:

https://gstreamer.freedesktop.org/data/doc/gstreamer/head/gstreamer-libs/html/GstBaseTransform.html

Description of the Sample Plugin: gst-dsexample

The GStreamer plugin itself is a standard in-place transform plugin. Because it does not generate new buffers but only adds / updates existing metadata, the plugin implements an in-place transform. Some of the code is standard GStreamer plugin boilerplate (e.g. plugin_init, class_init, instance_init). Other functions of interest are as follows:

GstBaseTransfrom Class Functions

start — Acquires resources, allocate memory, initialize example library.

stop — frees up resources and memory.

set_caps — Gets the capabilities of the video (i.e. resolution, color format, framerate) that flow through this element. Allocations / initializations that depend on input video format can be done here.

transform_ip — Finds the metadata of the primary detector.Call this function when the upstream element pushes a buffer.

sources/gst-plugins/gst-dsexample/gstdsexample.cpp directory in the SDK

We are using OpenCV library to save images in the directory, transform_ip functions invokes when the upstream element pushes a buffer. So we going to implement the image save logic in the transform_ip function. Please find code below.

In line 557 change the output directory path.

In line 639 change the class id to your preferred class id

To build custom gst-plugin:

cd /opt/nvidia/deepstream/deepstream-4.0/sources/gst-plugins/gst-dsexample/
make
sudo make install

If you get error during make command like below

update the CUDA version in Makefile.

To test the plugin functionality:

Change filesrc and change config path

gst-launch-1.0 filesrc location= <mp4-file> ! qtdemux ! h264parse ! nvv4l2decoder ! m.sink_0 nvstreammux name=m batch-size=1 width=1280 height=720 ! nvinfer config-file-path= <primary-detector-config> ! nvvideoconvert ! dsexample full-frame=0 <other-properties> ! nvdsosd ! nveglglessink

Run the following command in your terminal

gst-launch-1.0 filesrc location= /opt/nvidia/deepstream/deepstream-4.0/samples/streams/sample_720p.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! m.sink_0 nvstreammux name=m batch-size=1 width=1280 height=720 ! nvinfer config-file-path= /opt/nvidia/deepstream/deepstream-4.0/samples/configs/deepstream-app/config_infer_primary.txt ! nvvideoconvert ! dsexample full-frame=0 ! nvdsosd ! nveglglessink

Output

Thanks for reading.

If you have any suggestions/Questions kindly let us know in the comments section!