MaskOut — Heatmap

A Jetson based DeepStream application to identify areas of high risk through intuitive heat maps.

4 min readOct 12, 2020

Jetson eco-system is a great way of learning and developing AI applications on the edge. Nvidia recently announced Jetson AI Certification program which provides a curated list of video content and learning material to quickly get started with AI.

I have been a huge fan of community-based projects on these small but powerful devices. In this article, we will discuss the project I built as a part of the course for the assessment.

Inspiration

Hope can be a powerful force, especially in difficult times. Today, the world is facing the coronavirus crisis, a pandemic that has changed life for millions of people.

The most common project which started popping out recently

Face Mask Detection 😷

While solving the problem of localizing facemask is a challenging task the community has evolved to use transferable knowledge from similar domains to tackle emerging problems. But the problem doesn't end here, after localizing the target a lot can be done but requires a little bit of creativity and patience.

Idea

The principal mode by which people are infected with SARS-CoV-2 (the virus that causes COVID-19 ✳️) is through exposure to respiratory droplets carrying the infectious virus, which can be avoided to some extent by using facemasks.

Imagine if we could see through the time, It would be so easy to avoid paths where people have crossed recently without wearing a mask. Yes! ✊🏻, there is a way, that’s what this whole project revolves around.

Workflow

If a face without mask is detected then the region will be painted on the screen.
If more and more faces without mask cross through the same region the area will be painted with more intense colors.
If nobody has crossed for a very long time the color intensity will start fading out.

In other words, a Heat Map will be generated continuously representing regions where faces without any mask have been detected recently. Allowing us to see through the time.

The application is containerized and uses DeepStream as the backbone to run TensorRT optimized models for the maximum throughput. Built on top of deepstream-imagedata-multistream sample app.

How to setup

I have used Jetson Nano Devkit (2GB), you can set up on any other Jetson device.
Flash it with the latest JetPack.
Make sure nvidia-docker is installed on the device (it comes pre-installed with JetPack 4.3+).
clone the maskout repository.

$ git clone https://github.com/imneonizer/maskout.git
$ cd maskout

Build the docker container.

$ sudo docker build . -t maskout

Run the docker container.

$ export DISPLAY=:1
$ xhost +
$ sudo docker run --rm -it --gpus all \
    -v /home/$USER/videos:/videos \
    -e DISPLAY=$DISPLAY \
    -v /tmp/.X11-unix/:/tmp/.X11-unix \
    --net host --name maskout-ds-container \
    --hostname maskout \
    maskout bash

Once inside the container, you can run the DeepStream application using:

$ python3 maskout_app.py file:///videos/1.mp4
or
$ python3 maskout_app.py rtsp://<user>:<pass>@<camera-ip>

To view the Heatmap, open another terminal and execute below command.

$ sudo docker exec -it maskout-ds-container bash run_ui.sh

This will keep on fetching frames from the DeepStream container and serving to port 5000. You can keep this application running and restart the DeepStream application with different input streams. visit http://localhost:5000 to view heatmap.
Under high load, the RAM consumption went up to 1.2 GB + 600 MB used by Linux Kernel.

Demo

Conclusion

The region where more and more faces are detected in a very short period of time is painted in red, which essentially means those are the areas of high risk.
If a person doesn’t move, the painted regions grow and become intense in color and take more time to cool down.
If there is a cluster of faces then the intensity and radius of painted regions are much larger as compared to the heatmap generated for a single face.
All this valuable information allows us to visualize people moving around without facemask even when they are gone for a while.
This can be useful for monitoring people who are not following basic guidelines to help contain the coronavirus.

Resources

Author: Nitin Rai (AI Application Developer)