Overview
As a response to the coronavirus (COVID-19) pandemic, Neuralet has released an open-source application that helps people practice physical distancing rules in different places such as retail spaces, construction sites, factories, healthcare facilities, etc. This guide provides a technical overview of the approach to smart social distancing. Familiarity with deep learning computer vision methods and Jetson Nano is recommended to follow this guide.
Problem description
Social distancing (also physical distancing) is one approach to control and prevent the rate of infection of different contagious diseases e.g. coronavirus and COVID-19, which is currently causing one of the largest pandemics in human history. Current solutions to practice social distancing are either not practical or not viable in the long-term. Please visit this blog post to read more about why current approaches fail, and why we need a more robust approach.
Our solution
Our approach uses artificial intelligence and edge AI devices such as Jetson Nano to track people in different environments and measure adherence to social distancing guidelines, and can give notifications each time social distancing rules are violated. System design is illustrated in figure 1 and a high-level schema of the approach and the modules are depicted in figure 3.
Our solution can be modified to work in real-time by processing a USB or Camera Serial Input. The demo runs on a video that can be provided via the configuration file.
Setup
Neuralet GitHub repository
Neuralet is an open-source platform for edge deep learning models on GPU, TPU, and more. The source code of the Smart Social Distancing application is available on GitHub under the application directory. You can find the setup guide in the project Readme.
git clone https://github.com/neuralet/neuralet.git
cd neuralet/applications/smart-distancing/Jetson Nano
You will need an Nvidia Jetson Nano device (figure 2) to run this tutorial. We are making this application available on other devices, please find more details about compatible devices here. Jetson Nano is NVIDIA’s small and high-performance edge product that enables you to run a wide range of machine learning and deep learning applications.
To run this application you require to set up your Jetson Nano using JetPack 4.3 and have Nvidia-docker runtime available.
Object Detection Model
The core of this application uses Convolutional Neural Networks to detect pedestrians. We are working on adding more models, but as of right now, we use the pre-trained SSD MobileNet V2 which is trained on the MS COCO dataset for the proof of concept. We extract the “person” class of this model as the pedestrian detector.
We are running TensorRT runtime to get the best performance on the Jetson Nano. The model can be swapped easily in libs/core.py. You can implement other models by following our implementation example on libs/detectors/jetson.
Codebase Architecture:
In this section, we discuss our codebase structure and explain the main idea of each module. We highly recommend you to study this part thoroughly if you want to contribute or customize the behavior of the software for your specific application. In case you want to use the software as it is, you can skip this section.
In a high-level study, we can divide our code into two primary modules, which we review as follows:
Computer Vision Engine
The computational core of this application lies in this module. The Distancing class in the libs/Core.py file takes a video frame as the input and returns the coordinates of each detected object as well as a matrix of distances measured between each pair. The coordinates will further be used by the update method in the WebGUI class to draw bounding boxes around each person.
We will now discuss the primary building blocks of the Distancing class.
Data Pre-Processing
To prepare each frame to enter the object detection model, we applied some pre-processing such as re-sizing and RGB transformation to the frames.
resized_image=cv.resize(cv_image,tuple(self.image_size[:2]))rgb_resized_image = cv.cvtColor(resized_image, cv.COLOR_BGR2RGB)
Model Inference
In the constructor, we have a detector attribute that specifies the object detection model. We used SSD-MobileNet-v2 as the default model for this application. The average inference time on Jetson Nano was 44.7 milliseconds (22 frames per second).
The object detection model is built based on the config file and should have an inference method that takes a proper image as input and returns a list of dictionaries. Each dictionary of this list contains the information of a detected object, i.e., bounding box coordinates and object id.
tmp_objects_list = self.detector.inference(rgb_resized_image)Bounding boxes post-processing
Since we used a general-purpose object detector (trained on COCO with 80 different classes) that has not been trained for our specific task, the output of this model needs partial post-processing to be robust for our specific purpose.
We applied three post-processing filterings to the raw bounding boxes to eliminate large boxes, collapse duplicated boxes for a single object, and keep track of moving objects in different frames.
These post-processing filterings are applied in the following lines of code:
New_objects_list = self.ignore_large_boxes(objects_list)new_objects_list = self.non_max_suppression_fast(new_objects_list, float(self.config.get_section_dict(“PostProcessor”)[ “NMSThreshold”]))tracked_boxes = self.tracker.update(new_objects_list)
Compute distances
After post-processing, we need to calculate the distances between every pair of persons detected in each frame.
We use Python’s scipy library to calculate the distances between each pair of bounding box centroids.
centroids = np.array([obj[“centroid”] for obj in new_objects_list])distances = dist.cdist(centroids, centroids)
It is clear that the distances matrix is symmetric and has zeros on the diagonal.
User Interface (UI) and Web Application
The UI section is responsible for drawing bounding boxes around the detected persons at each frame. It uses different colors to show the distance between each pair of people.
The WebGUI object implements a Flask application and serves as an interface for the user. WebGUI constructor takes config and engine_instance parameters as inputs and acts as the central application for the output view.
Processing the video begins with the WebGUI start method. Within this method, the engine instance calls process_video to process the video frame by frame. This method returns a list of dictionaries for each detected object, a matrix of distances, and the image itself with the desired output resolution. These values are then passed to the update method of the WebGUI class that draws bounding boxes with proper colors around each object, i.e., person.
How to Use Our Application
Config file
The user can interact with the code by changing some parameter values in the config file. Some parameters might need modifications to meet the user’s specific purposes. To run our application on any video file of your choice, set the VideoPath parameter to point to the target file.
The ConfigEngine class is responsible for parsing the config file. Feel free to change or add other parameters to the config file if you want to develop your customized version of the software.
Docker container
After customizing the config file, you can run the web application and see the output in your browser by running the following commands. Please note that you should have Docker installed on your device.
cd neuralet/applications/smart-distancing/# 1) Download TensorRT engine file built with JetPack 4.3:./download_jetson_trt.sh# 2) build Docker image:docker build -f Dockerfile-jetson-nano -t “neuralet/social-distancing:jetson-nano” .# 3) run Docker container:docker run -it — runtime nvidia — privileged -p HOST_PORT:8000 -v /PATH_TO_CLONED_REPO_ROOT/:/repo neuralet/social-distancing:jetson-nano
This container executes the neuralet_distancing.py file that wraps up all of the modules we mentioned earlier and streams the output video on the HOST_PORT.
Smart Social Distancing roadmap
As the project is under substantial development, lots of ideas can help us improve application performance and add other exciting features. Smart Social Distancing roadmap can best explain our priorities in the future of this project:
- Evaluate and benchmark different models.
- Improve the distance calculation module by considering perspectives.
- Provide a safety score to a given site and calculate useful statistics about a specific time interval; for example, how many people entered the place, and how many times social distancing rules were violated.
- UI improvements: show statistical and historical data in the GUI.
- Aid the program optimization by profiling different parts of the software, from the CV engine to UI overheads.
- Provide the ability for the user to customize and re-train models with task-specific datasets.
Check out our other open issues on our GitHub repo.
Contribution instructions
Due to the tremendous social impact that this service can provide and the fact that this project is under intensive development, any contribution from the community is warmly welcomed by our team. In this section, we present some examples of how you can contribute to the project.
Getting feedback and sending issues
The easiest way to participate in improving this project is to report feedback. If you find any potential bugs or have feature suggestions, please check other issues in the repository before opening a new issue to avoid duplicate issues.
When creating a new bug report, please make sure to include all the necessary details and the code snippet that reproduces the bug to help our team and the community fully understand the problem.
Send PR on current issues:
If you find a bug or an enhancement interesting to work on, first, check the current issues to make sure no one else is working on the same problem. Then, tell others you are working on this issue by posting an announcement on the corresponding issue page. Before sending a pull request, make sure that you add a brief description of what your PR solves and refer to the issue you worked on.
Share your experience of using Smart Distancing with the community
Finally, if you have used this software in practice at your office, your warehouse, or any other place, you can share your experience with the community and let us know if you find this software useful in practicing social distancing.
Stay connected
Please visit our project at Neuralet website for more information and subscribe to our newsletter to receive the latest news and updates about our products. Don’t forget to check out our GitHub page for other models and applications.
