Detecting and Tracking Moving Objects with Background Subtractors using OpenCV
→ Keypoints: OpenCV & C++ , Python, Background Subtractors, KNN, MOG2, Object Detection & Tracking
Imagine a video that you take with your phone, you press a button, and the camera starts to record a video. Under the hood, the phone’s camera captures images very quickly, and when you watch the video, you see images one by one, but you don’t notice it because this process happens very quickly. In one second, your phone displays more than 30 images. If you compare these images and find the differences between them, you can detect moving objects, and this is exactly how Background Subtractors work.
- In this article, I will explain how Background Subtractors work, the different kinds of Background Subtractors, and how to use these subtractors. I will use Python as the programming language, and you can also find a C++ implementation of this project at the end of the page.
Ways of Detecting Moving Objects
1. Basic Motion Detection
The first and most intuitive way is to calculate the differences between frames, or between a frame considered “background” and all the other frames.
The idea is quite simple at the highest level: first, save the first frame. After saving it, compare it to the new frames. By comparing them pixel by pixel, simply subtract the two images. In this way, you are going to obtain moving objects.
- This technique is quite fast to implement but it is not that suitable for applications because you need to set the default frame as a background and probably background is not going to stay constant in your applications.
Imagine you are detecting cars. Setting a default background is not going to be effective because cars are constantly moving, and everything is changing. Lighting changes and objects are in motion. For instance, you set the first frame as a background image and there are 3 cars in the background image, but after just one second, they are not going to exist because they are moving. Consequently, the background image becomes inaccurate as everything is changing rapidly. Therefore, the algorithm is not going to be accurate, especially in environments with rapid changes.
Look at the images; the algorithm works, but it is not accurate. Look at the image on the left; there are meaningless areas. That is because in videos, the background is nearly changing every second, but in the algorithm, the background is constant.
I think you understand the main idea behind Basic Motion Detection. It may be useful depending on your expectations. If you don’t expect high accuracy, you can consider using it.
For solving the issues that I discussed above, Background Subtractors come into play. Now it is time to talk about Background subtraction and subtractors.
2. Background Subtraction
What is Background Subtraction ?
Background subtraction is a fundamental technique in computer vision for isolating moving objects from the background in a video stream. By comparing each frame of the video to a background model, areas of significant difference can be identified as potential foreground objects. This foreground information can then be used for various purposes, including object detection and tracking. Background subtraction is often a crucial step in many object tracking and detection algorithms.
How it solves background problem
In background subtraction, the background image is not constant; it changes over time due to various factors such as lighting variations, object movements, and scene dynamics. The goal of background subtraction algorithms is to adaptively model and update the background to accurately detect foreground objects in changing environments. In this way, the background problem is solved.
In OpenCV, Background subtractors can detect shadows, and with thresholding, they can exclude shadows from the objects that the subtractor detects. This is indeed a very important feature for accurately detecting objects because unidentified shadow areas may be mistakenly interpreted as separate moving objects by the subtractor, which is not desirable.
Using Background Subtractors with OpenCV
Opencv has several different background subtractors. I will use the two most famous of those subtractors :
- K-Nearest Neighbors (KNN)
- Mixture of Gaussians (MOG2)
I will only explain how MOG2 works for Background Subtraction but I will use both approaches for detecting and tracking videos.
Here is how MOG2 works for background subtraction:
- Initialization: Initialize a mixture of K Gaussian distributions to model the background of the scene. Each pixel’s background model is represented by a mixture of Gaussians, with K being a predefined parameter.
- Adaptation: Update the background model for each pixel over time, adjusting the parameters of the Gaussian distributions to adapt to changes in the scene.
- Foreground Detection: Compute the probability of each pixel belonging to the background based on the Gaussian mixture model. Pixels with low probabilities are classified as foreground.
- Update Background: For pixels classified as background, update the Gaussian distributions to incorporate new observations and adapt to changes in the scene
- Post-processing: Apply morphological operations(erosion, dilation..) or other techniques to refine the foreground mask and remove noise.
CODE / Detecting and Tracking Moving Objects
Inside the code, I have explained most of the process, but the best way to understand it is to copy the code and use the cv2.imshow function to observe every frame after each operation.
# import libraries
import cv2
import numpy as np
# KNN
KNN_subtractor = cv2.createBackgroundSubtractorKNN(detectShadows = True) # detectShadows=True : exclude shadow areas from the objects you detected
# MOG2
MOG2_subtractor = cv2.createBackgroundSubtractorMOG2(detectShadows = True) # exclude shadow areas from the objects you detected
# choose your subtractor
bg_subtractor=MOG2_subtractor
camera = cv2.VideoCapture("resources/run.mp4")
while True:
ret, frame = camera.read()
# Every frame is used both for calculating the foreground mask and for updating the background.
foreground_mask = bg_subtractor.apply(frame)
# threshold if it is bigger than 240 pixel is equal to 255 if smaller pixel is equal to 0
# create binary image , it contains only white and black pixels
ret , treshold = cv2.threshold(foreground_mask.copy(), 120, 255,cv2.THRESH_BINARY)
# dilation expands or thickens regions of interest in an image.
dilated = cv2.dilate(treshold,cv2.getStructuringElement(cv2.MORPH_ELLIPSE, (3,3)),iterations = 2)
# find contours
contours, hier = cv2.findContours(dilated,cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
# check every contour if are exceed certain value draw bounding boxes
for contour in contours:
# if area exceed certain value then draw bounding boxes
if cv2.contourArea(contour) > 50:
(x,y,w,h) = cv2.boundingRect(contour)
cv2.rectangle(frame, (x,y), (x+w, y+h), (255, 255, 0), 2)
cv2.imshow("Subtractor", foreground_mask)
cv2.imshow("threshold", treshold)
cv2.imshow("detection", frame)
if cv2.waitKey(30) & 0xff == 27:
break
camera.release()
cv2.destroyAllWindows()
