Intrusion detection using Python OpenCV

Analyze CCTV video to detect intrusion

Problem Statement

In this simple demonstration we will see how to analyze and identify object in video. The below video captures intruder and we will understand the basic processing technique to detect this.

What is OpenCV?

OpenCV is a Python library which is designed to solve computer vision problems. It’s an immensely popular image/video processing library.

OpenCV Python is nothing but a wrapper class for the original C++ library to be used with Python. Using this, all of the OpenCV array structures gets converted to/from NumPy arrays.

How does computer read an image?

Input to any neural network should be numeric and fortunately images are naturally represented as array of numbers.

• Images are made of pixels.
• A number between 0–255 represents the color intensity of each pixel.
• Each pixel in a color image is an array representing the intensities of red, blue and green. The red, green and blue (RGB) are called channels.
• A grayscale image (a black and white image), have only one channel.

Reading a gray scale and color image

The image will be read using OpenCV(cv2) and plot using matplotlib.

• ‘imread’ command will read image from “files” directory
• ‘cv2.IMREAD_GRAYSCALE’ will be use to read gray scale image

import numpy as np
import cv2
import matplotlib.pyplot as plt
import pandas as pd
## Reading a gray scale image
img = cv2.imread("files/sachin_gray_scale.jpg",cv2.IMREAD_GRAYSCALE)
plt.imshow(img,cmap='gray')

Sachin gray scale image

• OpenCv read color image in BGR color-space and matplotlib needs image in RGB color-space. To show image we have to convert the color from BGR to RGB using cv2.COLOR_BGR2RGB.

## Reading a color image
img = cv2.imread("files/sachin_color.jpg")
img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)

Sachin Color image

The shape of gray scale image is (206,160) and color image is (206,160,3). The last number shows channels.

Video Analysis

A video is basically a sequence of frames where each where each frame is an image. We will see how OpenCV can be used to process a series of image (i.e. video).

First load the video and find out total number of frames. We will also display one image from the series of the frames.

# Open a sample video available in sample-videos
vcap = cv2.VideoCapture(r'files/sample_video.mp4')
count = 0
#total number of frame in the video
vcap.get(cv2.CAP_PROP_FRAME_COUNT)
Out[32]: 2053.0

There are total 2053 frames in this video. ‘ret’ in below syntax is return binary value (0/1) and ‘frame’ return one frame from video.

ret,frame = vcap.read()
frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
plt.imshow(frame)

To convert video into series of frames — Read the sample video file and save image/frame at every 60th timestep in local folder. The loop will continue till vcap.read() return a frame. Save each frame in ‘.jpg’ format.

# Open a sample video available in sample-videos
vcap = cv2.VideoCapture(r'files/sample_video.mp4')
# read every nth frame
count=0
frame_num = []
while vcap.isOpened():
    ret, frame = vcap.read()
    if ret:
        frame_count = "frame_" + str(count) + ".jpg"
        cv2.imwrite(os.path.join('frames', frame_count), frame)
        frame_num.append(frame_count)
        count += 60
        vcap.set(1, count)
    else:
        vcap.release()
        break
# change color space from BGR to RGB
all_imgs = [cv2.cvtColor(cv2.imread(os.path.join("frames",x)), cv2.COLOR_BGR2GRAY) for x in frame_num]
# total number of frame saved
len(all_imgs)
Out[40]:35

• As per the above code there are 35 images saved in path provided in “os.path.join”.
• Count is the step after which we are saving image from video. We will now show first image from this list of images.
• Note that color image doesn’t make any difference in this analysis. So we will process the image in gray.

# show first image from the list. 
plt.imshow(all_imgs[0],cmap='gray')

• Our aim is to find out if any one has entered from the gate.
• If someone enters, subsequent frame will show difference as compared to previous frame.
• But after watching video carefully we can see the cars are moving on the road i.e. frames are changing with time. Also, the timestamp shown in CCTV video is continuously changing.
• We will mask these two area so that remaining area of each frame can be compared to detect any changes.

## masking the rectangular area near gate and timestamp for all images
for i in range(len(all_imgs)):
    all_imgs[i][0:350,700:1280] = 0
    all_imgs[i][0:100,0:700] = 0
plt.imshow(all_imgs[0],cmap='gray')

Now we will loop through all the images to find out the difference between two consecutive images. For this purpose I will use kernel/filter of size (10,10). If we use smaller kernel it will show difference in images for minor things also (like changes because of wind). Also, applying erosion (morphological operations).

# find the difference between consecutive image and save in a list. 
diff_from_ref = []
for i in range(1,len(all_imgs)):
    diff = all_imgs[i-1] - all_imgs[i]
    kernel = np.ones((10,10),np.uint8)
    diff_dilated = cv2.erode(diff,kernel,200)
    diff_from_ref.append(np.sum(diff_dilated != 0)/(diff.shape[0]*diff.shape[1]))

Convert the above list (diff_from_ref) in dictionary and sort in ascending order

# create dictionary
res = pd.DataFrame({"frame":frame_num[1:],"diff":diff_from_ref}) res.sort_values(by="diff",ascending=False)

frame 33 and 34 is showing highest difference. Let’s see 33rd frame image.

img_id = 33
diff = cv2.subtract(all_imgs[img_id-1],  all_imgs[img_id])
kernel = np.ones((5,5),np.uint8)
diff_dilated = cv2.erode(diff,kernel,12)
plt.imshow(diff_dilated,cmap="gray")

# to confirm check the 33rd and 34th image
plt.imshow(all_imgs[33],cmap="gray")
plt.imshow(all_imgs[34],cmap="gray")

We can see intrusion detected in frame 33.