An approach to Create a logging mechanism for real-time object detection using TDD

Original image: https://www.architecturaldigest.com/story/david-adjaye-spyscape-spy-museum-opens-in-new-york

Motivation

Recently I had a project about a real-time application which gets the video stream from surveillance system as input and returning bounding boxes drawn on the detections as well as logging them in json files. So I thought as my first post on medium, this simple project could be a good start. In this way, I used Test-Driven-Development to implement the code, so I really hope this post come in handy for the newbies who want to get familiar with unittest tools in python. You can check this link to get the source code. In this link you can also find out an object detection case usage of the code. The object detection code belongs to this awesome post by Adrian Rosebrock, one of my great motivators for computer vision.

Describe the system

To get to know what we are going to deal with, Let’s take a close look at this sample image:

Figure 1: Detection bounding boxes with labels and confidences in a frame. Source: pyimageresearch.com

This image shows a frame which contains two bounding boxes and each bounding box has got a label with specific confidence number. So, we have got a list of frames and in each frame, there are multiple bounding boxes and each bounding box has got label with its confidence. Usually in any object detection there are other labels with less confidence. We can let the bounding boxes to have a list of labels with their assigned confidences. we do that by choosing a parameter named as top_k_label and we expect all the bounding boxes to have exactly the same number of labels.

Here is an example json file for the figure 1 with top_k_labels equal to two.

{
“video_details”: {
“frame_width”: 1080,
“frame_height”: 720,
“frame_fps”: 20,
“video_name”: “test.avi”
},
“frames”: [
{
“frame_id”: 1,
“bboxes”: [
{
“bbox_id”: “0”,
“labels”: [
{
“category”: “car”,
“confidence”: 99.25
},
{
“category”: “truck”,
“confidence”: 57.14
}
],
“left”: 5,
“top”: 400,
“width”: 250,
“height”: 145
},
{
“bbox_id”: “1”,
“labels”: [
{
“category”: “car”,
“confidence”: 99.78
},
{
“category”: “bus”,
“confidence”: 65.23
}],
“left”: 650,
“top”: 450,
“width”: 300,
“height”: 150
}]
}]

I considered video details which holds the information about width, height, fps and name of the video file to make sure we can rebuild the bounding boxes when we have input video file again if necessary.

Initialization

I started the base code with 3 classes which contain the basic information for frames, bboxes and labels.

Basic components of framework

And To manage the output I created another class and named it as JsonParser :

Managing components with JsonParser class

First step: Adding frame

OK now that we have initial classes, let’s use the TDD software development method. I created another file and named it as test_json_parser and then put it in the tests folder.

Now we need a code which tests simply adding frame to the JsonParser.frames . Let’s initialize the TestCase with this code:

Initialization of tests on json_parser.py

I use the pycharm tools to test the code. the reason I added the sys.path.append(../..) is the fact that I can use the terminal commands to run the tests if pycharm was not available in your system. This is the test for adding frame:

testing simple frame insertion

In this test:

1- We make sure that output json contains frame numbers that were given in input.

2- If a repeated frame number was inserted, we expect an error to be raised.

Note that assertRaisesRegex function uses regular expressions to check the error message raised. the term (.*?) is a regular expression and in here I used it in case to ignore anything between the sentences (Frame id:)and already exists .

Now let’s develop the code to pass this test. We start with the JsonParser class. I added these 3 functions to it:

Adding these functions to JsonParser class will pass the tests

To access each class attributes in python, normally I use __dict__() function. But in order to access attributes which are list of objects like labels attribute in class Bbox or bboxes attribute in class Frame, we need to do a little trick to enable these parameters return the dictionary of attributes instead of returning a bunch of objects. Thanks to this awesome answer to my question in stackoverflow (feel free to upvote the question if this post is useful for you), I created a parent class which adds an extra attribute to classes Frame, Bbox and Label. I named it as BaseJsonParser. we set this class as the parent to Frame, Bbox and Label classs.:

Defining BaseJsonParser and `Label`, `Bbox` and `Frame` have yo state it as parent.

When we set the BaseJsonParser as the parent class, we set an attribute namely as dic for classes that inherit this class. This attribute will allow us to return the attributes in the nested classes.

Let’s add the output function for the `JsonParser` and run the test:

Add this function to `JsonParser` class

Now we run the test_add_frame.py function:

Test passed!

Step 2: Adding Bounding Box to each frame

Well done! Now we write the second test which is for adding Bbox to frames:

Now we have to pass this test for adding the bbox

Allright. In here we need to:

1- Create function for JsonParser to check if bbox exists before inserting the bbox
2- Create function to add bbox to a frame for Frame and JsonParser classes.

Now we add these functions to JsonParser class:

Add these to `JsonParser`.

and this one to Frame class:

add bbox function for `Frame` class

Now let’s run both tests again:

Step 3: Adding labels to bounding boxes

The next part we have to test adding labels to bboxes. What we have to test here, are these things:

1 — When we set the top_k_labels and we add the same number of labels to bboxes, we expect them to be added with no issue.
2 — Inserting labels more than top_k_labels will be abrogated with ValueError.
3 — During outputting the json we expect the program to check if the bounding boxes has the same number of labels as the top_k_labels allows.

Here it is the test:

Test for adding labels to a bbox

Now to pass the test we have to:

1 — Create a function that finds bbox in bbox list in a frame given the frame_id and bbox_id
2 — Create a function that adds a label to found bbox.
3 — Labels added to a bbox has to be as many as the top_k_labels is. Otherwise the error raises. we have to make sure about that during the outputting frames.

We add find_bbox function in JsonParser function:

find_bbox function added to JsonParser

Then we add add_label_to_bbox function to JsonParser in here:

Add label to bbox if bbox already exists

as you see there has to be a add_label function for bbox. I added that as well as a function which returns true if the labels number is as same as the top_k. I named it as labels_full :

We updated the class `Bbox` by adding `add_label` and `labels_full` function

And the value given in input for labels_full is going to be the top_k which we have already considered that in JsonParser. Now that we have to modify the code for output function in JsonParser:

Modifying output function in JsonParser class

Let’s run the whole tests once again:

Running all the tests

Final step: Automating Json outputs

Cool. We have our JsonParser. Now we can develop a function which helps us to automate the whole process of saving json files. The mechanism that I came up with is going like this:

1 — We set a interval parameter which could be between 1 second to 5 hours and it will indicate the how often json files will be saved.
2 — we set a starting time right before the process of 1st frame starts.
3 — We call the scheduler function in the loop which process is done for each frame. Then we can measure the time difference between starting time and the current time. Whenever this difference gets bigger than the time_to_save parameter, we can save the json file. We save the files by their corresponding timestamp.

Let’s create the test in another file named as test_scheduled_output.py and initialize the test with:

why mkdtemp ?

the function mkdtemp from tempfile library, uses a temporary directory in the system and makes a temporary folder. We use that folder to save the jsons in there for temporary test.

I need the process to take time for more than 1 second to see if automatically the json will be saved or not. I don’t want to use video for my test because in unit testing it’s more professional to design tests that are thorough and run faster.
So I simulate the action by using sleep function from time library.

Then we use the code below to pass the test:

Adding json_output and schedule_output

what is JsonMeta? The reason I used the JsonMeta class is the fact that I want to set limitation for all arguments in schedule_output function. (I don’t want the minutes or seconds to get more than 59 for example).

Let’s see if test is passed:

All tests are passed. Hooray!!!!

And for the end we can use below function to add video details to json:

Suggestions for further works

Allright, now we have a tool that is useful for saving jsons in real-time surveillance system. For practice, I recommend you to add these features:

1 — Create another class which is able to analyzes the json data and find insights from detections. for example to find how often a specific person passes the surveillance camera.

2 — Create a function that is able to return the detections in the video by reconstructing the bboxes on the video frame and show the result

3 — Use the code on an object detection code.

I already have done the 3rd suggestion on the code that belong to this post which is an awesome fast object detection code created by Adrian rosebrock. I uploaded the whole code in here