Improve the performance of your Object Detection model

TIDE(Toolbox for Identifying Detection Error)

Fig.1

Want to improve the performance of your object detection model ?

Let’s improve it by knowing the contribution of different types of errors that are affecting the overall map value.

Object detection and segmentation are the fundamental task in the computer vision, with the the various applications ranging from self driving cars to the tumor detection.

To perform object detection all you need is your training dataset. There are some open-source pretrained model like Tensorflow object detection API, detectron2 etc. You just need to download that pretrained models and retrain it using your custom dataset.

Now, coming to the evaluation of the model. After training of your model you need to check the performance of your model. There are some performance metrices like AP, map value through which you can get the accuracy of your model.

After all this training and evaluation of a model, you may get disappointed by seeing the numbers of AP and map as they are far away from your expectations.

There are different solutions by which you can improve the performance of your object detection model like improving your data set with the help of image augmentation, Pseudo labeling, Ensembling etc. You can refer this blog for the solutions to improve your dataset.

To increase the map value of your model, you’ll for sure want to know where we can improve it additionally. This is where TIDE comes into the picture. TIDE will help you pinpoint the exact type of error you should focus on improving in order to make your model run better.

TIDE is a general Toolkit for Identifying Detection and segmentation Errors. It is a technique for measuring the contribution of each error in a way that isolates its effect on overall performance of the model.

• Where we can use TIDE ?

If we take example of two models, model 1 is for self driving car and model 2 is for detection of tumor cells, having map value 42 and 45 respectively. In case of the self driving car classification is more important than the localization. Here, misclassification of the objects on a road is not acceptable as it may cause an accident. So, we will care more about recall and correct the classification than precise localization. Whereas, we consider the tumor cell detection model, localization is more predominant than classification. TIDE will give contribution of both classification error and localization error so that accordingly we can proceed for the model improvement.

• Different types of errors

Source image from[1]

Classification error: Object is localized correctly but misclassified.

Localization error: Object is classified correctly but localization of the object is incorrect.

Both classification and localization error: Object is misclassified as well as localization of the object is also incorrect

Duplicate error detection: For the same ground truth, model gives two bounding box.

Background error: Bounding box is detected by the model but there is no any ground truth value.

Missed GT error: There is ground truth, but model completely missed the detection bounding box.

• Implementation of a TIDE for COCO dataset

# Download the Mask R-CNN results to test on. Only for demonstration purposes.
import urllib.request # For downloading the sample Mask R-CNN annotations

bbox_file = 'mask_rcnn_bbox.json'   
mask_file = 'mask_rcnn_mask.json'
# Import the TIDE evaluation toolkit
from tidecv import TIDE

# Import the datasets we want to use
import tidecv.datasets as datasets

# Load the dataset
# This will automatically download COCO's 2017 annotations for you. You can specify your own COCO-style dataset too!
# See datasets.py for a list of all the supported datasets.
gt = datasets.COCO()
# Load the results in COCO format
bbox_results = datasets.COCOResult(bbox_file) # These files were downloaded above.
mask_results = datasets.COCOResult(mask_file) # Replace them with your own in practice.
# Create a TIDE object to use for evaluation 
tide = TIDE()
# Run the evaluations on the standard COCO metrics 
tide.evaluate_range(gt, bbox_results, mode=TIDE.BOX )
tide.evaluate_range(gt, mask_results, mode=TIDE.MASK )
# Summarize the evaluations run so far in the console
tide.summarize()

You can refer this github repo for the TIDE implementation.

TIDE will give the output for COCO dataset as follows:

TIDE output for COCO dataset

# Plot summaries for the evaluations run so far
tide.plot()

tide.plot() will give the visualization of the summary of the errors.

TIDE plots

• Applications of the TIDE

1. Cross model comparison: We can compare different models using TIDE very easily.
2. Cross data comparison
3. Comparing across scales

References

1. Daniel Bolya, Sean Foley, James Hays, and Judy Hoffman; TIDE: A General Toolbox for Identifying Object Detection Errors, ECCV 2020.
2. Tushar Kolhe; How to boost object detection accuracy by understanding data, https://blog.gofynd.com/boost-object-detection-model-accuracy-552586d698c
3. A better map for object detection https://towardsdatascience.com/a-better-map-for-object-detection-32662767d424

I hope this content will help you.

Read my previous blog

Text extraction for unstructured data using easy OCR

Thank you !!