Calculating mAP of models on Edge device

Calculating mAP

In this article, we’ll talk about how to calculate mAP for Edge device inference results.

Mean Average Precision (mAP) is a commonly used metric for evaluating object detection models and is utilized in various benchmark challenges, including Pascal VOC, COCO, and others. During the training of our object detection models on web devices, we don’t need to delve into the complexity of mAP calculations. Different frameworks provide the flexibility to calculate mAP simply by adding a parameter while running the training command at a predefined threshold.

But what if we need to calculate mAP for our mobile-optimized object detection models? Is there a method that allows us to calculate mAP directly for these models? Unfortunately, we were unable to find such a method. Calculating mAP separately for the mobile-optimized model and not relying on the mAP of its web counterpart is crucial, as we have observed a drop in accuracy for various reasons during the conversion to the mobile-optimized format. This article highlights how we calculated mAP and the challenges we faced.

We loaded our ML model into our mobile app and had the model run inference on the validation dataset, saving the bounding box information in JSON format for each image separately. Now we had the ground truth and predicted bounding boxes for our validation set ready. Reviewing the code in val.py from the YOLOv5 official documentation, we discovered that the mAP command in the yolov5 framework uses Pycocotools to calculate mAP values. So, we decided to use Pycocotools to calculate mAP values for mobile predictions.

Pycocotools requires the ground truth and predicted bounding boxes in COCO format. We had our ground truth in YOLO format and predicted bounding boxes in JSON format. We first converted them into COCO format.

JSON format conversion to COCO

First we define the coco dataset dictionary and categories of our dataset.

import json
import os
from PIL import Image
import glob


# Define the categories for the COCO dataset
categories = [{"id": 0, "name": "classname"}]

# Define the COCO dataset dictionary
coco_dataset = {
    "info": {},
    "licenses": [],
    "categories": categories,
    "images": [],
    "annotations": []
}

image_id = 0

Now, we loop through our images while simultaneously reading the corresponding JSON files containing the bounding box information, and we add the images to the Coco image dictionary.

# Loop through the images in the input directory
for image_file in glob.glob(input_dir + 'images/' + '*.JPG'):
    filename = ((image_file.split('/')[8]).split('\\')[1]).split('.')[0]
    # Load the image and get its dimensions
    image_path = os.path.join(input_dir, image_file)
    image = Image.open(image_path)
    # resizing the image.This step is not required if your images are already of the same size of validation set.
    image = image.resize([640,640])
    width, height = image.size
   
    # Add the image to the COCO dataset
    image_dict = {
        "id": image_id,
        "width": width,
        "height": height,
        "file_name": image_file
    }
    coco_dataset["images"].append(image_dict)

    # Load the bounding box annotations for the image
    with open(os.path.join(input_dir + 'labels/' + f'{filename}.json')) as f:
        annotations = json.load(f)

Once the images are added to the Coco dictionary, we proceed to convert the annotations into Coco-supported format and save the annotations in a JSON file

 # Loop through the annotations and add them to the COCO dataset
    for ann in annotations:
            x,y,w,h = ann['bounds']
            score = ann['score']
            x_min, y_min = int((x - w / 2)), int((y - h / 2))
            x_max, y_max = int((x + w / 2)), int((y + h / 2))

            ann_dict = {
                "id": (len(coco_dataset["annotations"]) + 1),
                "image_id": image_id,
                "category_id": 0,
                "bbox": [x_min,y_min,x_max - x_min, y_max - y_min],
                "area": (x_max - x_min) * (y_max - y_min),
                "iscrowd": 0,
                "score" : round(score,3)
            }
            coco_dataset["annotations"].append(ann_dict)
    image_id = image_id + 1
# Save the COCO dataset to a JSON file
with open(os.path.join(output_dir, 'predicted_annotations_final.json'), 'w') as f:
    json.dump(coco_dataset, f)

YOLO format conversion to COCO

The only part that changes is where we read the annotation text file and convert it into a Coco-supported format.

# Load the bounding box annotations for the image
    with open(os.path.join(input_dir + 'labels/', f'{filename}.txt')) as f:
        annotations = f.readlines()
    # Loop through the annotations and add them to the COCO dataset
    for ann in annotations:
        x, y, w, h = map(float, ann.strip().split()[1:])
        x_min, y_min = int((x - w / 2) * width), int((y - h / 2) * height)
        x_max, y_max = int((x + w / 2) * width), int((y + h / 2) * height)
        ann_dict = {
            "id": len(coco_dataset["annotations"]) + 1,
            "image_id": image_id,
            "category_id": 0,
            "bbox": [x_min, y_min, x_max - x_min, y_max - y_min],
            "area": (x_max - x_min) * (y_max - y_min),
            "iscrowd": 0
        }
        coco_dataset["annotations"].append(ann_dict)
     image_id = image_id + 1
# Save the COCO dataset to a JSON file
with open(os.path.join(output_dir, 'ground_truth_annotations_final.json'), 'w') as f:
    json.dump(coco_dataset, f)

Now that we have the ground truth and prediction boxes in Coco format, we run the Coco evaluator to calculate the mAP values.

# Setting iou threshold at which we need to calculate mAP values
iou_threshold = 0.5


coco_gt = COCO(path to ground truth annotation file)
coco_dt = coco_gt.loadRes(path to prediction annotation file)
coco_eval = COCOeval(coco_gt, coco_dt, 'bbox')

# Set IoU threshold for evaluation
iou_thresholds = [iou_threshold]

# Run evaluation
coco_eval.params.iouThrs = iou_thresholds
coco_eval.params.maxDets = [10,100,300]
coco_eval.evaluate()
coco_eval.accumulate()
coco_eval.summarize()

# Obtain mAP at IoU threshold
mAP = coco_eval.stats[0]

print(f"mAP@{iou_threshold}: {mAP}")

The output is generated in the below form:

The above results are for a sample dataset

The mAP values generated by the above process for mobile predictions differed significantly from the mAP values generated during training by YOLOv5 Ultralytics. Despite trying various options, we were unable to reduce the discrepancies. Since YOLOv5 also uses Pycocotools for calculating mAP values, this difference was unexpected. We submitted a request to the YOLOv5 GitHub repository, asking about the possible reasons for such a vast difference.

To which we got the response as :

Hence, to have an apple-to-apple comparison, we generated both the mAP values (for web and mobile) using Pycocotools and just did not rely on the val.py file, which also happens to use Pycocotools.

Here we conclude the article on calculating mAP values for mobile optimized models. Thank you for your time and patience. We hope you have gained some valuable insights.