Comparison of Web and Mobile object detection models based on inference results
This article is the fourth and last in a series of articles on object detection on Edge devices. The first three articles of the series can be found here:
- Building an object detection pipeline for edge inference
- Porting PyTorch based object detection model to mobile optimized format
- Running PyTorch based object detection model on Edge device with pre and post processing steps
In this article, we’ll talk about the comparison between web and mobile inference results.
Mean Average Precision (mAP) is a commonly used metric for evaluating object detection models and is utilized in various benchmark challenges, including Pascal VOC, COCO, and others. During the training of our object detection models on web devices, we don’t need to delve into the complexity of mAP calculations. Different frameworks provide the flexibility to calculate mAP simply by adding a parameter while running the training command at a predefined threshold.
But what if we need to calculate mAP for our mobile-optimized object detection models? Is there a method that allows us to calculate mAP directly for these models? Unfortunately, we were unable to find such a method. Calculating mAP separately for the mobile-optimized model and not relying on the mAP of its web counterpart is crucial, as we have observed a drop in accuracy for various reasons during the conversion to the mobile-optimized format.
To check the accuracy of our model on mobile, we calculated the mAP separately of the mobile version and then compared it with the web model mAP. We used Pycocotools to calculate the mAP values.
We saw around 6% drop in accuracy for the mobile-optimized version. Two reasons account for this decline:
- Although our pre-processing steps were similar, the libraries used on the web and mobile were different. Consequently, distinct underlying methodologies were employed for operations such as resizing.
- Conversion from .pt to .ptl format. (Note: We suspect this might also be a reason, but we lack data to prove this.)
Thank you for your patience and we hope you’ve found this blog informative and valuable. If you have any questions or would like to explore further, please feel free to reach out. Your feedback and engagement are greatly appreciated.
