Grocery Store Empty Inventory Detector using YOLOv5

The success of deep learning to solve complex problems is not hidden from anyone these days. Deep learning is helping automate problems in all walks of life. In this article, I present the application of YOLOv5, a deep learning based algorithm, to detect empty inventory in grocery stores.

Usually, when we go to a grocery store, and we see a shelf that doesnt have the product we need, then many customers would leave without asking the store workers if they have that item. Even if the store had that item in their warehouse. This can cause the store to lose out on potential sales for as long as the inventory remains empty. The system I designed using YOLOv5 can help stores replenish inventory quickly so that they dont lose customers and sales.

Requirements

For this project, we need:

1. YOLOv5, a great work by Glenn Jocher, present at https://github.com/ultralytics/yolov5
2. SKU dataset, present at https://retailvisionworkshop.github.io/detection_challenge_2020/
3. If you have a dataset of your store, even better.
4. LabelImg, a great and simple-to-use annotation tool, present at https://github.com/tzutalin/labelImg
5. Bounding box augmentation tool, (if you want to augment the already annotated images and not annotate more images), present at
   https://github.com/mukopikmin/bounding-box-augmentation

Process

First, we are going to use the SKU dataset. The SKU dataset is specifically for detecting items in grocery store shelves and refrigerators. SKU has all images and the annotations for each item on the shelf. But for our application, we don’t need the annotations of items on shelves. We want the opposite of that. Because we are only going to use those images in which the inventory was empty for any product(s) on the shelf.

Annotation

For annotation, we use LabelImg tool, to annotate the voids in any image of our application as shown below:

Augmentation

If you dont have abundant image samples, then use the augmentation code mentioned in the requirements above, which adds different kinds of augmentations and also augments the bounding boxes annotated in the image.

For augmentation, once you are done annotating the images using the above LabelImg tool, put all the images and their annotations (.xml) files in the “input” folder of the above downloaded Bounding Box Augmentation repository. Then run the following command:

pipenv run python augment.py

By default, this will augment each image and its annotation present in the “input” folder, to 10 times. To increase this, one can change the AUGMENT_SIZE in the augment.py file in the repository.

Training

To train the void detector model, I used YOLOv5 and trained the yolov5m model for 300 EPOCHS on a 12 GB NVIDIA GeForce RTX2080 GPU. The data I used included few grocery store shelf images and around 100 images from SKU dataset, that I annotated myself and augmented by a factor of 10. In total, I had 1254 images. The mAP@0.5 score came out to be 0.82.

Results

The inference results of shelf void detection are as follows:

Developers can also store the co-ordinates of voids in database and compare with new data lets say, after 2 hours, to find out if previous voids persist or not. If they persist then those voids should be prioritized and will let stores know what items need urgent attention and have not been replenished in a long time. For this task, simply new and previous void boxes’s IoU(Intersection over Union) needs to be computed and if its greater than 0.5 then that means the void is persisting and has been there for some time. Hope that makes sense.

Conclusion

Deep learning, despite of its large success, still has many uncharted territories to conquer. In this article, I presented one such application that can let stores know where they are out of inventory. This system when implemented, can help stores achieve higher sales, that they would otherwise lose, due to the late replenishment of inventory. Please show appreciation if you liked the article, by claps, so that I have more motivation to share my work.

Special Thanks

Special thanks to the people who designated me with this task and the developer of YOLOv5, Mr. Glenn Jocher.