Performance verification of anomaly detection model automatic creation platform with MVTec dataset

I verified the performance of the anomaly detection model automatic creation platform “ADFI” written in the previous article!
See the previous article for ADFI.

Screen of ADFI:

Experimental settings

Dataset

For the verification experiment, I used the MVTec dataset, which is a very famous image dataset for anomaly detection.
It contains datasets of 15 categories that are frequently used in verification experiments of anomaly detection related papers.

Dataset image example:

The number of data

Since the number of normal images and abnormal images differs depending on the data set, the below is used for each data set.
Training images were randomly extracted from the dataset. Test images are randomly extracted from images not used in Training images.

Normal training data: 50

Anomaly training data: 10

Normal test data: 20

Anomaly test data: 20

Verification items

• Learning time
  Time taken to train the model
• Test time
  Time from running the test to getting the results of all test data (40 images)
• AUC (Area Under the Curve)
  An evaluation index often used for anomaly detection.
  ROC (Receiver Operating Characteristic) The area corresponding to the lower part of the curve.
  The closer the AUC is to 1, the higher the performance of the model. (If predicted completely randomly, the AUC will be 0.5.)
  I downloaded the score result CSV and calculated the AUC.

Experimental results

The table below shows the results of creating a deep distance learning (DML) model with ADFI for all datasets.

The average learning time is about 11 minutes.
The average test time is about 4 seconds.
The average AUC is 0.947.

Very good results with AUC above 0.9 for datasets other than Cable and Screw.

Performance comparison with methods of deep metric learning

I compared the AUC of the model of ADFI and the AUC of the model created by each method of deep metric learning without using ADFI.

AUCs of each method in the MVTec dataset:

It is not possible to simply give a superiority or inferiority because the experimental settings are different.
But the AUC values of ADFI are the highest in many datasets.

See also the previous article for ADFI.

Image Anomaly Detection: How to create models quickly and easily | by Umapyoi | Mar, 2022 | Medium