Car Damage Assessment using Deep Learning

Overview:

In Car Insurance industry, a lot of money is being wasted on Claims leakage. Claims leakage is the gap between the optimal and actual settlement of a claim. Visual inspection and validation are being used to reduce claims leakage. But doing inspection might take a long time and result in delaying of claims processing. An automated system for doing inspection and validation will be of great help in speeding up the process.

Contents:

1. Business Use Case
2. Mapping to a Deep Learning Problem
3. Data Source
4. Existing Approaches
5. Improvement to existing approaches
6. Exploratory Data Analysis
7. First Cut approach
8. Deep Learning Models
9. Models Comparison and Selection
10. Yolo V3
11. Final Pipeline
12. Web app
13. Future Work
14. References

1. Business Use Case:

To reduce Claims leakage during Insurance processing. Visual inspection and validation are being done. As this takes a long time because the person needs to come and inspect the damage. We are trying to automate this procedure. Using this automation will result in Claims processing faster.

2. Mapping the Problem to Deep Learning Model:

We are trying to automate the Visual inspection and validation of vehicle damage. The input data we have are car damaged images.

For Validation of Vehicle damage we will divide the problem into three stages. 1. First we check whether the given input image of car has been damaged or not.

2. Second we check on which side (Front, Rear, Side) the Car in image has been damaged.

3. Third we check for the Severity of damage (Minor, Moderate, Severe).

This problem is a classic classification problem and Since we will be dealing with images as input, we will be using Convolutional Neural Networks (CNN).

Performance Metrics:

In this we are using Accuracy, Precision, Recall as performance metrics.

To calculate Accuracy, Precision, Recall we need to construct confusion matrix. Formulas for above are given below.

3. Data Source:

Since the dataset for damage in cars are rare. The car dataset is downloaded from.

https://www.kaggle.com/anujms/car-damage-detection.

The above dataset is divided according to the stages we need.

This consists of Train and Validation which each folder has Damage cars pics and whole car pics. A total of 2300 images are present in both train and validation combined.

4. Existing Approaches:

1. https://www.ee.iitb.ac.in/student/~kalpesh.patil/material/car_damage.pdf

In above mentioned paper the team collected the images and sorted the dataset into 8 classes ( Bumper dent, Door dent, Glass shatter, Head lamp Broken, Tail lamp broken, Scratch, Smash, No damage).

Since the dataset containing images are less so they synthetically enlarged the dataset 5 times using Augmentation of Random rotation between -20 and 20 with Horizontal flip.

They trained a Convolutional Neural Network and AE CNN from scratch. The accuracy did not achieve 75%. So they went and trained pretrained models. They used pretrained models Cars, Inception, Alexnet, VGG19, VGG16, and Resnet. They trained a linear svm from output of pretrained models and SoftMax with adadelta optimization, cross entropy loss. From the experimentation using SoftMax classifier is working better than Linear SVM and it is faster to train.

2. https://easychair.org/publications/preprint_open/RlQ2

In this paper the team collected the images and prepared the dataset for damaged cars. They divided the dataset into Bumper dent, Door dent, Glass shatter, Head lamp Broken, Tail lamp broken, Scratch, Smash, No damage.

They enlarged the dataset 4 times by applying rotation 20 degrees, shear of 0.2, zoom of range 0.2 and horizontal flip.

Instead of training CNN without pretrained weights. They went for Pre trained models (Alexnet, Inception V3, VGG19, Resnet50, Mobile nets). They trained only FC layers and All layers for every pretrained model.

In this they used YOLOV3 to localize the damage and highlight it using bounding boxes.

5. Improvements to existing approaches:

In all the above models only pretrained models are performing better than training from scratch.

I will be using VGG16, VGG19, Resnet. We can use Dense net as well and we can see how it performs over others.

6. Exploratory Data Analysis:

We have three types of Data:

1. Training and Test folders of Car damaged, not damaged images.
2. Training and Test folders of Damage on Front, Rear, Side.
3. Training and Test folders of Damage severity Minor, Moderate, Severe.

Stage 1 (Damaged or Not Damaged):

In EDA we will look at how many files each folder has and the top image height and width in our data.

Bar Plot:

Observations:

1. From the train plot we can see that the images we have for Damaged and Not damaged classes are equal.
2. There is no class imbalance.
3. Since the dataset we have is less we can enlarge the dataset using Data Augmentation.

Observations:

1. From the test plot we can see that the images we have for Damaged and Not damaged classes are equal.
2. Since the cars datasets are rare the test folder has less number of images.

Image Sizes plot:

Observations:

1. From the above plot we can see that the (194,259) image shape are more abundant than others in damaged folder and not damaged folder.
2. But in not damaged folder the counts of images with (194,259) and (480,640) are almost equal.
3. There are 132 unique image shapes present in train damaged folder.
4. There are 385 unique image shapes present in train not damaged folder.

Stage 2 (Front, Rear and Side):

Bar plot:

Observations:

1. From the train plot we can see that the images we have for Front, Rear and Side are not equal.
2. We have more front images and counts of rear and side are almost equal.
3. Since the dataset we have is less we can enlarge the dataset using Data Augmentation.

Observations:

1. From the test plot we can see that the images we have for front, rear and side classes are not equal.
2. Since the cars datasets are rare the test folder has less number of images.

Image plots:

Observations:

1. From the above plot we can see that the (194,259) image shape are more abundant in all classes.
2. There are 70 unique image shapes present in train front folder.
3. There are 67 unique image shapes present in train rear folder.
4. There are 60 unique image shapes present in train side folder.

Stage 3(Minor, Moderate and Severe):

Bar plots:

Observations:

1. From the train plot we can see that the images we have for Minor, Moderate and Severe are not equal.
2. We have more Severe images and less images in minor.
3. Since the dataset we have is less we can enlarge the dataset using Data Augmentation.

Observations:

1. From the test plot we can see that the images we have for Minor, Moderate and Severe are not equal.
2. Since the cars datasets are rare the test folder has less number of images.

Image plots:

Observations:

1. From the above plot we can see that the (194,259) image shape are more abundant in all classes.
2. There are 66 unique image shapes present in train front folder.
3. There are 63 unique image shapes present in train rear folder.
4. There are 70 unique image shapes present in train side folder.

7. First Cut Approach:

Since the dataset we have is small we will synthetically enlarge the dataset twice using Data Augmentation. Various papers shows different types of data augmentation. We will use below two types of data augmentation and see which performs better.

1. Random rotation between -20 and 20 degrees and horizontal flip transformations.
2. Random rotation between -20 and 20, shear-range of 0.2, zoom-range of 0.2 and horizontal-flip.

We will create required training folders to save the data augmented images.

Data Augmentation:

Since the data we have are less we will use Data augmentation to synthetically enlarge the dataset. As mentioned in existing approaches, Iwill be using the two types of augmentation and enlarging the dataset twice.

1. Enlarging the dataset twice using random rotation between -20 and 20 degrees and horizontal flip transformations.
2. Enlarging the dataset twice using random rotation between -20 and 20, shear-range of 0.2, zoom-range of 0.2 and horizontal-flip.

Visualization of Data Augmentation:

Data Augmentation 1:

Original

Data Augmented Image

Data Augmentation 2:

Original

Data Augmented

So we will be doing data augmentation on all data we have and create folders which consists of original image and generated images in a single folder.

I have created three types of data for each stage we have. Lets see the number of files we have for each stage separately.

Check Damaged or Not:

1. Original Data (1840 Train files, 460 Test files)
2. Original Data + Data Augmentation 1 (3680 Train files, 460 Test files)
3. Original Data + Data Augmentation 2 (3680 Train files, 460 Test files)

Check Front, Rear or Side:

1. Original Data (985 Train files, 179 Test files)
2. Original Data +Data Augmentation 1 (1970 Train files, 179 Test files)
3. Original Data + Data Augmentation 2 (1970 Train files, 179 Test files)

Check Minor, Moderate, Severe:

1. Original Data (979 Train files, 171 Test files)
2. Original Data +Data Augmentation 1 (1958 Train files, 171 Test files)
3. Original Data + Data Augmentation 2 (1958 Train files, 171 Test files)

After training models on above data I will be using YOLO V3 to categorize the image into one of the following using bounding boxes.

1. Glass and Light Broken
2. Car Dent and Scratch
3. Smash

8. Models:

Since pretrained models are working better than training models from scratch. We will be importing below pretrained models without Fully Connected layers.

Pretrained on Image net Dataset:

1. VGG16
2. VGG19
3. DenseNet201
4. ResNet50

We need a baseline model to compare our NN models. So I will be using Logistic Regression as baseline model with features extracted from above pretrained models.

I have used the pretrained model and predicted the class for all stages in a single ipynb. We have three stages

Stage 1 : Check Damaged or Not

Stage 2 : Damage Localization

Stage 3 : Asses Damage Severity

With Logistic Regression as Baseline model, I have trained above models by keeping layers as Non trainable as first model and keeping layers as Trainable as second model.

Since these models does not have FC layer, I have used below configuration for all models.

I have created functions which are generalized and can be used with any pretrained model.

Input data for NN and baseline models.

Baseline Model (Logistic Regression):

Below function gives an array of features predicted using pretrained model. I have created a x_train, y_train, x_test, y_test using above features.

Using above code for hyperparameter tuning of Logistic Regression. Using best alpha from above to create model and saving model to disk.

Confusion Matrix:

For binary values Stage 1(Check Damaged or Not)

For multiclass I will be showing Confusion, Precision and Recall matrix and compute average precision and Recall.

Code for Baseline models:

Neural Networks:

I have created below functions which are Training on FC layers only and Training on all layers.

After creating model using above functions I have compiled using Binary cross entropy loss for Stage 1 and Categorical Cross entropy loss for Stage 2 and Stage 3. Used SGD optimizer and accuracy as metric.

I have trained each model for 50 epochs and used Model Checkpoint to save best model.

Example using VGG16 for Stage 1:

I have trained a total of 108 models. Below is the scores I got and conclusion for each data we have.

9. Model Comparison and Selection:

Original Data:

Original Data + Data Augmentation 1:

Original Data + Data Augmentation 2:

Conclusion:

1. From above all models we can see that the Densenet (Trained on all layers) model which is trained on the Original data with augmentation 1 is performing best in Stage 2 and Stage 3.
2. For Stage 1 densenet (Trained on all layers) trained on Orginal data with Augmentation 1 is very close to the model trained on Orginal data and precision of both models are same. So we can use the Densenet (Trained on all layers) trained on Orginal data with Augmentation 1 instead of model trained on original data.

So Densenet (Trained on all layers ) with Augmentation 1 worked best for all 3 stages.

10. YOLO V3:

I have used below github link for reference in training yolov3.

robingenz/object-detection-yolov3-google-colab

I have used 3 classes for custom object detection.

1. Glass and Light Broken
2. Car Dent and Scratch
3. Smash

Since there is no data we can download for this, I have used LabelImg tool for creating bounding boxes and giving classes. Installed LabelImg by using below github link.

tzutalin/labelImg

After creating data for Yolo, I have created train.txt, test.txt, custom.data, yolov3_custom.cfg, custom.names files as specified for training yolo.

Since we have 3 classes we will be training yolo for 6000 epochs and weights will be saved for multiples of 1000.

I have used a 3 types of test images to see how each weights are performing. Below is the test images.

Glass and Light Broken example

Car Dent and Scratch Example

Smash Example

Conclusion:

1. After observing results for each file Yolo custom 4000 weights is giving best results than other models.

11. Final Pipeline:

Below code is a final pipeline. Given an Input image it will give the report and yolo prediction.

Below is output of above function:

I have show code for VGG 16 (Only FC layers) only. For other models and full ipynb files of EDA, VGG16, VGG19, Densenet, Resnet, Conclusion visit my GitHub Repository.

The above file is named as final_old.ipynb in my github repository.

12. Web App:

The above final_old file has been modified to return output to web app. I have created a templates folder which has app.html file contains the required web page.

Since we will be running in Colab notebook, I have used Ngrok to run flask. Visit my github repository to find the final.ipynb file which has flask with ngrok implementation and app.html file.

Below is a demo video of web app.

13. Future Work:

1. We could get more data by downloading images so that the training accuracy improves.
2. Using Cloud services to deploy the web app.

14. References:

1. https://www.ee.iitb.ac.in/student/~kalpesh.patil/material/car_damage.pdf
2. https://www.kaggle.com/anujms/car-damage-detection
3. https://github.com/robingenz/object-detection-yolov3-google-colab
4. https://github.com/tzutalin/labelImg
5. https://www.appliedaicourse.com

Profile:

GitHub Repository: https://github.com/bnspriyatham/Car-Damage-Classification

LinkedIn Profile: https://www.linkedin.com/in/priyatham-bhaskara-21a670ab/