CROP DISEASE DETECTION USING IMAGE PROCESSING TECHNIQUE AND CNN NETWORKS : Part2
Plant disease detection using real-time object detection models(YOLO, SSD, Faster R CNN)
In part 1, we discussed about Implementing CNN based image classification model for crop disease detection.
In this blog, we will implement object detection models for crop disease detection. Before going to model-building part, it’s better to know the detailed intuition of YOLO, Faster RCNN, SSD object detection model, and its basic working flow. The explanation is given Introduction to Deep Learning Object Detection. | by B.Thushar Marvel | Apr, 2022 | Medium
Now let’s start building an object detection model for our custom dataset. The first process after collecting images are data preprocessing part.
Data preprocessing
Data preprocessing is the concept of converting the raw data into clean data. Used 2200 images of potato disease plant images (Field potato images, plant village dataset, Plant Doc, other internet images). Data augmentation technique was applied like rotation, varying brightness, exposure, and saturation to increase the size of the dataset. Data augmentation is a process, basically used to improve the model performance by modifying the dataset with additional data. A single image can be converted into multiple images by using this technique. I have used 550 images for testing purposes.
After pre-processing, now we need to annotate every image. Manually we need to mark the bounding boxes.
Image Labelling
The bounding box coordinates of the images are marked manually using the graphical image annotation tool Label Image as per the model’s required format.
(source: https://github.com/tzutalin/labelImg).
Now the data set is ready for training object detection models.
YOLO
Creating files for training the yolo v3 model
In order to train the model, we need to create a text file of each image, which should contain bounding boxes of the objects present in it.
Every image has to have a text file with the same name as the image file.
Inside the text file, every line describes only one object and consists of the class number, object centre in x, object centre in y, object width and object height. [LabelImage software has text output with bounding boxes]
After all images are labelled, it’s time to prepare other files needed for training in the Darknet framework.
In order to train the model we need to create four files that are described below.
Create train.txt and test.txt files from image folder
import os
full_path_to_images = r'path\to\train images'p = []for current_dir, dirs, files in os.walk('.'):for f in files:if f.endswith('.jpg'):path_to_save_into_txt_files = full_path_to_images + '/' + fp.append(path_to_save_into_txt_files + '\n')p_test = p[:int(len(p) * 0.20)]p = p[int(len(p) * 0.20):]with open('train.txt', 'w') as train_txt:for e in p:train_txt.write(e)with open('test.txt', 'w') as test_txt:for e in p_test:test_txt.write(e)
Those 4 files are: -
1.labelled_data.data
2.classes.names
3. train.txt
4.test.txt
Labelled_data.data file should consist below details-
• classes = (number of classes)
• train = Path/to/train.txt (path to train.txt file that consists of full path to train images )
• valid = Path/to/test.txt (path to test.txt file that consists of full path to test images )
• names = Path/to/classes.names (path to file classes.names that has names of labelled objects)
• backup = backup (folder name where trained weights have to be saved)
classes.names in this project (potato disease):
• Early blight
- Late blight
After creating all required files for training, now edit the configuration files which are required for training the yolo v3.
In this project, darknet architecture is used as a feature extractor for yolo object detection. It is installed using the instructions given (source : Installing Darknet (pjreddie.com))
or you can just clone the darknet folder from below mentioned repository
[Note- These are heavy models, needs high processor /high computation, so
I have preferred Google colab for these task. Google colab gives free GPU and also running object detection models in colab is very simple]
!git clone https://github.com/AlexeyAB/darknetAfter cloning the repository, its better to store it in the drive. Because these cloned files will get erased once session gets completed in colab. So I prefer storing these folders and weights in drive.
Now the task to download initial darknet weights.
It can be download from here or one can get it from https://pjreddie.com
#code to download the weights in colab
!wget https://pjreddie.com/media/files/darknet53.conv.74After cloning the directory. Now it’s time to edit the configuration files
Go to
darknet-master\build\darknet\x64\cfgYou can see multiple configuration files there. Select yolov3.cfg file and edit it based on your data.
Create yolov3_train.cfg and yolov3_test.cfg file
Batch size =64 (You can alter it)Sub-division=32 (You can alter it)
its better to have sub divisions less than Batch size. Common pratice is sub division = batch size /2For test.cfg file keep batch_size and sub_division=1max_batches = 20000
[max_batches should be always more than classes * 2000]steps are calculated as 80% and 90% from max_batches.Steps=16000,18000Classes=2filters = (classes + coordinates + 1) * masks=(2+5)*3=21coordinates and mask value are kept the same.
We have 2 classes so we got a total of 21 filters.
Some of the parameters
- batch= number of samples that will be processed in one batch.
- subdivisions= number of mini-batches in one batch; GPU processes mini-batch samples at once; the weights will be updated for batch samples, that is 1 iteration processes batch images.
- width=every image will be resized during training and testing to this number.
- height=every image will be resized during training and testing to this number.
- channels=every image will be converted during training and testing to this number.
- momentum=hyperparameter for optimizer that defines how much history will influence further updating of weights.
- decay= decay the learning rate over the period of the training.
- max_batches = total number of iterations .
- learning_rate=initial learning rate for training.
- angle= parameter that randomly rotates images during training
- saturation= parameter that randomly changes the saturation of images during training.
- exposure= parameter that randomly changes the brightness of images during training.
- hue= parameter that randomly changes the hue of images during training
Update the number of classes in every of three [yolo] layers at the end of the configuration files. And also update the number of filters in [convolutional] layers right before such every [yolo] layers but not anywhere else.
Great!,
Now the time to install other dependencies. (OpenCV, CUDA, cuDNN etc)
%cd /content/drive/My Drive/path/to/darknet folder!sed -i 's/GPU=0/GPU=1/g' Makefile!sed -i 's/OPENCV=0/OPENCV=1/g' Makefile!make
Code to run cuda on darkent
!chmod -R 777 darknetSince we are using google colab , the train command is
!./darknet detector train /content/drive/MyDrive/path/to/labelled_data.data /content/drive/MyDrive/path/to/train config file /content/drive/MyDrive/path/to/initial weight/darknet53.conv.74 -map -dont_showThats it ..
The training will start..
[ Note — one can stop the training process at any time and resume it from last saved weights]
For this just we need to replace the initial weight “darknet53.conv.74” with last saved weights in backup folder.
[–map — don_show :- to avoid real time display]
Replace “detector train” with “detector map” to evaluate the model (mAP of the model)
./darknet detector map /content/drive/MyDrive/path/to/labelled_data.data /content/drive/MyDrive/path/to/train config file
backup/yolo_v3_20000.weightsAnd for testing replace train.cfg to test.cfg and add image file location.
#testing command example
./darknet detector test cfg/labelled_data.data cfg/yolov3.cfg weights/yolov3.weights data/test-image.jpgParameters used for training are steps=20000, learning rate=0.001, input image size= 416x416x3, batch =64.
The observations found after training the YOLO model for 20000 steps are given below.
Total training loss was found to be 0.2075 and Mean Average precision of 85.3% for the test dataset. Total training time to converge the model was 18 hrs. Below figure shows the graphical representation of the training process in YOLO. Accuracy increases with the number of steps. While training, continuous monitoring of the performance of the model is done. During monitoring it was found that for above 19000 steps, increase in accuracy for validation dataset was found to be constant. So at 20000 steps the model is said to be converged. Hence it is shown that at 416x416x3 input size image and batch size of 64 with 0.01 learning rate the model has converged at 20000 steps.
Run this file , if you want to input whole image folder / video to the model. Provide required input arguments for it
Quick summary
- Preprocessed the images using LabelImg (image and its bounding box text file)
- Stored all images with text files in a single folder
- Created 4 files labelled.data,classs.names, train.txt and text.txt files
- Cloned darknet folder and downloaded initial weights
- Installed all dependencies (OpenCV,CUDA)
- Edit config files (hyperparameters-epochs,number of layers,filters etc)
- Run the train command.
- Run evaluation command.
Training SSD and Faster R CNN model using Tensorflow API in google colab
Creating files for training
After labeling the images, create TFRecord files from the training files to train the model. Tensorflow object detection model takes input files/data in .record format. The train configuration files were edited based on requirements.
At first, we have to create TFRecord files from train and test annotations. TFRecord is a binary file format that makes dealing with large datasets more efficient.
To generate .record files, we need train images with its annotations in csv file. This can be done through LabelImg software.
[Note — Yolo needs annotations in .txt file where as tensorflow API models need TFrecord files which is generated by using images and annotations in csv file]
TFRecord files can be generated using this link
or just run generate_tfrecord.py
Process of generating train and test record files.
First split the entire dataset to 2 file, test and train along with annotation file
This generate_tfrecord.py needs 3 input arguments
1. Path to images (test images for test record file and train image location for train record file)
2. Path to annotation csv file ( test image csv for test record file and train image csv for train record file)
3. Labelmap.pbtxt file [ This file is class label file]
Labelmap.pbtxt file should contain class details. As you can see in the below image, we have 2 classes EB-early blight , LB- late blight.
input_path:”/content/drive/MyDrive/object_detection/train.record” path to train.record file input_path:”/content/drive/MyDrive/object_detection/test.record” path to test.record file
python generate_tfrecord.py --path_to_images ../dataset/train/images\
--path_to_annot ../annotations/train_labels.csv \
--path_to_label_map ../models/labelmap.pbtxt \
--path_to_save_tfrecords ../data/tfrecord/train.recordCreating configuration files and clone tensorflow repository
Before downloading model weights, we can clone the tensorflow repository.
[Due to high computation requirement, I have used Google Colab for this]
I had uploaded both test record, train record files to my drive
Now the time clone/download tensorflow repositiory.
Code to clone
!git clone https://github.com/tensorflow/models.gitAfter cloning , I just saved the whole tensorflow folder in my drive.
Because, those loaded folders would get erased once the session gets ended in colab.
After cloning the folder, just need to run the setup.
.
%%bashcd /content/drive/MyDrive/Object_detection/models-master/research/protoc object_detection/protos/*.proto --python_out=.# Install TensorFlow Object Detection API.
# Note replace tf1 instead of tf2, if your tesorflow version is 1cp object_detection/packages/tf2/setup.py .python -m pip install .
To train our model, we need to configure the files , here you can get detailed explanation for it. After creating the TFRecord file select the model weights and configuration files.
I this project, I have used SSD with mobilenet and faster R CNN with resnet. So let’s first download the pretrained model from model zoo, and use it as initialization to our model.
You can select any model weights, based on your requirement and system configuration.
cd models/
# download the mobilenet_v2 model or any model
wget http://download.tensorflow.org/models/object_detection/tf2/20200711/ssd_mobilenet_v2_320x320_coco17_tpu-8.tar.gz
# extract the downloaded file
tar -xzvf ssd_mobilenet_v2_320x320_coco17_tpu-8.tar.gzOr
Just manually download , your required model and upload it to drive .[Bit easier way, I used this way]
Configure the files
The configuration files are available in the tensorflow model directory.
# config file location
models-master\research\object_detection\configs\tf2Note : Select the files based on selected weights/model
Since I selected ssd with mobilenet and Faster RCNN with resnet, I selected its corresponding config files.
Parameters to be changed in config file.
My SSD config file and Faster RCNN for reference
num_classes= 2
batch_size = 32
total_steps: 35000 for SSD and 50000 for Faster R CNN
fine_tune_checkpoint:”/content/drive/MyDrive/object_detection/ssd_mobilenet/checkpoint/ckpt-0"
fine_tune_checkpoint : Path to initial downloaded weights check point
[If you remember, earlier we had downloaded model weights from model zoo.]
Inside train_input_reader : Add labelmap and train record file location
Inside eval_input_reader : Add labelmap and test record file location
label_map_path:”/content/drive/MyDrive/object_detection/label map.pbtxt”
input_path:”/content/drive/MyDrive/object_detection/train.record”
path to train.record file
input_path:”/content/drive/MyDrive/object_detection/test.record”
path to test.record file
Training the object detection models in google colab
After creating all files for training, now it’s time to train the models.
Model training
!python /content/drive/MyDrive/Object_detection/models-master/research/object_detection/model_main_tf2.py \--pipeline_config_path=/content/drive/MyDrive/Object_detection/ssd_mobilenet_v2_320x320_coco17_tpu-8.config \--model_dir=/content/drive/MyDrive/Object_detection/trainingssd_final\--num_steps=35000
[model_dir : path to store trained weights]
model_main_tf2.py : This file handles model training part
Just give stored weight directory for — checkpoint_dir
It will automatically calculate evaluation metric mAP
The SSD model is trained using the mobilenet framework in google colab platform using GPU.
learning rate=0.001, input image size= 300x300x3, batch =32. The model is trained for 35,000 steps. The Faster RCNN model is trained on resnet as a base model with the help of tensorflow object detection API.
Faster R CNN model also takes input training files as train.record format.
This model is trained using resnet framework in google colab platform using GPU. Training parameters are 50,000 steps, batch size is 12 and input image size is 640x640x3 with learning rate 0.0003.
%load_ext tensorboard%tensorboard --logdir '/path/to/stored/trained/weights'
/content/drive/MyDrive/Object_detection/trainingssd_final
We get detailed metrics of our model, after running above code.
SSD and Faster R CNN
Both these models are trained using tensorflow API.
The SSD model is trained for 35000 steps whereas the faster R CNN model is trained for 50000 steps. Figure shows loss curve in SSD and Faster R CNN. The change in number of iterations for object detection models mainly depends on batch size, input image size, base model and model architecture to get convergence. For SSD model mAP of 0.84 was found for test dataset and total loss of 0.2532
Similarly for Faster R CNN model Mean Average precision was found to be 0.802 and
total loss was found 0.204.Faster R CNN model used higher time to detect the image than the SSD and YOLO model.
During training the model, tensorflow saves the trained model as a checkpoint. This checkpoints cant be used directly for detection purpose we need to export the models to a format that can be used for inference, this final format usually called saved model or frozen model.
!python /content/models/research/object_detection/exporter_main_v2.py \--trained_checkpoint_dir path/to/stored/trained/model/directory \--output_directory /path/to/ save/the inference model \--pipeline_config_path /path/to/config file
!python /content/models/research/object_detection/exporter_main_v2.py \
— trained_checkpoint_dir /content/drive/MyDrive/object_detection/trainingssd_final \
— output_directory /content/drive/MyDrive/object_detection/my_model\
— pipeline_config_path /content/drive/MyDrive/object_detection/ssd_mobilenet_v2_320x320_coco17_tpu-8.config
We will get these folders in output directory
Now just run detect_objects.py file to get detected output. We need one more file detector.py to run detection . These files can be found in tensorflow cloned directory
or else you can just download it from detect_objects.py and detector.py
python detect_objects.py --video_input --threshold 0.5 --model_path path/to/exported_model/saved_model \
--path_to_labelmap path/to/labelmap.pbtxt --video_path path/to/video.mp4 fileBelow are the input arguments ,to be provided while testing the models
Great!
We have done with all 3 models
1. yolov3.weights
2. SSD inference model
3. Faster RCNN inference model
Evaluating the performance of object detection models with actual field images.
All three models were tested with 100 field images of potato crops. The results are as follows:
Yolo v3
The YOLO v3 model with darknet framework when tested with 100 field images captured at varied climatic conditions achieved 70.15% mAP . The model detects late blight disease better than early blight disease. Average precision for late blight and early blight disease was found to be 0.76 and 0.64 respectively.
SSD
The SSD model with mobilenet framework when tested with field images captured at varied climatic conditions achieved 65.9% mAP
This model also detects late blight disease better than early blight disease. When compared to the YOLO model, the performance of this model is low. Average precision for late blight and early blight disease was found to be 0.68 and 0.64 respectively
Faster R CNN
The Faster R CNN model trained with resnet architecture when tested with field images achieved 66.5% mAP . This model detects early blight disease well compared to late blight disease. Average precision for late blight and early blight disease was found to be 0.75 and 0.58 respectively
Comparison of different object detection models
Below are some of the disease detected field images from object detection models. By visualizing the images we can conclude that the model performs well in varied climatic condition
Please clap/follow if the content is useful
Thank you
