Classification peripheral blood cell
Authors: Silvia Garcia Hernandez, Jorge Gonzalez Sierra and Maria Ortiz Rodriguez
Introduction
Blood tests have become a direct means to detect or diagnose diseases. Within these, we can find several tests such as the blood smear, where different diseases such as jaundice, anemia, different types of cancer, etc. can be distinguished.
All these tests are performed by a complete blood count (CBC) which measures the amount of the different components of the blood by identifying and counting them. This is usually performed by a specialized hematologist using flow cytometry, investing a great deal of time and/or resources in the diagnosis in order to obtain reliable results. This is time consuming and the results can be fallible. [1]
With the development of new computational techniques such as machine and deep learning, the doors were opened to automatic blood smear counting models, with the limitation that the images may be of low quality and/or blurred.
However, technology is advancing and the development of convolutional neural networks (CNNs) successfully apply image recognition tasks, so they have become the most promising line of development.
CNNs have the advantage that they can be easily integrated into other systems or interfaces as will be developed later in this paper.
Motivation of the project
Every day, healthcare professionals spend a great deal of time and resources to diagnose diseases through this type of tests, which could be automated and/or expedited by properly trained artificial intelligence programs.
An early diagnosis is essential for a patient with, for example, leukemia, to receive the corresponding treatment and the heavy workload to which these healthcare professionals are subjected prevents them from being given in a period of less than 24–48 hours.
The first step in obtaining a hematological diagnosis is the detection, classification and counting of the cell types found in blood. Therefore, this project has focused on automating the classification of these cell types with an accuracy of over 99%.
Project structure
The dataset (https://data.mendeley.com/datasets/snkd93bnjr/1) has been used, which is divided into 8 classes, detailed in the next paragraph, and contains a total of 17092 images of individual cells divided into classes. The size of the images is 360 x 363 pixels in jpg format [2].
The dataset used consisted of 8 image subclasses or classes; basophils (basophil), eosinophils (eosinophil), erythroblasts (erythroblast), immature granulocytes (ig), lymphocytes (lymphocyte), monocytes (monocyte), neutrophils (neutrophil) and platelets (platelet).
Before starting to implement the model, the data set has been divided into training, validation and test; sets necessary both to train the model and to validate it later. This has been done with the splitfolders library, as shown below:
import splitfolders
splitfolders.ratio(‘../../PBC_dataset_normal_DIB’, output = “../../PCB_Dataset_Split”, seed = 42, ratio = (0.8, 0.1, 0.1))
Once the dataset has been studied and processed, a Transfer Learning methodology has been used for image classification, importing the pre-trained vgg19 model from tensorflow obtaining an accuracy higher than 99%.
Multi-class classifier using Transfer Learning
A Transfer Learning methodology consists of importing an already trained network into a deep learning model. It is generally imported without the last layer of the network, since the size of this layer varies according to the number of classes of the problem in question.
In this case, the VGG19 pre-trained network architecture has been imported; a variant of the VGG model, which consists of 19 layers (16 convolution and 3 connected layers, 5 MaxPool layers and one Softmax layer). The following image represents the structure of this network:
The VGG19 input is from (224, 224, 3) which was maintained, altering the size of the images through the Tensorflow generators. The example code can be found below:
train_generator = train_datagen.flow_from_directory(
____train_data_dir,
____target_size = (img_width, img_height),
____ keep_aspect_ratio = True,
____batch_size = batch_size,
____class_mode = 'categorical')
validation_generator = val_datagen.flow_from_directory(
____validation_data_dir,
____target_size = (img_width, img_height),
____keep_aspect_ratio = True,
____batch_size = batch_size,
____class_mode = 'categorical')
If there are more doubts about how to create these generators, this Medium article details in some depth how to use them.
A flatten layer was added to the model in the output and a dense layer, which is named “prediction” in the code.
vgg = VGG19(input_shape = [img_width, img_height] + [3], weights = ‘imagenet’, include_top = False)
for layer in vgg.layers:
____layer.trainable = True
x = vgg.output
x = Flatten()(x)
prediction = Dense(len(folders), activation = ‘softmax’)(x)
model = Model(inputs = vgg.input, outputs = prediction)
EarlyStopping, modelCheckpoint and ReduceLROnPlateu were used for training during 25 epochs:
->EarlyStopping: is one of the Keras callbacks that allows to automate an end of training when one of the monitored metrics stops improving.
-> modelCheckpoint: it is used in the callback for training. It allows to save a model or the weights every certain interval, so you can keep the model until a certain epoch and thus save the best result.
-> ReduceLROnPlateu: is another Keras callback like EarlyStopping that allows, in its case, to reduce the learning rate when a metric has stopped improving. This metric can be chosen.
Results obtained
The results obtained were quite promising with an accuracy rate of over 99% in all classes.
Explainability
Since deep learning models act as a black box, tools are applied to check what these models “observe” in order to classify them. In this case, the tool used is GRADCAM. This tool looks for which parts of the image have led a convolutional neural network to its final decision. This is done by means of heat maps representing the activation classes on the input images received.
These classes will allow to indicate the importance of each pixel with respect to the class in question by increasing or decreasing the intensity of the pixel.
An example of Gradcam applied to this work would be the following:
The class activation map attributes an importance to each position (x, y) in the last convolutional layer by calculating the linear combination of activations, weighted by the corresponding output weights for the observed class. This is shown with a heat map (a heatmap) which can be seen on the bottom left. You can see in the bottom right image, the original image with the heatmap superimposed, which shows us what the model is set to classify (right image).
If you want to know more about GradCam, here is a link to the documentation of this library in Keras.
Tensorflow projector
Tensorflow Projector (https://projector.tensorflow.org/) is a visual tool that allows us to interact and analyze multidimensional data (embeddings) and project them in a two or three-dimensional space. Each embedding is represented by a point that has a certain position in space and these will form certain clusters based on a similarity score. Thanks to this tool, we are able to observe how the model is able to distinguish the different classes (ig, leukocytes, etc), and where it has the most problems to distinguish them by the appearance of certain points of different classes within a cluster of a different class. In short, this visualizer is nothing more than a tool to be able to observe the confusion matrix.
In order to obtain the embeddings, the test images have been introduced in our vgg19 model and the data obtained in the layer before the flattening, i.e. in the penultimate layer of the model, have been collected, removing the dense layer that had been added as the last one. These data have 25088 dimensions, which makes their visualization impossible. Dimensionality reduction methods such as Principal Component Analysis (PCA), t-stochastic neighbor embedding (t-SNE) or uniform manifold approximation and projection (uMAP) included in Tensorflow Projector allow us to visualize our embeddings in three dimensions.
In this case, the t-SNE method has been used to visualize the confusion that may exist in the model when it comes to accurately predict to which group each cell belongs. This algorithm constructs a probability distribution over pairs of embeddings in space, so that the most similar ones are more likely to be included in the same cluster, reducing the dimensionality of the sample [3].
As can be seen in this figure, there are several insertions of certain groups within clusters belonging to other classes. In this case, the model is more confused giving a correct classification when dealing with neutrophils and immature granulocytes. Other notable insertions are erythroblasts, which are confused with platelets, neutrophils with basophils, and immature granulocytes with monocytes. Even so, the accuracy of the model in classifying the different cell types is very high.
Streamlit
Streamlit is an open source Python library that allows to create graphical interfaces in a simple way and to develop applications thanks to the interaction with other libraries to display web pages, calculators, programs and results of models used in machine and deep learning. The advantage of using this library, compared to other higher level libraries, is that no front-end knowledge is required.
The streamlit related to this work can be found in our github
The streamlit of this project was structured in several sections:
-> A main page where you can insert a picture for sorting.
When loading an image in the model tab, the result of the model will be displayed with a pie chart and numerical results of the prediction, along with a tab to switch from these results to the explainability part with the Gradcam library previously developed.
-> An “about the project” page where this project is put in context.
-> An “about us” page where information about the team members is given.
It also incorporates a “slide bar” that allows to change the language from Spanish to English and vice versa.
Conclusions, difficulties and next steps
As a conclusion, it can be observed on the Internet, that this type of work is widely studied.
The main difficulty that has been observed in this project has been the streamlit library, which despite having extensive documentation and examples, to improve the quality of the interface you need to integrate HTML.
For this type of models to be successfully deployed in specialized centers, they must have a blood cell detection and counting part. Therefore, the next steps will consist of a blood cell detection model that allows from an image with a multitude of cells, to isolate a particular cell and classify it, repeating the same process until all the cells available in the image are available. Subsequently, the cells will be numbered.
The code for this work can be found in the following github repository
References:
[1] Habibzadeh, M.; Krzyzak, A.; Fevens, T. White Blood Cell Differential Counts Using Convolutional Neural Networks for Low Resolution Images. En Proceedings of the International Conference on Artificial Intelligence and Soft Computing, Zakopane, Polonia, 9–13 de junio de 2013; pp. 263–274.
[2] Acevedo, Andrea; Merino, Anna; Alférez, Santiago; Molina, Ángel; Boldú, Laura; Rodellar, José (2020), “A dataset for microscopic peripheral blood cell images for development of automatic recognition systems”, Mendeley Data, V1, doi: 10.17632/snkd93bnjr.1
[3] Visualización de datos mediante t-SNE [PDF] Maaten, L.v.d. y Hinton, G., 2008. Journal of Machine Learning Research, Vol 9(Nov), pp. 2579–2605.
More Artificial Intelligence!
Saturdays.ai mission is to make AI more accessible (#ai4all) through collaborative learning where we build projects for good (#ai4good)..
If you want to learn and build AI, no matter what is your starting point, join our community at www.saturdays.ai , we root for you!
