Detection/Classification of Breast Cancer using thermograpy
October is the month of breast cancer awareness and prevention, marked by global mobilization. Each year, through various events and campaigns, Pink October draws the public’s attention to this disease that affects millions of women worldwide. It is a reminder that the fight against breast cancer is a collective challenge, demanding constant innovation in screening and diagnosis techniques.
One of these major advancements is the combination of medical thermography and artificial intelligence (AI) for early breast cancer detection. This fusion of science and technology offers new hope to women worldwide by improving the accuracy, convenience, and early detection of breast cancer, without the drawbacks of traditional methods.
Here are some scientific articles that deal with this subject :
- https://www.ingentaconnect.com/content/asp/jmihi/2014/00000004/00000001/art00015;jsessionid=g4sv3uvgrwl1.x-ic-live-02
- https://www.mdpi.com/1424-8220/21/14/4802
- https://www.sciencedirect.com/science/article/pii/S0010482521003474
In this October of 2023, amidst the Pink October campaign, I have embarked on an exploration of the subject of Breast Cancer detection through Thermography and Artificial Intelligence. My objective is to assess the efficacy of the models I can establish in this context.
Nvidia Jetson nano developer kit
For this Artificial Intelligence project for breast cancer detection on thermal images I used the Nvidia Jetson Nano developer kit.
The Nvidia Jetson Nano Developer Kit is a powerful and compact platform specifically designed for embedded artificial intelligence (AI) applications. Featuring a Quad-core ARM Cortex-A57 processor, an NVIDIA Maxwell GPU with 128 CUDA cores, and 4 GB of LPDDR4 memory, the Jetson Nano provides significant computing power for deep learning tasks and image processing. It is particularly well-suited for projects requiring embedded intelligence, such as object recognition, image classification, and other AI-related applications. The user-friendly nature and flexibility of the Jetson Nano make it a popular choice among developers looking to integrate AI capabilities into embedded devices and standalone projects.
Data resource
Dataset
In the present investigation, a publicly accessible dataset comprising thermographic images related to breast cancer was employed for diagnostic purposes. This dataset encompasses a total of 3,997images, featuring 3,024 images representing healthy cases and 973 images depicting breast diseases. These images are stored in JPEG format with dimensions of 640x480 pixels. Within the dataset, there are 179 images depicting healthy breast conditions and 101 images corresponding to patients with breast conditions. This dataset was sourced from http://visual.ic.uff.br/dmi/. Sample images from the dataset are provided in below.
Thermography
At specific temperatures, objects naturally emit thermal signals. The nature of these signals, as well as the temperature range, depends on the characteristics of the object. Under usual conditions, the human body, like other objects, emits Infrared (IR) signals. Due to the heat generated by different bodily functions, the emitted IR signals vary from one region to another. This principle finds frequent application in medical examinations, especially in the context of breast cancer screenings.
Breast cancer development is associated with inflammation and increased blood vessel formation, both of which generate higher temperature profiles. Thermography, often referred to as thermal imaging, offers a non-invasive method for capturing thermal maps of specific body areas in the field of medicine. This approach is contactless, non-destructive, and does not involve radiation, making it suitable for repeated use. In the context of breast cancer screening, a thermographic camera is employed to obtain thermal maps of the breasts and their surroundings, highlighting any anomalies.
The resurgence of thermography in medical applications has brought it into focus as an adjunct to image processing techniques, particularly for the diagnosis of breast cancer. Breast thermography capitalizes on the temperature differences beneath the skin between healthy and cancerous breasts. The presence of a breast tumor raises the temperature of the surrounding tissues. Typically, specialists conduct an asymmetric comparison of healthy and affected breasts.
The process of using thermography for breast cancer screening is relatively straightforward. It commences with a physical examination of the breast’s surface, allowing the physician to correlate any abnormalities with the thermal map. Subsequently, the individual is required to spend 15 minutes at room temperature to acclimatize. This is carried out in a controlled environment where both humidity and temperature are regulated. During this period, the individual needs to expose the upper body from the waist to the chin. Once the body temperature reaches equilibrium, the individual is asked to place their hands on their sides for the relevant surfaces to be observed. The imaging procedure is then initiated to complete the process.
Image processing
Histogram Equalization
Histogram equalization is an image processing technique designed to improve the distribution of grayscale levels in an image. By adjusting the distribution of brightness intensities, this method enhances the overall contrast of the image, thereby improving visibility of details and structures. It is particularly useful in correcting brightness and contrast disparities, enabling a better visual interpretation of images, and finds applications in various fields such as computer vision, object recognition, and medical imaging.
The use of histogram equalization on thermal images in the context of breast cancer detection is of crucial importance in the analysis and interpretation process. Thermal images, acquired through techniques such as medical thermography, capture subtle temperature variations on the skin’s surface, providing a thermal representation of the body. However, these images may exhibit disparities in contrast and brightness that can compromise the ability to identify significant thermal anomalies. Histogram equalization emerges as an essential method in preprocessing these thermal images, aiming to enhance the visibility of crucial thermal structures. By adjusting the distribution of grayscale levels in the image, histogram equalization strengthens subtle thermal features, thereby facilitating the detection of abnormal regions associated with breast cancer. This approach, integrated into the thermal image analysis process, can play a decisive role in improving the sensitivity and specificity of breast cancer screening methods based on thermography, contributing to earlier diagnoses and more effective interventions.
You will find here the link to the source code used for the image processing described above. https://github.com/JulienMilon/Breast-Thermogram-Cancer-Classification/blob/jmn_dev/Image%20Processing.ipynb
In this story, the images are used in their entirety. Another possibility could have been to cut out the breasts in the images to identify which breasts are healthy and which are sick. This work may be addressed in a future publication but not in this story.
Split Images 70% for training and 30% for test
In the field of image classification, creating a balanced and representative dataset is crucial for effectively training and evaluating models. Ideally, the total set of images is divided into two main parts: a training set, typically representing 70% of the total, and a test/validation set, representing the remaining 30%. The training set is used to train the model, allowing it to learn essential features and patterns in the images. Subsequently, the test/validation set, which has not been seen by the model during training, is used to assess the model’s actual performance. This separation ensures that the model can generalize effectively to new data, thereby enhancing its reliability and ability to make accurate classifications in real-world scenarios.
You will find here the link to the source code used for the split image process described above. https://github.com/JulienMilon/Breast-Thermogram-Cancer-Classification/blob/main/Split%20Images.ipynb
Classification of Breast Cancer using thermograpy using pre-trained models
In the realm of image classification, the adoption of pre-trained models represents an innovative strategy leveraging neural networks already trained on extensive datasets. This approach capitalizes on the capability of pre-trained models, such as ResNet18, to extract and comprehend intricate features from a variety of images. ResNet18, in particular, is built upon a residual architecture, introducing direct connections between layers to facilitate the learning of hierarchical representations. The use of pre-trained models allows harnessing prior knowledge gained from similar tasks, providing a more robust initialization for new classification tasks.
In this story, we delve into the various stages of integrating ResNet18, previously trained on extensive datasets, thereby enhancing its ability to capture intricate information. While we also employ other pre-trained models for image classification, we will confine ourselves to presenting and interpreting the results for these models. Indeed, the method and steps will remain identical to those outlined in the detailed presentation of ResNet18. This approach unveils significant potential across diverse applications, ranging from object recognition to the detection of medical pathologies, underscoring the substantial impact of pre-trained models in the field of computer vision.
Installation and prerequisites
On this project it is necessary to install and import the following Python libraries:
- os (already present when you install Python)
- random (already present when you install Python)
- time(already present when you install Python)
- copy (already present when you install Python)
- numpy
- pandas
- cv2 (opencv)
- matplotlib
- seaborn
- sklearn
- torch (Pytorch)
- torchvision
Advice : It is preferable to use Python environments such Anaconda to install your Python libraries and only use them when the environment is available.
In this story, I only explain how to install Anaconda and the Python libraries on a Nvidia Jeston Nano Developer Kit based on Ubuntu, but it is possible to do it on other devices / computer / operating system by other ways.
$ sudo apt update
$ sudo apt upgrade
$ sudo apt install python3-h5py libhdf5-serial-dev hdf5-tools
python3-matplotlib python3-pip libopenblas-base libopenmpi-devIn this tutorial, I used Archiconda. Run the following codes to download and install it.
$ wget https://github.com/Archiconda/build-tools/releases/download/0.2.3/Archiconda3-0.2.3-Linux-aarch64.sh
$ sudo sh Archiconda3-0.2.3-Linux-aarch64.sh
$ conda create -n myenv python=3.6 # myenv is the name of the environnement but you choose itTo enable or disable (close) the Python environment use the following commands :
$ conda activate myenv
$ conda deactivate myenvBefore installing the Python libraries, check that you have activated the desired Python environment to install the packages in the environment.
For the next commands you can use conda or pip3 :
$ conda install matplotlib pandas numpy scikit-image scikit-learn seaborn cython -c conda-forge
$ conda install -c conda-forge opencv
$ conda install -c conda-forge scikit-learn
$ wget https://nvidia.box.com/shared/static/p57jwntv436lfrd78inwl7iml6p13fzh.whl -O torch-1.8.0-cp36-cp36m-linux_aarch64.whl -O torch-1.9.0-cp36-cp36m-linux_aarch64.whl
$ pip3 install torch-1.9.0-cp36-cp36m-linux_aarch64.whl
$ wget https://drive.google.com/uc?id=1tU6YlPjrP605j4z8PMnqwCSoP6sSC91Z
$ pip3 install torchvision-0.10.0a0+300a8a4-cp36-cp36m-linux_aarch64.whlImport & setup configuration
Below is the list of Python libraries necessary for the implementation and evaluation of our breast cancer detection model based on the Resnet18 pre-trained model.
# Set up CUDA in OS
import os
os.environ['CUDA_LAUNCH_BLOCKING'] = '1'
# Import libabries
import torch
import torch.nn as nn
import torch.optim as optim
import torch.nn.functional as F
import numpy as np
import seaborn as sn
import pandas as pd
import torchvision
from torchvision import *
from torch.utils.data import Dataset, DataLoader
from torchvision.io import read_image
import torchvision.transforms as T
from torchvision import datasets, models, transforms
from sklearn.metrics import accuracy_score, classification_report, confusion_matrix
import matplotlib.pyplot as plt
import time
import copy
# Ignore warnings
import warnings
warnings.filterwarnings("ignore")To see what resources would be used GPU or CPU.
# Setup device agnostic code
device = "cuda" if torch.cuda.is_available() else "cpu"
device"cuda"Data Preprocess
Because Resnet accepts input image sizes of (224 * 224), the image must be resized to be (224 * 224), preprocessing for our data, which entails random horizontal flip, rotation, normalization, etc., needs to be defined at first.
# Create transform function
transforms_train = transforms.Compose([
transforms.Resize((224, 224)), #must same as here
transforms.RandomResizedCrop(224),
transforms.RandomHorizontalFlip(), # data augmentation
transforms.ToTensor(),
transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]) # normalization
])
transforms_test = transforms.Compose([
transforms.Resize((224, 224)), #must same as here
transforms.CenterCrop((224, 224)),
transforms.ToTensor(),
transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
])# Apply for training and test data
train_dir = "../../../DataSet/Transformed - DMR - Database For Mastology Research - Visual Lab/train"
test_dir = "../../../DataSet/Transformed - DMR - Database For Mastology Research - Visual Lab/test"
train_dataset = datasets.ImageFolder(train_dir, transforms_train)
test_dataset = datasets.ImageFolder(test_dir, transforms_test)
train_dataloader = torch.utils.data.DataLoader(train_dataset, batch_size=12, shuffle=True, num_workers=0)
test_dataloader = torch.utils.data.DataLoader(test_dataset, batch_size=12, shuffle=False, num_workers=0)Resnet18
By using models.resnet18(pretrained=True), we can call a pre-trained model of ResNet18 from The Pytorch API.
model_resnet18 = models.resnet18(pretrained=True)
model_resnet18ResNet(
(conv1): Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)
(bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
(maxpool): MaxPool2d(kernel_size=3, stride=2, padding=1, dilation=1, ceil_mode=False)
(layer1): Sequential(
(0): BasicBlock(
(conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
(conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
(1): BasicBlock(
(conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
(conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
)
(layer2): Sequential(
(0): BasicBlock(
(conv1): Conv2d(64, 128, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
(bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
(conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(downsample): Sequential(
(0): Conv2d(64, 128, kernel_size=(1, 1), stride=(2, 2), bias=False)
(1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
)
(1): BasicBlock(
(conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
(conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
)
(layer3): Sequential(
(0): BasicBlock(
(conv1): Conv2d(128, 256, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
(bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
(conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(downsample): Sequential(
(0): Conv2d(128, 256, kernel_size=(1, 1), stride=(2, 2), bias=False)
(1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
)
(1): BasicBlock(
(conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
(conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
)
(layer4): Sequential(
(0): BasicBlock(
(conv1): Conv2d(256, 512, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
(bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
(conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(downsample): Sequential(
(0): Conv2d(256, 512, kernel_size=(1, 1), stride=(2, 2), bias=False)
(1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
)
(1): BasicBlock(
(conv1): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
(conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
)
(avgpool): AdaptiveAvgPool2d(output_size=(1, 1))
(fc): Linear(in_features=512, out_features=1000, bias=True)
)This is a visualization of ResNet-18 architecture:
Customize model
The customization consists of freezing the base model and changing the output layer to meet our requirements.
Check the output feature and add a newly fully-connected layer from the pre-trained model.
num_features = model_resnet18.fc.in_features
print('Number of features from pre-trained model', num_features)Number of features from pre-trained model 512As we can see, there are 512 output features from the original model; therefore, we need to change this by adding a fully connected layer for our classification problem with only 2 target classes.
# Add a fully-connected layer for classification
model_resnet18.fc = nn.Linear(num_features, 2)
model_resnet18 = model_resnet18.to(device)Train model
As we have a customized classifier for our cancer problem, now we start to train our model with our data and Images.
To begin training, we define loss function and an optimizer first.
Using Cross Entropy Loss in the context of classification is justified by its effective ability to measure the divergence between model predictions and true class labels. This loss function facilitates precise learning of relationships between data features and their respective classes. When coupled with the Stochastic Gradient Descent (SGD) optimizer, this combination provides a robust framework for the iterative adjustment of model weights. The SGD, by progressively adjusting weights to minimize Cross Entropy Loss at each iteration, guides the model towards optimal parameters, contributing to an iterative improvement in classification performance.
# Define loss and optimizer
criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(model.parameters(), lr=0.0001, momentum=0.9)# Set the random seeds
torch.manual_seed(42)
torch.cuda.manual_seed(42)In the realm of deep learning and image classification, the choice of the number of epochs, or complete iterations through the training dataset, is a pivotal decision. The frequent selection of approximately 30 epochs in our programs arises from a delicate balance between model convergence and temporal efficiency. This value represents a judicious compromise, allowing the model to adapt to the complex features of the training data while avoiding excessive overfitting.
#### Train model
train_loss=[]
train_accuary=[]
test_loss=[]
test_accuary=[]
num_epochs = 30 #(set no of epochs)
start_time = time.time() #(for showing time)
# Start loop
for epoch in range(num_epochs): #(loop for every epoch)
print("Epoch {} running".format(epoch)) #(printing message)
""" Training Phase """
model_resnet18.train() #(training model)
running_loss = 0. #(set loss 0)
running_corrects = 0
# load a batch data of images
for i, (inputs, labels) in enumerate(train_dataloader):
inputs = inputs.to(device)
labels = labels.to(device)
# forward inputs and get output
optimizer.zero_grad()
outputs = model_resnet18(inputs)
_, preds = torch.max(outputs, 1)
loss = criterion(outputs, labels)
# get loss value and update the network weights
loss.backward()
optimizer.step()
running_loss += loss.item()
running_corrects += torch.sum(preds == labels.data).item()
epoch_loss = running_loss / len(train_dataset)
epoch_acc = running_corrects / len(train_dataset) * 100.
# Append result
train_loss.append(epoch_loss)
train_accuary.append(epoch_acc)
# Print progress
print('[Train #{}] Loss: {:.4f} Acc: {:.4f}% Time: {:.4f}s'.format(epoch+1, epoch_loss, epoch_acc, time.time() -start_time))
""" Testing Phase """
model.eval()
with torch.no_grad():
running_loss = 0.
running_corrects = 0
for inputs, labels in test_dataloader:
inputs = inputs.to(device)
labels = labels.to(device)
outputs = model_resnet18(inputs)
_, preds = torch.max(outputs, 1)
loss = criterion(outputs, labels)
running_loss += loss.item()
running_corrects += torch.sum(preds == labels.data).item()
epoch_loss = running_loss / len(test_dataset)
epoch_acc = running_corrects / len(test_dataset) * 100.
# Append result
test_loss.append(epoch_loss)
test_accuary.append(epoch_acc)
# Print progress
print('[Test #{}] Loss: {:.4f} Acc: {:.4f}% Time: {:.4f}s'.format(epoch+1, epoch_loss, epoch_acc, time.time()- start_time))Epoch 0 runningEpoch 0 running
[Train #1] Loss: 0.0463 Acc: 74.8130% Time: 1128.7280s
[Test #1] Loss: 0.0436 Acc: 75.2101% Time: 1260.4748s
Epoch 1 running
[Train #2] Loss: 0.0420 Acc: 76.4874% Time: 2042.1018s
[Test #2] Loss: 0.0395 Acc: 76.8908% Time: 2159.3111s
Epoch 2 running
[Train #3] Loss: 0.0392 Acc: 77.9124% Time: 2944.1674s
[Test #3] Loss: 0.0362 Acc: 78.1513% Time: 3062.6527s
Epoch 3 running
[Train #4] Loss: 0.0373 Acc: 79.5867% Time: 3847.8924s
[Test #4] Loss: 0.0324 Acc: 85.6303% Time: 3966.0144s
Epoch 4 running
[Train #5] Loss: 0.0338 Acc: 81.5105% Time: 4756.8147s
[Test #5] Loss: 0.0284 Acc: 85.1261% Time: 4875.7575s
Epoch 5 running
[Train #6] Loss: 0.0315 Acc: 83.2561% Time: 5669.3215s
[Test #6] Loss: 0.0276 Acc: 84.1176% Time: 5788.9542s
Epoch 6 running
[Train #7] Loss: 0.0295 Acc: 85.1443% Time: 6583.6435s
[Test #7] Loss: 0.0251 Acc: 86.0504% Time: 6703.5977s
Epoch 7 running
[Train #8] Loss: 0.0280 Acc: 85.7855% Time: 7499.8074s
[Test #8] Loss: 0.0207 Acc: 90.3361% Time: 7620.0305s
Epoch 8 running
[Train #9] Loss: 0.0268 Acc: 86.8899% Time: 8467.3604s
[Test #9] Loss: 0.0189 Acc: 90.8403% Time: 8588.3759s
Epoch 9 running
[Train #10] Loss: 0.0258 Acc: 87.3530% Time: 9381.6568s
[Test #10] Loss: 0.0191 Acc: 90.5042% Time: 9500.7094s
Epoch 10 running
[Train #11] Loss: 0.0252 Acc: 87.1393% Time: 10289.8637s
[Test #11] Loss: 0.0171 Acc: 91.7647% Time: 10408.4738s
Epoch 11 running
[Train #12] Loss: 0.0254 Acc: 87.2462% Time: 11197.5781s
[Test #12] Loss: 0.0165 Acc: 92.3529% Time: 11316.1268s
Epoch 12 running
[Train #13] Loss: 0.0227 Acc: 89.3124% Time: 12104.2549s
[Test #13] Loss: 0.0156 Acc: 91.8487% Time: 12222.4669s
Epoch 13 running
[Train #14] Loss: 0.0216 Acc: 88.9206% Time: 13016.0135s
[Test #14] Loss: 0.0146 Acc: 93.4454% Time: 13136.6813s
Epoch 14 running
[Train #15] Loss: 0.0212 Acc: 90.0606% Time: 13924.7683s
[Test #15] Loss: 0.0137 Acc: 94.0336% Time: 14043.0967s
Epoch 15 running
[Train #16] Loss: 0.0213 Acc: 89.6331% Time: 14829.3007s
[Test #16] Loss: 0.0129 Acc: 94.4538% Time: 14947.2660s
Epoch 16 running
[Train #17] Loss: 0.0188 Acc: 90.8799% Time: 15735.3024s
[Test #17] Loss: 0.0118 Acc: 95.3782% Time: 15853.8985s
Epoch 17 running
[Train #18] Loss: 0.0207 Acc: 89.7756% Time: 16641.0730s
[Test #18] Loss: 0.0118 Acc: 95.2101% Time: 16759.7112s
Epoch 18 running
[Train #19] Loss: 0.0191 Acc: 89.9893% Time: 17548.5152s
[Test #19] Loss: 0.0106 Acc: 95.6303% Time: 17666.6510s
Epoch 19 running
[Train #20] Loss: 0.0190 Acc: 90.8443% Time: 18454.4315s
[Test #20] Loss: 0.0109 Acc: 95.6303% Time: 18572.4112s
Epoch 20 running
[Train #21] Loss: 0.0178 Acc: 91.4499% Time: 19360.7066s
[Test #21] Loss: 0.0104 Acc: 95.1261% Time: 19478.9303s
Epoch 21 running
[Train #22] Loss: 0.0179 Acc: 91.9131% Time: 20266.1401s
[Test #22] Loss: 0.0106 Acc: 95.2941% Time: 20384.4787s
Epoch 22 running
[Train #23] Loss: 0.0190 Acc: 90.3812% Time: 21170.5141s
[Test #23] Loss: 0.0117 Acc: 94.7899% Time: 21288.6383s
Epoch 23 running
[Train #24] Loss: 0.0171 Acc: 91.7349% Time: 22073.5999s
[Test #24] Loss: 0.0107 Acc: 95.2101% Time: 22191.2150s
Epoch 24 running
[Train #25] Loss: 0.0172 Acc: 91.1293% Time: 22978.7546s
[Test #25] Loss: 0.0102 Acc: 94.7059% Time: 23097.5716s
Epoch 25 running
[Train #26] Loss: 0.0160 Acc: 92.2693% Time: 23882.2570s
[Test #26] Loss: 0.0106 Acc: 94.8739% Time: 24000.2315s
Epoch 26 running
[Train #27] Loss: 0.0169 Acc: 91.3787% Time: 24786.0575s
[Test #27] Loss: 0.0097 Acc: 95.3782% Time: 24903.6320s
Epoch 27 running
[Train #28] Loss: 0.0158 Acc: 92.0912% Time: 25682.4834s
[Test #28] Loss: 0.0104 Acc: 95.5462% Time: 25799.5185s
Epoch 28 running
[Train #29] Loss: 0.0163 Acc: 91.8062% Time: 26579.3982s
[Test #29] Loss: 0.0107 Acc: 95.2101% Time: 26696.4536s
Epoch 29 running
[Train #30] Loss: 0.0161 Acc: 92.3762% Time: 27483.3642s
[Test #30] Loss: 0.0095 Acc: 96.0504% Time: 27601.5414sWe must now save our trained model for testing and future use. For me the training took around 7–8 hours.
save_path = 'models/breast_thermo_cancer-classifier_resnet_18_final.pth'
torch.save(model_resnet18.state_dict(), save_path)Evaluate model
Analyzing the output logs reveals the commendable performance of our model on the new dataset (our dataset). Yet, for a comprehensive evaluation of its performance over time, let’s examine the corresponding plot.
# Plot
plt.figure(figsize=(6,6))
plt.plot(np.arange(1,num_epochs+1), train_accuary,'-o')
plt.plot(np.arange(1,num_epochs+1), test_accuary,'-o')
plt.xlabel('Epoch')
plt.ylabel('Accuracy')
plt.legend(['Train','Test'])
plt.title('Train vs Test Accuracy over time')
plt.show()
From the 16th Epoch, we have stable test accuracies of around 95%, while training accuracy keeps increasing until reaching a peak of around 91% from the 22nd Epoch.
# Get data to check on the performance of each label
y_pred = []
y_true = []
#model_resnet18.load_state_dict(torch.load('models/breast_thermo_cancer-classifier_resnet_18_final.pth'))
num_epochs = 30 #(set no of epochs)
start_time = time.time() #(for showing time)
# Start loop
for epoch in range(num_epochs): #(loop for every epoch)
model_resnet18.eval()
with torch.no_grad():
for inputs, labels in test_dataloader:
inputs = inputs.to(device)
labels = labels.to(device)
outputs = model_resnet18(inputs) # Feed Network
outputs = (torch.max(torch.exp(outputs), 1)[1]).data.cpu().numpy()
y_pred.extend(outputs) # Save Prediction
labels = labels.data.cpu().numpy()
y_true.extend(labels) # Save Truth# Visualization and result
# constant for classes
classes = test_dataset.classes
# Build confusion matrix
print("Accuracy on Training set: ",accuracy_score(y_true, y_pred))
print('Confusion matrix: \n', confusion_matrix(y_true, y_pred))
print('Classification report: \n', classification_report(y_true, y_pred))
# Plot
cf_matrix = confusion_matrix(y_true, y_pred)
df_cm = pd.DataFrame(cf_matrix, index = [i for i in classes], columns = [i for i in classes])
plt.figure(figsize = (7,7))
plt.title("Confusion matrix for Breast Cancer classification ")
sn.heatmap(df_cm, annot=True)Accuracy on Training set: 0.9605042016806723
Confusion matrix:
[[26160 720]
[ 690 8130]]
Classification report:
precision recall f1-score support
0 0.97 0.97 0.97 26880
1 0.92 0.92 0.92 8820
accuracy 0.96 35700
macro avg 0.95 0.95 0.95 35700
weighted avg 0.96 0.96 0.96 35700
<AxesSubplot:title={'center':'Confusion matrix for Breast Cancer classification '}>
The model demonstrates an accuracy of 97% in classifying class 0 — “Healthy” and a slightly lower accuracy of 92% for class 1 — “Sick”. These results are promising and indicate that the model can effectively differentiate between the two classes based on thermal images. The model’s evaluation through the F1 score is also satisfactory and encouraging, largely due to the low number of false positives and true negatives. These observations reinforce the model’s validity in accurately distinguishing between “Healthy” and “Sick” states from thermal data.
Criticism of the model and the dataset
The performance of the model based on ResNet18 shows promise, achieving a correct classification rate of approximately 95–96% (test and evaluation) . The confusion matrix further emphasizes the rarity of false positives and true negatives, reinforcing the model’s validity in accurately detecting breast cancer from thermal images.
However, a potential limitation of the model, serving as an avenue for improvement, lies in the imbalance between the number of “Healthy” and “Sick” images. This disparity may constrain the model’s performance, warranting further exploration to balance the classes and enhance generalization.
Another avenue for improvement involves adopting an approach that segments both breasts in each image. This strategy could be extended by introducing multiple classes, such as “Healthy Left Breast,” “Sick Left Breast,” “Healthy Right Breast,” and “Sick Right Breast,” providing increased granularity in the classification of thermal anomalies.
Results with other pre-trained models
Below, you will find the evaluation of models constructed based on the pre-trained ResNet50 and GoogleNet models.
Firstly, it is evident that the model relying on GoogleNet performs less effectively than the one utilizing ResNet18. On one hand, it exhibits lower precision, and on the other hand, it has a tendency to classify images more frequently as “Healthy” compared to the ResNet18 model. Its performance is notably impacted by the imbalance in the number of “Healthy” and “Sick” images, making it less adept at detecting breast cancers.
The model built on ResNet50, on the other hand, is equally effective, perhaps slightly superior, to the ResNet18-based model. It generally identifies more instances of illness than the ResNet18 model. However, only about a few of the identified cases are genuine illnesses, while more are false positives. This model also takes about twice as long during the training period compared to ResNet18, primarily due to a higher number of neurons across different layers. Considering the marginal improvement, ResNet18 remains a preferable choice in this context.
Several other pre-trained models are, of course, yet to be tested. Given the time-consuming nature of training, I am presenting this story by testing only these two models in addition to ResNet18. I will continue to augment and refine these findings with additional models and tests on my GitHub repository.
Conclusion
In conclusion of this story, it is evident that thermography effectively detects breast cancers, potentially in earlier stages than mammography. Leveraging the Nvidia Jetson Nano along with pre-trained classification models allows the construction of a reliable breast cancer detection model.
Within the scope of the study presented in this narrative, the ResNet-based model stands out as the most effective in my assessment. However, I posit that with a more balanced dataset, comprising 50% “Sick” and 50% “Healthy” instances, and a larger sample size, the results could potentially be further enhanced.
Moreover, there is a possibility of improving the model’s performance by implementing an image processing technique that segments the breasts. Additionally, by refining the classes to specify whether each breast is the right or left and its health status, the model’s efficacy could be augmented. This hypothesis may be explored in a subsequent narrative.
Furthermore, envisioning the integration of an Nvidia Jetson Nano with a mini-screen and a thermal sensor like FLIR PureThermal 3 (assuming a superior model), we could potentially create a compact, portable solution for breast cancer detection via thermography.
References
Silva, L. F.; Saade, D. C. M.; Sequeiros, G. O.; Silva, A. C.; Paiva, A. C.; Bravo, R. S.; Conci, A. (2014, Mar 1) A New Database for Breast Research with Infrared Image https://www.ingentaconnect.com : https://www.ingentaconnect.com/content/asp/jmihi/2014/00000004/00000001/art00015;jsessionid=g4sv3uvgrwl1.x-ic-live-02
Roger, R.; Lincoln S.; Adriel, S. A.; Petrucio, M.; Débora, M.-S.; Aura, C. (2021, Jul 14) Combining Genetic Algorithms and SVM for Breast Cancer Diagnosis Using Infrared Thermography https://www.mdpi.com/1424-8220/21/14/4802
Roger, R.; Lincoln S.; Adriel, S. A.; Petrucio, M.; Débora, M.-S.; Aura, C. (2021, Aug)A hybrid methodology for breast screening and cancer diagnosis using thermography https://www.sciencedirect.com/science/article/pii/S0010482521003474
Shreenidhi S (2017, Jul 10) Histogram Equalization https://towardsdatascience.com : https://towardsdatascience.com/histogram-equalization-5d1013626e64
Ramzan, F.; Khan, M. U.; Rehmat, A.; Iqbal, S.; Saba, T.; Rehman, A.; Mehmood, Z. (2019, Dec 18) A Deep Learning Approach for Automated Diagnosis and Multi-Class Classification of Alzheimer’s Disease Stages Using Resting-State fMRI and Residual Neural Networks https://www.researchgate.net :
https://www.researchgate.net/publication/336642248_A_Deep_Learning_Approach_for_Automated_Diagnosis_and_Multi-Class_Classification_of_Alzheimer's_Disease_Stages_Using_Resting-State_fMRI_and_Residual_Neural_Networks/citation/download
Bourke, D. (2022, Oct 5). 06. PyTorch Transfer Learning. Retrieved from https://github.com/: https://github.com/mrdbourke/pytorch-deep-learning/blob/main/06_pytorch_transfer_learning.ipynb
Munawar, M. R. (2022, Mar 23). Image Classification using Transfer Learning with PyTorch(Resnet18). Retrieved from medium.com: https://medium.com/nerd-for-tech/image-classification-using-transfer-learning-pytorch-resnet18-32b642148cbe
