Udacity AI Product Manager Program Review (Part II)
A Project Centric Review Approach
Overview
This article is the second episode of a three parts series on the review of the AI Product Manager Program offered by Udacity. Readers interested in part 1 can follow the link below.
As stated in the first part of this series, I will review through project assignments and publish each project in a different episode. Hence, the readers can read about the first project in part I, the second in part II, and the final in part III.
For each project, I will try to answer the following questions:
- What is the project?
- What is the outcome of the project?
- What are the sources and tools to complete the project?
- What is the solution?
- What are the problems faced during the project implementation?
- How can those problems be solved?
- How can the project be improved?
- What are the references for the project?
So let's start the second project.
Project 2 — Build a Model with Google AutoML
This project is a sequel to project 1, in which we have created directives for a labeling job that differentiates the healthy and pneumonia chest x-ray images.
In a typical machine learning (ML) workflow, the subsequent step after data gathering and preparation is the model-building step. Although this Nanodegree proposes a slightly different ml workflow that starts with the Business Goal and ends with Active Learning and Tuning, it is no different at this stage. We need to train and evaluate our model.
The Target
So, in this second project of the Nanodegree, we will build four different classification models that differentiate the healthy and pneumonia chest x-ray images. There will be no coding in this assignment. Instead, we will use Google AutoML. AutoML stands for Automated Machine learning. It is a tool to automate the ML workflow and find the best-performing model we can deploy from a given set of model algorithms for a given dataset. By using Google AutoML, we can build and deploy our models in the cloud.
Our project aims to build four models using Google's AutoML Vision platform and understand how data characteristics affect model performance. We will not deploy our model, but we will submit an AutoML Modeling Report for review in the end.
We will use the Kaggle chest x-ray dataset as in the previous project. We will select four different subsets of the dataset and build four ML models using each of those subsets. We will not label them using Appen's platform as in the previous project but use the original labels given in the dataset.
Google AutoML
To use Google's AutoML Vision platform, we will go to Google AutoML, click the TRY AUTOML button, and select AutoML Vision. After signing in/up to the account, we can create our project with any title. Please note that Google may charge you for this project according to your account status. But if you sign up, you'll be given a free $300 credit.
The Data
We will use four subsets of the Kaggle chest x-ray dataset, and I suggest preparing those subsets before modeling to ensure similar dataset usage for our models.
Create a binary classifier to detect pneumonia using chest x-rays
The first subset will contain 200 chest x-ray images, 100 from the 'normal' class and 100 from the 'pneumonia' class.
I will not go into the details of the training process except for one thing. If required, we can easily find help from Google and the Udacity project page. Instead, I would focus on the results and the modeling report we will submit for review.
I want to discuss the train/test split concept in the training process, which is very important in ML.
We can break up the model creation process in ML into three consecutive steps, which are the training, the hyperparameter tuning, and the evaluation steps. We need data in each stage to proceed to the next step. However, the data used in each phase should differ, and we must isolate these datasets from each other at the beginning. Otherwise, we will end up with a poor-performing model, which we do not want. So before creating a model, the first thing to do is split the dataset into three parts. These are the training set used in training, the validation set used in the hyper-parameter tuning, and the test set used in the evaluation.
So before the training, we will divide the dataset into 80%, 10%, and 10% portions, as in the image below. More data for training always means better model performance. So we will allocate as much data as possible for training (80%).
After we finish the training, we can evaluate our model by looking at the confusion matrix.
A confusion matrix is a grid that shows all the predicted 'labels' relative to all actual 'labels.' In our case, we have four cells in the confusion matrix below.
The explanation of each cell in a confusion matrix is as below.
- True Positives (Row 1 Column 1, TP in short): TP is the number of positive labels predicted as positives by the best-performing model.
- False Negatives (Row 1 Column 2, FN in short): FN is the number of positive labels predicted as negatives by the best-performing model.
- False Positives (Row 2 Column 1, FP in short): FP is the number of negative labels predicted as positives by the best-performing model.
- True Negatives (Row 2 Column 2, TN in short): TN is the number of negative labels predicted as negatives by the best-performing model.
As stated earlier, our test set has 10 'normal' and 10 'pneumonia' images. The confusion matrix tells us that our model has predicted the 10 out of 10 'normal' class images as 'normal' and 8 out of 10 'pneumonia' class images as 'pneumonia.'
In machine learning, we use various metrics to evaluate our model. One of the most popular metrics is model accuracy.
Model accuracy measures the percentage of accurate predictions against the total number of predictions for all classes. In mathematical terms:
However, relying on accuracy may be misleading, especially for unbalanced datasets. In this project, we will not use 'accuracy.' Still, we will calculate it twice to see how it misleads us. The accuracy for the first model is as follows.
To evaluate our model, we will use two metrics calculated from the confusion matrix: precision and recall.
These metrics measure the model to understand how it performs for an individual class and across categories.
Model precision measures the percentage of correctly identified class instances against that class's total number of predictions. In mathematical terms:
Model recall measures the percentage of correctly identified instances of a class against the total possible samples of that class. In mathematical terms:
So for model-1, which is trained with dataset-1, the precision and recall of the 'normal' class are as follows.
Create an unbalanced binary classifier
The second subset will contain 400 chest x-ray images, 100 from the 'normal' class and 300 from the 'pneumonia' class.
We will start by splitting our dataset into 80%, 10%, and 10% portions.
After the training, we will evaluate our model using the confusion matrix.
Our test set has 10 'normal' and 30 'pneumonia' images. The confusion matrix tells us that our model has predicted 6 out of 10 'normal' class images as 'normal' and 30 out of 30 'pneumonia' class images as 'pneumonia.'
Let's calculate precision and recall for the 'normal' class.
For the 'normal' class, precision increases from 0,83 to 1, and recall decreases from 1 to 0,6.
Model-2 predicts mainly the unbalanced class. In other words, model-2 tends to generalize the inputs to the majority class (Pneumonia). As a result, recall decreases for the minority class.
If we calculate the accuracy for this model, we can see why accuracy is not a good choice for unbalanced datasets. The accuracy calculation is as follows.
As we can see easily, although the accuracy for this model is the same as the clean/balanced case, this model performs much worse than it. Therefore, as I have said earlier, using accuracy for unbalanced datasets will mislead us.
Create a binary classifier with dirty data
For the third dataset, we will begin with the original dataset of 100 'normal' and 100 'pneumonia' images used in the first dataset. Then we will reverse 30 labels in each class and misclassify 30% of the data.
Again we will start by splitting our dataset into 80%, 10%, and 10% portions.
After the training, we will evaluate our model using the confusion matrix.
Our test set has 10 'normal' and 10 'pneumonia' images. The confusion matrix shown above tells us that our model has predicted 8 out of 10 'normal' class images as 'normal' and 9 out of 10 'pneumonia' class images as 'pneumonia.'
Let's calculate precision and recall for the 'normal' class.
For the 'normal' class, precision increases from 0,83 to 0,88, and recall decreases from 1 to 0,8.
We can easily see that the overall error (3) is larger than the clean/balanced case errors(2).
Moreover, there may be some dirty data in the test dataset, which makes the model unreliable, implying the real-life performance may be much worse than the evaluation.
Create a three-class model with' normal,' 'bacterial pneumonia,' and 'viral pneumonia.'
If we study the Kaggle chest x-ray dataset in detail, we'll see that viruses or bacteria can cause pneumonia. We can view this by looking at the image names in the dataset. If the image name contains 'virus,' it is viral pneumonia, and if the image name has 'bacteria,' it is bacterial pneumonia.
For the final model, we will create a three-class dataset with 100 images from the 'normal' class, 100 images from the 'bacterial pneumonia' class, and 100 images from the 'viral pneumonia' class.
We will start by splitting our dataset into 80%, 10%, and 10% portions.
After the training, we will evaluate our model using the confusion matrix.
This time we have ten 'normal,' ten 'viral pneumonia,' and ten 'bacterial pneumonia' class images in our test set. The confusion matrix tells us that our model has predicted 10 out of 10 'viral pneumonia' class images as 'viral pneumonia,' 10 out of 10 'normal' class images as 'normal,' and only 6 out of 10 'bacterial pneumonia' class images as 'bacterial pneumonia.'
We can calculate precision and recall for the 'normal,' 'viral pneumonia,' and the 'bacterial pneumonia' classes.
For the 'normal' class:
For the 'viral pneumonia' class:
And for the 'bacterial pneumonia' class:
It seems that the model confuses bacterial pneumonia with viral pneumonia. One method to solve this confusion is adding more data to training. So let's add 20 images for each class to create a new balanced dataset and retrain our model.
We have 120 images from the 'normal' class, 120 from the 'bacterial pneumonia' class, and 120 from the 'viral pneumonia' class. The new confusion matrix is as follows.
We've got a better result now. Let's calculate precision and recall for the last time.
For the 'normal' class:
For the 'viral pneumonia' class:
And for the 'bacterial pneumonia' class:
We can see that while the precision score for the 'viral pneumonia' class and the recall score for the 'bacterial pneumonia' class increases considerably, the recall score for the 'viral pneumonia' class and the precision score for the 'bacterial pneumonia' class decreases slightly. This phenomenon indicates better performance.
As a final stage, we will introduce a new metric to see this performance increase, the F1 score. We can calculate the F1 score using precision and recall, and the F1 score is the harmonic mean of precision and recall. In mathematical terms:
We will now calculate the F1 score of each class for the two datasets.
For the 'normal' class:
For the 'viral pneumonia' class:
And for the 'bacterial pneumonia' class:
The F1 score improves for both viral and bacterial pneumonia classes, suggesting a performance increase.
Troubleshooting
This project's most challenging part was creating an AutoML Vision account since Google doesn't accept debit cards quickly, and trying different cards generally solves the issue.
Another problem arises while uploading the datasets. Sometimes Google doesn't upload all the images while complaining about duplication. Manually adding new images will solve the issue.
Finally, sometimes the unbalanced model gives unexpectedly good results. Maybe Google AutoML is trying to filter the unbalanced dataset into a balanced dataset. Using a new dataset or retraining the model will solve the issue.
Conclusion
In this project, we have learned how data properties affect model performance. We have seen that dirty and unbalanced data significantly decrease model performance.
In addition, we have seen that using accuracy may mislead us. Instead, precision, recall, or F1 scores may give better results.
We have also learned how to calculate precision, recall, and F1 score using the confusion matrix.
As a final note, some of you may notice that although I promised to discuss possible improvements to the project at the beginning of this article, I haven't mentioned any since I will discuss these improvements in the next episode. Stay tuned for the final episode. Refer to my GitHub repository or my first episode below for the Udacity AI Product Manager Program if you liked this subject or are interested in it.
Thank you for reading!
References
