Demystifying the Confusion Matrix
Most aspiring data explorers have come across the confusion matrix and its components, but how do you actually read it and use it?
In this post, we’ll walk through the process of creating and interpreting a confusion matrix, using real-world examples to illustrate the concept. We’ll explore the various components of a confusion matrix, such as true positives, true negatives, false positives, and false negatives, and how they contribute to the overall performance of the model. Join me as we demystify the confusion matrix and gain a deeper understanding of how machine learning models work.
What is a confusion matrix?
A confusion matrix is a powerful tool that helps data explorers understand the performance of their machine learning models in classification tasks. It provides a clear visual representation of how well the model is able to predict the true labels of a dataset, and it is often used in conjunction with other metrics like accuracy, precision, and recall. But, before we get to that, let’s just understand what a confusion matrix is and how to construct one!
To really grasp the concept, we will use an example throughout this post. Let’s imagine we have six pictures of dogs and six pictures of cats, as we can see below.
Imagine we have already developed a super amazing machine learning model that can classify pictures of dogs and cats. Our model is so great that it gets every picture right! Let’s visualize this. The green is the “true label” or what we know is the right answer, and purple is the model’s prediction.
If we were to put the same information in a table, it would look like this:
Great! Our model picked all the right pictures for dogs and all the right pictures for cats. Congratulations! We just made our first confusion matrix!
Now let’s say our friend, let’s call him Marcos, loved our dog-cat image classifier so much that he wanted to try to build one too. Marcos’ model also says there are six dogs and six cats. He is so excited and can’t wait to show us. We notice that there are indeed six dog predictions and six cat predictions. However, we instantly realize there may be an issue. There are three dogs predicted as cats and three cats predicted as dogs!
If we were to put this into our confusion matrix, it would look some thing like this:
Which model would you trust more? Probably the first model because it got all the images correct, versus Marcos’ model that only got half correct.
Let’s look at one more example. Marcos takes our advice and reworks his code. He’s able to improve his model and comes back to show us. This time, his model identified all the cats correctly as cats Also, it identified two dogs as cats, and the other four dogs as dogs. What an improvement!
By looking at the confusion matrix for this model’s predictions we can more clearly see what is happening. We declare that his new model is good at predicting cats but sometimes gets confused at predicting dogs.
Essentially, a confusion matrix is a simple way to see exactly how the model is doing compared to the real “true” labels. Notice that the sum of all the values in the confusion matrix add up to the total number of pictures (or objects/values/etc.) that we are trying to predict. Because of this, we can take the results and find useful performance metrics such as precision and recall.
To finalize the 2x2 matrix, let’s see how it is usually displayed in textbooks and online. Don’t worry, we’ll cover the TP, TN, FN, FP variables in a minute. First, it’s important to note the rows and columns can be switched but will still show the same information but in a different order (i.e. “true/actual” can be columns and “predicted” as rows), so don’t be confused if you see one a little different.
It is convention when working with two labels (i.e. cats/dogs) to use positive and negative as the labels. In our dog-cat example before, we would just need to declare either cats or dogs to represent “positive” and the other to represent “negative”.
Let’s look at Marcos’ second model to better understand TP, TN, FN, FP from the confusion matrix above. Remember, his model classified 4 dogs as dogs, two dogs as cats, and six cats as cats. In our example, let’s say dogs are “positive” and cats are “negative”.
That is the true power of confusion matrices; you can take any of the true positive, false positive, true negative, false negative values and use them to calculate metrics such as accuracy, recall, precision, F1-score, or specificity (depending on what you need for your project). How exciting!
Beyond the 2-by-2 Matrix
You may be wondering, “What if I have more than two classes?” like cats, dogs, and birds? How can we put this in a confusion matrix? How would TP, TN, FN, and FP work with more than two class? Let’s demystify this as well. Let’s say we’re trying to predict three classes A, B, and C. We are most interested in classifying the Class B correctly. We can use the logic from before to find all the corresponding values as it related to Class B.
In contrast, let’s now say we’re interested in Class A. We redefine the TP, TN, FN, and FP placements as we see below.
It’s true that working with three or more classes can make it more challenging conceptually, but that is why we have computers! Usually, you would put all this into a computer and it would spit out the values for you. However, it’s good to know where they come from.
Metrics
We mentioned earlier that we can use true positive (TP), false positive (FP), true negative (TN), and false negative (FN) to find model performance metrics. We’ll quickly dive into calculating these values but will discuss the application of these more thoroughly in a separate post (here!).
Accuracy
Everyone talks about accuracy and we generally know that having a high accuracy is usually better. We define accuracy as the proportion of correctly classified values. It is used to measure the overall performance of a model.
There is a caveat to accuracy, though— if your classes are unbalanced then accuracy probably isn’t the best metric to use. For example, imagine our data consists of 100 dog pictures and 1 cat. If our model always predicts dog then we can say it has a 99% accuracy. This can be misleading for the model performance. For this reason, it is import to always know your dataset, goals, and how your model works.
Precision
As an alternative and additional metric, you can look at precision and recall. Precision is the proportion of true positives out of all positive results. It is used to measure the effectiveness of a classifier in identifying positive values.
Recall
Recall, also called sensitivity, is used to measure the effectiveness of a model in identifying all positive examples. Wait! That sounds like precision. Not quite. Although they are similar in that the both use the positive values, they are also different in what they tell us. I won’t go into more detail here, but be sure to check out my post on metrics (here).
F1-Score
Before calculating the F1-score, you’ll first need to gather your precision and recall scores. Once you do, you can find the f1-score, which is the “mean” of precision and recall. It is used to balance these two metrics to give a single value measure of the model’s performance.
Specificity
Lastly, we’ll look at specificity. This is the proportion of true negatives out of all negatives. It’s used to measure the model’s ability to correctly identify negative values.
I want to encourage you to check out my next blog post which dives more deeply into metrics, what we use them for, and how to calculate them. You can find the link here.
Thank you for reading! Find me on Linkedin ❤
I also would greatly appreciate any feedback, constructive criticism and input on what you read. Feel free to reach out to me via Linkedin or comment directly on this post.
