What is a Confusion Matrix?

A Confusion matrix is an N x N matrix used for evaluating the performance of a classification model, where N is the number of target classes.

The matrix compares not accurate but actual target values with those predicted by the machine learning model. This gives us a holistic view of how well our classification model is performing and what kinds of errors it is making.

For a binary classification problem, we would have a 2 x 2 matrix as shown below with 4 values:

Let’s understand matrix:

• The target variable has two values: Positive or Negative
• The columns represent the actual values of the target variable
• The rows represent the predicted values of the target variable
• We don’t know that what are these TP, FP, FN and TN here? That’s the crucial part of a confusion matrix. Let’s have a look for each term below.

Understanding True Positive, True Negative, False Positive and False Negative in a Confusion Matrix

True Positive (TP)

• The predicted value matches the actual value.
• The actual value was positive and the model predicted a positive value.

True Negative (TN)

• The predicted value matches the actual value.
• The actual value was negative and the model predicted a negative value.

False Positive (FP) — Type 1 error

• The predicted value was falsely predicted.
• The actual value was negative but the model predicted a positive value.
• Also known as the Type 1 error.

False Negative (FN) — Type 2 error

• The predicted value was falsely predicted.
• The actual value was positive but the model predicted a negative value.
• Also known as the Type 2 error..

Classification Rate/Accuracy:
Classification Rate or Accuracy is given by the relation:

Let’s see how our model performed:

The total outcome values are:

TP = 30, TN = 930, FP = 30, FN = 10

So, the accuracy for our model turns out to be:

96%! Not bad!

But it is giving the wrong idea about the result. Think about it.

However, there are problems with accuracy. It assumes equal costs for both kinds of errors. A 99% accuracy can be excellent, good, mediocre, poor or terrible depending upon the problem.

Recall:

Recall can be defined as the ratio of the total number of correctly classified positive examples divide to the total number of positive examples. High Recall indicates the class is correctly recognized (a small number of FN).

Precision:

To get the value of precision we divide the total number of correctly classified positive examples by the total number of predicted positive examples. High Precision indicates an example labelled as positive is indeed positive (a small number of FP).

High recall, low precision: This means that most of the positive examples are correctly recognized (low FN) but there are a lot of false positives.

Low recall, high precision: This shows that we miss a lot of positive examples (high FN) but those we predict as positive are indeed positive (low FP)

F-measure:
Since we have two measures (Precision and Recall) it helps to have a measurement that represents both of them. We calculate an F-measure which uses Harmonic Mean in place of Arithmetic Mean as it punishes the extreme values more.

The F-Measure will always be nearer to the smaller value of Precision or Recall.

Code : Python code to explain the above explanation

# Python script for confusion matrix creation.

from sklearn.metrics import confusion_matrix

from sklearn.metrics import accuracy_score

from sklearn.metrics import classification_report

actual = [1, 1, 0, 1, 0, 0, 1, 0, 0, 0]

predicted = [1, 0, 0, 1, 0, 0, 1, 1, 1, 0]

results = confusion_matrix(actual, predicted)

print 'Confusion Matrix :'

print(results)

print 'Accuracy Score :',accuracy_score(actual, predicted)

print 'Report : '

print classification_report(actual, predicted)

Output:

Confusion Matrix :
[[4 2]
 [1 3]]
Accuracy Score : 0.7
Report : 
              precision    recall  f1-score   support
          0       0.80      0.67      0.73         6
          1       0.60      0.75      0.67         4
avg / total       0.72      0.70      0.70        10

Confusion Matrix for Multi-Class Classification

How would a confusion matrix work for a multi-class classification problem? Well, don’t scratch your head! We will have a look at that here.

Let’s draw a confusion matrix for a multi-class problem where we have to predict whether a person loves Facebook, Instagram or Snapchat. The confusion matrix would be a 3 x 3 matrix like this:

The true positive, true negative, false positive and false negative for each class would be calculated by adding the cell values as follows:

That’s it. You are ready to decipher any N x N confusion matrix!

We can also get a normalized confusion matrix using:

df_conf_norm = df_confusion / df_confusion.sum(axis=1)

Predicted         0         1         2
Actual
0          1.000000  0.000000  0.000000
1          0.000000  0.333333  0.333333
2          0.666667  0.333333  0.500000