Evaluation Machine Learning by Confusion Matrix

Deep understanding on evaluation of confusion matrix .

Confusion Matrix is very important key feature to evaluate performance of machine learning . Main problem is , here , that evaluation approaches in various domain are different . For example , Same evaluation of business domain doesn’t fit on health domain because priority of every domain is not same . Even on same domain like business , we have to analysis too many metrics to understand of performance of machine learning due to complexity of data structure. I will try to explain how to evaluate via confusion matrix on every aspect as well as domain such as unbalance data , health domain , business domain, multiclasses , NLP etc.

Confusion matrix plays major role to understand accuracy and performance of machine learning of classification problem. Let’s start from beginning , confusion matrix can be imported from metrics modules of sklearn library.

from sklearn.metrics import confusion_matrix
from sklearn.metrics import classification_report

Here , I also imported another metrics , classification_report, to illustrate better way confusion matrix elements.

From blog , we will try to understand some following terms :

1. Terms and measures of confusion matrix.
2. How to calculate confusion matrix for a 2-class classification problem ?
3. Confusion matrix on unbalance data .

I want to display two picture downloaded from internet before go details about confusion matrix because we will get more confusion without detail understand of measuring terms and measures of confusion matrix.

Here , I am trying to say that if we search google about confusion matrix , we could get various tables . Some tables show colomns as Actual Values and rows as predicted values and vice versa. If we donot know detail understanding confusion matrix , then it gives us more pain to conclude evaluation of machine learning . Now , What standard table should we follow for remember , i would say nothing . However , On beginning time , we can keep remember columns as predicted values and rows as actual values because when we interpret on python code , it show us the standard like this :

For measures and terms of confusion matrix , Here below is good example figure .

The picture explains complete package of confusion matrix . We can follow the picture for reference and remember. Let’s try to understand with details .

A binary classifier predicts all data instances of a test data set as either positive or negative. This produces four outcomes-

1. True-positive(TP) — Correct positive prediction
2. False-positive(FP) — Incorrect positive prediction (Type I error)
3. True-negative(TN) — Correct negative prediction
4. False-negative(FN) — Incorrect negative prediction ( Type II error)

Basic measures derived from the confusion matrix

1. Error Rate = (FP+FN)/(TP+FP+TN+FN)
2. Accuracy = (TP+TN)/(TP+FP+TN+FN)
3. Sensitivity(Recall or True positive rate) = TP/(TP+FN)
4. Specificity(True negative rate) = TN/(FP+TN)
5. False Positive Rate (FPR) = FP/(FP+TN)
6. Precision(Positive predicted value) = TP/(TP+FP)
7. Negative Predicted Value = TN/(FN+TN)
8. F-Score(Harmonic mean of precision and recall) = (1+b)(PREC.REC)/(b²PREC+REC) where b is commonly 0.5, 1, 2.
9. False Positive Rate (FPR) = FP/(FP+TN)

Precision and Recall

Above lists are short form of all measures and terms of confusion matrix. Here , i am trying to come on main points, precision and recall , that how to recognize on any confusion table .May be it presents on various form. In simple word, precision means that how much machine learning predict true values and false values whereas recall means how much machine learning predict true values in term of total value of one class classification . We have to focus always precision and recall of both classes of classification. May be total number of TP and FP does not equal to total number of one class . We always should focus wrong and right prediction numbers in case of precision whereas we should focus rate of right prediction on total numbers . It means ,may be , machine learning predict only few right numbers and wrong numbers of total numbers of actual numbers in precision case . Let’s understand through example and pictures .

Here above picture of example of confusion matrix . Let’s try to interpret how to understand overall . We have to go always finding out how many total number of actual values . It gives us by adding total numbers of cells of matrix . So as matrix cells addition , we have total 13 animals , 8 cats and 5 dogs. After that , we should focus predicted values such as precision and recall . Remember don’t forget about each class precision and recall because it give us better understanding . Let’s calculate on above example , precision of cat is 0.71 or 71 % , TP / (TP+FP) i.e. 5/(5+2), and precision of dog is 0.5 or 50% , TN/ ( TN+FN) i.e 3/(3+3) . Similarly , Recall of cat is 0.62 or 62% , TP/(TP+FN) i.e 5/(5+3) , and Recall of dog is 0.6 or 60% , TN/(TN+FP) i.e. 3/(3+2).

Conlcusion:

Precision of cat and dog are 0.71 and 0.5 respectively . Here , There are big difference between them . Similarly , Recall of cag and dog are 0.62 and 0.5 and here no big difference rather than difference of precisions. So ,we should not check only precision of one class but also focus over all classes . Then we can evaluation proper way . It helps us by classification metrics measurement which mention above . Let’s see example how looks like

print(“\nClassification Report\n”)
print(classification_report(y_test,lg_pred))

Output:

I recommend to visit part2 blog on confusion matrix for other various condition like unbalance data , different domains.