The confusion plot: how to quickly analyze a confusion matrix

Classification is a leading field in statistics, and the research for ever more efficient methods is in full swing. It can consist in predicting a binary variable (e.g., loan repayment) or a variable with more than two categories (e.g., red, white or rosé wine). Numerous parametric models (such as logistic regression) or machine learning techniques (such as Random Forest or XGBOOST) are used to make these predictions. Most of the time, a training data set and a test data set are used. The training dataset is used to configure the algorithm parameters, and the test dataset is used to run the final algorithm on an independent dataset.

If the main objective of the algorithms is to make the least possible prediction error (which can be represented by the percentage of error), it is often necessary for the user to go into more detail about these errors, especially when the classes are unbalanced. Let’s take a simple example. A banker has 98% of his customers paying back their loans. A model that predicts 100% repayment will therefore have 98% success. But is such an algorithm satisfying enough for the banker? Of course not! He will give at least as much importance to the prediction of those who will not pay back the loan as to those who will pay it back, whatever the proportions of each class. Thus, in the context of binary data, we can study the sensitivity and specificity. Whether it is for a binary or non-binary response, the confusion matrix is one of the most used evaluation tools.

This matrix is very complete and contains all the necessary information. However, it is quite information-heavy, and it is difficult to summarize. Therefore, our teams have developed an original confusion plot that allows us to have a glance at the distribution of the predictions for each class (orange squares) as well as the total number of observations per class (grey squares). The values are displayed on the graph as well. Here is an example:

• a grey square that goes far beyond the orange square on the diagonal means that the class is poorly predicted
• the orange squares outside the diagonal allow us to quickly see how the prediction errors of each class are distributed. We can easily tell, for example, that observations of class 3 that are mispredicted, are most of the time assigned to class 4.

Thanks to this graph, XLSTAT users can draw conclusions about the reliability of their different methods at a glance!

Acknowledgements

Thanks to the XLSTAT founder Thierry Fahmy for this beautiful innovation!

References

Agresti A. (1990). Categorical Data Analysis. John Wiley and Sons, New York.

Biau, G., & Scornet, E. (2016). A random forest guided tour. Test, 25(2), 197–227.

Hosmer D.W. and Lemeshow S. (2000). Applied Logistic Regression, Second Edition. John Wiley and Sons, New York.