Bank Data: Accuracy | F1 Score | ROC AUC

On the bank data we saw that our dependent variable is imbalanced, and on the pervious blog we discussed that the metric that we will be basing our results on was F1 Score using the Confusion Matrix. This blog will discuss, in depth, why.

Accuracy

Accuracy score is the most commonly used metric when it comes to performance measurement, but sometimes this can be a bad metric to base your results on.

Accuracy measures how many observations, both positive and negative, were correctly classified.

This can be misleading when having an imbalanced dataset, because if you have an imbalance dataset where the dependent variable is binary, there are 80% 1’s and 20% 0’s, then our model will develop an accuracy score where it calculates most of predicted variables as 1’s, maybe giving it a 90% accuracy score.

So, when does it make sense to use it?

• When your problem is balanced using accuracy is usually a good start. An additional benefit is that it is really easy to explain it to non-technical stakeholders in your project,
• When every class is equally important to you.

F1 Score

F1 Score combines precision and recall into one metric by calculating the harmonic mean between those two.

This means that F1 Score is good at evaluating how the model did at predicting your positive class. This helps when you care more about your positive class.

When should you use it?

• Pretty much in every binary classification problem where you care more about the positive class.
• It can be easily explained to business stakeholders which in many cases can be a deciding factor.

ROC AUC

It is a chart that visualizes the tradeoff between true positive rate (TPR) and false positive rate (FPR).

• You should use it when you ultimately care about ranking predictions and not necessarily about outputting well-calibrated probabilities.
• You should not use it when your data is heavily imbalanced. False positive rate for highly imbalanced datasets is pulled down due to a large number of true negatives.
• You should use it when you care equally about positive and negative classes. It naturally extends the imbalanced data discussion from the last section. If we care about true negatives as much as we care about true positives then it totally makes sense to use ROC AUC.