Performance Metrics

Metrics are used to evaluate the performance of Machine learning algorithms, classification as well as regression algorithms. We must carefully choose the metrics because the measurement of the performance of Machine learning algorithms will be dependent entirely on the metric we choose.

Performance Metrics for Classification

Confusion Matrix

It is used in the classification problem to establish a relationship between predicted values and actual values. It shows how many values are predicted correctly and how many are predicted wrong for each class. We can derive the different type of metrics that will show how much good fit our model is.

For balance dataset

1. Accuracy

When the dataset is balanced we use Accuracy. Accuracy tells the percentage total correctly predicted value out of total values.

For an imbalanced dataset, we cannot use accuracy as it may not give a good result.

For Example: Out of 100 values if TP=90 TN=0 the accuracy still be 90%, which sounds good but not because TN is not detected correctly and output is more bias towards TP.

For imbalance dataset

1. Precision

Out of all the predicted positive values how many are predicted correctly.

Precision is used when a False Positive is more important like in spam detection if a mail is not spam and the model predicted it as spam, we will miss a very important mail because of this error.

2. Recall or Sensitivity

Out of all the actual positive values how many are predicted correctly. It is also known as True Positive Rate(TPR).

Precision is used when a False Negative is more important like a patient is covid positive but the model detected it as negative, it will cause a great impact on healthcare. If the patient is not covid positive still detected as positive, we can do further testing for that but vice vera shouldn't happen.

3. Specificity

Out of all the actual negative values how many are predicted correctly. False Positive Rate(FPR)=1-Specificity

4. F Beta Score

It shows a relationship between Precision and Recall.

When F=1, F1 will be harmonic mean of Precision and Recall i.e 2*pr*re/(pr+re)

If Precision is more important to us than put F>1 and if Recall is more important to us put F<1.

ROC and AUC curve

We need to try different threshold values to get the best TPR and FPR value. We can’t try all the values manually. For this, we use ROC and AUC curve.

We can try for different values TPR and FPR and we can also try for the different algorithm to find the best fit for our data. The model that fit most data under the curve is considered the best model.

Performance Metrics for Regression Problems

For Regression, problem Metrics are comparatively simple to understand.

1. Mean Absolute Error (MAE)

It is basically the sum of the average of the absolute difference between the predicted and actual values.

2. Mean Square Error (MSE)

MSE is like the MAE, but the only difference is that it squares the difference of actual and predicted output values before summing them all instead of using the absolute value.

I’m sharing Links for the reset of the two metrics i.e R-Squared and Adjusted R-Squared. I have explain them in more detail.

3. R-Squared

4. Adjusted R-Squared

Final Thoughts

Metrics are very important to understand as they tell us about the performance of our model. They can also misguide us if not used correctly. We need to understand data and pick a metric that is the best fit for our model.