Probability Calibration

In a Machine Learning model, it is not important to just predict the correct label. Obtaining the probability of the class label holds equal significance which provides ‘confidence’ on the model predictions.

Therefore, a well calibrated classifier can be defined as one which reflects the confidence of the model predictions.

For example:

A well calibrated (binary) classifier should classify the samples such that among the samples to which it gave a predict_proba value close to 0.8, approximately 80% actually belong to the positive class.

The following image extracted from the documentation of sklearn provides deep insights into the calibration of probabilistic predictions of various ML models.

Reliability Curves (Reliability Diagrams)

The position of the points or the curve relative to the diagonal can help to interpret the probabilities; for example:

• Below the diagonal: The model has over-forecast; the probabilities are too large.
• Above the diagonal: The model has under-forecast; the probabilities are too small.

Different Classifiers

The Logistic Regression model provides well calibrated predictions as it optimizes the log-loss, whereas others return biased probabilities. Notice the blue curve is running close to the black dotted curve indicating a perfectly calibrated model.

The Naive Bayes usually moves the probabilities to 0 or 1. Why? Because it assumes that the features are independent given the class, but it usually does not happen as the features are dependent in a dataset. That is why the counts in the histograms are high around 0 and 1.

Random Forest Classifier shows completely opposite behavior to Naive Bayes. It ranges from 0.2 to 0.9 whereas probabilities of 0 and 1 are rare.

Linear Support Vector Classification shows an even more sigmoid curve which is common for maximum-margin methods. Why? They focus on hard samples that are close to the decision boundary (the support vectors).

Before you leave,

If you enjoyed this post, please make sure to follow the NLP Gurukool page and visit the publication for more exciting tutorials and blogs on machine learning, data science and NLP.

Please get in touch if you would like to contribute to our publication.