Bumble Tech
Published in

Bumble Tech

Photo by Eduardo Sánchez on Unsplash

Evaluation of classification models on unbalanced production data

Taming production data disbalance on classification models comparison

Where did the task come from?

Unbalanced production data and how to deal with it

Common metrics for comparing classification models

Confusion matrix

Confusion matrix layout

Accuracy, Precision, Recall, F-score

Confusion matrix based metrics

AUC-ROC, AUC-PR

AUC-ROC, AUC-PR

How to deal with production unbalance?

Possible production distribution of cases (numbers are invented to demonstrate the example)

Collecting dataset in a “smart” way

Artificially-based distribution of cases saving the production distribution of confusion matrix cases. “Photo-based” distribution (numbers are invented to demonstrate the example)
Artificially-biased distribution of cases concerning Model A output. “Model-based” distribution.

Measuring quality and transferring errors

Error transfer chart
Error transfer chart in percentage from production distribution. As 90% of TN, a decently large part of production distribution was converted to FP (12.53%), and it seriously compromised the overall model quality.
Model A classification example. Red dots are the safe (negative) photos, blue ones are unsafe (positive) photos. The orange line (model A) separates the photos with some FP and FN cases.
Model A classification example with weighted confusion matrix cases. TN dots became thinner, as we’ve reduced the weight for TN, while FN, FP, and TP dots became thicker.
Model B classification example with weighted confusion matrix cases. We can see while we are reducing the impact of previously highlighted FP and FN, new FP and FN cases appeared.
Model B classification example on production distribution. We see, that while we’ve dealt with previously highlighted FP and FN cases, new ones have appeared, and become a large part of the whole production load.

Conclusion

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store