Q#103: Efficacy of a classifier

Classification please

Note: I believe it should be free and readily available for everyone to gain value from data, hence I pledge to keep this series free regardless of how large it grows.

Suppose you’re working for a financial services company that processes loans. It has a classifier that produces a continuous probability from 0% to 100% for a given loan application being fraudulent. Suppose we were to take the square root of the model’s score for each loan application that comes through. How would this impact the Receiver Operating Characteristic (ROC) curve?

TRY IT YOURSELF

ANSWER

OOH conceptual question, always something interviewers ask to see if you understand the basics. First lets go over what the ROC curve is, in an interview taking the initiative to explain it before directly answering the question is extra brownie points. Remember a key skill for Data Science is communication and the ability to explain things to key stakeholders and business leaders.

Understanding the ROC Curve

Before delving into the impact of the square root transformation, let’s briefly review the ROC curve. The ROC curve is a graphical representation of the trade-off between the true positive rate (sensitivity) and the false positive rate (1 — specificity) for different threshold values. A perfect classifier would have an ROC curve that passes through the upper-left corner (100% sensitivity and 100% specificity), while a random classifier would produce a diagonal line from the bottom-left to the top-right corner.

Square Root Transformation and its Impact

Now, let’s consider the scenario where we take the square root of the model’s scores for each loan application being evaluated for fraudulence. This transformation can be seen as a way to mitigate the impact of extreme scores while still preserving the relative differences between them.

1. Effect on Extreme Scores: Applying the square root transformation tends to compress extreme scores towards the center while expanding smaller scores. This means that the transformed scores will be more spread out, with a reduced range compared to the original scores.
2. Impact on the ROC Curve: The ROC curve is constructed by varying the decision threshold for classifying instances as positive or negative. When we apply a square root transformation to the scores, the relative ranking of instances remains the same, but the absolute values change. This alteration can lead to a change in the shape of the ROC curve.

• Sensitivity and Specificity: The square root transformation affects both true positive rate (sensitivity) and false positive rate (1 — specificity). Instances with lower original scores experience a greater expansion after the transformation, potentially leading to higher sensitivity values. Conversely, instances with higher original scores may experience a compression, impacting specificity.
• ROC Curve Shape: The exact impact on the shape of the ROC curve will depend on the distribution of original scores and the degree of transformation. In general, we might expect to see adjustments in the curve’s slope and curvature, reflecting the changes in sensitivity and specificity at different threshold values.

Applying a square root transformation to the model’s scores can introduce changes to the ROC curve by altering the relative distribution of scores. While this transformation may not fundamentally change the classifier’s ability to discriminate between positive and negative instances, it can influence the interpretation of sensitivity and specificity at different decision thresholds thereby impacting the shape of the ROC curve.

Plug: Checkout all my digital products on Gumroad here. Please purchase ONLY if you have the means to do so. Use code: MEDSUB to get a 10% discount!

Earn $25 and 4.60% APY for FREE through my referral at SoFi Bank Here