A New Way to Visualize Lift Charts with Shapely Values

Inventive new technique to combine Shapely values with your lift and gain charts to enhance model explainability

A Robot lifting weights in abstract form (Dalle-2)

Overview

I’m going to walk through a new way everyone needs to be visualizing their lift charts. But, before we get in to it, let me explain a bit about what lift charts are and why they’re useful.

A lift chart is a graphical representation used in machine learning to evaluate the performance of a predictive model, particularly in classification problems. It is useful for assessing how well a model is able to rank instances in terms of their likelihood of belonging to a certain class, especially the positive class.

To create a lift chart, we first sorted all of our observations by their predicted probabilities, from the highest to the lowest. We then divide the data into a predetermined number of equal-sized bins or groups. For each bin, the average predicted probability (for the model) and the average actual outcome (ground truth) are calculated.

The lift chart plots these average predicted probabilities and actual outcomes for each bin. The x-axis represents the bins, and the y-axis represents the average predicted probability or actual outcome. By comparing the two lines in the chart, one can gauge how well the model is capturing the relationship between the input features and the target variable.

Let’s take a look at an example lift chart for a churn model:

Lift chart

There are two things to note here.

• First, you can see the predicted probabilities (blue line) track the actuals (orange line) quite well.
• Second, you can tell that the model does a good job differentiating between clients likely to churn and clients likely to renew since the actuals range from 0% in the first bin all the way to 95% in the tenth bin.

The Benefit of Lift Charts 📈📉

Before I introduce how we can add Shapely values, let’s recap why anyone would want to use a lift chart in the first place. Lift charts are useful in several ways:

1. Model performance assessment: By visually comparing the predicted and actual lines, you can quickly determine if the model is performing well. If the lines are close together or follow the same trend, the model is performing well; otherwise, there might be room for improvement.
2. Model comparison: Lift charts can be used to compare the performance of multiple models. The model with a higher lift (the gap between the predicted and actual lines) in the top bins is considered better.
3. Targeting strategies: In marketing or sales, a lift chart can help identify segments of the population with the highest likelihood of responding to a campaign or purchasing a product. By focusing on the top bins with the highest lift, businesses can allocate resources more effectively and maximize their return on investment.
4. Identifying areas for improvement: A lift chart can reveal areas where the model performs poorly, which can help guide feature engineering or model refinement efforts to improve its performance.

The Twist: adding Shapely Values 💡

Lift charts are obviously helpful on their own, but what if we took them one step further? Let’s aggregate row-level Shapley values for each of the features in each of our bins. This way we can highlight which features are driving our predictions for the subset of observations that fall within each bin. Here’s an example of what this looks like:

Legend

What’s great about this approach is that you can drill into the riskiest bins to see what’s driving our predictions for customers that are most likely to churn. After all, these are the customers that we’re going to spend our time trying to save.

We can see that Net Promoter Score (the light green blocks) has a weak negative impact (i.e decreases the likelihood of churn) for customers in bin 2. This makes intuitive sense since the median Net Promoter Score is 9 (on a scale of 1 to 10), which is quite high.

Conversely, we can also see that Net Promoter Score has a strong positive impact (i.e. increases the likelihood of churn) for customers in bin 10. Unlike bin 2, the median Net Promoter Score in bin 10 is 5, which is a relatively poor score. It’s no surprise that this would drive up our churn predictions.

And that’s it. An easy, yet effective way to better shed light on how your model is arriving at its predictions and arm your end customers with additional information to make more informed decisions.

Go here to see the code and follow me on Medium and LinkedIn for more helpful data science tips. Thanks for reading!