Explainable AI (XAI) for Regression Problem
Practical Guide for many XAI tools
What does Machine Learning Explainability mean?
In 2016 Ribeiro et al. published their by now reputable paper: "Why should I trust you?": Explaining the Predictions of any Classifier
. The famous experiment of "Husky or Wolf"
points out that despite great efforts in designing an accurate model and training it properly, still the model can learn unpredictable correlations. For instance, the model learned to predict that it was an wolf in the picture if there was snow in the background. While it makes sense, it did not learn to distinguish a wolf form its characteristics, but from the background of the image.
AI applications are booming in today’s world and altogether the weight of decisions delegated to AI. Considering this enormous volume and also the scrutiny decisions need to go through, it has become paramount having clarity on why a certain decision was made. This gives way to Explainable AI (XAI). XAI focuses on making it clear (visually) what and by how much particular features influenced the prediction. Numerous tools and techniques are being developed in this space, and this post is going to take a look at them in practice.
This post delves deeper on integrating XAI tools in machine learning workflows. My goal is to give code samples and show results for a model that is not often covered in other posts or in the documentation of the tool.
Tools covered here are: LIME, SHAP, Seldon Alibi and InterpretML.
Quick word on ‘black-box vs. glass-box’
While it is not correct to put them against one-other, they are two different classes of models that result in different ML algorithms. Glass-box algorithms are the likes of Linear Regression (or other linear models), decision trees, Bayesian classifiers, that produce models, whose predictions can be inspected rather easily based on model’s components (weights assigned to features, or the path followed in a decision tree). The explanation for a specific prediction can be found by inspecting the combination of the components of the model.
The black-box models are commonly associated with deep learning algorithms. They tend to produce more accurate predictions but sacrifice clarity and traceability in their predictions. Common examples of black-box models are convolutional neural networks (CNNs), used extensively in computer vision, object detection in images and face recognition field.
The tools and techniques described in this post are model-agnostic, meaning that, independent weather the model is a glass-box model or a black-box one, they are able to provide clarity for the predictions being made. All the tools are setup to explain the same glass-box model (namely a TweedieRegression model).
In another post, I will focus on how these tools are used with black-box models.
Model
The model being used in this post is inspired by an example in scikit-learn documentation, namely TweddieRegression
. This post if focusing in the world of insurance, specifically claim amounts for car insurance. There are usually multiple ways of predicting the total claim amount expected per unit of exposure (meaning the duration of the insurance coverage of a given policy). One way is modeling the claims frequency, and the severance (claim amount) in two different models and calculating the total claim amount as product of them. Frequency usually is modeled via a Poisson distribution, since there are many policies which record 0 claims overall. And the severity is modeled with a Gamma distribution, as it is empirically shown to have such a distribution.
Alternatively, a Tweedie Distribution models the total claim amount directly when the power parameter is between 1 and 2. Grid search could be used to find the right parameter, but in this post 1.9 is chosen as the power value.
I decided to pick this model since:
- It is the model I wanted to play around with.
- It does simplify the content of this post.
- It captures the data quite well.
- It is not a typical model you see in online tutorials and it feels closer to real-world production scenarios.
Here is the full code to get the data, perform feature engineering, and finally train the model.
Feature Names
After applying the ColumnTransformer
, the feature names are no longer easy to read. The One Hot Encoding
(OHE) creates a whole set of new features. Also the KBinsDiscretizer
creates a new feature for every interval the continuous data are binned into. It gets a little challenging to figure out what is the feature name and what is the value of that feature for the current data point. In my other post, I write about how to get readable feature names from the ColumnTransformer
.
Explainable AI (XAI) tools
This post is focused in these four XAI tools: LIME, SHAP, Seldon Alibi, and InterpretML. For each of them, I will cover how to integrate it with the model and how to explain
a certain prediction for a given data point.
LIME (Local Interpretable Model-Agnostic Explanations)
LIME is a model agnostic tool which can give explanations for any given supervised learning model. It provides explanations that are locally faithful within the vicinity of the data point being observed. Its simplicity and intuitiveness makes it one of the most popular XAI tools out there.
The following code creates an explainer
and uses it to identify the main features that contributed to the prediction for a given data point (in the case of X_test[42]
).
The result looks like this:
Intercept 211.86354640053418 // Lime model intercept
Prediction_local [146.38162887] // LIME model prediction
Right: 133.80831374760626 // value from glm_pure_premium.predict
However this text represantation is not very descriptive. To visualize the prediction and the features that affected it the most, I need to run: explain_data_point.as_pyplot_figure()
or explain_data_point.as_list()
. The result shows visually (or in text) how the value of a certain feature for a given data point affects the prediction (positively or negatively).
explain_data_point.as_pyplot_figure()
explain_data_point.as_list()
[('Category__VehPower_4.0', -118.60886335511677), ('Category__VehBrand_B2', -59.89935149820192), ('Category__VehGas_Regular', -34.65470430935958),
('Category__Area_C', 31.8465417201858),
('Log__Density', 30.911940033440064),
('Segment__DrivAge: [39.0, 43.0)', 28.24619186526209), ('Category__Region_R24', -27.9788217749691),
('Segment__VehAge: [12.0, 15.0)', -19.304050151977467),
('BonusMalus', 0.02413232592200181),
('Segment__VehAge: [2.0, 4.0)', 0.0)]
SHAP (SHapley Additive exPlanations)
SHAP, uses Shapley values from game theory to explain models and their predictions. In the ML sense, the game is the prediction, and the players are the features and its values for one data point. SHAP explains the difference between the average prediction (the one usually from the training data) and the result of the prediction for the given data point (or set of data points).
SHAP library offers different ways to visualize how each feature and its corresponding value in the data point effect the prediction. Bar plot, waterfall plot, or decision plot, all are great visualizations of how the prediction was achieved. E[f(X)]
is an expected value calculated for the given model, basically an average value of all the target values from the training data set. For the given data point, its dimensions affect the expected value (some more and some less) resulting in the end with the f(x)
value.
shap.plots.waterfall(shap_values[i], max_display=12)
The expected value is usually explainer.expected_value
and a full list of all shapley values for the given data point can be retrieved via explainer.shap_values(X_test[i])
.
explainer.shap_values(X_test[i])
array([[-3.57367972e-02, 2.37260265e-05, -3.62400872e-03,
9.99498571e-03, 1.61319891e-02, -1.33205559e-02,
-1.70261858e-03, -5.77566345e-02, 2.10600674e-02,
-7.60667267e-04, 2.20831452e-02, 1.63428026e-02,
8.70301560e-03, 1.39718551e-01, -6.68123861e-03,
-1.28186673e-02, -9.83389108e-03, 2.70578702e-02,
-2.45288066e-02, 2.43979909e-02, -6.97570218e-04,
1.84981886e-04, 2.72899951e-02, 0.00000000e+00,
-1.75415087e-04, -1.24097019e-01, 1.17185801e-02,
-1.13515675e-02, -1.44770143e-02, 3.56013603e-03,
-2.78724434e-01, 3.47278038e-02, -2.96024571e-02,
1.28822536e-02, -4.03941874e-03, 7.75558604e-04,
4.69723415e-03, -3.04630041e-03, -6.58506613e-04,
0.00000000e+00, 3.73403677e-04, 0.00000000e+00,
-4.67284073e-02, -4.67924568e-02, -7.21878931e-03,
-0.00000000e+00, -8.58881460e-06, -0.00000000e+00,
-7.78920879e-02, 4.28093791e-03, 4.82906634e-04,
-2.98388445e-03, 8.30043843e-03, 0.00000000e+00,
-0.00000000e+00, 8.68300158e-03, -2.19710240e-02,
5.07694776e-03, -3.22590630e-04, -9.55616865e-05,
0.00000000e+00, -9.36496765e-05, -0.00000000e+00,
1.53178292e-03, -3.53638524e-02, 1.12146623e-04,
-2.42871808e-02, 9.61732427e-03, 5.28887846e-02,
1.01917176e-02, -3.19149583e-03, 1.07590850e-03,
4.24827082e-01, -4.42527041e-02]])
Seldon Alibi
Seldon Alibi, similar to LIME and SHAP, is able to provide explanation for almost any ML model. However, the GitHub repo does not seem to be very active. From my point of view, I developed a feeling that it is wrapping the SHAP tool, and some other explanation methods , with a more consistent and more development-friendly API, that is insipired by scikit-learn
. Therefore, I did not invest too much effort in it. Nonetheless, here is how you can integrate it and how to visualize the results:
InterpretML
InterpretML is an interactive tool, allowing users to visualize multiple dashboards and get insights during exploration phase. Furthermore, it offers also the data
method that returns the results in a text(dictionary) format, making it very handy for engineers.
InterpretML is an interesting tool since it already includes LIME and SHAP explainers under Blackbox Explainers and some more. It includes explainers for glass-box models (Linear Models, Decission Tree) and in the category of black-box ones it includes also Partial Dependence Plot (used to visualize dependency of the target and a (set of) input feature(s)), and Morris Sensitivity Analysis (used to screen which features are important enough for future analysis).
MLOps.toys
A great resource to learn more about Explainability tools in ML/AI and also many more cool tools that can be helpful in your MLOps journey is MLOps.toys. It is maintained by Aproia and it was one of my starting resources for this post.
Final thoughts
The tools mentioned in this post are mostly targeted toward data scientists to help with the data exploration, model design or trying to figure out which features are important and which ones are not so relevant. They make it easy to structure the output and visualize the information.
If these tools were to be used in production environments, where for all the predictions/decisions an explanation is generated, they need to me more development-friendly. This means that the output should be more in the form of dictionaries to make formatting, processing, and visualization easier via JS frameworks.
Exception is InterpretML, which provides the data used for dashboard generation also in dictionaries, and Seldon Alibi which made good efforts in providing a consistent API.
As with anything these tools are no silver bullet. They will not protect the model from being biased, they can only point that out. With enough effort misleading interpretations can be created, that can cover up the bias. A quick google search can show you how to use LIME and SHAP for adversarial purposes, however the point of this post is to make it easier to focus on the right direction.
An important point, to increase confidence in these tools and techniques is consistency. From my evaluation of LIME and SHAP, for the same model and the same data point, they provide different explanation as which feature (or the value of that feature for a given data point), had the strongest impact in the prediction made. But overall their explanations made sense and that is great.
Finally feel free to get in touch with me via LinkedIn.