Interpretability in Machine Learning
Machine learning is a very exciting field that is being used to solve business problems in various fields. Everyday new AI frameworks and tools are released, that is improving the model prediction accuracy for different types of data. But more the complex these frameworks and models are, more difficult it becomes to explain why a particular decision was taken by the model. This ‘black box’ model may create issue especially in Banking and Finance sector where things are rigorously audited and regularized.
In this post, we will deep dive into how to build explainable ML models and interpretation of the models. For humans to trust the ‘black box’ models, model ‘explanability’ is required i.e. to give insights about its decision making process. Also it need to defend its particular decision taken to auditors when needed.
Generally there is a trade-off between interpretability and predictive power of the model. e.g. A decision tree model is explainable but will not be as accurate as XGboost model and vice-versa. For an interpretable model, there are various questions to be answered like: What are the most important features/predictors? How the model works internally? Why a particular decision was taken by a model? How model will predict new cases that were nor part of training data?
- Interpretable ML algorithms:
Simplest way to get an explainable model is to create a model using interpretable ML algorithms such as Decision tree, linear regression, logistic regression, k-nearest neighbors, Naive Bayes, etc. e.g. you can understand a decision tree model as simple if-else-then rules.
- Feature importance:
For a complex model such as Random Forest, you can get the relative importance of each of the input variable. Feature importance estimates how much of the model prediction variance changes due to the exclusion of individual features. In any ensemble tree model, optimal splitting of the nodes happens while constructing the trees based on gini importance or Entropy. The feature importance is nothing but the total decrease in node impurity averaged over all the trees constructed in the ensemble. If the decrease in accuracy is low as we go down the tree, then less important the feature is.
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import make_classification
import pandas as pdX, y = make_classification(n_samples=1000, n_features=4,
n_informative=2, n_redundant=0)
rfc= RandomForestClassifier(max_depth=2, random_state=0)
rfc.fit(X, y)
print(dict(zip(X.columns,rfc.feature_importances_)))
feature_importances = pd.DataFrame(rfc.feature_importances_,
index = X.columns, columns= ['Feature_imp']).sort_values('Feature_imp',ascending=False)
- Permutation importance:
For feature importance, we look at how the model score such as F1-score, R² decreases when a particular feature/predictor is excluded from the dataset and model is retrained. This re-training will be resource consuming when done for each feature. So instead of removing the feature, we can replace the feature column with random noise(non-useful information) by shuffling the values of the feature. So Permutation Feature importance gives us the decrease in model score after randomly shuffling the single feature values. Impurity based feature importance as seen above can not be applied to unseen/test data while Permutation importance is calculated after fitting the model. So if we randomly shuffle a feature in test dataset which is not much important will not cause issue to the model prediction power but if we shuffle an important feature, we will get terrible predictions.
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import make_classification
import pandas as pd
import eli5
from sklearn.inspection import permutation_importancefrom sklearn.model_selection import train_test_split
from eli5.sklearn import PermutationImportanceX, y = make_classification(n_samples=1000, n_features=6,
n_informative=4, n_redundant=2)
train_X, test_X, train_y, test_y = train_test_split(X, y)rfc = RandomForestClassifier(max_depth=10, random_state=0).fit(train_X, train_y)
perm_imp = PermutationImportance(rfc, random_state=1).fit(test_X, test_y)
eli5.show_weights(perm_imp, feature_names = test_X.columns.tolist())
### OR using sklearn
result = permutation_importance(rfc, X_train, y_train,n_repeats=10,random_state=42)
perm_sorted_idx = result.importances_mean.argsort()
- Partial Dependence Plots(PDPs):
Permutation importance or Feature importance (based on Mean Decrease in Impurity) tells us which are the most important variables that affect the predictions while partial dependence plot shows how a particular feature affects the predictions. For a linear regression model, partial dependence plots will be similar to the coefficients(βs). So in PDP, we take a row in the dataset and change only one feature’s values and apply a trained model for the different values to get the predictions. PDP will be a plot of predicted output on Y-axis and features values on X-axis. They are ‘partial’ because they consider only one or two features at a time and do not consider the features interactions. sklearn supports PDP as well as permutation importance from V scikit-learn 0.22(sklearn.inspection).
from sklearn.datasets import load_iris
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.inspection import plot_partial_dependenceiris = load_iris()
mc_clf = GradientBoostingClassifier(n_estimators=10,
max_depth=1).fit(iris.data, iris.target)
features = [3, 2, (3, 2)]
plot_partial_dependence(mc_clf, X, features, target=0)
- Local surrogate models: LIME
LIME is model-agnostic and so can be applied on any ML model. LIME provides a local interpretabilty to determine which feature changes has the most impact on the predictions. Suppose you have a complex black box model and you want to understand why model generated a particular prediction then LIME will help you to understand that. In LIME, we will create simple interpretable models like decision tree or linear regression on the dataset created from permuted sample data and corresponding predictions of the complex model. This model gives a better approximation of the black box model locally but not globally.
Read LIME paper here: https://arxiv.org/abs/1602.04938
- SHAP values:
SHAP(SHapley Additive exPlanations) values can be used to explain each prediction by the model. Shapely values is a concept from game theory that tells us how to fairly distribute a particular prediction among the predictors.
import xgboost
import shap
X,y = shap.datasets.boston()
model = xgboost.train({"learning_rate": 0.01}, xgboost.DMatrix(X, label=y), 100)
# Explain the model's predictions using SHAP
explainer = shap.TreeExplainer(model)
shap_values = explainer.shap_values(X)
# Visualize the first prediction's explanation
shap.force_plot(explainer.expected_value, shap_values[0,:], X.iloc[0,:])
The above figure shows the features/predictors each contributing to push the model output from the base value to the model output. Features pushing the prediction higher are in red and those pushing the prediction lower are in blue.
So here we saw different methods to interpret complex ML or AI models. Interpretable ML models and tools are important because they help us in debugging a model, build a trust with user, explain the decision taken by model to users, auditors and regulators.
