Introducing Trelawney : a unified Python API for interpretation of Machine Learning Models
This article lays out the advantages of explaining black box machine learning models and explains how to do it with trelawney, a package that aims at simplifying the model interpretation workflow.
The importance of interpretation: why we built Trelawney
Interpretability : The final step of building Machine Learning Models
For a long time, models have focused on reaching high performances and not necessarily on explaining the causes of such results. For instance, computer vision problematics are more about being as precise as possible than about explaining what features led specifically to such outstanding precision… With the democratization of machine learning solutions, the need to understand the drivers of a specific model has become more and more apparent : many applications of machine learning algorithms cannot only rely on shiny figures but need the possibility to give easy access to their interpretation.
The interpretation first helps to assess the model relevance and to increase the confidence one may have in it. In that sense, it helps both technical and non-technical stakeholders. Indeed, most people will not blindly trust performance metrics, they will want that model and the product that is being built with it to “make sense”. This is the heart of explainability: making sense of black box models.
In addition, explaining how your model works enables non-technical stakeholders to derive more value from your model. Is the feature important ? If yes, in which direction does it drive the output? These are the type of questions we want to answer. For instance, in the original project for which we built Trelawney, the interpretation of the model was used to build a sales pitch for bankers.
Furthermore, being able to explain the predictions of your model will also enable you to build better products. For instance, if you are building a fraud detection system, being able to tell a user why their transaction was blocked might reduce friction and educate them.
Trelawney : Python Package for all interpretability methods in one place
This need for model interpretability led us to build Trelawney in collaboration with Quinten, a French consulting company specialised in Data Science and Data Translation. Indeed, Quinten works in applied machine learning, in particular in the healthcare and banking industries. For them, a good solution not only needs to meet analytical expectations, but should also be understandable and actionable on the field.
Our project during this collaboration was to give an appetence score for a banking product, for each given client, based on its profile.
In the context of selling a financial product, the banker would call a potential client to explain why the product would be relevant for them. An interpretation of the model therefore helps us build sales pitch (emphasising which characteristics in the client profile the banker should mention, and which one he/she should avoid).
We built several Machine Learning models, from the most interpretable ones (such as a single decision tree, or logistic regression) to less interpretable models (such as Random Forest, XGBoost and Neural Networks), and then applied several interpretability methods, model-dependent or model-agnostic, and global or local.
The whole philosophy behind the Trelawney package was to unify the different existing APIs of current explainability packages. Indeed, for the different methods we will present, there is an existing python implementation (usually provided by the researchers who introduced said methods). Our idea was to write a common API that would allow Data Scientists to learn how to use those methods once and focus on choosing the best one for their use case (instead of learning a new package each time). Furthermore, we wanted to create a unique go-to package for all-interpretability methods, with a unified syntax.
We named our package Trelawney in honour of the professor of divination of Hogwarts, known for her capacity to interpret blackbox tea-cups.
Interpretability methods, theory and implementation with Trelawney
Speaking of all the methods we are bringing under the same roof, let’s spend a bit of time explaining each of them and how to use them with Trelawney. This will be the most theoretical part of this article and you might want to skip if you already know it.
We will do some demonstrations by explaining a Random Forest trained on the Titanic Dataset. You can find the full notebook here .
We will first explain global interpretation methods with SHAP and surrogate models and then look at local explanation methods with LIME and SHAP again.
Global interpretation
The first type of explanation you will want to do is a general explanation of the model. The idea here is to tell which features are most influential on the model’s overall output. For this, there are two main methods that have been implemented in Trelawney: SHAP and using a Surrogate model.
SHAP
The first global interpretation method we will present is SHAP. SHAP (SHapley Additive exPlanations) uses game theory to explain the prediction by computing the contribution of each feature to the prediction. It computes a fair measure of each feature’s contribution to the final output (increasing or decreasing the prediction). This is divided between intrinsic contribution (if the feature was alone) and the marginal contribution (when the feature is combined with other features). Bear in mind that SHAP tries to approximate the prediction, and therefore boils down to building a simpler model on top of the original one.
If you sum the absolute value of all the SHAP values for each individual prediction, you get the feature importance.
Below is the application of SHAP for our Titanic Example :
>>> explainer = ShapExplainer()>>> explainer.fit(model, x_train, y_train)
We can then get the importance of each feature:
>>> importance_dict = explainer.feature_importance(x_train)Or we can directly use Trelawney’s graphs:
>>> explainer.graph_feature_importance(x_train)
In the above graph we can validate our intuition of what is important in order to determine if one survives the Titanic.
You might wonder what the ‘rest’ column is. Trelawney lets you limit the graph to the n most impactful features (the rest is the importance of all the other features summed).
Surrogate models
The second global interpretability method available in Trelawney is using a surrogate model. The main idea behind this method is to fit a fully explainable model (such as a decision tree or a linear regression) on the output of your black box model.
Here is the implementation with a single decision tree:
We fit our surrogate model:
>>> explainer = SurrogateExplainer(DecisionTreeClassifier(max_depth=3))>>> explainer.fit(model, x_train, y_train)
And we see how well it explains our model:
>>> explainer.adequation_score()0.968562874251497
This means a single decision Tree with a maximum depth of 3 explains 96.8% of our random forest’s predictions. The adequation score here defaults to accuracy but you can choose to use any sci-kit learn metric you wish (if you have an imbalanced dataset for instance).
The advantage here is that you can then visualize your model in a very intuitive way:
>>> explainer.plot_tree(out_path=’./tree_viz’)
This approach has the advantage of not being an approximation of a value for each of your features (as for SHAP) since it is a truly global approach that takes into account the strong correlation that might occur between features. Furthermore, in our experience, this method is also fairly simple to explain to non technical stakeholders, as people understand the principle of decision trees.
Obviously a simple decision tree cannot be as accurate as a more complex model, but we can compute to what extent the decision tree is sufficient, and explain that, in the rest of the cases, the prediction is too complex to be approximated by a simple decision tree.
Local interpretation methods
The other type of explanation methods that you might want to produce are local explanations. Here you are not trying to get a feel of your model but you want to know how it came to a result for a given sample. If we take our fraud detection system example from earlier, your user will not really care that X or Y influences to block transactions globally, he or she wants to know why he or she got blocked right now. This type of explanation are also generally quite useful for us data-scientists as debugging tools. Indeed, being able to peek at how our model made outrageous predictions might help unveil some overfitting issues with our model as a whole or biases in our training data.
LIME
The first local method that is implemented in Trelawney is LIME (Local Interpretable Model-agnostic Explanations). Lime was one of the first model agnostic explainability methods introduced by Marco Tulio Ribeiro, Sameer Singh, Carlos Guestrin in this paper. As one of the oldest methods it is fairly well known by data scientists but we will recap quickly. LIME works by building local approximations of your model using an explainable model (generally linear regressions). Basically, if you want to explain a prediction, LIME will sample points around the input of your prediction, run your model for those artificial points and try to approximate the output using an explainable model. The resulting model will give you the local approximation around the prediction of interest.
Here is how you can use Lime with trelawney:
>>> explainer = LimeExplainer()>>> explainer.fit(model, x_train, y_train)>>> explainer.explain_local(x_test.loc[most_probable, :])[{‘Age’: -0.029329040463817208,‘Deck’: 0.15191169970680513,‘Elderly’: 0.004273665522566747,‘Embarked’: 0.05382052701464114,‘Family_Size’: -0.020401593852417545,‘Is_Married’: 0.01739687755247181,‘Parch’: -0.033679921376068664,‘Pclass’: 0.11590397989964185,‘Sex’: 0.0012503083084444357,‘SibSp’: -0.0029291606226194682,‘rest’: 0.0}]
Here, the local explanation of our biggest false positive can be used to further improve our model as we’d think where a passenger embarked shouldn’t have influenced whether they survived the disaster or not that much.
SHAP for local interpretation
In SHAP for local interpretation, instead of summing the contribution for each individual sample to get the overall SHAP value for a particular feature (as we did above), we only look at the SHAP values at the observation level.
Here is the implementation:
explainer = ShapExplainer()explainer.fit(model, x_train, y_train)explainer.graph_local_explanation(x_test.loc[biggest_false_positive, :])
Final Thoughts and further improvements:
All in all, explaining your model can boost dramatically the value you are able to get out of it and we hope Trelawney will simplify this process for you (by unifying several packages with the same syntax and workflow). Explaining your models might also become a necessity in a lot of fields as regulations might require to explain the outputs of your models. For instance, the article 22 of the European General Data Protection Regulation states that “data subject shall have the right not to be subject to a decision based solely on automated processing”. The data scientist should then be able to provide explanations on the decision process. Such an opportunity is highly valuable for experts like Quinten as it allows to use black-box models when an increase in performance is needed without suffering from the lack of interpretability. Not to mention all the fields of application these new methods open.
Although we are proud of the work we have done so far, we still have a long way to go and have plenty of ways to improve Trelawney. Here are a few ideas of things that might come along in future versions:
- Support for regression problems (as Trelawney only supports classifiers as of writing)
- More model-focused methods (for intrinsically explainable models)
- Use more readable and faithful graphs
If any of those projects interests you or if you just see a bug/typo you would feel like fixing, we would love you to contribute !
co-authored by Antoine Redier and Amélie Meurer. Trelawney was built thanks to Ludmila Exbrayat, Inès VanAgt and Skander Kamoun and thanks to A. Grangetas for his support and advice.
