Explainable AI: The data scientists’ new challenge
Authors: Lamine Diop and Jean Cupe
Before we start, let me mention that in this article I will use without regards the words explainability and interpretability to describe the same concept.
That being said, let’s imagine a situation where someone would ask for a bank loan and the computer, powered by a specialised Machine Learning algorithm in credit risk assessment, denies the demand. The employee, being unable to explain the reasons of such a categorical answer, would surely be left in a very embarrassing position.
In a context where advances in Machine Learning are reaching critical areas such as medicine, criminal justice systems and financial markets, this kind of situation will surely tend to be more frequent. In fact there is a growing concern around the acceptance of AI agents and trust issues due to their lack of explainability. For many years, priority has been given to the performance over the interpretability leading to huge advancements in several fields including computer vision, natural language processing and sequence modeling. However, the crucial questions driven by the reluctance to accept AI-based decisions may lead to a whole new dynamic where explainability may be among the key measures for evaluating models. And in this article, I will discuss some concrete reasons why it would be a major differentiator. Some approaches to develop powerful algorithms while still retaining a good degree of interpretability will be also suggested.
“ Having an accurate model is good, but explanations lead to better products. “
Data science improvements have always been driven by the search of the model with the best performance to address any problem. From a mathematical point of view, the search of the model focuses on the minimization of a cost function or the maximization of a likelihood function. Thus the performance of the model is measured almost exclusively on the results we can get according to some rightly chosen metrics. This tendency has led to more and more sophisticated algorithms to the expense of explainability. In fact, some data scientists may even assert that “ What makes machine learning algorithms difficult to understand is also what makes them excellent predictors ”. They are complex and that is what makes them look like “ black boxes ” for most field practitioners.
The previous image depicts 2 levels of interpretability for machine learning algorithms:
- High interpretability: This level includes traditional Regression Algorithms (linear models for example), Decision Trees as well as Classification Rules. Basically they approximate monotonic linear functions.
- Low interpretability: This level includes advanced machine learning techniques such as SVMs, Advanced Ensemble Methods, and Deep Learning. Their complex lies
Between the two you may find some methods that can be classified in either one of classes depending on the constraints applied in the learning process (monotonicity constraints can be applied to Ensemble Methods and some implementations are already available).
Being explainability the measure of the degree to which a human observer can understand the reasons behind a prediction made by a model, it becomes increasingly more necessary to find the right balance with the accuracy. It could be the key to making algorithms as transparent as possible for day to day users. This change of focus towards “ user centricity ” may probably make the difference on the appropriation and acceptance of this technology throughout a variety of domains, granting them a more comfortable adaptation to carry out the tasks in a more realistic way.
Although many core aspects of applied machine learning are the same across different fields, they cannot be applied in every industry. The machine learning approach is very different specially in banks, insurance companies, healthcare providers and other regulated industries. The reason is mainly that they are prone to legal or even ethical requirements which tend to limit more and more the use of black box models. As an illustration, I may site the section 609(f)(1) of the Fair Credit Reporting Act:
“ The consumer shall be provided all of the key factors that adversely affected the credit score of the consumer in the model used, the total number of which shall not exceed 4. ”
Most importantly this clearly means one should provide insights about how the response of the model was obtained. The immediate consequence is reducing the scope of possible approaches to the simplest ones due to their easy explainability, unless we find ways to add contexts to the predictions of state-of-the-art algorithms.
Not to mention also that the tendency is more to the accentuation of regulation constraints rather than their slow-down.
All those factors contribute widely to the general reluctance of the industry to adopt and deploy advanced data products powered by Machine Learning systems. In front of such restraints, researches are more and more focused on the identification of ways to deal with the interpretability of models.
As far as I know, there are mainly two approaches towards achieving explainability of models: global explainability and local explainability. The former aims at making the entire process of decision making completely transparent and comprehensive while the latter focuses on providing explanations for each decisions.
“ When we start a Data Science project, it may now be a best practice to determine first the degree of interpretability we would like to achieve. “
In fact, deciding the degree of interpretaility at first place will guide the choice of algorithms and techniques we might implement. Concretely it means whether go for a simple model and make it more powerful (to achieve global explainability), or use a complex model that we could make more interpretable (to achieve at least local explainability). The following table presents a non-exhaustive list of techniques to deal with the interpretability of a model.
Maybe you have noted that there are more details about local explainability than global explainability. This is mainly due to the huge efforts that have been made to explain complex models but with a local boundary approach.
Finally, when we implement our algorithm with the right approach, the last interrogation lies on how to evaluate the interpretability of the model. There are 2 approaches that lead to a better representation of the degree of understanding granted by the final model:
- Simulate data with known characteristics and intrinsic implications to confront the interpretation of the model to the prior knowledge that we have in that context.
- Test the stability under data perturbations by putting a little noise in the explicatives variables. Trustworthy explanations likely should not change drastically for minor those minor changes.
Hope that you enjoyed your journey through this yet highly important “recent” concern. If you have any question or remark, I will be glad to welcome it for further discussions. In my next publication I will directly address this problematic through a real world use case to get a better understanding of all the implications above.
For further readings do not hesitate to consult the following links.
I would like to thanks my colleagues who contributed to this article too José Sanchez, Nelson Fernandez, Simon Palma.
https://arxiv.org/pdf/1606.03490.pdf
https://distill.pub/2018/building-blocks/
An Introduction to Machine Learning Interpretability by Navdeep Gill and Patrick Hall
