The Necessity of Explainability for Deep Learning

Deep learning algorithms aren’t that simple to explain, but in a nutshell, the model is fed with a variety of factors serving as inputs to find out scores (involving percentages, probabilities, etc.) that are numerical outputs.

Most times, the meaning of these scores clearly relates to a specific business problem, such as whether a score is high enough to issue a customer a certain loan or not or if the score is above a set threshold to predict which customers will churn out. Though what’s much less clear is how that score was derived and which factors impact it the most.

Explainability is the capability to understand how those scores were generated by the machine learning model.

Two aspects of deep learning make explainability particularly challenging. The first is its tendency towards non-linear pattern recognition, which exceeds that of traditional machine learning. Secondly, deep learning also does a fair amount of feature detection on its own. Thus, modelers might not know which features or patterns affect deep learning outputs which leaves them with little to no control over the outcomes of the model.

Without explainability, there’s little visibility into this process designed to create insights. By mastering explainability, however, data scientists can easily adjust deep learning models to perfect their outputs and the business processes depending on them.

Why is Explainability needed?

There are four key benefits driving the need for understanding how deep learning models create their outputs.

• Transparency: Transparency is required to understand and exploit the basic mechanisms of deep learning models. Knowledge of predictors or features enables data scientists to adjust their values to see their effects on scores. For instance, if they know one of the attributes is family size when attempting to predict which members of a certain population will buy a car in six months, they can increase that value to see how it affects the output.
• Verifying Intuition: Models don’t understand human intuition; they simply view variables in mathematical terms. When determining which customers to give a loan, for instance, people would assume a positive correlation between income and receiving the loan. However, models might create negative correlations between income and loans. Explainability is necessary to ensure models are drawing parallels consistent with human understanding.
• New Patterns: Whereas human thinking is limited to three dimensions, models can apply numerous dimensions to business problems. Explainability for non-linear patterns in fraud detection, for example, enables humans to learn new patterns from deep learning to incorporate into business processes.
• Regulatory Compliance: Regulatory compliance demands explainability in several industries. For example, financial companies need to provide reasons for declining customers, if customers want to know. Those explanations can’t be numerical model outputs, but rather their significance to financial factors which the customers (and credit card companies) understand.

So how do we improve Explainability in Deep Learning Models?

There are many ways to increase the explainability of deep learning models. Some of the less mathematically rigorous techniques include:

• Surrogate Modeling: Data scientists can create surrogate models from deep learning models to explain their results. Surrogate models are usually created by training a linear regression or decision tree on the original inputs and predictions of a complex model like predicting which customers will default on their loans. Coefficients, variable importance, trends, and interactions displayed in the surrogate model are then assumed to be indicative of the internal mechanisms of the complex model. The surrogate model results are easier to understand, helping explain the results of the initial neural network.
• Leave One Covariate Out (LOCO): This trial and error method is effective for seeing which variables impact models most. It requires removing single variables from deep learning models, one at a time, deploying the model, and seeing if the results dramatically change due to any specific missing variable. When they do, data modelers notate the difference in the model’s score and try to match it with their expectations for the difference in results based on that variable.
• Maximum Activation Analysis: This measure is effective for explaining patterns users are unfamiliar with, such as those contributing to fraud detection. When users detect unknown patterns contributing to high scores for fraud, they can give these patterns higher weights than others when optimizing models. The higher weights activate the patterns each time they occur, helping data scientists reach a conclusion about the nature of the pattern.

Explainability translates to better & more credible models

The true value of deep learning (and of artificial intelligence) is when we can have explainability associated with it. Explanations for model results can affect business outcomes, workflows, and even organizational objectives. Most of all, they indicate how best to adjust models to achieve goals for deploying predictive analytics.

Without explainability, users blindly follow models like they once blindly followed intuition. Explainability gives them the understanding to best apply accurate predictions.