A non-data scientist’s introduction to building trust in machine learning.

One of the barriers we see to the adoption of machine learning for critical decision making is that of trust. We’re often asked the question “Why should we trust the model?”. It doesn’t matter how well a machine learning solution performs if its output isn’t trusted by the users.

Trust and Data Science Models

If it’s not trusted, it won’t be used.

Our ability to help people trust machine learning models is as important as our ability to produce and run effective models.

You can start to build trust by testing a model’s ability to predict a known output. For example, if you’re predicting sales for a portfolio of products, you could show how accurately the model can predict last week's sales when it only sees data available before that period.

This approach is great at building trust for a model’s output before it starts to make any real world predictions. But what happens when the model is live and produces a prediction for sales (for one particular product) in one particular month that ‘feels’ off; doesn’t pass the ‘sniff test’? Here, historic accuracy metrics tend not to hold much weight. In order to get trust for a particular prediction, you need to do the same as a human making that prediction would - you need to explain your working; take your audience on the journey you took to get to that prediction. “I think we’ll sell more BBQs than normal this month because the weather forecast looks great for the next couple of weekends”

How can we do that with machine learning? Explainable AI (XAI) is a hot topic right now, and it looks to address a number of concerns with the lack of transparency of AI and, in particular, machine learning models. One approach we use is SHAP (SHapley Additive exPlanations). Before we dive into that, it’s worth a quick intro to the basics of supervised machine learning.

Supervised Machine Learning

Supervised machine learning is a family of machine learning approaches which learns associations between input data and output data. It does this by analysing a set of training data which contain the input data and the ‘answers’ (known outputs for the given inputs). In our sales forecasting example, this might be historic data about shopping trends, customer behaviours, etc. along with the actual sales made. Training the model is a process whereby the model learns the rules which determine how the input (all the various data points) affects the output (number of sales of product X). Once trained, the production model can apply these learned rules to a new (previously unseen) set of input data and generate predicted results. François Chollet sums it up nicely in this diagram:

The input data in a machine learning model is a set of “features” that represent the quantifiable elements of the business problem that are likely to influence the output. In our BBQ example, our features used to anticipate how many we might sell tomorrow, might include: weather forecasts for the next few days; day of the week; month of the year; and historic sales data.

The effectiveness of the model depends crucially on how representative the training data is of the unseen data. If there is a significant change in the operational input data, the model is unlikely to work well.

SHAP

Right, back to SHAP.

SHAP tells us the impact each feature has had on each prediction and is typically visualised in a chart like the example below:

Each row represents a feature the model used to make the predictions, and each dot on the row represents a prediction (for example a prediction for sales of a particular product).

The position of the dot along the row indicates how much that feature impacted the prediction: From a significant negative impact at the far left, through no impact in the centre, to a significant positive impact on the far right. The colour of the dot represents the value of the feature for that specific prediction. It can be a little tricky to get your head round at first, here are the predictions for next week’s sales of 3 different products:

The SHAP visualisation is a great way to get a macro level picture of the relative effect of different features on the predictions. But it’s the data behind the plot that is really valuable when building trust in your model. It provides the information to back up any prediction — it explains the working. It’s the model’s way of saying “My prediction of high BBQ sales is primarily due to the positive impact of a high average temperature forecast for the next 2 weeks”

As well as building trust by explaining its working, this transparency also allows greater scrutiny of the model’s behaviour. It may expose a really bad (as in, obviously wrong, from a human’s point of view!) justification for a prediction. Exposing these blind spots allows us to fix them, leveraging the experience of those closest to the problem. This, in turn, is another trust builder. People are more likely to trust a model when they can see how their own expertise has been directly involved in building it.

Surfacing all this information in the right format, in a timely manner, to the people who need it is a topic for another post. When done right, the information exposed by SHAP can help bring transparency and trust to what would otherwise be a black box prediction, resulting in a model which is used more and so provides more value.

Footnotes

If you want to read more about SHAP the original proposal can be found here: S.M. Lundberg and S.-I. Li. A Unified Approach to Interpreting Model Predictions. 31st Conference on Neural Information Processing Systems (NIPS 2017) and this is the SHAP GitHub repository.

For more thoughts on use of SHAP and its pros/cons, Christoph Molnar’s Guide for Making Black Box Models Explainable is a good read.