Podcast: Explaining the Predictions of Machine Learning Models (TWiML & AI)

Carlos Guestrin discusses ‘explainability’ in machine learning and an algorithm to display drivers of predictions called LIME.

Link: TWiML Talk #7 — Carlos Guestrin — Explaining the Predictions of Machine Learning Models | iTunes link

I’ve listened to a few of these talks (there are 11 in total) and thus far this is the most approachable and insightful for people new to the topic.

Carlos is a Professor at the University of Washington and sold his company Turi to Apple in August of last year. Around the same time, he co-authored a paper called “Why Should I Trust You?” outlining new algorithms his team developed that would help identify key inputs in models that led to a particular prediction (i.e. identify words in a paragraph that were most useful in identifying the topic of the paragraph). He discusses this work in fairly plain language (appreciated) in the second half of the talk.

The first half is a broader discussion around why Carlos believes its so important that predictive models, particularly complex ones, be capable of informing a user and affected individuals how it arrived at its conclusion. He buckets his rationale into three helpful categories with examples:

1. Public Perception and Trust: The end-users being served the prediction feel more comfortable when they can get a basic gist of how it arrived there (even if it only partially explains the outcome).
2. High-Impact Decision Making: For professionals accustomed to making high-stakes decisions based multiple inputs (i.e. doctor’s diagnoses, military missions), relying on one prediction alone feels insufficient. Understanding the key drivers reminds the decision-maker of their typical process and gives them confidence that the drivers sound reasonable.
3. Model Improvement: Giving the programmer of the model more insight into how predictions are generated and what variables are driving predictions allow them to diagnose issues more quickly and explain their work to others, technical and non-technical alike.

One more highlight: Carlos also mentioned an unfortunate trend that involves sacrificing accuracy for explainability. Carlos points out that in some industries, only specific models are permitted for use because more complex or less traditional models cannot be effectively explained. While I believe these types of restrictions are sensible — the so-called ‘burden of proof’ should fall on those introducing new alternatives — it’s a limiting factor for maximizing accuracy, which matters a great deal in many applications.

It’s very clear to me that this area is deserving of more research and products aimed at solving the ‘black box’ perception and reality of predictive models.