Breaking Down the Black Box
In other words, how to make machine learning intelligible to humans
If It Works, It Works, Right?
I remember sitting back after finishing my first Kaggle competition on machine learning, marveling at what essentially boiled down to chunks of unintelligible linear algebra and a few statistics describing how nicely the output aligned to our expectations. I thought I knew what it was doing- I could explain how the janky neural network operated (well, at least in theory), or go through the TensorFlow documentation and at least recognize most of the parameters and its role in the process.
But if you’d asked me why in the world it had made the predictions it did, I couldn’t have been bothered to give more than a careless shrug. The model works, so what does it matter anyway?
Turns out, it matters quite a lot. Say you’re a machine learning specialist developing an in-house model for a large hospital system. You have access to reams of raw medical data and patient metadata, and over the course of months, have trained a nifty model that displays realtime probabilities of mortality, indicating that a patient may require further surgical intervention.
The issue isn’t simply that the model may sometimes contradict the physician’s judgement. If…
