Understanding Black Box Algorithms
In 2018, shortly after a self-driving Uber car killed a pedestrian, an article ran in The Guardian, titled, “Franken-Algorithms: The deadly consequences of unpredictable code.” The article went on to explain that Uber engineers don’t know why the AI guiding the car ran afoul. Since then, the tech community and the news media have generally decried the existence of “black box algorithms.”
If you’re in the business of overseeing the development of an AI product, one thing you’ll need to take into consideration in choosing what kind of model to use is whether having explainable outputs is important, and if so, to what extent. This isn’t an easy decision in many cases, and that’s because explainability doesn’t come for free.
AI — or more specifically, ML (machine learning), works by feeding an algorithm tons of examples (aka data). You give it 10,000 pictures of your spouse, it “learns” what your spouse looks like by noticing patterns among thousands of of pixels and the relations among those (clusters of) pixels, and if it did a good job, when you upload the 10,001st picture of your spouse it sees the “spouse pattern” in that new picture and categorizes it appropriately.
The quantity of data points your algorithm is combing through and the relations between those data points is vast and complex. Too complex, in fact, for mere mortals like us to understand what pattern it’s recognizing among those 10,000 pictures. The same goes for if we’re talking about more consequential applications of ML: why this resume got a red instead of a green light for an interview, why this person was denied a mortgage, housing, or credit, why this person was rated as being at high risk for committing a crime, and so on. We can’t understand either the general rules of the game the model is applying to inputs to turn them into outputs (what’s known as global explainability), or why this particular set of inputs led to this particular set of outputs (what’s known as local explainability).
I said that explainability doesn’t come for free. That’s not just because it takes additional resources (e.g., time) to think through whether explainability is important for a given model, but also because increasing explainability tends to decrease accuracy. That’s because, all else equal, the more complexity you give an ML the more accurate it gets. But the more complex it is, the harder it is for us to understand; turning up the volume on accuracy means turning down the volume on explainability, and vice versa.
As someone leading the development of an ML product, you’ll need to think about whether explainability is important for your project and what its level of importance is relative to accuracy. There are no strict rules here for how to make these decisions (any more than there are strict rules about how you choose your objective function). That said, if you’re not making any predictions about people — perhaps your model makes predictions about energy consumption, for instance — you can probably safely ignore explainability for the sake of accuracy (at least as far as ethical concerns are involved; you may also want explainability so that you can tweak your model). If, on the other hand, your model will be used to make life-impacting decisions on people, you might owe them an explanation.
As someone leading the development of an ML product, you’ll need to think about whether explainability is important for your project and what its level of importance is relative to accuracy. There are no strict rules here for how to make these decisions (any more than there are strict rules about how you choose your objective function). That said, if you’re not making any predictions about people — perhaps your model makes predictions about energy consumption, for instance — you can probably safely ignore explainability for the sake of accuracy (at least as far as ethical concerns are involved; you may also want explainability so that you can tweak your model). If, on the other hand, your model will be used to make life-impacting decisions on people, you might owe them an explanation.
