Exploring how AI models can avoid “confident incorrectness”
Much of the conversation around AI models today is healthy skepticism and centers around raising proactive questions about risks and harms — including the risk of models confidently offering incorrect predictions. It’s important to raise these questions early, so that the challenges that arise can be explored more deeply and addressed, and to avoid unintended outcomes such as predictions that might be understood as unfair bias. At PAIR, we’ve been writing AI Explorables — a series of visual, interactive essays to help people understand emerging issues in AI development. Our latest, looks at how and why AI models sometimes misbehave. We also offer details on research on what can be done about this.
“From Confidently Incorrect Models to Humble Ensembles” focuses on a root cause for model misbehavior: when a model encounters new data that’s different from what it has been trained on. When this happens, a model might classify the data as something it isn’t, simply because the model only recognizes what it has been trained on, which is usually a fixed set of data types.
For example, a model that has been trained to identify letters might mis-classify a number as a letter. The model might even assign a high confidence score to this misclassification because it was the best choice among the types of data it was trained to predict, even if it is incorrect.
In our recently published Explorable, we discuss an approach to help create an AI model that isn’t as “confidently incorrect.” It’s a technique called “ensembling,” which works by averaging the outputs of multiple AI models. In the Explorable, we offer interactive demonstrations that illustrate how using the ensembling technique with more and more models can reduce how often a model is confidently incorrect. In the figure below, for example, you can explore how increasing the number of models broadens the decision boundary, decreasing the model’s confidence on ambiguous examples. With only 1 model, there is a strict boundary between red and blue dots:
But with the ensembling technique, there is more of a gray area, where there might be a less clear decision:
We also discuss other ways to address the issue of AI models’ confident incorrectness, by changing the training paradigm, loss function, or expanding the training data.
Enjoy our latest Explorable. And let us know your feedback via comments to this post.
– Nithum Thain, Adam Pearce, Jasper Snoek and Balaji Lakshminarayanan, with Reena Jana
