Deep learning sometimes makes strange mistakes
To err is human, but to make really interesting mistakes you need deep learning.
Deep learning can be amazingly powerful, but it can also make surprising mistakes that no human would make.
When computer vision started to take off, it was observed that deep learning based image classifiers were somewhat fickle. An image classifier might, for example, correctly recognize a picture of a school bus but mistake it as an ostrich when a minute amount of noise (carefully selected to confuse the algorithm) was included.
Deep learning can also sometimes see things where there are none. A group of researchers have devised a way to generate images that a computer vision algorithm categorizes as familiar objects with a very high confidence even though the images actually consist of just wavy lines or other abstract patterns.
These kinds of problems occur also when computer vision is applied at the real world. By attaching small colored patches to a traffic sign, a computer vision algorithm could be made to misrecognize a stop sign as a speed limit sign.
The problem is not limited to just computer vision. Amazon noticed this with their experimental computer program for evaluating job applicants’ resumes. The experiment was shut down after the developer team noticed that the program was not basing its decisions on words that describe candidate’s actual work experience. Instead it was favoring non-relevant words that male engineers tend to use proportionally more often than women. Essentially, the program had learned to predict applicant’s gender instead of suitability to the job. This bias arose because the system was trained on resumes submitted to the company in the past, and because men have applied and been hired to engineering positions more often than women.
Why do computers make stupid mistakes?
How come an image of a school bus gets classified correctly when an essentially identical copy is mistaken as an ostrich?
Deep learning models are complicated mathematical black boxes. Their decision logic is often intractable. Human brains are also complicated, but there are no reasons why two complicated things would perceive the world similarly. It’s perhaps not surprising that humans and machines have a radically different perception of what makes two images similar.
Humans and AIs see the world very differently. When a human sees an image of an ostrich, they recognize it, associate it with the concept of an ostrich and recall related facts: a fast-running flightless bird, a long neck, lives in Africa, etc. An AI, on the other hand, does not even know an ostrich refers to an animal (or, indeed, what animals are). To an AI an ostrich is just a statistical pattern of pixels (in computer vision applications) or characters of text (in natural language processing applications). AI applications are missing the common sense knowledge shared by most adults.
Thinking in terms of statistical patterns is very unintuitive to humans. Even developers of AI applications often struggle to understand what kind of patterns the system has learned.
To make matters worse, statistical models might learn completely different patterns than the ones developers intended. Statistical methods take shortcuts and learn the simplest patterns present in the training data. If most of the images labelled as polar bear include ice and snow, a model might incorrectly learn that polar bear means a white background.
Training better models
There are ways to mitigate problems but no complete cure. Conceptually the simplest fix is to collect more training data. However, covering all corner cases would require impractically large training sets. A training data for a self-driving car can only ever contain a small fraction of possible situations the car might encounter in the real-world traffic.
A related approach is to synthetically extend the training set by generating slightly modified copies of the training samples by cropping, rotating or augmenting images with low levels of noise. This is a way to indicate to the model what level of variation in images humans still perceive as the same object. The predictions from a model trained in this fashion are found to be more robust and more consistent with the human perception.
Problems could be alleviated if prediction models could notice when they are confused. A reason for the confusion might be, for example, that the input is significantly different from anything the model has seen during the training. An active track of research is aiming to figure out how to make algorithms measure their own confidence.
The ultimate solution would be some way to include human common sense knowledge in the prediction model. Bayesian modeling provides principled means of integrating prior knowledge. However, expressing one’s knowledge in the required probability distribution form is difficult in complex problems.
Impact on AI development
An AI system can work well most of the time but still make surprising mistakes occasionally as discussed above. This has implications on the development of AI systems.
First and foremost, developers need to be mentally prepared for unexpected predictions. They need to assess the stakes of the potential errors and provide the users suitable ways to recover.
Comprehensive testing can provide some degree of confidence that the system is behaving as intended. However, testing will never be able to cover all cases. If there exists even a few rare inputs that lead to strange predictions, users are bound to stumble upon them sooner or later. Therefore, AI systems should be designed to degrade gracefully in the case of surprising predictions.
The nature of the application and its position on the continuum of automation also affects the impact of peculiar predictions. In some applications an AI is merely offering suggestions that are always reviewed by a human. Gmail autocomplete is an example of this kind of application where an occasional strange suggestion might cause hilarity but rarely have grave consequences. If, on the other hand, an AI system is expected to make decisions autonomously it should be designed with a sufficient margin of error. Users should additionally be given control to inspect and reverse the decisions retrospectively.
Finally, odd results are not always to be shunned! Unexpected outputs might be a cradle for creativity. AI dungeon is an open-ended text adventure story generator. Its attractiveness is primarily based its ability to generate locally coherent sentences. At the same time, its limited capability to track the full story context tends to lead to dreamlike, anything-can-happen storylines.
