Detecting corner cases for visual perception in autonomous driving
In recent years, automated driving has emerged as a revolutionary technology, capturing the interest of both the research community and the media. While significant progress has been made in the field, detecting corner cases remains a critical challenge for ensuring the safety and reliability of autonomous vehicles.
Understanding corner cases
Corner cases refer to unexpected and unknown situations that can arise while driving, situations that may not have been encountered during the training of automated driving systems. These scenarios pose a significant safety risk and are often challenging to detect using conventional visual perception methods. Examples of corner cases include a person suddenly running onto the street from behind an occlusion, a ghost driver, or lost cargo on the road.
In 2019, a Tesla in Autopilot mode failed to recognize a stopped truck on a Florida highway. In investigations following the crash, it was clear the system’s cameras captured the truck, yet it was not properly sensed, causing the car to continue driving regardless of the obstruction. Following this incident, Tesla noticed certain weaknesses in its technology’s ability to detect perpendicular trucks. Instances like this and other AV accidents in the headlines highlight how crucial the accurate detection of corner cases is.
While cases like this have occurred and autonomous vehicles are still rather new to the market, the technology has demonstrated significant safety advantages compared to human drivers. As AVs become more widely adopted, it is expected that there will be a reduction in human deaths caused by vehicular accidents. In a study conducted by Waymo on their fleet, they found a 76 percent reduction in property damage in AVs compared to human drivers.
In recent years, there have been significant technological strides in AV development, with Level 4 autonomous vehicles being deployed in city streets. That said, as their presence increases on roads, more and more instances of technological shortcomings have been brought to light as autonomous vehicles encounter new and unforeseen challenges. To ensure the safety of such systems, it is crucial to understand the various technological limitations and weaknesses that can be confronted on the road. With the assessment and categorization of these limitations, it is easier to understand how exactly systems must be trained to ensure the utmost safety of AVs.
The importance of detecting corner cases
The reliable detection of corner cases is crucial for minimizing accidents involving autonomous cars and continuing to fortify the safety of AV technology.
Detection methods play a vital role in both online (in vehicle) and offline (during development) applications. In the online application, the corner case detector acts as a safety monitoring and warning system, identifying critical situations as they occur. In the offline application, the detector is applied to comprehensive datasets to select suitable training and test data for the development of new visual perception algorithms.
Systematization of corner cases
To address the challenges associated with corner cases, we have taken as reference the definition and classification shown in Corner Cases for Visual Perception in Automated Driving: Some Guidance on Detection Approaches, corner cases are present when “there is a non-predictable relevant object/class in a relevant location”. A systematization has been introduced, categorizing them into different key groups ordered by detection complexity, going from higher to lower complexity of detection: scenario level, scene level, object level, domain level, and pixel-level corner cases.
Source: arXiv:2102.05897
Let’s look at some examples.
- Representative example of corner case at pixel level (global outlier): navigating through a tunnel on a bright sunny day, our camera images experience overexposure at the tunnel’s exit due to the intense sunlight.
- Representative example of corner case at domain level (domain shift): navigating through a bustling city street, the atmosphere is defined by rain and darkness. Adding to the complexity, there is a substantial amount of oncoming traffic.
Here you can see a complete list of representative examples of each corner case level.
Detection Approaches
Researchers introduce various detection approaches, with a focus on deep learning methods. These approaches are categorized into five concepts: reconstruction, prediction, generative, confidence scores, and feature extraction. Each approach has its unique strengths and applications in addressing specific corner case levels.
- Reconstruction approaches: based on autoencoder-type networks, these methods reconstruct normality more faithfully than anomalies. They are applicable across various corner case levels and are particularly effective for global outliers.
- Prediction-based approaches: primarily found at the scenario level, these methods predict future frames and compare them with actual frames to detect anomalies. They are trained in a supervised manner, assuming all training samples are normal.
- Generative approaches: closely related to reconstruction methods, generative approaches consider the discriminator’s decision or the distance between generated and training distribution. They prove effective for detecting unknown objects in object-level corner cases.
- Confidence scores: categorized into learned scores, Bayesian approaches, and post-processing, confidence scores are useful for assessing the uncertainty associated with corner cases. They can be applied across various levels, providing valuable information for detection.
- Feature extraction approaches: utilizing deep neural networks to extract features from input data, these approaches either directly classify corner cases or use extracted features for decision-making. They are effective for object-level corner cases and domain adaptation.
The challenge of gathering corner case data
Gathering corner case data is challenging. On the one hand due to the lack of large-scale datasets containing different types of corner cases, and on the other due to the almost infinite list of possible corner cases that can occur.
Future research should focus on developing more complex and specialized training datasets to address the wide range of possible corner cases, and to achieve this, the addition of synthetically generated data is highly advantageous.
Synthetic data helps researchers to generate the corner case data they lack. It allows them to replicate any scenario, including the rare and challenging situations we are addressing throughout this article that are difficult to encounter in the real world.
This permits gathering great data diversity and variability to train robust autonomous driving algorithms to develop market applications that are trustworthy in all real-driving situations. Synthetic datasets may bring the ultimate goal of achieving autonomous vehicles as safe or safer than vehicles commanded by humans closer.
Conclusion
In conclusion, the journey towards fully autonomous driving hinges on overcoming the challenges posed by corner cases. The systematic categorization of these rare situations and the introduction of diverse detection approaches provide a roadmap for researchers and developers. By understanding the nuances of each corner case level and leveraging advanced detection methods by gathering the right training data, we can pave the way to train corner case-proof autonomous driving algorithms for safer and more reliable automated driving technology.
