On a Formal Model of Safe and Scalable Self-driving Cars (Response)
This paper explores the formal process in which self-driving cars can be programmed to avoid accidents. It lays out, in a rather simple language, the conditions a self-driving car would need to follow to be “safe”. It explores the idea of safety and responsibility from a legal standpoint, citing tort law and people’s perception of self-driving vehicles. The paper establishes early on the assumption that AVs must be at least three orders of magnitude safer than a human driver to be realistically viable and trusted. This incorporates people’s general cynicism, fear, and pushback against AVs and makes them only viable in the case of extreme advantage over human drivers.
The paper begins with a few broad definitions of the key components of AVs (sensing, planning, acting) and analyzes current safety standards on AV decision making. Different methods are deconstructed and debunked (e.g. “Miles driven” approach makes it too hard to prove statistically) and sets the stage for the main goal of the paper: defining a framework for programing safety into AVs. After expressing the need for this framework, named RSS, the paper wastes no time in defining its main objectives. These objectives are basically: laws have implemented the way a human would understand them, laws lead to a ‘useful’ driving mechanism where cars are not overly defensive or obstruct other cars, and finally, the implementations of RSS are efficiently verifiable to be correctly done.
When thinking of vehicle safety, there are two levels of analysis. Firstly, there is that of the actual vehicle in question or functional safety, and then there is that of the decision making “brain”. This paper effectively only covers the latter. The decision-making capacity of the RSS model is built from the ground up with the simplest driving scenarios dealt with before the more complex ones.
RSS is built around the legal notion of “being careful”. The entire premise of this system rests on the idea that if an agent follows pre-defined qualitative rules such that any reasonable human would determine them to be careful, then that agent is not responsible for any accidents that happen. This is because, in a world where humans and AVs share the road, it is impossible to avoid eventual crashes. The most important thing this paper does is decompose the idea of being careful into 5 actionable qualitative rules then take the time to formally define them quantitatively. This is done by considering the simple single-lane road example, then more and more complicated road structures up till completely unstructured spaces (e.g. parking areas). Through these examples, this paper establishes definitions of important concepts that build on each other. These concepts include the idea of safe distance (initially defined as the minimum distance between cars such that a reasonable breaking deceleration from the front car can elicit a proper reaction from the back car to avoid collisions), safe time, proper response, danger threshold time, etc.
These fundamental concepts, proven one by one through rigorous proof, build a robust framework to understand responsibility. To oversimplify it, if an agent never takes an action to put it in a dangerous situation, and when forced into one, always takes the best action to exit that dangerous situation as long as it doesn’t put it into another dangerous situation, then that agent cannot be responsible for an accident.
While the paper does in fact consider more complex cases where accidents might happen: occultations, sensing problems, extreme irrational behavior, etc. It does not consider the ethical ramifications of an accident caused by those problems. For example, assume a very foggy environment where a specific two-year-old sensor, which holds during testing, partially fails enough to affect reaction time enough to create real danger. Since this is technically a functional safety issue instead of a nominal safety issue, one could argue that it is no one’s fault as much as failing breaks are. But consider this, with failing breaks, we can, as a society, understand the full reason behind the accident. But when a car seemingly acts against its programming to cause a crash even by our own robust RSS definition, what do we do then. This idea is unsettling to many because this is where the AV stops acting like a human. If a car in front of a human driver suddenly stops on a highway lane, the human driver will usually break aggressively and swerve because they will not have time to react and think about all the implications of their decision. However, an AI-controlled vehicle can make a decision every 0.1 seconds, and it can thus consider all of its surroundings. It would follow the “If you can avoid an accident without causing another one, you must do it” rule, but then if it cannot avoid an accident, which one would it get into? If the car tries to break in-lane it would bump into the erroneous human-driven car in front of it, which would violate the rule “Do not hit someone from behind”. However, if it swerves into another lane, it violates the “Right-of-way is given, not taken” rule. Through this example, we see that however robust RSS is, there are still gaps in its procedure. Because as I’ve just shown, there are examples where there needs to be a trolley-problem-style decision made where a human would decide something wholly irrational. So, is it ethical to make an AV to make those irrational decisions? If not, what are the ethics of deciding who to implicate in an accident? These are tough questions we must explore before letting AVs roam around unsupervised.
