A Closer Inspection of Tesla’s Autopilot Safety Statistics
The automotive industry is at the beginning of a grand experiment. If completely successful, humanity could be ushered into a new economy where driving is a hobby, only for sunny days along clear roads with a view. The struggles and tedium of the daily commute could be handled by autonomous vehicles, traffic accidents could fall to nil, passengers could focus on working and relaxing in their mobile offices, and the elderly, disabled, and blind could have considerable mobility and autonomy. If a complete failure, automobile companies would have invested billions of dollars in computer vision, sensors, and automated driving systems only to have no effect on or actually increase the number of traffic accidents and fatalities by introducing new risks. This would cause a public backlash, and requiring regulators to impose a slow, costly review process that slows the pace of innovation so that after an initial roll-out to a few hundred thousand vehicles, further roll-outs are halted. Then, autonomous vehicle technology may follow the same path as the U.S. nuclear power industry, which has stopped building new power plants since the Three Mile Island accident in 1979. Which scenario or whether something in between unfolds depends on good design, as well as careful understanding and communication of the safety of autonomous driving technology and the path from partially autonomous to fully autonomous vehicles. And understanding the safety of autonomous vehicles (AV) is a very thorny statistics problem.
Recent fatal and injury-producing crashes involving vehicles with Tesla’s Autopilot and Uber’s self-driving pilot have led to significant disagreement among experts, reporters, automakers, and regulators about safety statistics for partial autonomy technologies[1]. Tesla, in particular, has made recent headlines after two crashes and 1 fatality with its Autopilot-equipped partial autonomy vehicles in the past few months. Tesla claims that its technology is 3.7x safer than the existing U.S. vehicle fleet, stating a fatality rate of 1 death per 86 million miles for conventional vehicles versus 1 death per 320 million miles for Autopilot-equipped vehicles, but many experts question the methodology and data behind these statistics. In this article, I’ll review the data, methods, and the three main criticisms of Tesla’s methodology for conventional vehicle fatality rates, provide my best estimates, and make recommendations for regulators and automakers on the safety of autonomous vehicles. I don’t have access to data to verify the fatality rate for Tesla Autopilot-equipped vehicles but the company has promised to release public Autopilot safety data in future quarters.
1. What’s an Autopilot mile?
One informal complaint I’ve heard among analysts is the question of which miles should be included as an ‘Autopilot mile’ in Tesla’s statistic of 1 fatality per 320 million miles. Some analysts argue that one should only compare miles driven in a vehicle with Autopilot engaged to manually-driven vehicle-miles to obtain a fatality rate. Instead, Tesla’s methodology includes all miles driven with an Autopilot-enabled vehicle, whether or not the functionality was engaged.
I agree with Tesla’s methodology on Autopilot mileage because the road conditions under which a partial autonomy system is rated for operation (highways, clear lane markings, etc) are systematically different from manually-driven miles. If one only used Autopilot-enabled miles in the fatality rate calculation, a comparable baseline of miles for a manual vehicle driven under similar road conditions would be difficult to obtain and there are already considerable gaps in the vehicle mileage data needed to compute good partial autonomy safety statistics (more below).
Because the characteristics of manually-driven miles in Autopilot-enabled vehicles are very different than the miles driven in a manually-driven vehicle — more curves, poor lane markings, rain or poor-visibility weather, etc. — it could be possible that crashes are more likely to occur when an Autopilot-enabled vehicle turned over operation to the driver, because road conditions were worse. If that hypothesis were true, these types of crashes should be included as an Autopilot crash, as it pertains to the road coverage of Autopilot and the hand-off between autonomous and manual control, which is related to Tesla’s design choices.
So, unless the owner of an Autopilot-enabled vehicle never or rarely chose to enable the functionality, the proper comparison for fatality rate safety statistics should be made between Autopilot vehicles and all other vehicles.
2. What are comparable vehicles and fatalities?
Another criticism of Tesla’s Autopilot safety statistics is aimed at its choice of comparable baseline vehicles in the 1 fatality per 86 million miles statistic. Analysts believe this statistic was obtained from the Insurance Institute for Highway Safety (IIHS)’s general statistics on fatal crashes, which includes all fatalities (driver, passenger, pedestrian) from accidents by all vehicle types (automobiles, pickups and SUVs, trucks and buses, motorcycles, etc), to arrive at a 2016 total rate of 1.16 fatalities per 100 million miles, or 1 fatality per 86 million miles. Criticisms that this is an ‘apples to aardvarks’ comparison are fair.
There are two main factors that contribute to vehicle fatalities: the number of crashes per vehicle, and the number of passengers or pedestrians involved in each crash[2]. While the number of crashes per vehicle is related to Autopilot design, the number of passengers in the vehicle is essentially random. The ideal calculation of the fatal crash rate statistic should be the ratio of any crash with one or more driver, passenger, or pedestrian fatality to the number of miles traveled per vehicle. In addition, vehicles of similar classes should be compared, i.e. large luxury sedans vs. the Model S, large luxury SUVs vs. the Model X, and mid-size luxury sedans vs. the Model 3, as the demographics and driving patterns of drivers of these sub-classes of vehicles should be similar.
Unfortunately, these ideal data are not reported by the IIHS, which either reports all fatalities, not broken out by vehicle sub-class, or driver-only fatality rates by vehicle models and sub-classes. If one uses the overall rate of 29–32 driver deaths per million vehicle-years for all vehicles, and 11,000 miles driven per year per vehicle, that corresponds to a driver fatality rate of one per 340 million to 380 million miles[3]. However, the fatality rate for any crash that included pedestrian, passenger, or driver deaths could be higher. According to the 2016 IIHS crash death by type statistics just under 60% of vehicle fatalities involve a driver, when including all vehicle types but excluding fatalities involving truck drivers/passengers, bicyclists, and motorcyclists. If so, the fatal crash rate could be one per 210 million to 230 miles if we included vehicle crashes with other types of fatalities. But we don’t know for sure because the IIHS doesn’t track fatal crash rates by vehicle, only driver death rates. By this last estimate, Autopilot has about the same to 35% lower fatal crash rates than any conventional vehicle at this time.
Why does my estimate differ so much from Tesla’s reported baseline for conventional vehicles of 1 in 86 million miles? Partly because Tesla includes all deaths for all vehicle types for conventional vehicles which is not a fair comparison with Autopilot-equipped vehicles, and partly because that simple arithmetic calculation should not be used because it doesn’t tell the whole truth and misses the bigger business and public policy problem.
3. Lies, Damned Lies, and Statistics
If life is a simulation, a simple arithmetic calculation of the miles per fatality statistic is the outcome of a single run, a single roll of the dice. If a butterfly flapped its wings in China and an autonomous vehicle accident occurred earlier or later in time, a snapshot of the miles per fatality statistic calculated immediately after an accident would cloud our crystal ball. That one fatality per 320 million miles statistic for Autopilot should really be calculated using the same methodology (called Poisson regression) that IIHS used to determine the driver fatality rate of 29–32 per million vehicle-years for conventional vehicles. A Poisson regression is used to model independent random events in time, like fatal accidents, as a function of exposure, such as miles traveled per vehicle to obtain a probabilistic estimate of fatality rate. However, IIHS was only able to obtain a narrow range of driver fatality rate estimates (29–32 per million car-years) for the entire population of vehicles in the U.S. The driver fatality rates for some less-common vehicles makes and models (including some of the large luxury vehicles that could be compared with Tesla Autopilot-enabled vehicles), while shown on the IIHS website, have such large (confidence interval) ranges that the average driver fatality rate figures are useless for decision-making.
Policymakers, automakers, and the general public need to get comfortable with the uncomfortable fact that over 8 billion of miles will have to be driven with partially-autonomous vehicles before we have statistical confidence that autonomous vehicles are 20% better than humans, according to RAND[4]. In the short term, we cannot be certain that partially autonomous vehicles are safer than drivers, but automakers and customers who think the technology can improve over time will take a risk to invest in it or use it.
There are consequences in delaying the roll-out of autonomous vehicle technology. RAND has developed a useful decision analysis tool to allow anyone to compare the timing of a partial-autonomy roll-out, the safety of the technology (from half-as-safe to almost perfect), and the lives saved over the course of several decades, assuming that by 2060, almost perfect fully-autonomous vehicles are rolled out to the majority of the vehicle fleet in the U.S. This model shows that rolling out just as safe or a little safer partially-autonomous vehicles by 2020 will save 160,000 more lives over 50 years than a scenario that waits until 2025 to roll out almost perfect autonomous vehicles. Delaying the roll-out of partially-autonomous vehicles costs lives. This conclusion assumes that (1) automakers make steady progress in improving the safety and reliability of their partially autonomous vehicles and (2) drivers are comfortable enough with monitoring the partially-autonomous vehicles so that new sources of error associated with the transition to and from manual and autonomous control do not increase fatality rates. Automakers and regulators should do everything to verify and ensure that these assumptions are true. RAND finds that the most lives can be saved if partially autonomous vehicles are rolled out after they are 10% better than human drivers, and from the publicly available data, Autopilot appears to be performing at least at that level, if not better.
Recommendations for Regulators and Automakers
NHTSA has been a strong supporter of the transition to autonomous vehicles throughout the years because of its potential benefits for safety and productivity for all and increased mobility for the disabled, blind, and elderly. It would be highly beneficial to development of autonomous vehicle technology if NHTSA coordinates the collection and analysis of safety statistics using the Poisson regression methodology used by IIHS for conventional and partially autonomous vehicles. These statistics should be developed for the specific vehicle classes that are likely to see partial autonomy features in the coming years, i.e. mid-size and large sedans and SUVs. These data are vitally important so that the automotive industry and the general public know where partial autonomy technologies stand relative to conventional vehicles, so that individual drivers can make informed purchasing decisions.
Similarly, automakers should collect and report on safety statistics on partially-autonomous vehicles, including counts of fatal and non-fatal crashes, airbag deployments, fender benders, near-misses, manual overrides of vehicles, as well as miles driven with autonomy-enabled vehicles and non-autonomy-enabled vehicles. Over time, as vehicle fleets gain enough mileage, safety statistics could be computed using the Poisson regression methodology.
According to RAND’s study, ‘developers of this [partial autonomy] technology and third-party testers cannot drive their way to safety’ with small fleets of ~100 vehicles driving 24 hours a day. Large-scale trials of 100s of thousands of vehicles would have to operate for multiple years before to obtain the mileage necessary to have confidence in better than human performance of a partial autonomy system. Researchers have proposed simulation and testing approaches of hardware and software to obtain autonomous driving safety data without on-the-road mileage. For example, Mobileye has been developing open-source rules for how a partially-autonomous vehicle should handle the 37 main pre-crash cases in NHTSA’s accident database. Functional testing to ensure that partially autonomous systems follow these rules would help to clarify system capabilities for drivers and limit manufacturers’ liability (though some legal experts question this approach). Regulators will have to work with manufacturers to develop and review these alternative safety tests and adapt regulations accordingly on the road to a self-driving future.
Notes: The author is a data scientist, former Tesla employee with no involvement with Autopilot development, and holds TSLA stock. This article was written with no input from Tesla. All views are my own.
[1] Some companies (e.g. Waymo/Google) argue that partial autonomy systems will never be safer than human driving and promote the development of fully autonomous vehicles, complete with no steering wheel, only. That debate is outside the scope of this article.
[2] Treatment of crashes involving pedestrians is not just a statistics problem, it is a legal problem outside the scope of this article. With conventional vehicles, the driver is fully responsible for pedestrian crashes, but with partial- and fully-autonomy vehicles, the liability and insurance costs is a shared responsibility between drivers and automakers depending on the level of autonomy claimed or achieved. This is an area that requires a lot of further study and the development of custom insurance products.
[3] Note that the correct number to use of this type of analysis is vehicle-miles traveled per vehicle, as opposed to vehicle-miles traveled per licensed driver as in Figure 4–4 in this FHWA reference (because people could drive multiple cars), or vehicle-miles traveled per capita (because not all people drive), to quantify the exposure to the risk of a fatal car accident per vehicle.
[4] Less on-the-road mileage is needed if safety statistics on other types of road accidents, like non-fatal crashes, fender benders, and near-misses can be obtained. However, data on minor accidents and near-misses are spotty.
