Tesla’s Autopilot Safety Statistics Are Extremely Misleading

Elon Musk claims that Autopilot is nearly four times safer than human-driven cars. This is based on flawed statistics.

Elon Musk loves to boast about how safe Tesla’s Autopilot is. There are two statistics he often cites. The first is a supposed finding by the National Highway Traffic Safety Administration (NHTSA) that Autopilot reduces crash rates by 40%. The second is that Autopilot is 3.7 times less likely to be involved in a fatal accident than the average car.

The 40% crash rate reduction statistic was already debunked earlier this year when the NHTSA said it was derived from Tesla airbag deployment data, not Autopilot safety assessments. In this article, I will show why the 3.7 times safer statistic is also flawed.

Musk calculated this number by simply dividing fatalities for all vehicles (one in 85.5 million miles traveled) by fatalities for Autopilot-equipped Teslas (one in 320 million miles traveled). There are two major problems with this. The first, and perhaps most obvious, is that it’s dividing apples by oranges. In other words, there are big differences between Autopilot and human drivers. For example:

• Autopilot can only be used in good weather on roads with low-to-moderate traffic and clear lane marking, it even says so in the owner’s manual; however, human-driven cars must operate in all sorts of weather, traffic, and road conditions — hence why they cause more fatal crashes.
• Total traffic fatalities include motorcycles, golf carts, trucks, buses, and tractors. In fact, only about 40% of vehicles involved in fatal crashes are passenger cars. But even this adjusted comparison skews the numbers in Autopilot’s favor as most of these aren’t luxury cars with advanced safety features.
• Expensive autonomous Teslas are typically driven by wealthy middle-aged people — the demographic with the safest driving record. Not many are piloted by teenagers, drunks, or 100-year grannies — groups with some of the highest crash rates.

In addition to comparing apples to oranges, the other major flaw in Musk’s Autopilot is statistically safer claim is the small sample size. Since fatal crashes are so rare, the amount of data required to derive statistically robust risk estimates is enormous.

According to one study, in order to be 80% confident that autonomous cars are 1.9 times safer than human-driven cars, they’ll need to be driven 11 billion miles in a variety of real-world conditions. Another study suggests that even hundreds of billions of miles might be required to fully demonstrate their safety.

Tesla doesn’t even come close to this. Virtually all of the miles engaged Autopilot has logged — so far around 2–4 billion, according to various sources — have been driven in unrealistically ideal highway-type conditions, where all the car has to do is stay in its own lane and not get too close to the vehicle ahead. Autopilot is pretty good at those kinds of simple tasks — but then again, so are humans.

Where things get tricky is driving in congested parking lots, on unmarked rural back-roads, through obstacle-filled construction zones, confusing intersections, and in other areas where visibility is poor and the rules and boundaries not clearly defined. It’ll take many years before we have autonomous cars that can safely operate in these more complex environments. For now, contrary to the promotional statements made by Musk, Autopilot is just an unproven and unreliable prototype.

This presents a very serious problem for Tesla as safety has been the key selling point for this technology. Given the big spike in Autopilot-caused crashes (at least four over the last month alone) and mounting Autopilot-related lawsuits and complaints (like the recent $5+ million class action settlement), this selling point has proven to be — as consumer advocacy groups recently put it — “deceptive and misleading.”

It’s certainly possible, and even likely, that this crisis will hurt demand for Autopilot-equipped Teslas going forward. Considering that these are generally the highest margin cars Tesla sells, it’s a very real question whether the company can achieve Musk’s ambitious goal of becoming profitable in the second half of this year. This huge risk has yet to be priced into Tesla’s $50 billion market cap.