Why testing self-driving cars in SF is challenging but necessary

A typical day in San Francisco, captured by one of our self-driving cars.

Anyone who has visited San Francisco knows driving here is kind of ridiculous. Our vehicles encounter challenging (and often absurd) situations up to 46 times more often than other places self-driving cars are tested. Perhaps for this reason, nobody else is regularly testing self-driving cars in SF. And while we’re generally drawn to tough problems, we test in SF only because we have to. We believe it’s the fastest path toward deploying self-driving cars at scale.

Our path to scale starts with fleet deployments in the most dense urban environments, where high utilization rates generate sustainable unit economics even at low initial vehicle volumes. As we ramp up volume, we’ll drive down costs and gradually roll out our technology to suburban and rural areas where ride sharing is less common today. It takes scale in order to reach these areas and achieve a significant impact on society, so for us anything less than that is failure (and probably an unsustainable business).

Let’s take a look at some metrics from our test fleets in San Francisco and suburban Phoenix. Each cell shows the number of times our vehicles encountered a particular situation or had to execute a maneuver per 1,000 miles of autonomous driving in each location, normalized to time of day. Driving in an urban environment is almost nothing like driving in the suburbs.

Instances of various maneuvers per 1,000 miles of autonomous driving. Source: Cruise internal metrics.

Testing in the hardest places first means we’ll get to scale faster than starting with the easier ones. This may seem counter-intuitive, but by testing in densely populated areas we expose our software to unusual situations at a much higher rate, which means we can improve our software at a much higher rate. Based on our experience, every minute of testing in San Francisco is about as valuable as an hour of testing in the suburbs.

The counter argument would be that testing and ultimately deploying in suburban areas would accelerate the timeline to deploying in more challenging places. If dense areas were perhaps 20% more challenging than suburban areas, we might subscribe to this view. But in some cases we’re talking about a 4,658% jump! We believe software designed and tested in the suburbs requires a complete tear-up in order to succeed in dense urban areas.

In addition to being our home, SF conveniently has by far the highest population density of any place self-driving cars are being tested today:

Population density of common self-driving test areas. Source: US Census Bureau 2010.

Dense urban areas contain more people, cars, and cyclists that our vehicles must pay attention to at any given time (and sometimes even raccoons). Our vehicles predict the future motion of each object and simulate how those objects might interact with others. For example, we’ll predict that the car in front of us will brake if it appears it‘s about to get cut off by a cyclist or that a car making a left turn in front of us will probably yield to a pedestrian in a crosswalk.

San Francisco’s population density is 17,246 people per square mile, which is about 5 times greater than in Phoenix. However, vehicles in our SF fleet predict an average of 32 times as many possible interactions as those in Phoenix. This task gets much more difficult as the number of objects increases, since there are exponentially more possible interactions between objects.

How our self-driving cars see the world. The predicted future path and multiple possible interactions of every person, car, and cyclist are calculated 10 times per second.

That’s just the object tracking side of things. Let’s take a look at some of the typical situations our vehicles face and the maneuvers they make when driving through San Francisco and in the suburbs.

Driving in SF (1 of 4): Traffic lights aren’t always red, yellow, or green

Self-driving cars must be able to read traffic lights, even in bad lighting conditions, but sometimes the lights don’t even work. Our vehicles have to reason about these kinds of situations and determine how to proceed. We encountered a six-way intersection in the Castro neighborhood of San Francisco with flashing red lights in all directions and successfully navigated through it.

Traffic lights out at 6-way intersection. All driving completed autonomously.

Driving in SF (2 of 4): People break all the rules

People put junk in the street. They park everywhere. People don’t obey crosswalks. Our vehicles must be assertive, nimble, and sometimes a bit creative. In this case, while driving through Chinatown, our vehicle had to avoid people moving pallets in the street and then wait for a bus to pass before moving around an open box truck. In SF it’s very common to see maneuvers stack up on top of each other, unlike in suburban environments where vehicles usually only execute one maneuver at a time.

Passing a box truck in Chinatown. All driving completed autonomously.

Driving in SF (3 of 4): Frequent construction creates outdated maps

Construction zones are unusually difficult. They rarely follow any strict convention and often result in changes to lane markings or road topology. This means software that relies too heavily on a pre-existing map of the world will fail. Sometimes our vehicles must follow paths delineated by cones or flares, and sometimes they must obey hand signals from workers (as seen in the video below). Every construction zone is unique, and we interact with them 39x more often in SF than in suburbs.

Humans controlling traffic near construction. All driving completed autonomously.

Driving in SF (4 of 4): Cyclists are everywhere

Cyclists are tough to predict. They’re small, can reach speeds of 30mph or higher, sometimes break traffic laws, may or may not use bike lanes, split lanes and weave between cars, and can change direction far more quickly than vehicles. Our vehicles encounter cyclists 16x as often in SF than in suburbs, so we’ve invested considerably in the behavior of our vehicles when a cyclist is nearby. In the video below, we narrowly avoided a cyclist in all black. He made a sharp left turn across traffic without having the right of way.

Cyclist cutting off a car at night. The driver took over, but post-analysis shows our vehicle was already braking and would have stopped in time to avoid a collision.

Driving in Phoenix: Wide lanes and smooth sailing

Many companies, including Cruise, test self-driving vehicles in Phoenix and its suburbs due to the ideal weather conditions, well maintained roads, minimal pedestrian / cyclist / vehicle traffic, and favorable regulatory environment. The unprotected left turn below is about as hard as it gets in these less dense areas. While we focus primarily on our performance in San Francisco, it’s still valuable for us to be testing in Phoenix and Detroit. We test in multiple places to ensure our software adapts to varying population densities, weather, and local driving styles.

Unprotected left in traffic. All driving completed autonomously.

Notice anything different about the Phoenix clip? You can get almost anywhere in the city without deviating outside a lane boundary, navigating through construction zones, or even passing a cyclist. While it seems crazy to test in an absurdly complex place like San Francisco, it’s absolutely necessary. We believe it’s the fastest way to achieve the level of performance and reliability needed to deploy self-driving cars at scale in a sustainable way, and that’s the only thing that matters.