Continued Momentum Through Simulation
By: Robert Morgan, Director of Engineering and Sameer Qureshi, Director of Product Management
We’re excited to announce that our autonomous vehicles (AVs) are back on the road — and that during the shelter in place we continued to make progress by doubling down on simulation. Simulation is an important part of our testing program, enabling us to test beyond road miles.
Road Miles are not Enough
Testing AVs in the real world is necessary, but can also be limiting. Training inputs like weather and pedestrian behavior are limited to what’s happening in the world at each moment, and it can be unpredictable when you encounter a rare obstacle a second time. If reliant upon on-road miles, it may take some number of billions of miles to test everything. Simply put, the scale makes it impractical to rely only on road miles.
Therefore, we supplement our on-road testing with simulation which gives us a cost effective way to create additional control, repeatability, and safety. It also allows us to test our work without vehicles, without leaving our desks, and for the last few months, without leaving our homes.
Simulation Takes us Further
Simulation is an expansive domain: the focus of the simulation must pair with what you’re trying to develop. At Level 5, we’re developing an AV and have a purpose-built simulation catered to supporting our autonomous technology. This includes two dominant modes:
- Log replay: Replays the data from previous road testing to see how new software behaves using real data from a real vehicle. We leverage “closed loop” replay (within its limits) to test the effects of the vehicle commands that are sent by the software in simulation.
- Synthetic simulation: Creates a virtual world for the autonomous vehicle that includes the static world as described by maps and dynamic elements such as pedestrians, vehicles, and traffic lights. Synthetic simulation allows us to create an exact situation and explore a specific scenario we may not have encountered on the road.
With these tools, we can provide actionable feedback to the development team, ultimately resulting in greater confidence each time we push a release to the vehicle.
Using Rideshare Data to Power the Machine
The real amplifier is the data used to feed the simulation. Without the right data, teams might focus on solving problems in the wrong order, or the wrong problems entirely. At Level 5, we are able to use data from Lyft to develop accurate and useful synthetic simulation, enabling us to focus on solving the right problems in the right order.
- Learning from real rides: We know the routes, road complexity, and more about daily trips, focusing our feature development and capabilities on what’s most useful for people. This will allow us to get closer to developing a consumer product, sooner.
- Cameras: We placed cameras on a subset of rideshare vehicles in order to capture real-world data about the scenarios drivers and AVs face daily and how often they occur. For example, to better our planning, we can learn how often other drivers “cut in” for a last minute lane change at a particular intersection. The rideshare camera data also enables us to discover rare long tail events at scale to improve our system.
Improving Simulation with Feedback
We can combine all of these efforts to create a virtuous feedback machine that validates simulation performance or identifies opportunities for improvement.
The feedback pinpoints:
- Coverage: Our insights from Lyft data inform us of additional test space and the frequency of occurrence. This feedback helps us discover any unknown unknowns.
- Confidence: Comparing results between simulation and on-road AV performance allows us to understand the correlation and the predictive power of our simulation. This drives improvements in the fidelity of our simulation technology.
- Capability: The measurements of our AVs’ behavior in both simulation and the real world drive refinement and improvement of our AV software.
Maintaining Level 5’s Momentum
While road testing remains a critical aspect of our program, simulation allows us to leverage existing on-road data in many more ways, and multiple times over, to help improve and validate our software. With Lyft’s unique data and Level 5’s advancements in simulation, we believe we’re reducing the road miles needed by several orders of magnitude. Our focus on simulation over the last few months allowed us to maintain Level 5’s momentum toward our goal to improve access to safe and reliable transportation for millions of Lyft riders everywhere.
For more information about Lyft’s self-driving approach, follow @LyftLevel5 on Twitter and this blog. To see open roles at Level 5, visit our careers page.
