To Make Self-Driving a Reality, Dispense With the Safety Debt

By Kartik Tiwari

For the last four years, I have been leading the work to make the dream of driverless technology a reality at Starsky. In this blog post, I have tried to condense my learnings about autonomous vehicles, the industry, and its biggest blindspot : Safety.

It has been more than a decade since the DARPA Grand Challenges kickstarted the self-driving industry. Back then, experts thought autonomy was right around the corner. Today, if anything, we’re less optimistic about the deployment timeline. Commercial deployment of full autonomy seems as far off as it’s ever been.

An obvious question that one can ask an industry expert is what are the things stopping successful deployment of these vehicles on public roads. The answers to that range from “regulators need to get their act together” to “we need to run more miles” to “sensors need to be more up to spec w.r.t price and functionality”. For me the problem is more basic than that and these are only symptoms not the root cause. Self driving vehicles can be deployed on public roads when developers can PROVE that they are safe. The holy grail is provable safety. It cannot be done by just collecting autonomous miles or by getting ok from regulators for a limited testing. This is the big challenge because as of today, it is not clear how to even define what safety is.

But that’s puzzling. Building a safe system is not a new thing. Aerospace, automotive, military — engineers have been able to build safe and complex systems for a long time. There’s a disciplined process required to build and validate such systems (detailed in Starsky’s VSSA). It involves systems’ engineering approaches to break the problem down into smaller components, performing Functional Safety design, including an FMEA and the testing of each individual failure mode along with their mitigations, followed by a Safety Of the Intended Functionality (SOTIF) process. Following this process ensures that safety can be proved to all the stakeholders. What this amounts to is a three step process:

• Design and build the system based on a assumed use-case
• Verify the performance based on that assumption
• Validate that the assumption matches with the actual use-case

Take the task of safely constructing a complex system like a bridge. One can theoretically construct an entire bridge all at once, as a single piece, so that it satisfies all the design requirements. But to prove that the bridge meets those design requirements up to an appropriate safety standard, an almost unimaginably large number of input combinations would need to be addressed and tested at the bridge level. Does it collapse under the weight of a car? A dozen cars? What about fully laden trucks? Or the mechanical resonance of marching soldiers? Or cars, trucks and soldiers? And what about in high winds? To validate the bridge’s safety, the designers would have to physically test every conceivable combination — which would lead to an intractable safety validation process.

For people involved in the AV industry, this should sound eerily familiar. Developing a system that can perform all aspects of the driving task is a complex and safety-critical problem. Machine learning (ML) is an essential tool to solve this. But because machine-learning algorithms are black boxes, and it is impossible to understand why it makes the decisions that it does, you have to validate a self-driving system that uses machine learning against an exponentially large set of Autonomous Driving Scenarios (ADSs). The problems don’t stop there. Each time a developer changes anything significant about the system (or retrains the ML models), they have to start the entire process all over again. Building self-driving capabilities with machine-learning without a robust safety baseline amounts to building the entire bridge all at once. That is why even though different teams can showcase extremely complex driving behaviors, they still have safety drivers (who are expected to react in a fraction of a second) because they can’t prove that the system is actually safe.

Of course, bridge designers do not actually build bridges in a single piece. Instead, they break the design requirements into smaller components, like steel beams and rivets. That way, it’s easy to verify the functional safety of the entire structure by testing the safety of small sets of individual components with simpler test cases. With that, no matter what the ultimate condition is (from high winds to marching soldiers to multiple cars all at once), individual components (steel beams and rivets) have a small set of testing requirements (max stress and strain levels etc). This method follows the well-understood system for verifying functional safety that we’ve used for decades.

At Starsky Robotics, we employed a steel beams-and-rivets approach to building a self-driving system. We set our preliminary goal as the safe operation of a truck on a highway with remote supervision. We have spent considerable time and effort to achieve this preliminary goal by using deterministic components for our inherent safety architecture. The idea was that over time, we plan to use machine learning for complex decision making, but it won’t sacrifice the inherent safety of our system because of the foundation on which it was built.

Each of our deterministic software modules controls a particular driving task, such as lane-keeping, changing lanes and the act of braking in response to a slower vehicle ahead. Think of these modules like the steel rivets and beams of our bridge. And like them, our deterministic components have simpler failure cases, and thus are easily testable. By avoiding building complex decision making from the get go, our approach exponentially decreases the number of potential test cases. Because we understand the failure modes of the components we’re using, we have greater confidence in the reliability and safety of the overall system. So we can get much faster to the point that our system is considered safe, and ready to be used by the public — what the startup community refers to as a “minimal viable product,” or MVP.

Most people are familiar with concepts like management debt and technical debt. Think of a college engineering team that hacks together a demo for a contest on a weekend — but still has an enormous amount of work to do on that demo before it can become a viable product. That work amounts to technical debt.

Self-driving development has given rise to a similar concept I call safety debt. It happens when developers are building self-driving systems with new capabilities without completing disciplined safety-engineering processes. At some point in the future, the developers will need to validate those capabilities through the functional safety process we discussed earlier. The teams that have built a large number of functions that are demo-able, but not safe, are actually further behind teams that have focused on deploying fewer, safer, functions.

Those who have been following our journey know that Starsky Robotics has accomplished an enormous amount in a short time with a team that’s a fraction the size of our colleagues’. In June 2019 we became the first to remove a human driver from a truck operating at highway speed on a public road. We were able to achieve that milestone because we focused more on paying down safety debt than building additional features. Which is only possible because we understand how our system works — every aspect of it.

So why’s it taking so long? Most of the industry is working to develop self-driving systems with impressive capabilities — and ever more daunting amounts of safety debt, which require an intractable amount of resources to dispense. Any new revolutionary technology should focus on shipping the minimal viable product first.

Kartik Tiwari was the chief technology officer and co-founder of Starsky Robotics.