The Curious Case of Operational Design Domain: What it is and is not?
In a recent blog for Zenzic UK, I talked about why absolute safety is a myth and that there is a need to create Informed Safety to ensure safe deployment of Connected and Automated Vehicles (CAVs). The concept of Informed Safety is very apt in the current COVID-19 situation. The easing of lockdown rules doesn’t mean that it is absolutely safe to be going around mingling with people. However, it does mean that given we follow a set of rules (and we know why these rules exist), we may be safe. If anyone is guaranteeing your absolute safety in these times, they would be lying to you!
Similarly, the thought of proving or claiming absolute safety for CAVs is unreasonable. Safe deployment not only needs safe technology but also safe use of the technology, which can be achieved by imparting Informed Safety (which prevents misuse and disuse). Informed Safety essentially means the user is aware of what a system can and cannot do. An aspect of Informed Safety involves understanding ‘conditions’ in which the CAV is capable of operating safely. The CAV industry calls these ‘conditions’ — the Operational Design Domain (ODD). For example, the operating conditions (ODD) of a low-speed shuttle could include a city centre or a business park, and not a four-lane highway!
By defining the ODD, one is essentially in-part defining a CAV’s capabilities and limitations. I would like to point out here that defining the ‘limitations’ is more important (than capabilities). Why are limitations so important? Because limitations illustrate the boundaries of the ODD and when the CAV can no longer operate safely. Thus, the role of ODD is especially important for higher levels of automation — SAE level 3 and SAE level 4.
The Motivation
Be it the creation of the safety case for trials (or commercial deployment), or identifying test scenarios for CAVs, defining the Operational Design Domain (ODD) remains the first crucial step of the process. However, over the last six-seven months I have had innumerable discussions (both nationally and internationally) with industry, government, and academic experts about the concept of Operational Design Domain (ODD). On one hand, I feel super happy and on the other, I feel a bit frustrated! I feel happy because finally (after four years of hype) people are talking about ODD (and asking the right questions) which is fundamental to the safety of CAVs. But I feel a bit frustrated because everyone has their own understanding of what is an ODD, to fit their products or pre-conceived notions (a perfect example of their confirmation bias).
After speaking with much of the CAV community, I believe there is a need to address some of this seemingly apparent void in the industry and add clarity to the question: ‘what is an ODD?’
The Definition
Let us start with the original definition of the term ODD. The Society of Automotive Engineers (SAE) in their SAE J3016 (first published in 2014) introduced the term ODD in the context of CAVs. Since then it has been updated twice with the latest update being done jointly with ISO, which is happening as we speak. According to the 2018 version:
SAE J3016 defines ODD as “Operating conditions under which a given driving automation system or feature thereof is specifically designed to function, including, but not limited to, environmental, geographical, and time-of-day restrictions, and/or the requisite presence or absence of certain traffic or roadway characteristics”.
While some might say the definition is too wordy, it still provides what we need to know about an ODD. The key thing to keep in mind here is the words “operating conditions”.
The Confusion
While the definition of the term ODD does exist, in the last months, I have heard a variety of interesting ideas from many organisations about things that an ODD definition should include. Some of the interpretations did surprise me! From probabilities of pedestrians jumping in front of CAVs to the ability of a CAV to do an unprotected right (or left) turn manoeuvre, from ODDs for simulation to saying ODD is the same as test scenario: it is an interesting mix of ideas.
To be honest, ODD is not the first such instance when the industry and research community have been unable to agree on a definition. A classic example of such confusion is another widely used term — ‘scenario’. During my PhD, I created a glossary of terms after reading many international standards. To my surprise, I found eight (!) different definitions of the term scenario in as many standard documents. So much for standardisation…
Coming back to ODD: ODD is a characteristic of the system (a system property). The keywords in the SAE J3016 definition of ODD are ‘operating conditions’. A manufacturer would choose the specification of its CAV’s sensors depending upon its desired operating condition. For example, the sensors required for a motorway chauffeur system operating at 70 miles per hour on motorways, will be very different from the sensors required for a shuttle operating at 12 miles per hour, ferrying passengers in business parks, and which will be different from sensors required for operating in wintry conditions (like the one below).
The thought of defining probabilities of ‘things’ happening (e.g. pedestrian jumping on the road) as a part of an ODD is a bit of an oxymoron. As ODD defines the operating conditions, the choice of sensors on the CAV should not be based on jumping from an assumption that there is a 50% probability of a pedestrian on the road, to a conclusion that only 50% accurate detection is required. That would be an unsafe system!
On the subject of CAV behaviour (e.g. vehicle manoeuvres): The behaviour of the CAV is not a part of the ODD but rather the outcome of the CAV responding to the driving environment (operating conditions) it encounters, based on its performance or technology limitations. However, when we define the maximum allowable speed of a CAV (not instantaneous behaviour), in terms of the performance limitations of the automation, that would be an ODD limitation. On the other hand, when the top speed was a limitation based on tire performance, it would be a vehicle limitation and thus not an ODD constraint.
ODD ≠ Scenarios (be it functional, logical or concrete)
To keep it simple — ODD is not the same as test scenarios.
If there is one message I would like you to take away after reading this blog, it is this. There is a sizeable section of engineers who are starting to use ODD and scenarios interchangeably. There is no doubt that ODD and scenarios are related and they absolutely should be. However, being related doesn’t mean that they are the same. For some reason, this is a fact I am finding the industry is having a hard time accepting. It might possibly be because the industry is getting indoctrinated with the scenario way of thinking (and rightly so when it comes to testing), and thus have a natural tendency to badge everything we do (or talk about) with the tag of ‘scenario’ — (fans of Daniel Kahnemann (like myself) will call it — the system 1 thinking).
This is possibly the biggest source of confusion leading to the concepts of manoeuvres, probabilities, use in simulation etc. within the concept of ODD. Rather these concepts (scenarios, simulation, and probabilities) are derived from ODDs.
So what is an ODD?
ODD essentially defines the operating environment for which a system is designed for. It may also be seen from the perspective of the end-user (e.g. city council authority) as the operating environment in which a system should be able to operate safely. It is essential that there is an overlap between the two perspectives of the ODD, manufacturer (or the system designer) and the end-user, for ensuring the safe deployment of CAVs. More often than not, one of these perspectives is forgotten. However, I must admit that SAE J3016 doesn’t provide this clarity either. Latest intel (from informed sources) suggests that a new SAE document will address this.
One thing we as an industry have to accept is that an ODD definition will never be exhaustive. One can make a list of ‘n’ attributes used for an ODD definition, but there will always exist a ‘n + 1th’ attribute. Like all safety processes, our goal as engineers is to ensure we do our best! And we do (well, mostly!).
One of the first such attempts at defining a standardised way of specifying the ODD has been BSI PAS 1883 and SAE AVSC Lexicon. Inspired by the work carried out by PEGASUS project (Germany), NHTSA, University of Waterloo and WMG’s role in Streetwise project (UK) and OmniCAV project (UK), BSI PAS 1883 specifies a taxonomy for ODD attributes and has three high-level attributes:
- Scenery: non-movable elements of the operating environment
- Environment: weather and other atmospheric conditions
- Dynamic elements: all movable objects and actors in the operating environment
An obvious question would be — why would anything I say, override the pre-conceived notions. Let me address this head-on, and I must make an honest confession here.
ODD as a term was coined by the SAE J3016 document, created by a team of experts on behalf of SAE’s ORAD standards committee (back in 2014). Initially, I too shared a lot of the confusion and it was not until two years back that I got the clarity in my head. I had the good fortune of having access to some of the authors of the original SAE J3016 and discussed the subject of ODD at length with them. In subsequent years, we have worked on the SAE J3016 revisions together, with most recently being in the joint ISO/SAE working group creating ISO 22736 (the ISO version of SAE J3016).
And as I thought I had mastered ODD, in comes the term ODC — (Operational Design Conditions).
Introducing ODC (Operational Design Condition)
In my more recent (last month) discussions, I have been introduced to the concept of Operational Design Condition (ODC) which is being advocated as a superset of ODD. I feel the concept of ODC, helps clear some of the confusion surrounded around ODD. ODC is said to consist of three things:
- ODD
- Subject vehicle (ego vehicle) capabilities (instantaneous)
- Driver capabilities
ODC is a superset of conditions required for the CAV to operate safely. If we don’t have a detailed understanding of your system’s ODD and ODC, we are essentially creating an unsafe system.
Standardisation galore!
As I mentioned earlier, up until seven months back, there wasn’t much of a discussion around ODDs. All of a sudden, not only there are a lot of discussions, even the standardisation landscape has become a crowded field, with various national and international standardisation bodies wanting a piece of the pie (with their own interpretation of ODD and its usage). Be it ISO or BSI, SAE, or UL or even ASAM (standardisation bodies), everyone is interested in standardising something or the other for CAVs and ODDs.
It is absolutely fantastic to see so much interest in standardisation. However, in our recent BSI CAV Standards Advisory Board (SAB) meeting, it was evident that we need to be careful about things we standardise, and not to standardise immature technology or in an area where we (as an industry) still don’t know enough. For example, in some of the forums I have been involved in, we have spent days trying to get experts to understand what is an ODD before we actually start a standardisation project.
While BSI PAS 1883 defines a taxonomy for ODD definition, ISO 34503 adds a high-level definition format for ODD using the taxonomy, and ASAM OpenODD aims to provide a simulation level format for ODD. UL 4600 provides some considerations for ODD definition and SAE AVSC provides a lexicon for the ODD definition. The subject of ODDs and scenarios is large enough for everyone to have their niche areas. One word of caution here is that we need to avoid creating conflicting standards. Someone needs to play the role of coordinating these activities to ensure that there are no conflicts.
The Hope…
With all of the standardisation activities and confusion, what is refreshing to note is that people are starting to accept that there is a lack of understanding. ODD is key to CAV safety, and if we don’t have a common understanding of it, there will always remain a clinch in our logic for CAV safety. The key to the success of the CAV industry is knowledge sharing and collaboration. I personally have had discussions with SAE members, ISO and ASAM. And the great thing about the CAV industry is that we are willing to learn and accept, given logical arguments. With my mission to ensure the safe realisation of the CAV technology, that’s what keeps me going in this industry and in standardisation…
And while there is hope, great things can always happen…
I serve as the work item leader for the ISO standard (ISO 34503) on Taxonomy for an Operational Design Domain which is building on the work I led for BSI and CCAV on PAS 1883. A big note of thanks to Dr Steven Shladover (UC Berkeley, PATH Program), Ryan Lamm (Southwest Research Institute, TX, USA), Dr Peter Burns (Transport Canada), David Webb (CCAV, HM Government), Kenji Okamura (JSAE, Japan), Dr Edward Griffor (NIST, USA), Dr George Economides (Oxfordshire County Council), Dr Dave Jones (UK’s Met Office), Dr David Bott (WMG) and Professor Paul Jennings (WMG) for their comments on this blog, and for all the discussions we have had over the past many years.
Update: 17 Aug 2020: Added links to PAS 1883 published version
