Teleoperation: Safety Principles and Indirect Methods of Control
Ensuring safety in a teleoperation platform is the most important element in Ottopia’s mission. Safety is a topic that arises with all of our partners and customers. In order to share our approach with a broader audience, the Ottopia team is publishing a series of posts on the topic of teleoperation safety.
In an earlier article, we shared a brief overview of our approach to autonomous driving and teleoperation safety. In this text, we take a deep-dive into the limitations inherent to the remote assistance of a vehicle, outline important principles for safe teleoperation, and describe one very important application of these principles: indirect methods of remote control, or what others call “tele-assist”.
The 4 limitations of teleoperation
In teleoperation, a human operator situated in a remote location controls a vehicle from a distance. The teleoperator perceives the vehicle’s surroundings through real-time high-definition video and additional sensory data that originate from the vehicle. The operator analyzes the situation and establishes the optimal solution, proceeding then to assist the vehicle.
There are specific limitations inherent to remote human assistance and control, which need to be effectively addressed by any safe teleoperation system, namely:
- Situational awareness: The sensory information available to a remote operator can approximate, but never really match, the exact set of information that is available to in-vehicle drivers.
- Network latency: In order to exchange information between a vehicle and a remote operator, a data connection is needed. However that connection is made, it will be subject to some lag between what would be observed from inside the vehicle and what is observed remotely.
- Network reliability: Similarly, any system or technology used to carry data can be subject to variable performance, whether it’s 4G, 5G or even 6G cellular networks. Any network has limitations, whether it be bandwidth, reliability or competing traffic.
- Human error: Just like the millions of in-vehicle drivers on roads today, that remote operator can experience fatigue, distraction or simply make mistakes.
Ottopia’s 4 principles for teleoperation safety
Due to the complexity of autonomous driving safety and the unique challenges teleoperation needs to address, there are four key principles that Ottopia has identified as a foundation for safe teleoperation.
Principle 1: Vehicle-side calculations
Due to the network limitations that teleoperation encounters in any realistic setting, it is important to ensure that any safety-critical calculations do not depend on network communications. This is especially important when the connection between vehicle and operator takes place over a public network with competing traffic such as LTE or 5G.
Any final instruction from the system to the vehicle should take place on a compute platform that is located in the vehicle. The reliability and precision of these calculations are thus independent to network conditions.
Principle 2: Take full advantage of system capabilities
Active safety measures meant to prevent a vehicle from collisions or other dangerous events should be suited to the vehicle’s expected operating environment. This includes an adequate sensor stack and algorithms that have been properly developed and vetted for said environment, among many other things. In the case of autonomous vehicles, vast resources are dedicated to ensuring that all the key components of the system are optimized for safety and performance in any given domain.
It follows that any assistance provided to a vehicle remotely should also take advantage, to the greatest extent possible, of such capabilities. Doing so can compensate for the differences in situational awareness compared to in-vehicle driving.
Simply put, if the vehicle has smarts (e.g., high-end sensing and computing capabilities), a teleoperation solution must use them.
Principle 3: Algorithm hierarchy
In active safety systems designed for in-vehicle driving (e.g., Advanced Driver Assistance Systems, or ADAS), the human driver usually has the upper hand over the collision avoidance system. For example, when an ADAS collision avoidance system is activated, the human driver can decide to override that response and press on the gas pedal. The vehicle will then ignore the collision avoidance commands and probably collide with the obstacle that triggered the system.
Due to the challenges discussed in the first section, a safe teleoperation system needs to assume that a remote human operator is inferior to the vehicle-located system. Unlike ADAS, safe teleoperation should be designed so that the vehicle-side system can supersede operator commands. Should a risk of collision arise during a remote control session, the system is programmed to take control and ignore the operator’s commands. The operator’s commands are, at all times, overseen by the system to ensure maximum safety of the passengers and the vehicle’s surroundings.
Principle 4: ODD-specific safety protocols
When discussing autonomous driving safety, it is common to anchor around the most headline-grabbing application of autonomy: self-driving taxis that will one day zip us from home to work and back at the touch of a button.
But, simply put, autonomy comes in many shapes and sizes. Hub-to-hub trucking has very different requirements (and almost certainly, regulatory requirements) than, say, sidewalk delivery robots. The same can be said of autonomous haulage in mines, automated guided vehicles (AGV) in container terminals, or autonomous forklifts in fulfilment centers.
These operating environments — or operational design domains (ODD), to use an industry term — are so diverse that the optimal way of performing safe teleoperation as broadly as possible is to tailor tools and principles as required.
Applying the principles: Indirect methods of control
There are two very specific applications of these principles that form part of Ottopia’s core competency. The first application is covered in this article: teleoperation via indirect methods of control.
Before moving on, it makes sense to clarify terminology. Teleoperation can be interpreted broadly, to cover multiple methods of control or assistance to a vehicle. These methods of control can be direct or indirect.
Direct methods of control require a remote operator to engage in “hands on the wheel” driving, providing direct steering, acceleration and braking commands to the vehicle. Direct control also applies to using a joystick to remotely control an autonomous forklift’s forks.
In contrast, indirect methods of control eliminate the need for such direct commands. Instead, they allow a remote operator to issue commands at a higher level of abstraction (e.g., “Follow this path”, or “Stop and wait”, or “Ignore this obstacle and proceed”), and allow the vehicle to execute. Other companies have called these methods “tele-assist”.
Teleoperation is sometimes interpreted quite narrowly to include only direct methods of control. Ottopia uses a broader definition of teleoperation, encompassing direct and indirect methods of control.
An example scenario
Imagine you hail an autonomous taxi on the way back home. Outside the vehicle, it’s a busy afternoon as most workers leave their downtown offices and begin their commute back home. Pedestrians and cyclists zip to and fro, mostly along crossways and dedicated lanes, but not always. The taxi drives up to a construction site, which is forcing some pedestrians and cyclists to weave in between cars and traffic cones. The cones, some trodden on or knocked over by previous cars, haphazardly spell out a new lane, contradicting faded lane markings.
As a safety worker holds up a sign and gestures with an upraised hand to stop, the sign catches the setting sun just enough that the glare renders the letters “STOP” effectively illegible.
The speed limit in this public road may be 30 miles per hour, but only a few yards past the construction site, a motorcycle zips by at 40 miles per hour. (In just 200 milliseconds, that motorcyclist will have traveled over 8 feet.)
Inside the vehicle, high-end hardware allows the vehicle to sense this environment, and go well beyond that. With the use of LiDARs, radars, cameras, and multiple other sources of information like high-definition maps of the environment, the vehicle is prepared to respond to almost anything that crosses its path.
This technology will probably allow the vehicle to drive unassisted 90% or even 99% of the time. But it may not know how to navigate a complex scenario as described above. Once it recognizes that fact, the vehicle begins to slow down, and triggers a request for help from a remote operator, who promptly establishes a connection to the vehicle within seconds. This “human backup” is the go-to protocol in many autonomous sectors: delivery robots, taxis, shuttles, busses, forklifts, freight trucks, yard trucks, haul trucks and others.
The solution
The inherent limitation that the autonomous vehicle itself faces has relatively little to do with awareness of its surroundings. In fact, it’s the awareness and complexity of its surroundings that prompts it to request help. Its sensing capabilities are remarkable, as is its computing power. However, it faces such a rich combination of challenges, that it decides to err on the side of caution and request the insight or permission of a human operator.
It would be a mistake for this human operator to take control of the vehicle in such a scenario, and substitute the vehicle’s precise planning and execution of paths with direct steering, acceleration and braking commands issued from a remote station. These commands would normally be safe enough for an average driver sitting inside the vehicle, but the commands from the operator would come with the added latency of the remote connection. Latency affects what the operator perceives and reacts to, and there is always a risk that the connection may deteriorate just as the operator executes a maneuver or path.
It is precisely in these kinds of complex environments that indirect methods of control are by far the safest and most effective way for the remote operator to intervene.
Indirect methods of control require that the operator not just be able to receive video and audio from the vehicle’s surroundings. It also requires communication, or perhaps translation, between what the vehicle sees and interprets — like the path that orange cones seem to be delineating for vehicles — and the remote operator.
By providing this translation, the remote operator can “inject” the human insight that a regular driver would have in this situation into the vehicle’s autonomy stack.
The remote operator provides real-time insight and instructions, which the vehicle translates into specific actions. An example of this system is path choice, in which the vehicle sends all executable paths for the teleoperator to choose from. Another method is path drawing, in which teleoperators draw out a desirable path for the vehicle to then validate and execute.
The vehicle receives a set of inputs that, together with other calculations from its stack, give it the clarity for execution that it previously lacked. With full sensors engaged, and able to react in real time to moving or undetected obstacles with virtually zero latency, the vehicle proceeds. It navigates past the traffic cones, at the appropriate time, deftly brakes if a cyclist or scooter cuts through, and resumes its trip.
It may be clear by now how such a method follows the safety principles we have laid out in the past.
- The vehicle’s execution took place in the vehicle itself. Although insights or permissions came from a trained remote operator, all calculations required for navigating the vehicle past the construction site came from vehicle-side hardware and software
- This method minimized any degradation from the system: all sensors and safety measures remained engaged as the vehicle executed all maneuvers.
- Throughout the intervention, the vehicle’s algorithms were allowed to respond to any obstacles, moving objects or new events occurring in real time, regardless of the initial input from the remote operator.
Conclusion: In some ODDs, indirect methods of control are the only way to go
By adhering to Ottopia’s safety principles, indirect methods of control are the most appropriate mechanism for providing remote assistance to an autonomous vehicle in complex environments such as public roads. Public roads come with higher levels of complexity, unpredictability, and liability. In other environments, such as a mine, direct methods of control may be safely utilized. In Ottopia, we enable all methods of remote control as each customer has different needs.
It is no surprise that for the most complex scenarios our customers and partners prefer an integration with their autonomy stack to enable indirect methods of control, combined with a highly reliable and unparalleled network, video and cyber-security layers — all of which are enabled by Ottopia’s software. Learn more here.
