Driver-in-the-Loop Simulation for Guardian and Chauffeur
By: Andrew Best, Senior Research Scientist, Project Lead
Jon Anderson, Senior Research Hardware Engineer, Hardware Lead
Andrew Patrikalakis, Senior Software Engineer, Software Integration Lead
Located at our headquarters in Los Altos, California, the Driver-in-the-loop Motion Simulator (DIL Motion Simulator) at Toyota Research Institute (TRI) is a high-realism, immersive driving experience that was created to enhance the interface between the human and autonomous vehicle system. Our team at TRI is exploring novel algorithms for controlling a car, understanding the driver, and facilitating smooth interactions between the two.
While the potential benefits of autonomous vehicles (AVs) are endless, there are still developmental challenges slowing their wide scale deployment and adoption. In order to spread the potential advantages of autonomy sooner, and cater to those who prefer to drive themselves — all while improving safety — there is a need for intelligent interaction and collaboration between increasingly automated vehicles and humans. At TRI, we call this Human-Centric Intelligent Driving (HCID).
HCID has many challenges to solve — some of which are shared with traditional AVs.
- Prototyping is cumbersome: Real-world on-vehicle testing of these interaction concepts is typically cumbersome due to the difficulty in quickly prototyping and testing arbitrary scenario and interaction concepts.
- Testing can be hazardous: The nature of advanced safety systems and the real-life dangers they protect from means we are testing in risky, dynamic scenarios.
- Simulation lacks fidelity: Easily adaptable simulation environments generally lack the realism of on-vehicle driving, which can lessen the impact of results and their insight into true human response.
- The human and vehicle operate as a team: The future vehicle and its driver will need to establish mutual understanding and operate as a team. The vehicle must understand the driver’s attention and capability in an intelligent manner and communicate with the driver. Effective interactions are critical.
TRI’s motion simulator allows us to address the above challenges in a configurable test environment that bridges the gap between on-vehicle testing and traditional static simulators. Our goal is to ensure that our new technologies help — rather than hinder — the driving experience. Each new feature has the potential to make driving safer and more fun, but also if configured poorly — more frustrating and confusing. Managing the driver’s expectations, cognitive load, and attention level requires exploring how different integrations of new technology help or hinder the human/machine team. We designed our simulator for this exact purpose.
The simulator
The DIL Motion Simulator is an immersive installation including a 250-degree, 26-foot diameter screen; five-channel 4k projection system; six degree-of-freedom motion base; and custom flexible vehicle cabin, designed in partnership with Toyota Racing Development USA (TRD USA). We have created a modular, extensible, simulator software based on open-source technology to create an unparalleled level of customization in the simulation.
Our cabin design allows us to customize the available interfaces and level of realism for individual experiments. We can easily mount new displays, haptic devices, monitoring equipment, and other cabin affordances. Our team can swap the steering and pedal system or remove it altogether, and we provide onboard networking and power at multiple voltages. We have also designed the cabin to allow us to mount and remove dashboard and interior layouts. This means we can customize the look and feel to meet the needs of our specific studies with ease.
Our flexible cabin and modular software design allow us to rapidly prototype new scenarios and new in-cabin devices.
The team has designed a simulation framework to allow for extreme flexibility in the system components as well. Building on open-source technologies (including CARLA and ROS2), new software, computing power, or interface devices can be added as needed. We can input new in-cabin displays, active-safety, and augmented-reality alerts on emulated mirrors, and even redesign human vehicle control interfaces. The framework is modular and scalable. One of the other features of this setup is the ability to integrate new devices or design new experiments at a desk, test them on static simulator benches, and graduate them to the full installation without significant downtime. The vehicle characteristics and control response can be dynamically modified in software, allowing for reconfiguring the vehicle during testing. The system also captures extensive telemetry from the driver, platform, interface devices, and simulated world. This data collection pipeline was built to support new devices with minimal overhead and simplify data alignment.
We have built partnerships within Toyota to accelerate our development. Our relationship with TRD USA has been essential to our success. TRD USA has provided us with guidance, advice, and engineering services leveraging over 10+ years of experience in race car simulation. We have been able to form a deep partnership with an experienced simulation group and synergize within Toyota, which we believe is a true differentiator in our research and a key factor to our success.
Our Research
Our research team inside of TRI consists of software engineers, roboticists, computer scientists, machine learning experts, behavioral and cognitive scientists, and human-factors engineers. We designed our simulator from the ground up to intentionally account for inputs from such a varied team. By working in collaboration with a multi-disciplinary approach, we aim to address as many factors as possible that drivers of the future might encounter. We are currently pursuing projects in areas including better understanding driver behavior, predicting hazards on the road, developing novel ways to interact with the driver, and designing new control algorithms for the car.
Additionally, we are researching the efficacy of simulation platforms in general and how we can most effectively use the data we are gathering going forward. No simulator is perfect, and we recognize that there are differences between a driver’s behavior in a simulator versus on the road. By exploring these differences, we can anticipate how new technologies will transfer between the simulated vehicle to a prototype, and ultimately into Toyota passenger cars. We are approaching this in a formal, human-subjects research model targeted for publication.
Flexibility for the Future
Our simulator is in active development at TRI. As of the beginning of 2022, we have completed version 1 of the simulator installation. This includes building and calibrating our immersive projection system, installing our flexible cabin, and calibrating and testing our simulated side-view and rear-view mirrors. We have integrated a prototype instrument cluster in partnership with our Guardian team and have completed safety testing of our simulated cabin to enable human subject research. We are now working to identify experimental confounds and tune the simulated vehicle, and we are also collecting internal experiment data for the first set of research questions.
In the last year, we have learned many lessons about designing a truly flexible framework. One of the challenges was to figure out a way to reduce the time it takes to get a new device integrated into the platform and build tools to enable software and hardware testing independently, as well as in an integrated fashion. Additionally, we have learned more about driver attention and how to develop more realistic and plausible scenarios.
Iteration is a key factor in our integrated simulation framework. Each new driver has a unique set of feedback and attention. Each new test moves us one step closer to a simulator that delivers a truly believable driving experience. We’re learning more about how individual differences in driving style affect the challenge of driving in a simulator. Using recruited, regular drivers — instead of professional ones — adds new layers of complexity critical to further developing our technology, and our team is focused on addressing these problems and sharing what we learn.
Why Open Source?
One of the guiding philosophies at TRI is to advance the scientific disciplines in which we work. The goal of using open-source tools such as CARLA and ROS2 also serves as a significant force multiplier. The future of autonomous driving and advanced safety systems is being dreamed up across the spectrum of OEMS, startups, and universities. Our tools enable us to collaborate with universities, publish our research results, and share code or data as needed to move towards replicability without significant license barriers. These resources will also facilitate intaking research results from universities and other partners and integrating novel devices, algorithms, and approaches with a mind toward publishing replicable results. Finally, by using open source systems, we developed the ability to test the spectrum from fully driver-controlled to fully autonomous, integrating sensors and supporting computation as necessary.
What’s next?
We are extremely excited for the year ahead! By summertime, we anticipate being fully integrated with several of TRI’s most exciting research projects, providing them invaluable access to a safe and reliable platform to begin answering how vehicles of the future will interact with their operators. Data collection with human participants is scheduled to begin this summer as well with participants from the public providing feedback on the simulator itself and our safety technologies under development. And our plan is to publish what we learn. Ultimately, the goal is to apply our learnings and technology within Toyota vehicles, so we can continue to make cars safer, easier, and more fun to drive!
Our team has been supported by contributions from teams across TRI. We are extremely grateful for the ongoing support from our colleagues.
