Driving Sensei: Unlocking Driving Mastery with AI

This is the first post in a three-part series on key elements of TRI’s Human Interactive Driving technologies.

Autonomous vehicles promise a litany of potential benefits, from increased safety to reduced congestion and even environmental benefits. However, their many challenges have made wide-scale deployment and adoption slower than originally anticipated. With fully autonomous systems out of reach for most consumers in the foreseeable future, there is another way we are exploring bringing the advantages of autonomy to drivers. Treating the vehicle as an intelligent companion and partner, TRI is developing technology where AI serves to amplify drivers and help them gain mastery over driving skills. This concept is known as Driving Sensei and hopes to unlock a person’s full driving capability while simultaneously making driving safer and more enjoyable.

To explore the Driving Sensei concept, TRI uses performance driving as a challenge problem. In this context, performance driving refers to advanced techniques typically used on race tracks or drift driving, where drivers can push their skills to the limit in a controlled environment. Driving in these situations requires the very highest level of skills, from excellent vehicle control to quick and insightful scene understanding and decision-making. These skills are helpful beyond the track in real driving scenarios as well.

According to a 2018 National Highway Traffic Safety Administration survey, drivers were found to be the critical reason for 94 percent of accidents. The majority of these accidents occurred due to drivers’ errors in recognition, decision-making, and/or performance (control). By helping drivers gain mastery at the performance driving level, Driving Sensei gives drivers the confidence and skills to handle many of the challenges in the real world.

How Is It Done Now?

Our goal is to improve driver’s skills. First, we examined how professional driving instructors train their students. Teaching takes place in many different ways, including classroom lectures, demonstrations, ride-alongs, and feedback based on data collected during laps. In all of these different methods, instructors communicate with students mainly by language with some use of gestures.

While training with human instructors is an effective approach, it is essential to recognize that it may come with some limitations. For example, it can be time-consuming, may not be easily accessible to everyone, and needs to be scalable to meet the needs of many students. By acknowledging these limitations, our research aims to make quality training available to broader users and also explore aspects of training that are unique to AI.

We are exploring verbal and non-verbal feedback training approaches based on what we learned from human instructors. Before looking into our training approaches, let us look at our research platforms first.

Our Platforms

One of the challenges in studying performance driving is the extreme conditions that can cause fear and are inherently risky. Therefore, it is crucial to have the appropriate research platforms.

We use two types of platforms to perform studies safely: driving simulators and GRIP (Global Research Innovation Platform) with low-μ emulation.

Driving Simulators

Driving simulators are human-in-the-loop simulators that allow participants to drive on a simulated 3D race track without the risk of physical harm. We use various driving simulator setups to study race track driving training. The setups range from simplified and scalable versions to one with high fidelity, with surrounding projection screens that provide an immersive experience and a motion platform that enhances the physical fidelity of the motion. Driving simulators allow us to safely and easily replicate various driving scenarios.

GRIP and Low-μ Emulation

GRIP (Global Research Innovation Platform) is an electric vehicle TRI created from scratch. It is a research platform designed to have large flexibility for testing new interaction concepts. Among the most exciting features of GRIP is its overactuated steering, which enables both front and rear tires to be steered. Rear steering can create a sideslip effect, mimicking the behavior of a vehicle on ice; we call it low-μ emulation due to the low friction of the road surface. The low-μ emulation allows participants to practice drift driving safely and at a slow speed.

The use of these two types of platforms enables us to explore the training of performance driving while ensuring the safety of our participants.

Training with Verbal Feedback

The majority of communication by human instructors is using language. Our first approach involves leveraging the principles of human instructors’ verbal interactions at different times of training.

Verbal Feedback During Driving

We collected data on human instructors’ directions during training sessions and used imitation learning techniques. We trained a machine-learning model to determine what to instruct and when to say it based on the data of the vehicle’s position, dynamic status, and the instructors’ directions.

The instructions were categorized into five categories that pertain to lateral and longitudinal vehicle control. The model was trained to predict the probability distribution over these five categories in the upcoming four seconds based on the positional and vehicle dynamics status history and map information for the past few seconds.

Conversational Feedback After Driving

In addition to verbal feedback during driving, human instructors provide interactive feedback after driving. We developed a system that records students’ driving data and performance metrics and generates natural language feedback using an LLM (large language model). The driver can ask the model for more advice on improving their driving.

The challenge is to ensure the feedback is appropriate and specific to performance driving. In addition to fine-tuning and few-shot prompting using sample data and corresponding ideal responses, the system compares the student’s driving data and performance metrics against optimal actions to provide appropriate feedback. Knowing the optimal actions to take is critical to this approach. Our expertise in autonomous drifting cars plays an important role in providing the optimal solutions.

Training with Non-Verbal Feedback

Communication modalities by a human instructor are limited. Especially for the feedback during driving, verbal communication is one of the few options available.

On the other hand, AI can utilize various modalities. We explore the methods that are unique to AI to expand the existing motor learning process.

Augmenting Sound and Haptics Cues

The key to successful training is to pay attention to appropriate environmental cues and associate them with specific actions. For drift training, the screeching sound from tires and the subtle torque changes of the steering wheel are essential cues to know how much the vehicle is sliding and when to apply counter-steer. We implemented simulated tire screeching sounds and steering haptics to exaggerate these cues so that the student can perceive them quickly and learn to pay attention to these modalities at the right time. This is made possible by the vehicle dynamics expertise of the AI system, which knows when and how to provide these augmented cues.

Difficulty Modulation

Changing the vehicle’s dynamics and environment is difficult for instructors to do.

As briefly described in the platform section, our research vehicle GRIP has the ability to simulate different friction of the road surface and also in the vehicle’s weight distribution. Both factors have a large effect on the drift capability and, therefore, can modulate the difficulty of the drift training. This feature enables the AI to adjust the difficulty level of drift training in real time, providing a more personalized and effective learning experience for the student.

What’s Next?

While we explore potential assistance by AI in improving the driving skills acquisition process, there are some challenges we need to address.

Reduce Dependency

One of the main challenges is the dependency problem, as suggested by Schmidt’s guidance hypothesis. The hypothesis states that having too much assistance can make the student depend on it, which can hinder the motor learning process. For instance, using training wheels for bicycle riding can be beneficial at first, but it’s important to gradually reduce the assistance level as the student’s skills develop.

Elicit Motivation

Another challenge is ensuring the learning process is enjoyable and motivating. According to Csíkszentmihályi’s flow theory and Yerkes and Dodson’s inverted-U theory, a good balance between task difficulty and skill level can facilitate intrinsic motivation and improved performance. By estimating the student’s skill and engagement level, AI can potentially tailor the training to their skill and learning style. This personalization ensures the training is engaging enough to keep students motivated, promoting a sense of accomplishment and satisfaction that encourages continued training.

Additionally, AI potentially encourages students, helping them stay motivated and engaged throughout the training. Positive reinforcement and constructive feedback further enhance the experience, leading to increased enjoyment and motivation.

As we age, driving abilities decline due to physical and cognitive changes, leading to decreased mobility and independence. Driving Sensei’s approach may help elderly drivers maintain their independence and confidence behind the wheel and help younger generations become motivated to be skilled and responsible drivers as well.

By leveraging AI, Driving Sensei offers dynamic and effective training to gain mastery over their driving skills and promotes a lifelong motivation to learn regardless of age or skill level.