Yandex publishes industry’s largest AV dataset, launches prediction challenge at NeurIPS

Self-driving technology revolves around a task that is too complex to be solved by heuristics alone, which is why machine learning is used to create many of its different components. Autonomous vehicle technology benefits greatly from machine learning, which is also the type of scientific approach that presents some challenges. One of them is the so-called distributional shift.

Distributional shift is one reason why it’s challenging to scale AV technology, because ML-trained models must perform equally well in various conditions — even in new conditions where the model hasn’t been trained. This could be the ability of perception algorithms trained in sunny California to detect pedestrians in Detroit after heavy snowfall; or of prediction models trained in Las Vegas to guess the intentions of drivers in Miami and so on.

Here is a crossroad in Moscow in both summer and in winter. You can see that in summer, drivers cross it faster and take risky maneuvers. While in winter, they act more cautiously on the snowy road.

For a long time datasets for this type of research were very limited. As a company testing its AV technology in six cities, three countries and in all types of weather conditions, Yandex can offer researchers and developers all over the world an impressive dataset of a massive scale and diversity.

That’s why Yandex Self-Driving Group just released the largest AV dataset in the industry to date. It includes 600,000 scenes (or more than 1,600 hours of driving) collected through testing our self-driving technology in different cities across the US, Israel and Russia in good weather, rain and snow. Releasing such a comprehensive and expansive real-life dataset publicly will help boost scientific research and speed up the scaling of AV technology.

The data features HD-maps, traffic signals and tracks of all cars and pedestrians in the vicinity of the vehicle, including parameters like location coordinates, velocity, acceleration and more, but does not contain any imagery showing personal information.

Here are two examples of busy unregulated crossroads in Ann Arbor, MI and Moscow. Apart from the differences in the rules for crossing such junctions in the USA and Russia, traffic at the crossroad in Ann Arbor is calmer, while in Moscow drivers can be quite assertive, even if doing so may lead to a violation of the traffic rules.

The release of this dataset is a part of a bigger initiative. Together with scientists from Oxford and Cambridge we are launching a global Shifts Challenge as part of NeurIPS conference on machine learning. These types of challenges can help uncover real-life industrial problems and develop solutions.

We have chosen one of the most interesting, yet challenging components of AV technology — predicting the trajectories of others on the road. Participants are tasked with developing their own trajectory prediction models, ensuring they will be able to work equally effectively in road and weather conditions that differ from those they were trained on. The challenge is split into two stages: development and evaluation:

July — early October 2021: Development Stage

• Participants train their models to predict the possible trajectories of surrounding cars and pedestrians, as well as assess the uncertainty in each prediction. The available training data includes hundreds of thousands of scenes from real self-driving trips around Moscow, collected during the daytime and in good weather conditions. Challenge participants also have access to a ‘development’ dataset, which includes scenes from other cities across Israel and Russia and features a variety of weather types. Using this data, participants can see how well their models cope with distributional shift and make predictions across a range of unfamiliar environments.

October 17–31, 2021: Evaluation Stage

• Participants are provided a new ‘evaluation’ dataset which contains examples that match the training data, as well as examples that are mismatched to both the training data and the development data from the previous stage. During this period, participants will adjust and fine-tune their models, with the final deadline for submission on October 31. At the end of the evaluation phase, the top scoring participants will present their solutions to competition organizers, who then verify that models comply with competition rules.

Organizers verify submissions during November with competition results announced November 30, 2021. Evaluation set references and metadata will also be released at this point. Those judged to have created the best models will be awarded with cash prizes.

The models will be assessed both in terms of their accuracy, and their ability to evaluate uncertainty in their predictions. The second parameter allows models to express their confidence in their decisions and is equally as important as the accuracy of the model. Whether the model can assess uncertainty hinges on the model’s ability to accurately predict the size of the distributional shift it is encountering. Models that can do this are robust and reliable, which is especially important for technology such as autonomous transportation.

Even in very similar situations, driving habits can differ from country to country. In Tel Aviv, for example, drivers are more likely to enter a roundabout in haste, while in Innopolis they are more likely to be in a rush to exit it.

Here at Yandex, we came across the issue of distributional shift when we first started developing self-driving technology. We felt the weight of its presence even during our very first year, when, the night before the first demonstration of our technology, we were treated to significant and widespread snowfall, changing the landscape beyond recognition. This was the first time our cars had seen snow, but they navigated the situation and successfully completed their tasks. We continue to observe distributional shift, testing our self-driving cars in different cities, countries, weather conditions and even on new platforms, such as our autonomous delivery robot.

We are excited to share our experience with researchers all over the world and invite them to join this unique competition. The dataset for training the models is available on the challenge website and participants may join the competition right away.

Two other competition tracks are also available for those interested in trying their hand in testing their models for machine translation and weather forecasting. The machine translation track will make use of data from Yandex.Translate, with participants tasked with creating a model capable of dealing with both literary text and everyday internet speech. Models trained for the weather forecasting track will be tested on data taken from different times of the year. This data will be provided by the Yandex.Weather service.

We wish all Shift Challenge participants the very best of luck!

Paper on arxiv.org

GitHub

Written and published by Yulia Shveyko