Automotive Simulation Meets AI

Published in

aiMotive

6 min readSep 24, 2024

Welcome to the third edition of our newsletter: Automotive Simulation Meets AI. Read Tamás Matuszka’s blog about aiMotive’s research contributions to global conferences. If you like this blog and enjoy reading technical texts on AI, neural rendering, simulation, and other automated driving-related technologies, consider subscribing to this newsletter!

Leading the way: aiMotive’s research contributions to global conferences

Research is at the heart of everything we do at aiMotive. As automated driving and AI technologies are one of the fastest-changing industries, we are constantly exploring new ideas and striving to improve our products by developing innovative solutions. This commitment to scientific inquiry is evident in our contributions to conferences, where we not only share our latest findings but also engage with the global research community. By prioritizing research, we ensure that our products and services are not just industry-leading but also grounded in the latest scientific advancements. This blog post gives a short overview of our work published at scientific conferences.

Semi-Pseudo-Labeling

Training neural networks require a large and diverse set of annotated data. However, obtaining high-quality, sufficiently large datasets can be expensive and sometimes impossible due to human and sensor limitations. In our ACIIDS 2022 paper (available on arXiv and SpringerLink), titled “A Novel Neural Network Training Method for Autonomous Driving Using Semi-Pseudo-Labels and 3D Data Augmentations,” we address this challenge in the context of far-field 3D object detection, a problem particularly affected by the scarcity of quality training data. We developed a method called semi-pseudo-labeling, which allows us to leverage 2D bounding box annotations for training 3D object detectors. By combining semi-pseudo-labeling with 3D data augmentations, we significantly extended the detection range of the neural network beyond the original training data distribution. The CVPR 2024 publication of a similar approach by researchers from NVIDIA, UC Berkeley, Caltech, and CUHK at CVPR 2024 further underscores the ongoing relevance and difficulty of distant object detection.

Multimodal dataset designed for robust autonomous driving

Our paper, “aiMotive Dataset: A Multimodal Dataset for Robust Autonomous Driving with Long-Range Perception” (arXiv, OpenReview), published at ICLR 2023 SR4AD Workshop, was inspired by the challenges mentioned earlier. Several public multimodal datasets were available at the time of publication, but they primarily featured only two sensor modalities (camera and LiDAR) without radar sensor data. Additionally, these datasets lacked far-range annotations, making it challenging to train neural networks for highway assistance functions in autonomous vehicles.

To address these gaps, we released a multimodal dataset designed for robust autonomous driving with long-range perception. It includes scenes captured with synchronized and calibrated LiDAR, camera, and radar sensors, offering a full 360-degree field of view. The data was collected on highways, in urban and suburban areas, under various conditions — day and night, as well as in the rain — and is annotated with 3D bounding boxes with consistent identifiers across frames. By making our dataset and models publicly available, we aimed to advance research in multimodal sensor fusion and robust long-range perception systems. For more details on the dataset, baseline models, implementation, and experimental results, please visit our GitHub repository.

Reducing the computational load of active learning

One common challenge in autonomous driving development is selecting the most valuable data from massive collections of recordings. Active learning, a powerful machine learning approach, helps cut labeling costs by choosing the most informative samples from an unlabeled dataset, making it a promising solution. However, traditional active learning methods can be resource-intensive, limiting scalability and efficiency. In our NeurIPS 2023 RealML Workshop paper, “Compute-Efficient Active Learning” (arXiv, OpenReview), we tackle this issue with a new method designed to reduce the computational load of active learning on large datasets.

Our approach introduces a simple yet effective method-agnostic framework for strategically selecting and annotating data points, optimizing the process for efficiency without compromising model performance. Through case studies, we show how our method can significantly lower computational costs while maintaining or improving model outcomes. For further details, please check the corresponding NeurIPS page and GitHub repository.

The industry hot topic, neural rendering

Neural reconstruction is undoubtedly one of the hottest research topics in 2024, and we are actively contributing to this field. Our work, “Controllable Neural Reconstruction for Autonomous Driving,” was showcased at CVPR 2024 and will also be part of the ECCV 2024 Demonstrations. Additionally, it was presented as a poster at SIGGRAPH 2024 and featured in the ‘Video Generation’ Technical Papers session. In this project, we developed an automated pipeline for training neural reconstruction models using sensor streams from a data collection vehicle.

These models are then used to create a virtual replica of the real world, which can be replayed or manipulated in a controlled environment. To bring these scenes to life, our simulator adds dynamic agents to the recreated static environment, handling occlusion and lighting effects. This highly flexible simulator allows us to tweak various parameters like agent behavior and weather conditions to create diverse scenarios.

https://youtu.be/Ad4dFNxt1Q8

The first large-scale 3D traffic light and sign dataset

Our latest work, “Accurate Automatic 3D Annotation of Traffic Lights and Signs for Autonomous Driving” (arXiv), will be presented at the ECCV 2024 VCAD Workshop. We introduce the first large-scale 3D traffic light and sign dataset with far-field annotations, created entirely through our automated annotation method without any manual work. This novel method generates precise, temporally consistent 3D bounding box annotations for traffic lights and signs, effective up to 200 meters. These annotations are ideal for training real-time models in self-driving cars, which require vast amounts of training data.

Our scalable approach uses only RGB images with 2D bounding boxes of traffic management objects, automatically obtained using an off-the-shelf image-space detector neural network and GNSS/INS data, eliminating the need for LiDAR point cloud data. The quantitative results confirm the accuracy and feasibility of our method. We hope this dataset will benefit the community and support the development of 3D traffic lights and sign detectors. For more information, visit the corresponding GitHub repository.

In addition to publishing and showcasing our work at conferences, aiMotive researchers actively serve as reviewers for several events, including the NeurIPS Datasets & Benchmarks Track, the CVPR Workshop on Autonomous Driving, and the ECCV Workshop on Robust, Out-of-Distribution, and Multi-Modal Models for Autonomous Driving. By participating as reviewers, we believe we can further contribute to the research community and support these technological pioneering events.

For more details, visit our website or email us.