How Accurate Are HD Maps for Autonomous Driving and ADAS Simulation?
In our mission to create digital twins of real world roads, our team at atlatec has taken on a number of HD Mapping projects all over the world, delivering HD maps and 3D models for autonomous vehicle operations and AV/ADAS simulation.
Along the way, we’ve discovered a number of topics and questions that are of relevance to almost all project partners involved — and we want to take the opportunity to discuss some of these in more detail. To start, we’ve decided to answer one of the most prominent and frequent questions we get:
“How accurate are HD maps?”
Maintaining high accuracy is one of the biggest challenges in building HD maps of real-world roads — and a rather complex one. Let’s dive right in and start by looking at what accuracy means in this context:
What does “accuracy” mean for HD maps?
With regard to accuracy, there are two main focus points that determine the quality of an HD map:
- Global accuracy (positioning of a feature on the face of the Earth)
- Local accuracy (positioning of a feature in relation to road elements around it).
It is important to note that, in terms of road mapping, accuracy is an index that cannot be derived from a single variable. With regard to our mapping technology, accuracy is directly dependent on 3 potential sources of error:
- GPS-based global referencing (error in tracking global sensor position)
- Local referencing/drift (error in capturing the pose and motion of a survey vehicle)
- Local sampling (error in labeling road elements in a processed 3D model)
Let’s look at the different aspects in more detail — we’ll also provide some specific numbers, derived from projects that atlatec has completed in the USA, Europe and Japan.
Global/GPS error
Global accuracy of maps is generally bound by the accuracy of GPS: This challenge is the same for map providers all over the world. With regard to this type of error, then, the main cause is poor GPS signal quality. It is most often affected when driving a survey vehicle in areas that are covered by roof-like structures (most commonly under bridges and through tunnels), as well as surrounded by tall buildings (within street canyons).
The challenge to accurately determine the position of ones sensor pod is a very old one. It’s basically the same that seafarers used to have when navigating by the stars: In order to accurately pinpoint your goal and chart a course, you need to first determine where you are located. Similarly, in HD mapping, you can’t answer the question “Where is this sign we’re detecting?” unless you first answer the question “Where are we currently positioned?”.
As a result, your ability to accurately survey a road and its surroundings is directly dependent on your ability to first pinpoint the position and pose of your survey vehicle — along all trajectories it was driven. Any errors in determining a sensor’s position and pose will subsequently result in a global accuracy error of the map created from this sensor’s data.
To maintain a high degree of global accuracy, our sensor pods contain survey-grade differential GPS sensors: This ensures optimized signal reception and allows to supplement the real-time satellite signal by using correction data from GPS base stations, which exist almost all over the world. In combination, such correction data significantly enhances the accuracy, compared to using only GPS satellite signals.
Local error
Before a survey vehicle enters a tunnel and after it comes out at the other end, the differential GPS receiver usually provides accurate global coordinates to determine its position. However, as mentioned above, attempting to track its movement on a global scale whilst it is driving through a tunnel produces error — there is no GPS satellite signal underground.
This is where the importance of the stereo cameras comes in: Imagery that we collect from the two, calibrated cameras whilst driving through a tunnel allows us to compute and track the pose and motion of a survey vehicle by using computer vision technology. To further supplement accuracy, we add another, redundant sensor in the form of a motion sensor, or IMU (inertial measurement unit).
When it comes to the processing stage, then, we use sensor fusion to combine the data from the cameras, GPS and IMU to successfully reconstruct the trajectory of a survey vehicle and its surroundings, maintaining high accuracy throughout the entire data set. The advantages of using computer vision technology stand out in contrast to other systems that are mainly IMU-based: Their main side effect is that, in areas with no or poor GPS, the trajectory of a vehicle can go off (drift) and may only be corrected once GPS signal is recovered. In the context of autonomous driving, such errors can not be afforded.
Using imagery collected from the stereo cameras allows us to recreate a very consistent trajectory, even in GPS-denied areas. If the GPS signal is lost for a very long distance, though, drift/local referencing error will eventually occur, as is the case for all known approaches.
The benefit of using a camera-based approach — also called visual odometry — over an IMU-based system lies in the nature of how the error accumulates over time: whereas the total error of an IMU accumulates in a cubic fashion (at a factor of x³, with x being the distance travelled), atlatec’s vision-based approach only makes for linear accumulation of error.
Sampling error
This type of error is caused by incorrect calculation of distance between a point of interest (for example a stop line or a traffic light) and a sensor pod camera. Local sampling, or annotation takes place after the collected data is translated into a 3D model and is the process of labelling features within this model, thus making them identifiable to simulation tools or autonomous vehicles. In other words, annotation is the process of translating 3D imagery, which humans can easily understand and process, into a vectorized “digital twin” which can be processed by algorithms and AI.
In order to annotate road objects accurately, we use a combination of AI and manual work, which will be discussed in more detail later on.
What is atlatec’s approach to creating accurate HD maps?
Attempting to deal with all three causes of accuracy error in the practice of road mapping poses a number of challenges both in terms of software and hardware development. Our mapping technology employs a number of tools and solutions which allow us to achieve high HD map accuracy in a cost-effective manner.
Portable, camera-based mapping setup
At atlatec, we use a sensor setup that is mainly camera-based. Having two cameras and a GPS receiver at a fixed distance from each other in a small, portable box allows us to map roads worldwide with very little logistical difficulty: The metal case containing all sensors is the size of a suitcase and can be set up on any car in a matter of minutes.
Leveraging the survey-grade differential GPS and our computer vision expertise as explained above, we manage to accurately recreate all trajectories driven during data acquisition. As both our hardware and software are developed inhouse, the sensor pods’ configuration and the pipeline for processing the data from them are heavily optimized for each other.
Loop closure
Strong emphasis on achieving extremely accurate loop closure is a crucial step in creation of coherent datasets. Our survey methods include driving on every lane of a road we set out to map, extending initial driving duration but ensuring higher data quality (and eliminating occlusion issues). The main reason why this increases mapping accuracy is that, by driving on every lane of the same stretch of a road, the same road object can be detected multiple times, enabling us to determine its global position more accurately. The process of bringing the sensor data from these multiple survey trajectories together into one consistent result is what’s called loop closure.
To exemplify this, let’s say a vehicle equipped with an atlatec sensor pod drives on a lane framed by dashed lane borders. The survey vehicle will drive on that lane at least once (the example of driving past a desired point on the road twice is represented in the schematic image below as trajectory a and trajectory b). Moreover, the vehicle will also drive on its neighboring lanes (if there are any) as part of the same survey session which starts and ends at the same location. In turn, once it comes to the annotation stage, we will be able to represent, for example, a corner of any individual dash as a point in a 3D map.
In complex cases such as sharp turns where a certain point can be absent in some trajectories, then, we will still be able to determine the position of a dash accurately. The reason for it is that, thanks to loop closure, our data sets are very coherent. That makes it possible to connect the data acquired from both stereo cameras and track key points from multiple trajectories in which they are visible.
Human-assisted AI
Our third and main strength is our software. Data retrieved from the stereo cameras, the GPS receiver and the IMU is first pre-processed in order to accurately reconstruct driving trajectories and mitigate potential incoherences from driving in areas with poor GPS signal. Following this stage, we use a combination of AI and manual work to reconstruct a broad spectrum of road objects in a virtual environment. Although our software can detect and identify a wide range of road elements accurately, integration of manual work is an important step in ensuring high accuracy and consistency throughout the entire map.
How accurate are atlatec HD maps?
Based on thousands of kilometers of HD maps we’ve created all over the world and the results of various tests and audits, we conclude that accuracy errors will be lower than the following for 95% of atlatec HD map coverage:
Global/GPS accuracy
In areas with good GPS reception we achieve a global accuracy of less than 3 cm deviation using satellite signals and correction data from base stations.
In GPS-denied areas, however, inaccuracy rises with distance traveled through the area, being largest in its middle. This means that the maximum GPS error can be expressed as a percentage of the distance traveled through a GPS-denied area: We have quantified this through repeated tests which indicate that this value is less than 0.5%.
For instance, if we drive through a tunnel that is 500 meters long, our GPS-based estimation of the global position of a survey vehicle will not deviate more than max. 1,25 m from the truth in the middle of that tunnel.
As this is still a relatively high margin of error, we leverage computer vision as discussed above to mitigate the error on a local level:
Local accuracy (drift)
By using computer vision technology to reconstruct the trajectory driven on any route we can work around GPS, keeping consistency and accuracy at a high level even in tunnels and urban canyons.
As mentioned above, the error that occurs when relying on visual odometry accumulates far slower than e. g. IMU-based approaches: Within a certain horizon (h) around a survey vehicle, the drift of the reference trajectory will contribute to an error of less than 0.1%*h.
For instance, a feature located at 20 meters distance from a survey vehicle will not be displaced by more than 2 cm due to local drift of the reference trajectory.
Sampling accuracy
Inside of a corridor of 10 meters width around the mapping trajectory, features in the finished 3D model can be surveyed with less than 4 cm deviation. At a larger lateral distance, precision will drop.
Which kind of accuracy matters most for HD maps?
We have taken on a number of mapping projects all over the world so far, a typical customer use case being the creation of 3D models for (ADAS) simulation. With that in mind, it is important to note that, when it comes to virtual testing environments, the relevance of the accuracy errors mentioned above can differ.
Usually for simulation use cases, a low local and sampling error are of highest significance. Meanwhile, global accuracy and GPS positioning are often irrelevant in this context. In fact, GPS receivers weren’t even a part of our sensor setup in the beginning: This is due to the nature of virtual testing, where what matters is that the local environment is reproduced accurately — e. g. in the process of simulating lane-keep assistance on a digital twin of real-world lane geometry. As long as the positioning of the vehicle in relation to road elements is correct, it usually does not matter where on the globe these road elements are located. We will discuss map development for simulation in more detail in a separate article.
If you want to see for yourself how atlatec data can boost simulation, you can download a free sample map of Downtown, San Francisco here — provided in the OpenDRIVE format, as supported by a growing number of simulation tools.
