“LiDAR is a fool’s errand. And anyone relying on LiDAR is doomed.”
This was said by Elon Musk (cofounder and CEO of Tesla) during Tesla’s Autonomy Day in April 22, 2019 (the statement can be viewed here).
Within the autonomous vehicle (AV) space, Tesla has a very public anti-LiDAR position. They argue that computer vision systems / algorithms can provide all that is needed — Andrej Karpathy, former director of Artificial Intelligence (AI) and Autopilot Vision at Tesla, highlights advancements in deep learning / neural networks and emphasizes that humans and animals are able to navigate the world primarily using their eyes and visual perception. This is quite a contrast from many rival AV companies (e.g., Waymo, Volvo, and General Motors) that leverage LiDAR to enable their self driving capabilities. For example, in San Francisco, California and in Tempe, Arizona, Waymo vehicles, with their large LiDAR sensors, can be spotted driving around.
However several people have seen Tesla vehicles sporting LiDAR sensors. For example, Twitter user Grayson Brulte saw a Tesla Model Y equipped with LiDAR in May 2021. Another individual spotted a Model X with a LiDAR rack (their video can be found here).
Furthermore, according to Bloomberg, Tesla has contracts with Luminar Technology to use their LiDAR sensors. In this article, we will explore what LiDAR is and how it is used for autonomous driving, Tesla’s reasons against LiDAR, and Tesla’s future with the technology.
What is LiDAR and how is it used?
LiDAR stands for Light Detection And Ranging. It works by shooting out light, which then bounces off surrounding objects and returns to the sensor. The sensor determines direction and distance based on the angle and time it took to return. Modern LiDAR sensors can send out light many times within a short time frame, allowing us to create very accurate maps of our environment. For a more visual walkthrough, check out the visualizations on Velodyne Lidar.
Some of the reasons why LiDAR is used for autonomous driving are (note: this list is non-exhaustive):
- Depth perception: LiDAR can provide accurate data on the distances of objects, which is great for detecting and avoiding obstacles. The range for LiDAR sensors varies, but it can go as high as 250 meters (e.g., Blickfeld’s Cube Range LiDAR)
- 3D Point Cloud: the data from LiDAR can be used to construct a 3D Point Cloud of the surrounding environment. This can be useful for simultaneous localization and mapping (SLAM), navigation, and motion planning.
- Real-time data: LiDAR collects a lot of data and does it very fast. For reference, according to TechCrunch, Luminar’s LiDAR sensor can send millions of light pulses per second — this can enable cars to have a larger time buffer to react / make decisions.
Tesla’s arguments against LiDAR
At Tesla, they focus on using camera and vision-based techniques. In May 2021, Tesla removed radar from their Model 3 and Model Y in their efforts for vision-based autonomy (which they call Tesla Vision). (Consequently, according to Reuters, the National Highway Traffic Safety Administration, NHTSA, changed their rating on these models to not include having some advanced safety features).
What are Tesla’s reasons for using vision and excluding LiDAR from the picture? Below are some (non-exhaustive) reasons Tesla have argued:
- Costs: LiDAR is much more expensive than cameras. In 2019, when Elon Musk said “LiDAR is a fool’s errand,” these sensors had a hefty price tag — for reference, a high end LiDAR from Velodyne was $75,000 and Waymo’s more affordable LiDAR was $7,500 (source: TechCrunch). Adding a LiDAR sensor at that time would require Tesla to increase the price, affecting the affordability of their cars. In comparison, camera sensors can be as cheap as $10 to $20 (source: Forbes). Although promises of sub-$1,000 LiDAR were present in 2019, Tesla could not wait since they were actively deploying vehicles to customers. Fast forward to modern day (to 2022), Luminar is currently producing LiDAR for $500-$1000 per production vehicle with Volvo (source: TechCrunch). It’s clear that costs will continue to decrease, however it is still uncertain if Tesla will start using LiDAR for their production vehicles.
- Not Human: The way we, humans, navigate the world is using our eyes. We are able to detect obstacles, determine depth, and map / localize all from the images our eyes perceive and brain processes. This is very contrary to LiDAR which basically shoots out light pulses. Given that roads are designed for humans and vision based navigation, Tesla argues that LiDAR wouldn’t be sufficient.
- Cameras Still Needed: Furthermore, cameras are still needed with LiDAR to navigate the roads. For example, LiDAR might tell how far the traffic light or road sign is, but doesn’t provide information on what the light color is or what the sign indicates— a visual system is needed. Tesla argues that cameras are still required, and if all the capabilities of LiDAR can be done by vision, then LiDAR becomes redundant.
- Advancements in Computer Vision: There’s lot of progress in deep learning, neural networks, and object detection. Tesla argues that their vision algorithms are able to generate 3D worlds from their camera sensors and video clips. Eventually, Tesla argues that computer vision can do what LiDAR provides.
- General Computer Vision: Moreover, Tesla believes vision-based algorithms are more versatile and can be applicable to the general vision problem. They argue their vision algorithms can be applied more broadly whereas LiDAR systems for AV wouldn’t. Currently, LiDAR is used to create 3D HD maps of the environment, which is then used to navigate. The challenge with these 3D HD maps is that it requires meticulously collecting and validating data on a location (for example, the first step for Waymo to make these maps is to drive around manually). As a result, LiDAR based vehicles are limited to the geography that data is collected in (more technically called geo-fenced). For Waymo, that means their vehicles are geo-fenced to San Francisco, California and Tempe, Arizona.
- Sleek Design: LiDAR can be bulky and may require adding a rack(s). For Tesla, including a LiDAR sensor might require changing the design and silhouette of their cars.
Tesla’s future with LiDAR
With Tesla’s strong stance against LiDAR, why are Tesla vehicles with LiDAR being spotted and developed? Although the exact details are not public, it appears that LiDAR is being used for developing and testing their vision-based algorithms. This is further supported by the fact that there are no production Tesla vehicles with LiDAR.
Reviewing Tesla’s recent patents (specifically, their patent titled Multi-channel Sensor Simulation for Autonomous Control Systems, patent US 2022/0043449 A1), it appears, for now, that LiDAR will enhance their simulation and machine learning capabilities. In essence, LiDAR will help improve their training / validation of their vision based algorithms (e.g., depth perception, 3D mapping, localization, etc.). Tesla is using LiDAR to improve their vision capabilities and learn against LiDAR sensors, and in effect, removing the need for LiDAR.
Conclusion
With the autonomous driving space, there appears to be two camps: LiDAR based autonomy and vision based autonomy. Each have their strengths and weaknesses. It’s unclear (as of writing this article) which will dominate and usher forth Level 5 autonomy, but for the foreseeable future, Tesla will continue working hard on their vision systems.
