The road ahead for Autonomous Vehicles
Part 1: Open Strategic Questions Facing the Industry: Who will win the autonomous vehicle wars? Will automakers become mapmakers? Will self driving cars be good for Uber and Lyft? Will GM go the way of Samsung?
This is part 1 of a 4 part series on autonomous vehicles, a complex, fascinating and fast changing industry. Part 2 of this series is at https://medium.com/@murthynukala/a-constellation-of-innovations-is-needed-for-autonomous-driving-4b3cf6f98148#.jld382twz.
I have personally been interested in cars forever. My Masters research at MIT was conducted in collaboration with General Motors. I am an investor in Cruise Automation, Beepi and Instamotor. Control theory and robotics were my favorite topics studying engineering at IIT and MIT. More recently, I have been working on a specific angle related to autonomous vehicles which has driven my research and interest.
The automobile has had an outsized impact on society. It is one of the largest industries on earth — with about 2 billion cars on the road and more than $20T in worldwide value of autos. They drove the design of cities and commerce (suburbs and shopping malls owe their existence to the modern automobile). At the same time, 2 million people worldwide die in auto accidents, 33,000 of which are in the US.
The auto world is changing rapidly along three interlocking vectors — (a) gas vs electric (b) ownership model — owned versus shared (c) human driver versus autonomous driver. There are other changes such as direct versus dealer networks (Tesla goes direct), but I shall not cover those. The full effects of autonomous vehicles — its impact on urbanization, or on the insurance industry, are fascinating — but out of scope for this series.
CB Insights has a survey showing the efforts of 30 different companies. The list is striking in its diversity — in addition to all major automakers, auto parts manufacturers (Bosch, etc) and internet players (Google, Apple) are all investing in this market. The list is also impressively global. Industry changes have driven M&A and investments, with GM buying Cruise Automation and investing in Lyft. Countries such as Singapore and Dubai have explicitly set near term targets to achieve at least partial autonomy in their countries.
How do Autonomous Vehicles work?
Simply put, Autonomous Vehicles use a range of sensors to map their immediate environment. They create a composite picture of the environment (called a ‘scene’) and compare it to a previously created reference map (called a ‘prior map’). They then triangulate all of the readings to adjust their reading of their location. They then compute the next set of moves to get to their destination. Each car also shares the data it collects so other vehicles can learn. Each of these is described in more detail below.
Autonomous vehicles and sensors
Modern autonomous vehicles use a range of sensors and devices. For localization, GPS signals are combined with inertial measurement units (that use gyroscopes) and wheel odometers (number of times the wheels have turned) to come up with localization estimates. They also use LIDAR, radar, stereo cameras and ultrasound to map the environment around them in fine grained detail. These approaches can vary by vendor — Google uses LIDAR extensively and Tesla uses stereo vision cameras from MobileEye.
A typical AV with LIDAR, RADAR, Camera and Ultrasound. Source: Texas Instruments.
Prior Maps
One critical component is called a ‘prior map’, which serves as a reference. It is computationally a lot easier for a car to look in a certain direction expecting to see a pole with three circles that is labeled as a traffic light, than to see it for the first time and have to figure it out in real time. Comparing what the car is seeing in real time to a reference enables continuous correction so that errors don’t build up and cause drift.
Prior maps that are usable by autonomous vehicles are much more granular than the maps we are used to on the mobile phone. The mobile mapping industry co-evolved with phones, and are intended for a human user. They are rich with annotations that humans like, and don’t need to be very precise with location since humans can make local decisions. But the bottom up maps of the environment that AV’s need have to be a lot more granular.
Google described its effort to build this map in its most recent monthly update on the self-driving car program.
“Before we drive in a new city or new part of town, we build a detailed picture of what’s around us using the sensors on our self-driving car,” Google writes. “Our mapping team then turns this into useful information for our cars by categorizing interesting features on the road, such as driveways, fire hydrants, and intersections.”
Google’s self-driving cars are able to navigate city streets pretty well even without this kind of detailed map. But when people’s lives are at stake, “pretty well” isn’t good enough.
The human-annotated map provides an extra margin of safety, allowing a car to know its location within about 4 inches. And identifying permanent, immovable road features ahead of time, the map allows a car’s onboard software to quickly focus in on objects that aren’t labeled in the map. These new objects tend to be people, animals, or vehicles that are likely to move, requiring the car to be extra cautious.”
The need for high definition 3D maps is pretty well understood, and drove the $3B acquisition of HERE maps by Audi, BMW and Diamler Benz. Uber acquired Bing’s mapping assets and the team, and also hired the star team from Google Maps. This is serious business.
Prior maps can be very expensive to build and maintain at scale. Google, TomTom and HERE Maps all have many hundreds of sensor laden vehicles building reference maps. Mapmaking is also intensely capital intensive, and 3D maps are even more so. Google Streetview has had 1000+ cars driving around all over the world capturing imagery. HERE Maps has over 100 cars and so does every mapmaker.
Autonomous vehicles as data collectors
Source: Google. Bottom up scene from an autonomous vehicle.
However, autonomous vehicles introduce a new twist — sensor laden autonomous vehicles are actually the best data collection probes. They create a high fidelity map of the environment, and also have the most current view of road conditions and changes.
Apparently, the Google car cannot recognize a temporary Stop sign. So it is not sufficient to get the data once, it needs to be continuously updated and the best way to do so is to use every car as a probe. Tesla is following precisely this strategy with their highway driving cars, which have already driven 50 million miles on the road. But this creates a bootstrap problem — AV’s need prior maps so they can drive and contribute to mapmaking!
Simultaneous Localization and Mapping (SLAM)
One of the hardest problems is to locate the car with high precision (ideally a few centimeters). An obvious question is “why not just use GPS”? Well, it turns out that GPS is only accurate to about 2m-10m, does not work well in urban canyons and tunnels, and is not that accurate at global referencing (ie, things are not where it says they should be). Global referencing is a very hard problem — the GPS system assumes a shape for the surface of the earth (it is an oblate ellipsoid), which is largely but not exactly true. Further, tectonic plates move around every year, which further causes errors.
Every measurement the autonomous vehicle receives has errors. Errors compound and build if they are not corrected, causing ‘drift’. So as the car navigates the unfamiliar environment, it solves two problems simultaneously — (a) localizing itself within the environment and (b) building a map of the environment. The technical approach is called SLAM — for ‘Simultaneous Localization and Mapping’, which has a rich body of research over the past two decades.
A really cool trick that the cars use is called ‘loop closure’, where a car travels in a loop and revisits the same spot again, the error in the location measurement is distributed over the route. This can get computationally intensive when a map has multiple loops, and the errors for an entire ‘connected’ area have to be computed together. I will go into more technical detail in future posts.
Global referencing and global brains
Architecturally, the best computational model is a ‘global brain’ — where the system learns from every probe which contributes data and benefits from data that other probes collect. This makes the ‘data exhaust’ from autonomous vehicles a very valuable asset indeed! Tesla calls this ‘fleet learning’ — currently 100,000 cars collect data on about 1.5M miles driven every day.
A related architectural point is whether the top down and bottom up maps need to be reconciled. There is disagreement in the industry about whether global referencing is important. I think it is possible to create demos where a small number of vehicles navigate a curated terrain without global referencing, but it is crucial for crowdsourcing as you need to know the precise location from where the data was collected.
To recap, to have a scalable AV program, one needs (a) High definition prior maps (b) sensor laden autonomous vehicles with on-board intelligence and collision avoidance to © compute SLAM algorithms to localize the car and compute local maps (d) global brain architecture so each vehicle can learn from the fleet and (e) globally referenced ground truth incorporated into (a).
Phew, this sounds hard. And it really is hard.
With this background, the points of industry tension become clear.
Question 1: Who Owns the Driving Data?
As semi- and fully-autonomous vehicles drive around, who owns the data collected: The auto maker? The fleet owner? Or the user? Why does the user pay $100,000 for a car and have the data belong to the auto manufacturer?
As we have seen, the data collected by AV’s is a critical asset, and we should expect to see major fights over ownership. Indeed, a deal being negotiated between Apple and both Daimler and BMW seems to have fallen through over this critical point.
It’s early days, but it appears like the battle lines are drawn between the traditional automakers and the internet companies. Google, Apple and Tesla seem to believe that the data belongs to them — Tesla makes owners sign over all data rights to the company and uses the data extensively. On the other hand, the Alliance of Automobile Manufacturers, with 20+ global automakers just agreed to “Consumer Privacy Protection Principles”, where they make clear that users own the data and are free to do what they want with it.
Question 2: Do automakers become map makers?
Where do the lines get drawn between data owned by vehicle manufacturers versus mapping companies? And where do technology providers such as MobileEye and NVIDIA fit in?
From a strategic standpoint, Autonomous vehicles and HD Maps are complements. Vehicle manufacturers would love nothing more than access to ubiquitous, high quality and cheap maps. They would like a ‘commodity complement’. This tension arises because a non-commodity complement threatens to eat into the profit share of the industry. Chris Dixon has a great discussion on “Commoditize your complement”. Tesla announced that they would make the maps they make from their vehicles available to others.
The key issue is that mapmaking is a capital intensive business. By some estimates, it takes $2B per year or more to run a mapmaking operation — which includes 2000+ people in post processing operations, usually in India. This implies that HD Maps are unlikely to be a commodity.
Similarly, mapmakers would love to have access to the data from each car, which provides them ubiquitous data that they would otherwise have to spend (a lot) of money to acquire. It is rumored that the buy-in to join the HERE maps consortium is $500M! But it’s unclear what their insertion point would be. There are already deals between mapmakers and automakers, and there are likely to be more of these.
OpenStreetMaps is a viable open source mapping system, with a rich set of contributors. However, the licensing requirements of OSM require any modifications or changes to be given back to OSM, which is likely to be a hurdle.
Another alternative is that the industry could vertically integrate, with vehicle makers buying mapping companies either directly or as part of consortia. This will likely accelerate, given that there are relatively few map companies. But HD Cartography is a beast with its own challenges, and auto makers have their hands full solving the on-vehicle systems challenge.
Question 3: What consumer ownership model is likely? What do AV’s mean for fleet operators?
If a consumer is able to access a vehicle whenever they want, why own a vehicle? Do automakers also setup fleets and become service companies? Or do they sell vehicles and systems to the fleet owners? What does this mean for fleet owners — do they evolve to become like airlines and trains (valuable but not highly profitable)?
I attended an NHTSA public hearing on autonomous vehicles, where the Mayor of Beverly Hills, CA talked about autonomous minivans being offered instead of buses as a public service. I have heard of other cities and municipalities with similar plans. We’re likely to see many fleets with unique strategies and markets — municipal fleets that replace public buses, horizontal fleets like Uber and Lyft, and vertical fleets that target specific segments (like elders, students or women). The number and diversity of fleets will only increase.
So what will be the impact of AV’s on fleet owner’s economics? The central issue, which Ben Thompson of Stratechery touched in one of his articles, is that autonomous vehicles actually destroy the network effect that drives these powerful businesses.
Consider a single metro like Boston, with approximately 1600 taxis and 10,000 Uber drivers registered. Say 1000 cars operate in Boston at any given time to meet a pickup SLA of, say, 5 minutes. A finite number of vehicles satisfy demand.
Say GM and Ford decide to offer an on-demand service, and they each put 1000 cars on the street in Boston. Assuming $10K cost, that is about $10M in capital expense to capture a decent chunk of the market. The point is that Uber and Lyft derive their network effect from the human driver who decides to contribute their vehicle and more importantly, their time to the network. Take this away, and you have a business that more closely resembles an airline, train or bus company. Could still be valuable, but scarcely as profitable.
In summary, data ownership and ‘global brain’ strategies are core to enabling autonomous vehicles. High definition maps are complements to autonomous vehicles. HD mapping will increasingly be a by-product of vehicles driving around and acting as probes, not from survey vehicles.
Auto manufacturers are wise to the challenges and autos will not evolve in a similar fashion to mobile phones, where phone manufacturers using Android were forced to use PlayStore and Maps from Google, reducing them to hardware manufacturers. GM does NOT want to become Samsung.
Game on.
Stay tuned for Part 2, where I will dive into the details of the sensors, control and computational systems that are on board each Autonomous Vehicle. Part 3 will focus on mapping the real world and the challenges thereof. In part 4, I will offer my view on a potential path for how the industry could evolve.
