Image for post
Image for post
Photo by Charlie Deets

How over half a million Teslas help train neural networks for autonomous driving

Here are the five pillars of Tesla’s large-scale fleet learning approach to autonomous driving, as I see them:

  1. Automatic curation of rare, diverse, and high-entropy training examples for fully supervised learning of computer vision tasks (i.e. when a human annotator labels images or videos). Curation techniques may include deep learning-based queries, automatic object discovery, human interventions (e.g. Autopilot disengagements), disagreements between a human-driven path and the Autopilot path planner, and disagreements between different neural networks.

Image for post
Image for post
Photo by Stefan Lehner

Automatic labelling and automatic flagging of interesting training examples obviate the need for human labour and make data valuable

Lately I’ve been watching talks from the International Conference on Machine Learning (ICML) on this topic of autonomous driving. I’m going to jot down some thoughts that these talks have stimulated.

Tesla has approximately 650,000 HW2/3 cars on the road. These vehicles are driving approximately 1,000 miles per month each or 650 million miles as a fleet. Waymo, by comparison, drives about 1 million miles per month. So, Tesla’s vehicles are driving 650x more.

Does this matter? It doesn’t if Tesla needs a human to label every frame of video for these miles to be useful. …

Image for post
Image for post
Tesla Model 3. Photo by Taun Stewart.

Copying drivers’ behaviour using neural networks

How can Tesla best leverage its fleet of approximately 600,000 cars equipped with “Full Self-Driving” hardware? More specifically, how can it leverage this fleet for imitation learning: training neural networks to emulate human behaviour. Simply collecting all the data from all the cars all the time wouldn’t be helpful. After a certain point, neural networks will master following lane lines on straight stretches of highway. Adding more examples of that to the pile only waters down the dataset and biases the neural networks toward thinking they should drive straight ahead. We need to narrow down what we collect.

There are all kinds of ways to trigger an upload. When a driver brakes hard or jerks the wheel. When a neural network detects a certain kind of object like a horse or a certain kind of scene like a construction zone. Another idea is to train neural networks to drive via imitation learning and then run them passively in the car whenever a human is driving. Anytime the neural networks’ output is an action different from what the human driver actually did, trigger an upload. Elon Musk has alluded to Tesla’s ability to passively run self-driving software in cars, calling it “shadow mode”. The stated purpose of shadow mode is to compare the software’s output to human action. So, choosing what data to collect for imitation learning seems like a perfect application for shadow mode. …

Image for post
Image for post
Sowing seeds. Photo by Dương Trí.

Capital markets are needed to invest wealth wisely, even if workers seize the means of production

Karl Marx argued in Das Kapital that the profit a company generates after it has paid its workers — which Marx called “surplus value” — is wealth that the capital class steals from the working class. Imagine, as many Marxists do, that workers owned and controlled companies, including the surplus value they generate.

Let’s suppose there are around 7.5 billion workers on Earth. Earth’s annual income is about $130 trillion (in U.S. dollars). That’s about $17,500 in annual income per person. Let’s imagine that, in the wake of a global Marxist revolution, the full $130 trillion, including surplus value, is taken under the control of the working class. …

Image for post
Image for post
Photo by Donald Giannatti

A strange and wonderful truth has settled uncomfortably into my stomach: the universe is magic and magic is science. Magic is the process of arranging unmagical atoms into patterns that – often in ways we don’t yet understand – give rise to meaning, consciousness, emotion, purpose, struggle, and mystery. Magic is not strictly in the particle, but in the INFORMATION that groups of particles instantiate. Through this process of knowing the arcane patterns of physics, chemistry, biology, and cognition and arranging atoms according to their design, we see phenomena conjured into existence that are completely scientific in their basis yet utterly supernatural in their effect. The truth is that magic literally exists – our world is teeming with it – and yet it doesn’t take magic to make magic. …

Image for post
Image for post
Photo by David Lytle.

The science of romantic fluidity

How should we interpret stories of falling in love with someone of a surprising gender? Stories like this:

…I found myself in unfamiliar territory when I – the open guy, the “figured out” guy, the unquestionably straight guy – realized that I was in love with my best friend, a man. A man I had known for seven years. A man I had never before even thought of in a romantic way. But, there I was, in love.

One possible explanation is that the person was gay or bi or pan all along, and they just didn’t know it. But many people report feeling no previous attraction to men and then falling in love with a man, or feeling no previous attraction to women and then falling in love with a woman. When a person comes out to themselves as gay, bi, or pan, often in retrospect they can find gay attraction in their past. …

Image for post
Image for post
Photo by Matt MacGillivray.

“I’m not worthy,” she said, kneeling before God at the exit of the universe.

God laughed with warm delight — a laugh that rung out like heavy brass chimes.

God reached a tendril of light across the room and touched her. Suddenly, the wall in her mind was gone. She remembered the beginning of time, when she broke herself into a zillion pieces — gemmules of light scattered across the cosmos.

Her mind filled with the image of her swaying tendrils constructing the walls that the hid the gemmules from themselves and from each other. …

Image for post
Image for post
Photo by Steve Jurvetson.

Responding to popular arguments about lidar, disengagements, AI talent, and so on

Neural networks are needed to solve the problems that a self-driving car needs to solve: computer vision, predicting the behaviour of road users, and the planning and execution of driving tasks. More training data makes neural networks perform better. Among self-driving car companies, only Tesla has the capability to train neural networks at the scale of billions of miles. No other company comes close. So, it stands to reason that Tesla will make more progress on self-driving cars than any other company.

When I make this argument, in response I hear a lot of counter-arguments about Waymo. …

Image for post
Image for post
The CARLA Simulator. Image courtesy of the CARLA Team.

Large-scale real world data is the only way

Why use a simulation to train neural networks, rather than collect data from robots in the real world? Three reasons:

  1. To avoid real world danger.

All companies working on self-driving cars use simulation. Simulation is a useful tool for testing software, and in some cases for training neural networks. However, simulation can’t substitute for large-scale data collection in the real world.

The reason is this: a simulation doesn’t contain the same empirical knowledge that the real world does, and some of that empirical knowledge is necessary for driving. In particular, a simulation lacks empirical knowledge about the behaviour of other road users, namely vehicles, pedestrians, and cyclists. …

Image for post
Image for post
Photo by Chuttersnap.

What Tesla can do that Waymo can’t

Training data is one of the fundamental factors that determine how well deep neural networks perform. (The other two are the network architecture and optimization algorithm.) As a general principle, more training data leads to better performance. This is why I believe Tesla, not Waymo, has the most promising autonomous vehicles program in the world.


Yarrow Eady

spiritual machine

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store