Photo by Charlie Deets

The Five Pillars of Tesla’s Large-Scale Fleet Learning

How over half a million Teslas help train neural networks for autonomous driving

Here are the five pillars of Tesla’s large-scale fleet learning approach to autonomous driving, as I see them:

  1. Weakly supervised learning of computer vision tasks (e.g. semantic segmentation of free space) using behavioural cues from human driving to automatically label images and videos. An upload may be triggered where there is a conflict between the vision system’s output and a behavioural cue.
  2. Self-supervised learning of computer vision tasks (e.g. depth mapping) or self-supervised pre-training for tasks that are fine-tuned with fully supervised learning. Similar curation techniques to (1) may be used. (Self-supervised learning on video is probably what Tesla’s Dojo computer is intended to accelerate.)
  3. Self-supervised learning for behaviour prediction tasks (e.g. predicting cut-ins). The future automatically labels the past. An upload may be triggered when the system’s prediction is wrong.
  4. Imitation learning for planning tasks (e.g. path prediction), probably combined with an explicit, hand-coded planner, and possibly used to bootstrap some form of real world reinforcement learning. Human interventions and human-Autopilot “disagreements” are possible curation techniques.

spiritual machine | she/her