FSD now, with Tesla Vision and Teleops

Introduction

This article advocates for and predicts the use of teleops (remote drivers) as a fallback for Tesla’s current vision based FSD system, in order to deliver full autonomy much sooner. It makes the case that this isn’t just technically and financially viable but is a good direction for Tesla with significant advantages.

By the way, to try to make this a quicker read, we’ve marked sections that can be skipped (like sidebars, but better for small screens) by starting them with “by the way”, and we’ve bolded key parts.

FSD is “Full Self Driving” and refers both to a feature that Tesla already sells for its vehicle and to their vision of fully autonomous vehicles. The gulf between these is unfortunately still large and is quite controversial, but the gulf could be bridged and that vision could be enabled now using teleops (remote control) in combination with their existing Tesla Vision based system.

The Potential for Progress is Immense

Autonomous vehicles hold incredible promise, to save millions of lives, to free up countless driver hours, to transform transportation and cities, and to reduce service costs (eg. for deliveries). Given the theme of this publication (Press Progress) it will be no surprise to hear that we feel strongly that delaying this revolution unduly would be a tragedy, and we appreciate that Tesla and Musk have pushed forward with their vision rather than giving in to the nonsensical criteria thrown up to slow it down, such as the idea that a single death is too much in spite of the huge number of lives that could be saved — this sounds harsh, but it must be accepted that autonomous vehicles will cause deaths in order to unlock the vastly greater number of lives saved.

Musk referenced some of the potential in the Q1'21 Earnings Call…

“And so if you have a system which has very good eyes, you can see in all directions at once, you can see three focal points ahead or forward, but it never gets tired. It’s never sort of texting. It has redundancy and its reaction time is super human. Then it seems pretty obvious that such a system would achieve an extremely high level of safety, far in excess of the average person.”

By the way, we have a personal perspective on this safety discussion. 1.4 million people die in car accidents every year, but even that isn’t the whole story. Ten years ago Nella, who is otherwise healthy and fit, was involved in a car accident. She and the other driver both walked away from the accident and do not show up in any statistics of fatalities or injuries, and yet she is plagued by back pain to this day. So, as consequential as that 1.4 million death figure is, we think it is just the tip of the iceberg of the harm caused by car accidents.

The Problem

Tesla FSD is not near to achieving level 4 (L4) autonomy — the minimum level that would be necessary to realize these dreams.

By the way, these “levels” of autonomy are defined, roughly, as follows. (For more, search on SAE J3016; the official document is here.)

• L2 — Tesla autopilot (an advanced driver assistance systems)
• L3 — Tesla “feature complete” FSD of 2021 (conditional automation, requiring human supervision)
• L4 — Full autonomy (no driver required) in certain locations and weather conditions. Waymo’s service, which has no safety driver, but only operates in “easy” locations and can be impacted by adverse weather conditions, is an example of this.
• L5 — Full autonomy under any conditions — no one is close to this or is even on the path to this (consider the traffic in the most chaotic city you’ve visited!).

We know that many readers who only have time to follow the area casually have very different expectations, but I’m not aware of any serious analysis with projections that match the statements of Musk or the expectations of many in the Tesla community (see the ‘analysts’ sidebar).

You may judge that your Tesla vehicle seems to be very close, but the experience of the most advanced players in this area (Waymo and Cruise) shows us that once you get extremely close to solving the problem, you still have years to go before you get to 100%. Tesla’s data is a great strategic advantage in several areas, but it isn’t the magic that some of the panderers make it out to be. (Perhaps you think we are just advocating for the adoption of a different sort of magic here — if so, may the best magic win!)

By the way, to put a positive bound on the analysts’ projections consider ARK Invest, which is an investment management firm that is one of the most bullish and high-profile Tesla supporters (and who have based much of their pro-Tesla argument on the potential of Tesla’s FSD). They recently declared that they were upping their estimated odds of Tesla achieving fully autonomous driving by 2025 to 50%. Since then ARK have taken note of Tesla’s decision to stop augmenting their vision system with radar, and predicted that this change will delay that timeline.

This leaves Tesla in a real predicament, with huge unrealized potential, unfulfilled promises and even liability concerns. As well, there is growing concern about the dangers of being in this ‘not quite finished’ state, with many customers overconfident in its abilities. We believe that many of the incidents trumpeted in the media are the result of drivers being irresponsible to a degree that could cause trouble in any type of car, and that even in its current state Tesla’s autopilot/FSD are already preventing accidents. On the other hand, the number of accidents it is preventing now is a small fraction of what it will prevent once it is mature, and there is also a legitimate concern about what happens when you give customers a product with incredible potential that is close to being realized particularly when, for many of them, it falls short of what they understand it to be; this could well become a bigger problem as it gets closer to being finished.

By the way, for those interested in TSLA (Tesla’s stock) it is important to note that the bull case centres around autonomy/FSD: selling it to customers and using it to enable the Tesla ride-hailing network. Competitors such as Waymo (a sister company of Google) and Baidu in China have already launched their own small, commercial robotaxi services, and may add their robotaxi’s to existing ride-hailing networks. If Tesla could realize their robotaxi service soon it would easily eclipse these potential competitors who have (by necessity) started small and slow. Waymo has shown how difficult and expensive it can be to ramp up such a service even with great technology. Tesla could leapfrog them easily, but if they have to wait several years then it’s a very different story. At that point, the incremental cost of their lidar based solutions will be minimal, they will have ramped up, and safety may become the dominant consideration for a public that seems to get freaked out by a single accident if it involves new technology. We would suggest that, in deciding whether to buy or sell TSLA the most important question is what are the prospects for them moving from L2/L3 to L4/L5 autonomy.

What’s holding Tesla back?

The main problem is edge cases. One can view endless videos on Youtube that make it appear that Tesla has solved autonomy because they aren’t showing you the edge cases. Even once the ‘feature complete’ version of their FSD is working well (what Tesla is working towards this year), the experience of their competitors in this space have shown that the edge cases are extremely hard to address.

The Solution — Tesla Vision plus Teleops

This problem can be resolved by complementing Tesla Vision with teleops, used only as required for edge cases. Teleops is short for Teleoperation (or remote operation), which is human operators remotely controlling the vehicle.

When an edge case arises, eg. a difficult intersection during rush hour or a strip of a road in front of some bars at night, the vehicle uses its cellular connection to relay some of its video feeds to a human operator who temporarily assists in driving the vehicle — this can range from taking complete control to just assisting in the interpretation of the video.

Note that the onboard FSD will always be on and available — it will just be in something like shadow mode (the mode currently used when the human driver is in control of the vehicle) if a remote operator is actively driving the vehicle, but doing some additional processing geared mainly towards minimizing the amount of data that needs to be sent.

By the way, teleops is already widely employed, eg. for piloting drones, and is even already used in a way very much like I’m proposing here, by Voyage, Inc.. They developed a successful (technically and operationally, at least — not sure about commercially) business around robotaxis in large retirement communities and towns, but their aspirations were always that this should be a stepping stone to a regular (full speed) service. Last year they introduced teleops to augment their vehicle’s onboard autonomous driving technology ( for more see Introducing Voyage Telessist). They have since been purchased by Cruise, and we don’t know what plans Cruise has for Voyage’s teleops, but using teleops in this manner actually makes considerably more sense for Tesla than for Voyage or Cruise since Tesla’s sensor network is vision only.

The Tesla Vision + teleops solution would be most useful for location + time based edge cases: driving segments that involve difficult situations or uncertainty, usually at particular times of day. Where possible, routing will avoid these (a good idea from a safety perspective anyway) but where that is impossible a remote driver who is experienced in the scenario at hand can be brought online seconds ahead of time. If there is a communications issue then the FSD takes control and goes into a “caution” mode, behaving as conservatively as possible and possibly allowing the sort of decelerations that autonomy engineers are trying to avoid and possibly even pulling over to the side of the road. Obviously this must be made as rare as possible and we explain below why this will in-fact be less common than you think.

So the onboard FSD is acting as a fallback to the “teleops enhanced” FSD which is a fallback to the onboard FSD — so let’s just say they complement each other very well (we don’t think teleops for cars even makes sense without some onboard autonomy).

For other types of edge cases, most importantly, sudden and unexpected events, there would be a delay before the remote operator became available during which the on board FSD would again be in a “caution” mode. The remote operators would need to be skilled at orienting themselves extremely quickly, but the skill exhibited by video gamers suggests that many people are remarkably good at this, and scale would allow a degree of scenario specialization that would make this easier, eg. an operator could specialize in handling roundabouts, flat tires or snowy conditions.

Nevertheless, teleops would be less effective at this type of edge case (compared to the location based ones) and Tesla would have to focus early efforts on training the caution mode. These events should be much less common then the others (that’s part of why they are unexpected) but this is important so we will come back to it later.

As the onboard FSD becomes mature, eliminates the sorts of edge cases discussed, and approaches the sort of complete L4 system that the other vendors are working towards now (with their more sophisticated and expensive sensor systems), there will still be the issue of the large area between L4 and L5 which includes more challenging weather conditions and locations. Here again, the addition of teleops as a complement to Tesla’s onboard FSD will give them a significant advantage.

By the way, let us consider one more significant upside of going this route… once Tesla vehicles are uploading data and getting back driving instructions to augment their FSD, all in real-time, this opens up new possibilities: the server could resolve some of the situations without resorting to human operators, eg. using what it knows from other Tesla vehicles in the area and by applying additional computational resources. This capability would be improved with time and with increasing density of Tesla vehicles, and one would expect that the areas where they offer their ride-hailing service would have lots of Tesla’s, and the places where FSD needed to resort to “outside help” would usually be dense urban areas with high vehicle density. we would expect that this is a long-term end goal for most of the autonomous driving systems anyway: that they evolve from a bunch of vehicles making their own decisions to a network of connected vehicles with a degree of collective intelligence (mediated through the Tesla motherships that is — peer-to-peer vehicle communication is a whole other topic).

Possible Issues

So then why would Tesla not pursue this solution?

This hybrid solution doesn’t have the technical “purity” of achieving the holy grail: an L4 autonomous system based purely on Tesla Vision — some might see it as a costly detour on the road to that goal. And yes, it would certainly involve extra cost and complexity relative to the purely onboard FSD system based purely on Tesla Vision, so it would not even be worth pursuing if it weren’t needed. If Tesla really only needs another year to get there then this hybrid solution would not be worth adopting, but we think we’ve been clear that we don’t believe that is the case and this pessimism is very much in line with industry consensus which is that they are many years away from even L4.

Some might see telops as Elon Musk swallowing his pride and admitting that the vision he has espoused and predicted is not possible in the near term. We don’t see it that way at all. Musk is about making things happen (which is often at odds with technical purity anyway) and it would be quite understandable that Tesla would not have revealed that they had an interim solution such as this up their sleeves. Frankly, it would put all of Musk’s predictions in a much better light.

By the way, Waymo has taken pains to make it clear that while their ‘driver’ (their autonomous driving system) may have to rely on human intervention for strategic planning, they are not using teleops to drive the vehicle so their system should be considered fully autonomous — a semantic and pedantic distinction that we don’t think is even valid. If this distinction were for the regulators then they wouldn’t be exhibiting the same need to get the word out publicly. Perhaps this is pride? Perhaps they fear that teleops would allow competitors to suddenly close the gap that they have gained with such enormous investment.

All of these points of pride and purity are beside the point, what matters is whether the solution proposed is viable and can actually solve the problem at hand, so we will now address some of the reasons people might put against that. We think that cellular networking deficiencies, and costs will be the first that come to mind for many readers, but in fact the latter considerations below are more pertinent.

Cellular Communications

We think the primary issue that people will bring up to argue against the use of teleops will be the reliability and latency of the cellular communications. I’m not going to get into a detailed, technical discussion of the requirements and capabilities at this moment because cellular technology is a quickly moving target.

In particular, 5G can provide lower latency than 4G and it can allow for reserving slices of spectrum with defined characteristics (more here), at difficult locations. AT&T is Tesla’s provider in the U.S. (We focus on the U.S. here because that is likely where they would roll out this service first), and they are rolling out some network slicing in 2021 and they have “plans to expand its 5G footprint into the world of autonomous vehicles” this year. While 5G is often over-hyped, it has key advantages for autonomous vehicles that will make teleops considerably more viable in 2022 then in 2020 when Voyage launched Teleessist.

The fact that AT&T has plans to cater to autonomous vehicles this year is noteworthy. We think it is likely that all of the autonomous vehicles will want to maintain a real-time, cellular link even if they aren’t using teleops. Waymo is the one we know the most about since it is the only one with a commercial service, and they do indeed fall into this category. And this is not just an interim bandaid — onboard autonomous systems will get better but they are so far from L5 that they will benefit from realtime augmentation for the foreseeable future (forever in my opinion). Developing this part of the teleops stack — a reliable, low latency, real-time communications link — will be a requirement for autonomous vehicles regardless of whether teleops is being used or not.

So the feasibility of using cellular is a matter of when and where, rather than if, and further evidence that ‘when’ is soon is that remote controlled robots (as drones, industrial robots, and vehicles) over cellular links are already here — even with onboard autonomy that is inferior to Tesla’s and without 5G network slicing.

By the way, supporting 5G on Tesla vehicles would require upgrading a daughterboard radio (and SIM card) that is located in the MCU (this is based on Model S, but having designed it for simple upgradability we doubt they would have gone backwards on that). Supporting mmWave 5G would be a lot harder but it sounds like AT&T is being the least aggressive of the carriers on that.

The Cost

The cost is another moving target, with the number of edge cases that require teleops and the cost of cellular data continually decreasing. Every use of teleops will be valuable training data that would improve the fallback performance of the onboard FSD and eventually eliminate the edge case. Also keep in mind that teleops will only be used for a small portion of each trip since the routing software would avoid those tricky intersections, construction zones, busy streets etc. whenever possible.

The cost of the operators might remain fairly steady initially — increasing as more vehicles start using teleops but decreasing as the number of edge cases remaining decreases.

The technology will need to be developed and though this isn’t trivial, neither is it cutting edge work like their onboard FSD — after all, in 2020 Voyage, one of the smaller companies in this area — released something very similar to what Tesla would need to do. As noted in the description of the system above, even after Tesla’s onboard FSD realizes the current dream of L4 autonomous driving without teleops, this investment in teleops will remain relevant for many years by helping bridge the gap between L4 and L5 — so this investment in adding teleops will have many years to pay for itself.

Most importantly, though, this concern about the cost must also be balanced against the revenue it enables, in selling the FSD option to vehicle owners (and avoiding possible class action lawsuits from those who have already paid), in enabling the Tesla robotaxi network, and in the publicity and prestige it would bring to Tesla as the first company to sell true autonomous vehicles to consumers and the only company that can do so at a reasonable price (since their sensor array is so much simpler and cheaper than that of their competitors).

By the way, in spite of these costs, we think that in the long run, as Tesla’s competitors find that they too have to strengthen and optimize the real-time cellular connection with their vehicles because they operate better with some shared intelligence, as they consider how the extra cost and complexity of using lidar and radar as well as vision has left them without the larger fleets and extra revenue that Tesla’s approach has made possible, and as they investigate how to move on from L4 towards L5 autonomous driving, they may wish they had accepted the costs of teleops too.

Keeping Sensors Clean

Well, this concern is rather mundane compared to the others, but it doesn’t seem to us that Tesla has made adequate provision for keeping the cameras clean in adverse conditions. However, this concern is not specific to this solution — it is a concern about Tesla Vision in general. If an inability to keep the sensors working properly is a deal breaker for teleops then it is a serious problem for FSD and for Tesla in general, and then I’m baffled as to why it would not be the current focus of attention and of incremental hardware improvements.

Here are two considerations that might ameliorate the problem in the context of their robotaxi service. We anticipate that there would be Tesla employees at their charging stations to clean their robotaxis while they are charging, and they would also clean the cameras. We also expect that Tesla will start their network in locations with consistent good weather.

By the way, as a Canadian this is difficult to face, but to some degree, full autonomy and the ride-hailing network will simply not be available to customers in bad weather in the near future. Obviously this is a major drawback, but we believe Waymo has had to accept this limitation as well — to a smaller degree in Waymo’s case but nevertheless, autonomous driving will be weather dependent for a very long time. This is one of the reasons why robotaxi networks are usually forecast to start as hybrid networks so a decision can be made between autonomous and regular drivers for each individual trip depending on routing possibilities and weather. This could give Tesla an advantage over a competitor such as Waymo if Tesla is prepared to launch a hybrid network (though one avenue Waymo is exploring to address this is to add their robotaxi’s to Lyft’s network, as well as on their own Waymo One network.)

Regulatory Issues

Regulatory approval is an important consideration and potential impediment for any autonomous driving company or network. Adding teleops to the Tesla technology mix, while it makes Tesla’s technology more complicated, it should not make the regulatory proposition more complicated. After all, the fact that Tesla has now reduced their sensor stack to just cameras doesn’t make it simpler for them to get approval relative to companies that also use lidar and radar. They all must prove that their technology is safe — regardless of what that tech stack comprises — so if adding teleops makes Tesla’s FSD better then it will enhance their potential for regulatory approval, and if it doesn’t then they won’t adopt it.

Unfortunately, all of the autonomous driving companies must also manage fickle public perceptions of these new technologies, and this will influence their chances of regulatory approval. For this, teleops is a bit of an unknown. One of the reasons we are eager to start the discussion about Tesla relying on teleops and to respond to some of the likely concerns, is to address this. Keep in mind though, that the current in-between state of Tesla’s FSD technology has also not been helpful. The quicker they can jump out of this “uncanny valley” between L2 and L4 the better.

Technology Mismatch

The fundamental proposition here is that teleops, provided when needed rather than continuously, will be able to address most of the edge cases that would otherwise significantly slow down Tesla’s progress. As well, some of the remaining edge cases could be addressed simply by being prioritized over the ones that teleops can handle, and those that remain are rare enough they can be accepted as emergency situations in which the onboard FSD enters a cautious driving mode.

If, on the other hand, the critical path edge cases are the determination of people’s intentions (which is certainly a challenge for these systems) and this must be done suddenly and in frequently occurring and unexpected locations, then teleops employed in the proposed manner will not help.

We have speculated and reasoned about the validity of that proposition, and given my opinion, but ultimately only Tesla will be able to make the determination. Obviously we would love to know Tesla’s position on this!

Conclusion

In conclusion of the discussion about potential issues, keeping the sensors clean and regulatory issues are part of the broader discussion — they are not specific to the technology proposition outlined here. And while we think it will surprise many readers, in my opinion the dependence on cellular communications and the costs are (or soon will be) acceptable, so the viability of the solution proposed here comes down to the technology match — whether teleops is the right complement to address the current limitations of Tesla’s onboard FSD.

We concede that we have not seen this approach discussed in the media, in spite of the avalanche of Tesla speculation that is created every day (have we missed it?), but neither have I seen anything that would preclude it, or any analysis rejecting it. As far as I can tell, it is simply ignored, and that is why I wrote this.

Nevertheless, we are confident enough that I’m predicting that Tesla will adopt teleops in 2022. The forthcoming ‘feature complete’ version of Tesla’s onboard FSD (now renamed to v9 and likely for Q3 of 2021) is a precondition for the hybrid system to work as we have depicted it here.

Actually, we had rather assumed that this was Tesla’s plan: that it is one of the reasons they stuck to their simpler (cheaper) sensor stack while still exhibiting such great confidence in the near future potential of their FSD to achieve autonomy. The recent tweet by Musk (since re-affirmed) about them excluding even radar from the latest incarnation of FSD reinforces this impression. And if they aren’t planning to utilize teleops (and have no other magic in their back pocket) then we consider Musk’s claims about L5 last year to be truly jumping the shark.

Going beyond this attempt at a rational analysis and prediction, since this publication (Pressing Progress) is concerned with promoting technological progress for the benefit of all, and with the potential of autonomous driving so enormous in that regard, it must be our fervent hope that Tesla will lean into its extraordinary tradition of making change happen and find a way to bring full autonomy to its vehicles more quickly than currently seems possible.

By the way, we do see plenty of discussions of them possibly launching a human driver based service. This would have fewer technical and regulatory dependencies than teleops, but it would obviously be an enormous undertaking and commitment with a much higher cost structure, and would only make sense as a step towards a hybrid network so the pressure to add the autonomous component of that hybrid would remain. As well, that does not in any way address the need to provide the FSD that Tesla owners believe they have purchased.

If you have any questions or comments please leave them below. Thanks for reading!

Disclosure

Nella is a Tesla customer and a (small time) stockholder, but we have no other connection with Tesla. Neither of us are in possession of any inside information pertinent to this post, and we have no association with teleops companies.

This article expands on a previous one that Tom wrote on the same topic. We must admit that we got a very negative reaction to the idea, but since then we discovered that Voyage had introduced a similar system in 2020, and seeing how similar it was to what we had in mind convinced us to try to make the case again.