Full Self Driving Is Possible With Existing Tesla Hardware

Here’s everything that the car can see at junctions

Lewis Gavin
Nov 24, 2020 · 5 min read
Image for post
Image for post
Tesla front fisheye and left pillar camera (Image source)

The Full Self Driving beta has been out for a few weeks now and it already looks amazing. The capabilities already showcased are impressive and much further along than I originally expected.

It is far from perfect, the best analogy I have seen so far is that it is similar to a teenager taking to the road for the first time. It can deal with simple tasks fine. However, more complex ones cause it to become more hesitant and it may require assistance.

Even with the overwhelming number of videos out there showcasing the Full Self Driving (FSD) beta’s capabilities in a wide range of scenarios, I am still frequently seeing comments from naysayers stating it just isn't possible with existing hardware, especially when it comes to junctions.

This post will hopefully show that the current hardware is capable of capturing the data that is required to deliver Full Self Driving.

“The Car Can’t See Traffic Properly At Junctions”

Most people confidently claiming that FSD won’t work do so after watching videos of the car making left turns (in America), where it must wait for oncoming traffic from the left and right. This is because the current beta has been hit and miss in these scenarios, sometimes attempting to cross traffic when it isn’t safe or being overly hesitant causing it to remain in the way of oncoming traffic for too long.

If you feel this way then hopefully the images in this post will help ease some of your concerns.

Image for post
Image for post
Tesla FSD camera images (Image source)

The above image and the one used at the top of this post were both taken from videos provided by @greentheonly (on Twitter), who has lots of videos from the different Tesla cameras on his YouTube channel.

The two images show what is captured by the left pillar camera and the front fisheye camera whilst the car is on Autopilot driving through Tokyo. The purpose of these images is to simulate what the cameras see when at a junction.

In the image at the top of this post, the car can clearly see down the street on the left identifying a pedestrian, the car and you can already see a lorry that is just starting to make its way into the image.

The same is true for the second image (just above). It can see down the street as much as you or I could when driving, certainly enough to make a decision on whether it is clear to continue.

Where things get interesting is when the view is obstructed as shown in the following image.

Image for post
Image for post
Tesla right pillar view obstructed (image source)

The right pillar camera can see some of the road to the right, but its view is obstructed by a car, pedestrians and the building. This same would be true for a human driver too.

For argument's sake, if this junction wasn't protected by traffic signals and the car would need to decide for itself if it is safe to proceed then how would it deal with it?

Well the FSD beta testers have already showcased such a scenario, where the car slowly creeps forward until it is able to see more and therefore make a more informed decision (as shown in the video below).

Now while this isn’t “proof” that the car can fully navigate each and every junction, what it does show is that the cameras are providing enough data that given enough training, the car should eventually be able to.

The reason I think this misconception exists about the car's visibility is that it’s rare to actually see footage from the B pillar cameras. Whether viewing the backup cameras in the vehicle or looking at sentry clips, you only see the front, rear and rear-facing side cameras, not the B pillar images. Because of this, it’s hard to imagine what they actually see and in some instances are probably forgotten about altogether.

So What Is Next for Full Self Driving?

New versions of the FSD beta are being delivered weekly each of which bringing great improvements according to the beta testers on Twitter.

When quizzed about when the beta will be released to a wider audience, Musk said that if next weeks release looks good (release on the week commencing 23rd November 2020) then they will widen the beta.

I don't believe this means a full rollout to all owners who have the FSD package in the states, just that it will be expanded to even more people than the select few that have it at the moment.

Elon has also confirmed that an end to end neural net that spits out steering, braking and acceleration as outputs from the camera feed is in development. Currently, the camera feed is fed through neural nets for perception but rules around acceleration are built using conventional code.

Image for post
Image for post

All of this has been confirmed possible with the current hardware. Again, it appears that the hardware isn't an issue at all when it comes to solving the FSD problem and that the team only needs to nail the software.

When thinking about this it makes sense. They have given the car a better field of view than a human with extra sensors too. This is fairly trivial to solve as we know and understand the field of view of humans when driving a vehicle. What we don’t understand is how the human brain translates that input into driving commands.

This is the big problem and the only thing holding FSD back, not the hardware.

The Startup

Medium's largest active publication, followed by +755K people. Follow to join our community.

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store