The Full Self Driving beta has been out for a few weeks now and it already looks amazing. The capabilities already showcased are impressive and much further along than I originally expected.
It is far from perfect, the best analogy I have seen so far is that it is similar to a teenager taking to the road for the first time. It can deal with simple tasks fine. However, more complex ones cause it to become more hesitant and it may require assistance.
Even with the overwhelming number of videos out there showcasing the Full Self Driving (FSD) beta’s capabilities in a wide range of scenarios, I am still frequently seeing comments from naysayers stating it just isn't possible with existing hardware, especially when it comes to junctions.
This post will hopefully show that the current hardware is capable of capturing the data that is required to deliver Full Self Driving.
“The Car Can’t See Traffic Properly At Junctions”
Most people confidently claiming that FSD won’t work do so after watching videos of the car making left turns (in America), where it must wait for oncoming traffic from the left and right. This is because the current beta has been hit and miss in these scenarios, sometimes attempting to cross traffic when it isn’t safe or being overly hesitant causing it to remain in the way of oncoming traffic for too long.
If you feel this way then hopefully the images in this post will help ease some of your concerns.
The above image and the one used at the top of this post were both taken from videos provided by @greentheonly (on Twitter), who has lots of videos from the different Tesla cameras on his YouTube channel.
The two images show what is captured by the left pillar camera and the front fisheye camera whilst the car is on Autopilot driving through Tokyo. The purpose of these images is to simulate what the cameras see when at a junction.
In the image at the top of this post, the car can clearly see down the street on the left identifying a pedestrian, the car and you can already see a lorry that is just starting to make its way into the image.
The same is true for the second image (just above). It can see down the street as much as you or I could when driving, certainly enough to make a decision on whether it is clear to continue.
Where things get interesting is when the view is obstructed as shown in the following image.
The right pillar camera can see some of the road to the right, but its view is obstructed by a car, pedestrians and the building. This same would be true for a human driver too.
For argument's sake, if this junction wasn't protected by traffic signals and the car would need to decide for itself if it is safe to proceed then how would it deal with it?
Well the FSD beta testers have already showcased such a scenario, where the car slowly creeps forward until it is able to see more and therefore make a more informed decision (as shown in the video below).
Now while this isn’t “proof” that the car can fully navigate each and every junction, what it does show is that the cameras are providing enough data that given enough training, the car should eventually be able to.
The reason I think this misconception exists about the car's visibility is that it’s rare to actually see footage from the B pillar cameras. Whether viewing the backup cameras in the vehicle or looking at sentry clips, you only see the front, rear and rear-facing side cameras, not the B pillar images. Because of this, it’s hard to imagine what they actually see and in some instances are probably forgotten about altogether.
So What Is Next for Full Self Driving?
New versions of the FSD beta are being delivered weekly each of which bringing great improvements according to the beta testers on Twitter.
When quizzed about when the beta will be released to a wider audience, Musk said that if next weeks release looks good (release on the week commencing 23rd November 2020) then they will widen the beta.
I don't believe this means a full rollout to all owners who have the FSD package in the states, just that it will be expanded to even more people than the select few that have it at the moment.
Elon has also confirmed that an end to end neural net that spits out steering, braking and acceleration as outputs from the camera feed is in development. Currently, the camera feed is fed through neural nets for perception but rules around acceleration are built using conventional code.
All of this has been confirmed possible with the current hardware. Again, it appears that the hardware isn't an issue at all when it comes to solving the FSD problem and that the team only needs to nail the software.
When thinking about this it makes sense. They have given the car a better field of view than a human with extra sensors too. This is fairly trivial to solve as we know and understand the field of view of humans when driving a vehicle. What we don’t understand is how the human brain translates that input into driving commands.
This is the big problem and the only thing holding FSD back, not the hardware.