To transfer this to a real car, you will probably need to do some amount of domain randomization. This worked very well in my experiments with Duckietown, even though my simulator looks cartoony: https://pointersgonewild.com/2018/05/21/building-a-simple-self-driving-car-simulator/

Without domain randomization, I find that neural networks will try latch onto irrelevant features of the input. A VAE may ameliorate this, but I suspect it won’t be enough, you will need to make sure the input has enough variability that a neural network can’t easily overfit.