Cloning Driving Behavior by augmenting steering angles
In the recent Udacity’s Self-driving Nano-degree project, Behavioral-Cloning, where I’m tasked to train a neural network, by providing sample images mounted from inside the car and the actual steering angles at that moment, to train a car to drive in a simulation. I used the NVIDIA’s neural network model to drive on a never before seen test track while staying on the road without veering off to the ledge or exhibiting any behavior deemed dangerous if humans were riding within.
To collect data for input data, I recorded my driving the car using my keyboard’s arrow keys to move it forward and turn left/right. I noticed that when the car tends to steer a bit discretely on the track.
I read in our forums that if you have a joystick, your steering angles tend to be less jerky.
The collected data from the three virtual cameras inside look like this.
At this instance, the steering angle recorded is 0.15
I soon realized my data was insufficient to train the model as I didn’t collect enough good data for training. I elected to use Udacity’s provided data. In the class provided data, the car is going mostly straight so the steering angle is zero or near zero. I have an unbalance data issue as when I trained on that data, the car will continuously turn slightly right or left and eventually veer off the road. See my first simulation drive where my car going off-road into the woods and through trees!
To combat this problem, I removed zero degree turn, down-sampled near zero degree and up-sampled larger turn data. Even with this data sampling, there wasn’t enough data to train on. Many in the class suggested to augment the image data by rotating, flipping, sheering and translating the image (thanks to October cohort for providing sample code) to generate new data. I’ve tried this for a week but I felt that I didn’t get all the angle adjustments just right and my model worsened with these types of image augmentation.
I decided to augment the angle instead of performing image augmentations. Why you ask? If you see the sample training data above, it was turning at 0.15 but is 0.15 the only “correct” steering angle for this situation? In a real driving situation, you could have turned 0.145 or 0.155 or any number of angles close to 0.15 and it would have been an acceptable driving behavior. That’s why I decided to create new data by duplicating existing images by perturbing the angles a bit. I hope the neural network would learn that there are many acceptable steering angles for a given image and thus also minimize over-fitting the data.
See my sample of generated data from one original source image below.
new_angle = angle * (1.0 + np.random.uniform(-1, 1)/30.0)
# Randomly select a percent change
change_pct = random.uniform(0.4, 1.2)
# Change to HSV to change the brightness V
hsv = cv2.cvtColor(image, cv2.COLOR_RGB2HSV)
hsv[:,:,2] = hsv[:,:,2] * change_pct
#Convert back to RGB
img_brightness = cv2.cvtColor(hsv, cv2.COLOR_HSV2RGB)
def flip_image(image, angle):
img_flip = cv2.flip(image,1)
angle = -angle
return img_flip, angle
center_image = plt.imread(X_train_data["center_image"])
angle = X_train_data["steering_angle"]
plt.rcParams["figure.figsize"] = [40, 20]
for i in range(0, 10):
plt.subplot(4, 5, i+1)
if i == 0:
new_img = change_brightness(center_image)
new_angle = angle
if np.random.randint(2) == 1:
new_img, new_angle = flip_image(new_img, new_angle)
new_angle = perturb_angle(new_angle)
I didn’t abandon all image augmentation though. Thanks to Vivek Yadav, Denise R. James, Kaspar Sakmann and many others who thoughtfully wrote up their experiences from which I learned. I kept image flipping and brightness adjustments. With this data in hand, I trained the NVIDIA’s model to drive the car on the training and the unknown test tracks.
By applying some adaptive throttling, the car can drive up hills on the test track. It zooms around at near 30 mph, top speed in the simulation. Even though the car exhibits some jittery moves at high speeds where I could apply less throttling, I like to go fast!
Often, there are numerous ways to achieve the same goal. In this project, you can augment the input variables and collect more data. In my case, I also augmented the output variable, the steering angles, to achieve the desired result.
- Dealing with unbalanced data: Generating additional data by jittering the original image by Vivek Yadav
- Learning human driving behavior using NVIDIA’s neural network model and image augmentation by Vivek Yadav
- Udacity Self Driving Car Project 3 — Deep Learning to Clone Driving Behavior by Denise R. James
- Behavioral Cloning — make a car drive like yourself by Kaspar Sakmann
- NVIDIA’s End to End Learning for Self-Driving Cars by NVIDIA Corporation