Cloning Driving Behavior by augmenting steering angles

In the recent Udacity’s Self-driving Nano-degree project, Behavioral-Cloning, where I’m tasked to train a neural network, by providing sample images mounted from inside the car and the actual steering angles at that moment, to train a car to drive in a simulation. I used the NVIDIA’s neural network model to drive on a never before seen test track while staying on the road without veering off to the ledge or exhibiting any behavior deemed dangerous if humans were riding within.

Car on the Training Track Simulation

To collect data for input data, I recorded my driving the car using my keyboard’s arrow keys to move it forward and turn left/right. I noticed that when the car tends to steer a bit discretely on the track.

I read in our forums that if you have a joystick, your steering angles tend to be less jerky.

The collected data from the three virtual cameras inside look like this.

At this instance, the steering angle recorded is 0.15

I soon realized my data was insufficient to train the model as I didn’t collect enough good data for training. I elected to use Udacity’s provided data. In the class provided data, the car is going mostly straight so the steering angle is zero or near zero. I have an unbalance data issue as when I trained on that data, the car will continuously turn slightly right or left and eventually veer off the road. See my first simulation drive where my car going off-road into the woods and through trees!

My first attempt using class data

To combat this problem, I removed zero degree turn, down-sampled near zero degree and up-sampled larger turn data. Even with this data sampling, there wasn’t enough data to train on. Many in the class suggested to augment the image data by rotating, flipping, sheering and translating the image (thanks to October cohort for providing sample code) to generate new data. I’ve tried this for a week but I felt that I didn’t get all the angle adjustments just right and my model worsened with these types of image augmentation.

I decided to augment the angle instead of performing image augmentations. Why you ask? If you see the sample training data above, it was turning at 0.15 but is 0.15 the only “correct” steering angle for this situation? In a real driving situation, you could have turned 0.145 or 0.155 or any number of angles close to 0.15 and it would have been an acceptable driving behavior. That’s why I decided to create new data by duplicating existing images by perturbing the angles a bit. I hope the neural network would learn that there are many acceptable steering angles for a given image and thus also minimize over-fitting the data.

See my sample of generated data from one original source image below.

def perturb_angle(angle):
new_angle = angle * (1.0 + np.random.uniform(-1, 1)/30.0)
return new_angle
def change_brightness(image):
# Randomly select a percent change
change_pct = random.uniform(0.4, 1.2)

# Change to HSV to change the brightness V
hsv = cv2.cvtColor(image, cv2.COLOR_RGB2HSV)
hsv[:,:,2] = hsv[:,:,2] * change_pct

#Convert back to RGB
img_brightness = cv2.cvtColor(hsv, cv2.COLOR_HSV2RGB)
return img_brightness
def flip_image(image, angle):
img_flip = cv2.flip(image,1)
angle = -angle
return img_flip, angle
center_image = plt.imread(X_train_data["center_image"][0])
angle = X_train_data["steering_angle"][0]
plt.rcParams["figure.figsize"] = [40, 20]
for i in range(0, 10):
plt.subplot(4, 5, i+1)
if i == 0:
plt.title(str(np.round(angle,5)), fontsize=30)
new_img = change_brightness(center_image)
new_angle = angle

if np.random.randint(2) == 1:
new_img, new_angle = flip_image(new_img, new_angle)
new_angle = perturb_angle(new_angle)
plt.title(str(np.round(new_angle,5)), fontsize=30)
First is the original image and the rest is generated images with steering angle as header

I didn’t abandon all image augmentation though. Thanks to Vivek Yadav, Denise R. James, Kaspar Sakmann and many others who thoughtfully wrote up their experiences from which I learned. I kept image flipping and brightness adjustments. With this data in hand, I trained the NVIDIA’s model to drive the car on the training and the unknown test tracks.

Simulation drive with generated date on the Training Track
Simulation drive with generated date on the Test Track

By applying some adaptive throttling, the car can drive up hills on the test track. It zooms around at near 30 mph, top speed in the simulation. Even though the car exhibits some jittery moves at high speeds where I could apply less throttling, I like to go fast!


Often, there are numerous ways to achieve the same goal. In this project, you can augment the input variables and collect more data. In my case, I also augmented the output variable, the steering angles, to achieve the desired result.