Self Driving Car Nanodegree Experience So Far….
To start with, like others, I was one of those lucky students accepted in the first cohort of the Self-Driving Car Nanodegree. Inspired by my fellow students, I am using this blog to write about my experience in Udacity Self Driving Car Nanodegree (SDCND)
To begin with, I have around 8+ years of experience in Computer Vision and Machine Learning. Of late, before joining this Nanodegree, I have also taken many online courses like cs231n, Robotics Perception, etc. The main reason for enrolling in SDCND was to keep track of advancement in the technology happening around especially in Self Driving Car field.
To begin, So then why did I choose this course over other available courses? “The main reason was that I have experience in ADAS so this course was a perfect fit for my career passion”. Also, it was like a monopoly. But that alone can’t be a reason for pursuing this course.
An another reason being, the course content. It was giving me exposure to deep learning projects and as well as perception problems using Radar and Lidar. Even before the course announcement, I started to learn things on my own, especially things related to self driving car. But I found it difficult to search and gather things around and learn. Like I had to spend weeks to know about Lidar processing and spend loads of time reading different blogs.
The third reason being, since I don’t work with deep learning stuffs at my work, it was a bit difficult to get real-time experience in deep learning projects. When I was thinking about how to improve my skills further in deep learning, I came across Udacity Challenge #2 and started working on the challenge.
With the help of great people around in Slack Channel, I learned Tensorflow and ROS and started to work on this challenge on my own. With my knowledge (experience from other courses taken before), I was not able to come close to even getting a bare minimum result in that challenge. Even though, I know about the theoretical aspects, I was not able to use that to get the challenge working. I understood that the deep learning is all about experience. After trying out for a few weeks, I know it would be difficult to complete this challenge. Meanwhile, I got my admission into first Cohorts. With literally bad experience in Challenge #2, without any second thought, I made up my mind to join SDCND.
Like others, I was eagerly awaiting the start of the Nanodegree. Meanwhile, I brushed up python knowledge. As days came close by, my excitement level was at its peaks and was kind of self assurance that made me think that I am ready for this course. So, let’s see, how things spun out for me in the first two months of the Nanodegree.
Project #1 - Lane Detection
Just as a warm to the course, the first project was to detect the lane lines. Since this project deadline was in 7 days after enrollment, I was in a sense of panic initially. So I started working on the project from day one. One good thing that happened was, some months before to explore deep learning, I bought GTX 980 and configured my system to make it work. This helped me to focus more on projects and course content and not think about environment setup. Within a few days, I completed the project. The following are some of my observations on getting around the project, especially in challenge video.
Tips and Tricks: (Not mandatory to follow this. Just my opinion)
- Work with trapezoid region of interest
- Using Hough parameters filter out not redundant line. For example, use slope values to eliminate line which are vertical or horizontal
- In some frames, there might be a situation when there is no output from Hough transform. In those cases, use the cached result from previous frame
- If possible, use color information for getting results on challenge video, especially in the grey part of the road
Key takeaways from project:
- Basic usage of OpenCV, Python and video processing
- Usage of color information for efficient lane detection
- Understanding the difficulties in lane detection like curvy road, non-clearly visible lane detection to name a few
Project #2 — Traffic Sign Recognition
After a successful project 1 submission, I excitedly started the Deep Learning section of the program. In a few days, I completed the Tensorflow lab and started the traffic sign recognition project. As most people did, I too started off using a linear model initially and then moved to convolution models. With Yann LeCun’s papers as base, I implemented the data augmentation and used a Multi-Scale Features model as shown below.
But I was not able to produce the original result. The following might be the reasons for not achieving the results mentioned in the paper.
- Exact model mentioned in the paper was not used
- Could not replicate the exact pre-processing steps. Only Y channel was used in my implementation
- The mean value of the training data was not used in testing time. So mean preprocessing was not used
Having learned Keras since, I plan to revisit this project to better my previous test result of 97%. Please note, I tried to use as small a depth as possible as mentioned in the paper. I am of the assumption that, a bit improvement to the accuracy is possible by going one or two levels deeper.
Key takeaways from project:
- Exposure to Tensorflow and data augmentation
- Importance of data pre-processing
- Python skills for displaying results and plots
- Writing own model and training/testing activity and fine tuning hyper-parameters
Project #3 — Behavior Cloning
This has been by far, the most challenging project until now. The project involves predicting the steering angle of the car. Using this predicted angle, the car should drive autonomously around the track. This is similar to the challenge #2 mentioned before.
This project is completely different from most of the example tutorial available online . It is a regression and not classification problem. To explain, Traffic sign is classification problem since labels are given and we need to predict exact labels. So even though soft-max layer at the end gave us probability, it is converted into output label using arg-max (Maximum probability index). It is then compared against training label and loss obtained is used to train the network. In project #3, since steering angle is predicted, which is a float value, it becomes regression problem. So to train this model, the output from last layer using single neuron is directly used (soft-max not used since it will truncate the value between 0 and 1) and compare against given steering angle (Label for regression problem). For loss MSE (Mean Squared Error) is used.
Initially I started off using Nvidia and Comma.ai model. Even though I was able to drive through the track, steering movement was not visually pleasing. The initial model trained, worked well on track 1 but failed to generalize on track 2. Then I started to work more on data augmentation part to make it work on track 2 using images trained from track 1. Also, I started to work on my own model. With data augmentation, the model started to drive smoothly in track 2 as well. But at the later part of the track, when there was a tight right, it failed.
The data augmentation was obtained by modifying the in-build Keras Image Generator. It was elegant and easy to use. Also, it had functionality of flow_from_directory, where you can specify the image path and it will load directly batch by batch. It will really helpful, it you have limited memory or you want to use huge data.
Later drawing inspiration from one of colleague, I used traffic sign recognition model with regression head. Even that small model was able to drive across both the tracks.
Finally, since this project came from the back drop of transfer learning, I used trained VGG16 model as the base, and modified it to detect steering angle. For this transfer learning, I reused last three layers from VGG16 and added three more custom layers at the end followed by fully connected layers to make the model as regression type.
I was really excited to see these results after hours of hard work. Now with the experience gained, I am planning to give a go at challenge #2 again.
Tips and Tricks: (Not mandatory to follow this. Just my opinion)
- If you are planning to use a lighter model with fewer layers like Nvidia, data plays a major role. So, record more data to make it work
- Use left, right and center image for training.
- Use data augmentation as shown above (Not mandatory, but good to learn about it)
Note: I used my own data set for training, captured using keyboard controller. No images from track 2 was used to train the model. It is not a straight forward project where you implement the model and you get the result. A fair bit of effort is required.
Key takeaways from project:
- Importance of data and data augmentations
- Keras and usage of data generator
- Training a regression model
Udacity Learning Experience:
According to me, the best-selling point of this Nanodegree course is the group of people with whom we work with. The community in slack has some of the best minds in the world and constant interaction with them and knowing about their thought process is invaluable and cannot be replaced. I work in an environment where most of the people have little exposure to recent technologies like deep learning. So, interaction with like-minded people makes a huge difference. There might be some people who might think the course content needs improvement and be much deeper. I too agree with that. But it is always difficult to teach each and every concept deeply. Until now, I feel extremely satisfied with what I have learned so far. If you are excited about self driving cars, no need to think further, this course will surely give you the right foundation.
Acknowledgment:
Finally, I would love to acknowledge few people who helped me a lot in this learning process like John Chen, Vivek Yadav, Patrick Kern, Paul Balança, Justin Heaton and many more in the slack channel. Thank you guys you are simply awesome. And also I would like to thank Udacity for providing this Nanodegree and selecting me for the first cohorts. Finally, I would love to thank Dhruv Parthasarathy for proof reading this article.
