Udacity’s Self-Driving Car Nanodegree — Term 1 Review
Udacity’s launched its Self-Driving Car Nanodegree program late last year (2016) and I had been toying with the idea of enrolling after a good friend of mine recommended the program to me. I finally got around to taking the plunge and signed up for the May 2017 cohort and was accepted into the program. The first term wrapped up 12 weeks later at the end of August and what a ride its been! I’d like to share my thoughts, experiences and challenges of working through the incredible material in this term put together by the folks at Udacity.
First, a little background. I’ve spent the better half of the last decade working as an engineer on various transportation systems (open-road tolling, traffic management, public transit systems, traffic signaling, etc.) under the umbrella of what’s known as Intelligent Transportation Systems. This industry, in my opinion, is one of the many that will be heavily affected by the inevitable transition to both autonomous and connected vehicles which was one of my main motivations to delve into the world of the self-driving car.
Preparing for the term
The folks at Udacity have put together a high-level guide (in the Application Process section) for the uninitiated to get up to speed with what is required of the program. There’s also this excellent overview of the first term by David Silver, one of the nanodegree instructors. The Udacity recommendations included three additional Nanodegree programs, as well as some Udacity courses. This seemed a bit much to cover in the couple of weeks I had left till term started so I took a more realistic approach given the time I had juggling my day job responsibilities. I’d highly recommend covering the following material prior to the start of the course or in conjunction with the first term depending on how much time you have:
- Coursera: Andrew Ng’s Machine Learning Course. This is a fantastic entry-level machine learning course that covers the fundamentals of supervised machine learning (linear/logistic regression, neural networks, support vector machines), all of which is introduced in some form within the Deep Learning lessons of Term 1. The theoretical background (and math) from this course complemented the applied learning approach that Udacity takes in introducing certain topics and getting you straight to the relevant frameworks to write code. The course lasts 11 weeks and I ended up running through it in parallel with my nanodegree which added 4–5 hours to my weekly time commitment. The investment was definitely worth it and it gave me a much better understanding and intuition when approaching the two deep learning projects.
- Udacity: Programming Foundations with Python. I ended up skimming through the lectures in this course to refresh my python syntax. I’m sure there are better introductions out there but this was a quick and easy course and was one of the Udacity recommendations for the program. It’s not implausible to complete this entire course in a couple of days if you put in some solid time during the evenings.
- Coursera: Introduction to Data Science in Python. This is another low overhead course taught over 4 weeks that gives a decent overview of NumPY and Pandas if you’ve never worked with them before. There’s two great thing about this course. First, it gets you started with Jupyter Notebooks which you’ll be using heavily for several projects during Term 1. Second, it’s a good introduction into handling data with two of Python’s most popular data science libraries, NumPy and Pandas. The NumPy material, however, was a bit light and I found myself learning NumPy via the Udacity projects as the term progressed (and I continue learning NumPy to this day…)
- Coursera: Applied Plotting, Charting & Data Representation in Python. I wish I had stumbled on this course earlier. I actually haven’t completely gone through this course yet but I highly recommend it. It provides a great overview of Matplotlib, used to create all sorts of fancy plots in Python, which is incredibly useful for the term projects.
Term Breakdown
The first term is divided into two primary sections (Deep Learning & Computer Vision) consisting of five projects.
Detecting Lane Lines
This is an introductory project to whet your appetite and familiarize you with some of the tools (OpenCV, NumPy, Jupyter Notebooks) that’ll be used throughout the term. The material is presented in a clear, concise way with plenty of boiler plate code available to ease you into the program. A number of OpenCV functions are explored to process a recording of highway driving for lane line detection using Canny edge detection and Hough Transforms to draw straight lines.
This was a great starting point to learn how to load images, understand how they’re represented in arrays, manipulate color spaces and use OpenCV functions to transform the images. Here’s a link to my take on this project.
Traffic Sign Classification
Unlike the first project, there is a significant build up to the second project. The Udacity instructors do a great job of introducing linear/logistic regression, gradient descent and back-propagation. We then jump into using TensorFlow and are introduced to activation functions and regularization techniques (Dropout, L2-norm). This is proceeded by an explanation of convolution layers, max pooling and finally, an introduction to LeNet-5. There are several quizzes and mini-projects along the way to help drive in the concepts, however, it was here that I felt I benefited most from the supervised learning section of Prof. Ng’s Machine Learning course referenced earlier. It was great to have an understanding of cost functions, optimization algorithms, validation/test splits and back-propagation when reviewing the lectures leading up to this project.
This is the first Deep Learning project. It involves building a Convolutional Neural Network (CNN) of your choice using TensorFlow and training it on the German Traffic Sign dataset to perform traffic sign classification on a test split and eventually on images downloaded from the internet. I opted to use a variation of LeNet-5 to complete this project with promising results. Here’s my write-up on GitHub describing my approach.
Behavioral Cloning
The next Deep Learning project was preceded by an introduction to Keras and notes on transfer learning. The Keras material is pretty well put together and I found myself going back to it several times over the term. Also, for someone that was new to deep learning, it was fascinating to see how easy it is to build, train and evaluate different architectures in Keras. It abstracts a lot of supporting code required to construct, run and analyze TensorFlow sessions into simple statements and allows users to focus on the model architecture and objectives.
This project was very exciting. Udacity provides a simulator built on Unity to navigate a vehicle through a test track and collect training data with the objective of being able to navigate the test track autonomously using a trained model. There’s also an optional challenge track if you find the test track too easy :) The models are built using your new found knowledge of Keras and I would recommend making use of the $50 AWS credit available to SDC students to train the model on a GPU-equipped machine. This was another project where Prof. Ng’s course was invaluable, particularly his advice of getting simple architectures up and running quickly and then expanding them as opposed to starting with a complicated architecture. This was also David Silver’s approach in the project lessons.
I ended up taking a different approach on this project where I trained my model on the challenge track and attempted to generalize it enough to work on the test track with some decent results. The complete write-up is available here.
Advanced Lane Line Finding
With the deep learning modules completed, the focus is switched to Computer Vision problems with the Advanced Lane Finding project. We take a deeper dive into the capabilities of OpenCV, performing camera calibrations to undistort images, transforming perspectives and experimenting with different color spaces and Sobel gradients to improve our lane finding techniques introduced in the very first project.
There’s plenty of boiler plate code provided with in the project lessons to do some of the heavy lifting for lane finding once you’re satisfied with the color/gradient transformations. However, there’s a lot of room for improvement if you want to address the challenge videos. Here’s my submitted solution for this project.
Vehicle Detection
The last project of this term is the Vehicle Detection project which follows after a brief introduction of the Naive Bayes classifier, support vector machines (SVMs) and decision trees. Once again, Udacity’s approach is to briefly introduce the concepts behind these tools and get you right into the code (in this case Python’s SciKit-Learn module) to better understand how to implement them. I found that the SVM background provided by Andrew Ng’s course (tired of me bringing this up yet?) greatly helped in answering some of the questions I had while going through the videos.
This last project uses a combination of computer vision and machine learning techniques to perform vehicle detection on the project video introduced in the lane finding projects. Several concepts, including spatial binning, color histograms and Histogram of Oriented Gradients (HOG) features are introduced. A feature vector is manually crafted for windows extracted from the image and fed into a classifier of your choice to perform detection.
This was a great project to understand and contrast various approaches to solving a problem. The classifiers provide a speed improvement over traditional CNNs but come at a cost of having to manually prepare feature vectors. I found this project to be a great way to get students to research the tools available for doing general object detection and where the industry stands today. I ended up implementing both an SVM and CNN approach to compare and contrast the two for speed and accuracy, the results of which can be found here.
One of the primary limitations of these approaches is the inability to do real-time detection with either of these models, I felt this was one of the points the Udacity instructors were trying to drive home with this project. This got me researching a number of alternate approaches including YOLOv2 and TensorFlow’s recently released object detection API.
I’m currently in the process of putting up a revised implementation of my combined and more robust lane-finding/vehicle detection pipeline so stay tuned for a post on that.
Update: Here’s my link to my post on lane-finding/vehicle detection
Reflections
The first term of Udacity’s SDC nanodegree program has been an incredible learning experience. I’ve had to commit around 20–25 hours a week and sometimes more near the project deadlines so this isn’t a course to take lightly. Here are some general take-away points that really helped me during the course of the term.
- Build yourself a routine so you get into the habit of putting in the effort required to complete the projects by the recommended deadlines. This is important to stay on track, especially if you’re juggling commitments at work/school and to your friends & family. This will also help you understand how much time you have to allocate to certain tasks.
- Eat right, exercise and sleep on time. You’ll be learning a lot of new things through this term in a condensed timeline and want to absorb and retain that information. I learned this the hard way where I had to write-off what could have been several productive evenings because I lost track of time working on a project the night before.
- Use GitHub to submit and showcase your projects. If you don’t know how to use GitHub, here’s what you need to know courtesy of Udacity. Create good READMEs for your repos.
- Use one of the many cloud computing platforms that offer GPU-equipped machines, especially for the behavior cloning project. This will greatly reduce the time evaluating tweaks made to your model. Udacity offers a $50 AWS credit but there are other options such as Paperspace or Google’s Cloud Platform.
- Visit and use the discussion forms, the mentors there are knowledgeable and quick to respond. I found the discussions there invaluable whenever I’d come across a stupefying issue.
- Supplement the topics introduced in the lectures with additional research so you can build up some intuition and get a better understanding of how to solve your problems.
- Read the documentation. Here are some links to add to your bookmarks: Numpy, Scikit-Learn, Keras 1.2.2 (the term 1 environment doesn’t use Keras 2.0.2 yet), TensorFlow, OpenCV3 (tutorials).
- Here’s a link to a fantastic collection of cheat sheets for many of these modules and frameworks used throughout Term 1.
I’ve particularly enjoyed Udacity’s learning-by-doing approach although, depending on how deep you want to go down the rabbit hole, I do recommend going through some of the more academic courses on deep learning available on Coursera and other platforms. My current deep learning to-do list includes the following items to get a deeper (pun intended) understanding:
- Coursera: Neural Networks for Machine Learning by Geoff Hinton
- Coursera: Andrew Ng’s Deep learning specialization
- SU: Convolutional Neural Networks for Visual Recognition
I found the projects interesting and enjoyed working on the challenge portions of the projects which I highly recommend completing. You can always come back to the projects between terms to improve on both them and your own understanding.
I’m now in the middle of a well-deserved break between Term 1 and 2 and am using this time to improve some of my Term 1 projects, brush up on my C++ and maybe start learning about some of the topics in David’s description of Term 2. It’s been an exciting journey so far and I’m eager to get my hands on the Term 2 material.
