Self-Driving Car Engineer Nanodegree - Is it worth it?

There has been a lot of fuss about the Udacity’s Self-driving Car Engineer Nanodegree program. I joined the program in November 2016 and I graduated in October 2017. In this post, I will share my experience throughout the Nanodegree and I’ll try to cover as much as possible to clarify anything about the program.

Syllabus

The SDCND (short for Self-driving Car Nanodegree) is divided into three terms. Each term is three months long and provides content specific to a certain technical stack used in the self-driving industry. There is a break of at least two weeks between two consecutive projects and in order to graduate, you need to complete all the projects by the end of the program.

Term 1 — Computer Vision and Deep Learning: Term 1 focuses heavily on Computer Vision, primarily with Python as the programming language used. There are a total of five projects in this term. These are:

1. Lane Finding: This acts as a sort of intro to computer vision. In this project, we had to build a system from scratch that can be used to find lane markings on the road using OpenCV .
2. Traffic Sign Classifier: In this project we were required to build a DNN (short for Deep Neural Network) classifier in TensorFlow for classifying the different traffic signs as per the German Traffic Sign data set.
3. Behavioral Cloning: This is the most exciting project in the first term. Here we had to build an end-to-end DNN that can be used to train a car to drive itself like a human being does.
4. Advanced Lane Finding: This project gives in-depth insights of classical computer vision techniques like Sobel, color masking, etc. that can be used to find lane markings more accurately. In my opinion, classical approaches are fine but a DNN would outperform them in these areas.
5. Vehicle Detection and Tracking: Here we were supposed to build a system that can detect and track cars in a video. Udacity teaches classical approaches like HOG for this project but we were free to explore and use advanced techniques for object detection and tracking. For example, many students used YOLO and SSD networks.

Term 2 — Robotics: This term is all about robotics, especially about Sensor Fusion, Localization and Control. This term heavily uses C++ as the main programming language. So, before starting this term, you should brush-up your C++ concepts. This term was a little bit difficult for me as I didn’t know anything about robotics and I had not used C++ for almost three years (being a Data Scientist I heavily use Python and I am totally in love with it). Again, this term consists of a total of five projects:

1. Extended Kalman Filter (EKF): As the name suggest, here you learn about Sensor fusion (combining data from multiple noisy sensors) and then applying that knowledge to built an EKF from scratch in C++.
2. Unscented Kalman Filter (UKF): In this project, we learn about the shortcomings of EKF and how UKF can overcome all those shortcomings. We were supposed to build a UKF that can take care of non-linear motion.
3. Kidnapped Vehicle: What if the vehicle is kidnapped or what if it is lost? How does the vehicle find out where it is? Yes, localization is what all this project is about. The project is to build a Particle filter for a SDC which can help it do this.
4. Proportional–integral–derivative (PID) control: This project is similar to the Behavioral cloning project except that instead of data driven approach, here you are required to build a PID controller. After completing this project, you will realize why robotics is a much easier way to go in some situations as compared to the data driven approaches.
5. Model Predictive Control (MPC): Similar to the Behavioral cloning and PID project but with extra complexity and fun. The aim of this project was to build a MPC controller from scratch so that the car is able to navigate around the track itself.

Term 3 — Path Planning, Concentrations and System Integration: This is the final and the toughest term of the SDCND program. It consists of a total of three projects. The programming languages used are C++ and Python. The projects included in this term are:

1. Path Planning: This is the toughest project among all three terms. As far as I know, almost every student struggled a lot with this project. Path planning is a crucial concept and is sometimes known as the brains of a self-driving car. This project was so tough that Udacity extended the deadline of this project by a week. Considering the complexity of the project, Bosch, in partnernship with Udacity, announced a challenge for designing a path planner. The top-25 participants were eligible for a direct interview at Bosch. You can read about the challenge here.
2. Semantic Segmentation/Functional safety: These are the two electives from which we had to complete any one for graduation. I chose semantic segmentation. Although object detectors like YOLO and SSD work fine, bounding boxes have certain limitations and that’s where semantic segmentation comes in handy. In this project, we learn how to segment out road pixels. As an extra and optional challenge, we were required to build an end-to-end segmentation DNN for different categories like buildings, roads, pedestrians. One of the most important aspect of this project was to learn how to freeze a computational graph of a DNN and optimize it for inference time.
3. System Integration: With this project, we finally got a chance to work in teams. Students were allowed to form a team with fellow students in a group of four-five students per team. We were required to build different modules for Carla (Udacity’s self-driving car) including a traffic light detector and classifier, path planner, etc. You can read about Carla here.

How much time is required for the Nanodegree?

A lot. Udacity themselves expect almost 15 to 20 hours/week for the ND but I think it takes more than that. I have seen people in term 1 spending almost 30 to 40 hours/week on the projects. Also, if you want to stand out from other students while doing amazing things with the projects as a bonus challenege, then you can expect more than 40 hours/week.

It’s tough, I know. This was the first time I spent so many months on an online course. While doing this program, I was also working at a startup. During daytime, I used to work at the office for 9 hours and 6 days/week so during the night, I did the ND. It requires a lot of hard work, focus and patience to cope with a program for so long.

How is the content in all three terms?

Udacity has done a terrific job in their partnership with Mercedes-Benz, Uber and Nvidia for teaching the concepts in the different terms. But not everything was satisfactory. For example:

• A lot of content in term 2 was taken from Sebastian Thrun’s Artificial Intelligence for Robotics course at Udacity. Also, for people who are totally beginners in the field of robotics, it’s can be hard to get a good grasp from the lessons and you will be required to put extra effort by reading some extra material to get a better understanding.
• It’s good to get an understanding of the classical approaches as taught in Term 1 but I think the projects like Advanced lane finding and Vehicle detection should have been done with DL approaches.
• Most of the people struggled a lot and wasted a lot of time while doing the setup for the course. Udacity should provide a better setup like a docker or a VM image to solve all these problems, especially for terms 2 and 3.

I was in an early cohort, so these problems would be solved over the time for sure but Udacity needs to pick up the pace for it. While the content does not make you an expert in this field, it certainly gives you more than just a headstart.

What else should you keep in mind before taking the Nanodegree?

Though you can learn almost everything required for the Nanodegree while doing it, the people in the ND slack are very helpful and the forums are too good, you should consider the following before joining the program:

• Brush-up on your Python and C++ concepts. You will be required to use both languages heavily during the program.
• Make sure you know how DL libraries like TensorFlow and Keras work. Also, make sure you should at least take Andrew Ng’s course for Machine Learning and CS231n for Convolution Neural Nets. You don’t have to do these in-depth but you must at least understand how things work.
• If possible and if you can afford to build your own system for doing DL, do it. You learn more by doing more and more experiments which you can’t do in the cloud.
• Commitment, focus and patience is all you need. For the first 10 months, I did not get a break. I even worked on Sundays to complete the projects. I used to sleep for only 4 to 6 hours as during the daytime, I was involved in my job and in the night I used to study for the Nanodegree. It is hard and frustrating but it is worth it. I love doing Kaggle and learn a lot from it but during this program, I hardly got any time to do it.

Did I get any job offers during or after the Nanodegree?

Though there are a dozen of hiring partners associated with this program, it doesn’t necessarily mean that you will get a job during or at the end of the program.

I am based in India and we all know that Self-driving cars are something that aren’t going to be a reality in India for at least the next 10–15 years. But then why did I spend so much of money ($2400) on this course? Even though I spent all my earnings of this year on this program, I am happy that I did. Learning is what all matters in the end. I know that being from a middle class family in India, I can’t afford to do an M.S from a good university but I wanted to get a sneak peek of the niche industrial content so that I don’t lag behind, hence I put my money in this program.

If you ask me about the nature of learning curve in the program, it’s exponential. Thanks to my friend Mikel who stopped me from opting out at the beginning of ND.

Should you take this program?

If you can afford to pay for the program, you should definitely take it. It’s an amazing experience. Not only you learn about the industry, but you get to know the importance of a community, a team and how it feels to be responsible while working with some of the amazing and experienced team members.

Thanks to Gianni Jacklone, Bin Liu, Paul Heraty and Soon Cheong for giving me an opportunity to be part of a great team for the Capstone project and thanks to Udacity for providing such an awesome course.