Let’s dive in! Intro to Self-Driving Cars

A detailed review of Udacity’s recently introduced Intro to Self-Driving Cars Nanodegree program

In this post, my second public blog post at all, I want to give you a detailed review of the impressions I got within the two month journey of my first Nanodegree, the “Intro to Self-Driving Cars”.

This is a new Nanodegree which has just been announced by Udacity at the end of September 2017 and started with it’s first cohort in October. It’s target is to lay a solid knowledge foundation for students who want to start a career as Engineer for Self-Driving Cars via the following Self-Driving Car Engineer degree or to just get an impression of the fascinating world of Self-Driving Cars.

As of writing a student who successfully graduates in this Nanodegree will also gain automatic admission to the “Self-Driving Car Engineer” Nanodegree for which you normally have to apply and hope first.

But for now — Let’s finally dive in!

Udacity… I have heard of this “Udacity” a couple of times already in the past, but because I already had a little tower of books about Kotlin and Swift impatiently waiting for me at work, next to some other ones about microelectronics, Raspberry and Arduino at home I did not pay any further attention to the regular advertisements.

In 2017 this changed though. One of the reasons was out of question that I became curious about all the talk about Big Data, Deep Learning, Cognitive Computing, SLAM etc. with which they “bombarded” you on the CeBIT in Hannover which I visit each year.
 The more important reason was though that I also got more and more tasks at work which could of course still be solved with the “old” methods, but which could often have been solved far more efficient with the techniques named above… as I realized very quickly after diving more and more into these subjects.

droidcon 2017

On 4th of September I met a team of Udacity at the droidcon 2017 in Berlin where it had a little promotion booth. We had a great talk about Self-Driving Cars, Deep Learning and Nanodegrees in general. After a very long evening sponsored by Google….

… as you can see Google preferred promotion wheelbarrows instead of booths :)… I became really curious about this possibility to learn the really relevant knowledge “directly from the best” at Udacity.

Let’s do it!

After informing as much as possible in the web about all the Nanodegrees offered afterwards I decided the next day to give it a try and to extend my already started research with the help of a Nanodegree in Deep Learning.

So I registered at Udacity and wanted to sign up for the Deep Learning Nanodegree, the problem:
 You could not join any nanodegree of Deep Learning in September 🙂 — they did not even let you know when the gates of Siraj Raval’s kingdom would open again ;). All you could do was to let yourself add to a waiting list… in theory, because the form did not work as well… at least for one week till they finally fixed it, but that did not change anything of the problem itself… no available seats :/.

Because I wanted to finish the “short” Deep Learning degree at all events before starting the “huge” 9-month Self-Driving Car degree I decided to simply wait, sit it out and to hope the gates will open again soon. As already mentioned I anyway still had a lot of books waiting for me so “too much time” was one of my lesser problems ;).

So I continued learning with publications and videos of Brandon Rohrer, Andrew Ng, Siraj & Co. till on September 19 the following e-mail arrived:

The promotion promised you will get the foundation skills for the large Self-Driving Car Nanodegree and a guaranteed admission within four months!

4 months? At least one month till Siraj’s gates will open again?

As I’m in general very fast in things such as coding and learning I thought:
 “Ok, ready — set — go, you will have to do this in ONE!” 🙂

So I still on the same day signed up, totally excited and counting the days of the remaining 3 weeks till my first Nanodegree would finally start on 10th of October.

While waiting I peeked into a couple of the for free courses and finished the Python Introduction course, which really made a very qualitative impression and even more curious about how a real Nanodegree would look like.

The curtain opens

On the 10th of October the Nanodegree was finally accessible.

It is as of writing subdivided into the following chapters:

  1. Introduction
  2. Bayesian Thinking
  3. Working with Matrices
  4. C++ Basics
  5. Performance Programming in C++
  6. Navigating Data Structures
  7. Vehicle Motion and Control
  8. Computer Vision and Machine Learning

Unfortunately the majority of these 8 chapters was still locked… so far about my plan to finish it quickly within a month before the Deep Learning degree would start ;).

Chapter 1 — Orientation

The first chapter starts with a brief Introduction by Sebastian ‘Godfather of Self-Driving Cars’ Thrun himself who gives a brief overview of the topics of this Nanodegree.

Right afterwards you get introduced to the team, as of writing consisting of Sebastian, Anthony Navarrow (Product Lead), Andy Brown (Curriculum Lead), Cezanne Camacho (Course Developer), Elecia White (Author/Instructor), Tarin Ziyaee (Voyage/Instructor), Jessica Lulovics (Program Manager) and Erika Alonzo (Student Experience Coordinator).
 You get a quick course about how Nanodegrees work, which student services such as mentorship, Slack chat and Waffle Udacity provides and what you might in general expect of this Nanodegree.

Right after this theoretical introduction you are ready to make your hands dirty for the first time and get taken into the world of Self-Driving-Cars in a really awesome designed interactive comic story called “The Carla Chronicles”.

In 14 chapters with lots of humor you for the first time get an understanding of why a Self-Driving Car is not built overnight, that even slightest changes such as the air pressure of the tires have huge impact on the sensing of the current position, because less air pressure also means smaller diameter of the wheels and a smaller diameter of the wheels less meters of movement per wheel rotation. And so on and so on.

If you have not been motivated enough to become a Self-Driving Car engineer before — now you are for sure!

To make the waiting for the unlock of the second chapter a bit more comfortable the first chapter ends with a large list of recommendations of courses and subjects which will in any case help on your way of this Nanodegree.
 As well you get a brief introduction in Jupyter notebooks, a web browser based Python source code editor, which you really get to like very fast and in which the majority of the exercises of this nanodegree are solved.

And the absolute highlight at the end of the first chapter: An one hour video of the DARPA Challenge 2005 in which Sebastian Thrun and his team at Stanford with theirs car Stanley for the first time absolved a tour of the insane distance of more than 200 kilometers completely autonomously in the Mojave desert in less than 7 hours.

Chapter 2 — Bayesian Thinking

As already very well pointed out in the first chapter of this Nanodegree and in the DARPA challenge video in which Stanley nearly accidentally committed suicide because one of the sensors did not behave as expected — in the world of Self Driving Cars just one thing is really certain: That nothing is certain.

No sensor is really 100% precise and even if it were, who guarantees the rain does not influence it’s measurement, it did not get loose, moved for one millimeter and now points at an angle of 21 degree instead of 20 degree as previously assumed, that the GPS signaled is disturbed?

In the second chapter the team from Udacity shows how to solve this problem with the help of Bayes’ law by combining the information of multiple sensors and multiple measurements to minimize this uncertainty as much as possible and to make the best of the available data.

But before you have to enter the “Wonderland of Bayes’s law”, you are allowed for the first time to steer your own personal self-driving car in the first rated project of this Nanodegree called JoyRide. The task of this project is to reverse park a car as precisely and centered as possible without crashing the one’s of your neighbours:

Chapter 3 — Working with Matrices

Let’s “Enter the Matrix” and Kalman filters are the topic of the third chapter, you won’t even need a landline phone to do so.

The goal of this chapter is to make the students more familiar with tools such as matrices, vectors, Gaussians and and how to use this know how in the final project to implement a full functional Matrix class from scratch on your own which can then be used for a Kalman filter to predict the movement and velocity of another car… so a typical situation for a self-driving car which all the time needs to be aware of it’s surroundings.

Next to the basics the students also get a brief introduction in how to read equations such as the one of a Kalman filter and how to translate such an equation to also effectively usable Python code.

Chapter 4 — C++ Basics

C++ basics. When I read the announcement of the 4th and 5th chapter I had a very broad smile on my face and thought “That’s going to be interesting, hopefully you won’t behave worser than your wife”.
 Here needs to be mentioned that my wife worked in the Kindergarten for many years and when she had a talk at the Kindergarten about our son with other nursery school teachers she sometimes really “destroyed” them, because the situation (far too many children / less teachers) was really awful and I sometimes had liked to be able to become invisible in these moments.

And with 20 years of experience in C++, actually my major skill, you now enter a class room about “C++ Basics”, becoming a smartass like my wife was basically inevitable.

The chapter started far better though than I had awaited while it was still unlocked. Sebastian, Elecia, Andy and Anthony give you a great introduction about why languages such as C++ are needed in general, for devices such as nVidias PX2, and what major differences to Python need to be considered.

In 5 chapters and overall 52 subchapters they provide a great transition from Python to C++ and you are made familiar with all you need to know:

When to use C++ and when to use Python, static vs dynamic typing, object oriented programming in C++, compilers and linkers, how to translate a source code you prototyped in Python to C++, which important differences you need to consider and of course how to successfully compile and run a program locally on your own computer.

The problem:
 I was done with chapters 4 and 5 within 8 hours. And within these 8 hours I stumbled over dozens of “improvable” points and even 3 evil bugs which for example made two projects unsolvable if someone used Microsoft Visual Studio. Because I did not get any fast response via the contact formular I decided to give this “Slack” a try. I just heard about it on the droidcon and it sounded really promising. So I joined the ISDC slack…

Becoming a “voluntary mentor”

Actually being likely even more annoying than my wife at the Kindergarten I basically bombarded Slack with bug reports and suggestions for improvements.

And to my surprise I got at first contacted by Andy Brown (Curriculum Lead) and soon after by an “undercover” content developer named Andrew Paster who saw all my posts and said thanks for them. Instead of the typical behavior my wife experienced at the Kindergarten (no time, too small staff, no space) I started a great and very constructive discussion with both of them.

Because they were so open for suggestions and friendly in response (instead of being offended like some likely would be) I decided to really review these two chapters once again intensively and wrote a 13 page review of really every small bug, typo etc. and believe it or not: Just two weeks later you could not find any of them anymore, awesome. — No wrong pride first, students first — really impressive.

Another side effect of all my “smartassnes” on Slack was that many other students who read my postings who struggled with C++ contacted me directly, also because of the huge time shift between Germany and Westcoast.

So if they could reach no one at 3am in the night in San Francisco, in Germany it was 12am already and in 95% of the cases I could answer theirs questions within a minute. The majority of the students was really very thankful and within two weeks I suddenly knew three dozens of people from all over the world I did not know before and really had awesome talks with many of them and we even founded a little inofficial ISDC team. 🙂

Being able to contribute even as student

One of the problems the most students struggled with were the huge amount of operating systems and IDEs. After it became sort of annoying to respond the same copy and pasted text over an over again, I decided to write a guide for my IDE of choice for these small projects, XCode:

Setup guide for Intro to Self Driving Cars project 3 in XCode

and short afterwards also added one for Visual Studio:

Setup guide for Intro to Self Driving Cars project 3 in Visual Studio

The feedback was great and to help newer students Andy Brown even linked to them directly from the first chapter of the C++ Basics part of the Nanodegree pages, great, students first! 🙂

The effect of this was that I afterwards of course got even more questions than already before, but also a lot of motivating feedback:

It was and is really an awesome experience, I had never imagined to “meet” so many so awesome and motivated people from everywhere in this world within such a short amount of time.

The chapter 4 ended with the 3rd of the 5 rated project, a translation of a Python histogram filter code to C++.

Chapter 5 — Performance Programming in C++

Chapter 5 is opened by Sebastian personally again and he reminds the students that it’s of course nice to have a language such as C++ which is by nature far faster than Python, but at the end every tool is just as good as what it’s user makes of it — majorly in low level languages such as C++.
 After all the resources and systems in a Self-Driving car are far more limited than in any AWS farm in which you can easily spawn your 32-core server with two clicks.

So in 2 chapters and overall 31 subchapters you get taught all of the common “tricks” every C++ developer simply needs to know such as static cached variables, minimizing of copying overhead, passing data by reference, when to use 32 bit and when 64 variables, how C++ code becomes executable cpu op code etc.

As performance optimization is one of my major tasks at work I assumed I could not learn anything new in this chapter…

Speed optimization contest

The last subchapter and optional project of this chapter 5 was a history filter optimization contest — as I not really awaited otherwise at this subject I at first “steamrollered” all other “competitors” with my first try of optimization.

That was until another guy from Germany called Carsten G. entered the arena as well and steamrollered me too. 🙂 It became a really awesome and demanding contest and in the end he and me really had to tune “every bit” :)… been a long time since I had to go so low-level.
 After both of us saw no further optimization potential we decided to “Let 5 be even” and to not enter “level 2” with weaponry such as SSE, AVX and multithreading and to continue with chapter 6 of this Nanodegree :).



Chapter 6 — Navigating Data Structures

The topic of the 6th chapter is a bit misleading and this chapter is basically split into 3 subchapters:

  • A step by step guide how to solve and approach new problems in general and to plan the solution and solve them step by step
  • Efficient use of the right Python classes for different problems such as a ticketing system, when to use sets, when to use dictionaries or simple lists and how to use and combine them efficiently.
  • The presentation of several routing algorithms in detail and how to use them to for example solve a problem such as calculating the fastest route from San Francisco to Los Angeles.

I could not really find anything interesting or new in the first subchapter, but I was really totally excited by the two other ones, majorly the detailed presentation of the routing approaches of subchapter 3 was really capturing.

The final project and the 4th rated one of this Nanodegree was the implementation of the A* algorithm which is able to find the shortest path for from every node to every other node of the network shown below:

Chapter 7 — Vehicle Motion and Control

As chapters 4, 5 and 6 were all three feeling a little bit “far away” from the world of Self-Driving cars itself and more general and theoretic I really fell instantly in love with chapter 7:

Odometers, Speedometers, Accelerometers, estimating trajectory of a “real” car again or of an elevator — finally back to metal!

The whole chapter is dedicated to the handling of sensor inputs and how to use methods like calculus in form of derivatives, integrals and trigonometry to either make of much data even more data or of less data as much as possible.

If I had to name one of the 8 chapters my favorite it would definitive be Chapter 7, because it was simply the one closest to the real world with many possibilities to experiment and to get a great understanding again for all the small problems of the engineering of a self driving car.

The final (optional) project of this chapter was the reconstruction of a car’s position and direction with the help of sensor data (excluding GPS of course):

Chapter 8 — Computer Vision and Machine Learning

This last chapter is separated into the insane amount of 49 subchapters including 12 Jupyter labs in which Cezanne Camacho with support of Sebastian Thrun and Tarin Ziyaee from Voyage introduce the student step by step into how he can make a car “see” with the help of computer vision and also more general applications like for example how to remove and replace a shadowed green screen in a video or image.

Some of these steps are standardizing the data with the help of cropping, color conversion and resizing and to then find the patterns in data such as the single RGB or HSV channels to for example categorize images between day and night:

In the final 5th of the overall 5 projects you have to combine all the tricks of these 49 chapters to write a traffic light classifier which never misclassifies a red light as green (would be deadly in reality after all) and which at least classifies 90% of the provided images correctly.

The optional part I used to implement a variant using Deep Learning:

Personal mentor

On November 6, about a month after the start, a mail from Udacity arrived in which it announced that ISDC students will now also get support of a personal mentor instead of live support.
 The major difference is that you can tell your mentor about your targets for the next week, over which problems you stumbled and as he comes to know you, he can of course more intuitively help you over time.

I had the luck that my mentor was Sebastian Trick, one of the graduates of the first cohort of the Self-Driving Car Engineer Nanodegree working at the Karlsruhe Institute of Technology, short KIT, which researches SDCs and at which I just a couple of years ago spent a week in a conference for parallel computing.

I did not really need the personal support here, but we had a couple of great talks about the world and hurdles of Self-Driving Cars, the graduation party in Silicon Valley etc. — was really nice and likely the best assignment I could get. 🙂

From other students and from my second Nanodegree in progress, Deep Learning Foundations, I can tell though that mentorship in general works pretty well if you ask — waves at Soon Yau.
Many people though seem to prefer to “hang around” in Slack. I also count myself to the later ones, because it’s not a 1:1 conversation anymore, many people can give you feedback, you can start discussions about solutions etc.

One sketchy problem which fast comes into play here is unfortunately the Udacity honor code. So for example in the current Nanodegree there was a lot of discussion about the last rated project (the one with the traffic lights mentioned above) in which people began to post partial solutions of the test and theirs results.

At a later job that’s out of question the best way to solve it, work in a team, exchange etc., but if at the end suddenly 10 different students submit completely identical solutions with identical thresholds for color ranges and then you have to set your checkmark at the line

“I confirm that this submission is my own work. I have not used code from any other Udacity student’s or graduate’s submission of the same project”

…then I assume you are actually walking on very very thin ice :). I think it’s also very hard to find here a golden middle between team work and testing each student individual, majorly if your team consists of 3 people and more it happens fast than one becomes a “passive observer”.


On December 20th, about two and a half months after start, I finally graduated — and very likely even as world first, 9 hours after the unlock of the last two chapters.
The traffic light project was really awesome and made a lot of fun, I really forgot time and space completely till I got closer and closer to the 100% testing accuracy for the right traffic light color (of at least 90% required for passing). 🙂

At 1am in the night I could finally submit the last of the 5 projects and then fought me till nearly 6am in the morning from chapter to chapter while waiting for the review result — simply too close now to the first Nanodegree to still be able to go to bed ;-).

At nearly 6am then the big blue graduation button finally appeared and there it finally was, my first Nanodegree:


The whole Nanodegree was really an exciting two month journey through the world of Self-Driving cars, starting with a really awesome made comic introduction adventure and ending with a demanding last project and chapter.

A bit critical I see the large portion of Python and C++ and there were also a couple of controversial discussions about this in Slack.

If you are completely new to programming, so barely got in contact with C++ and/or Python, then Udacity absolutely keeps it’s promise that the Intro to Self-Driving Cars Nanodegree is a really great foundation for the huge SDCE degree. (Actually I personally even had called it Self-Driving Car Foundation Nanodegree)

And from the talks with the other students there were really many who had to fight hard with the C++ and the Python part of it and were totally excited about being able to learn it — with the mentorship, slack support etc. in difference to the free courses where you had to learn it all by yourself.

If you though just finished your master degree, are totally skilled in C++ and Python as well and like to solve the 3rd grade derivatives for a back propagation while watching Netflix then this course might be exciting as well, but you will likely be too unchallenged.

I had the “luck” that I never got in contact with Python before, although I even learned 10 other languages, including even ones like Lua meanwhile, but simply missed Python all the time somehow.
So the world of Jupyter notebooks and all the possibilities tools such as Pandas, Numpy, Plotly and Co. provide were completely new to me as well. The same was true for a couple of the math topics such as derivatives. Trigonometry, matrices, vectors, sure, my daily tools in the world of geographic data… but derivatives? Integrals? At least nearly 20 years ago. 🙂

To come to the point…
If you are skilled in both languages and math: Evaluate the Deep Learning Foundations Nanodegree or apply for the SDCE ND directly. You won’t regret DLF for sure though and I recommended it a couple of students, because it well help you in every job in which you have to predict data, if of sensors or financial sorts… and in any case a lot in the first of the three parts of the huge SDCE degree.

If one or both languages are completely new to you and college/highschool are some years ago: Good choice and you will have an exciting journey into the world of Self-Driving Cars.

If you are somewhere in between: Throw a “fair coin” ;).

Thank you very much for reading, now let’s dive in!

Udacity logos and exercise/project data shown Copyright © Udacity

Thank you very much at this point for the legal clarification @ Rachel and Jonathan

Originally published at www.PathToLevel5.com on January 1, 2018.