Udacity’s Sensor Fusion Nanodegree Program!
Udacity’s Sensor Fusion Nanodegree Program launched yesterday! I am so happy to get this one out to students 😁
Goal
The goal of this program is to offer a much deeper dive into perception and sensor fusion than we were able to do in our core Self-Driving Car Engineer Nanodegree Program. This is a great option for students who want to develop super-advanced, cutting-edge skills for working with lidar, camera, and radar data, and fusing that data together.
The first three months of the program are brand new content and projects that we’ve never taught before. The final month, on Kalman filters, comes from our core Self-Driving Car Nanodegree Program. The course is designed to last four months for new students. Students who have already graduated the core Self-Driving Car Engineer Nanodegree Program should be able to finish this specialized Sensor Fusion Nanodegree Program in about three months.
Curriculum
Course 1: Lidar
Instructor: Aaron Brown, Mercedes-Benz
Lesson: Introduction. View lidar point clouds with Point Cloud Library (PCL).
Lesson: Point Cloud Segmentation. Program the RANSAC algorithm to segment and remove the ground plane from a lidar point cloud.
Lesson: Clustering. Draw bounding boxes around objects (e.g. vehicles and pedestrians) by grouping points with Euclidean clustering and k-d trees.
Lesson: Real Point Cloud Data. Apply segmentation and clustering to data streaming from a lidar sensor on a real self-driving car.
Lesson: Lidar Obstacle Detection Project. Filter, segment, and cluster real lidar point cloud data to detect vehicles and other objects!
Course 2: Radar
Instructor: Abdullah Zaidi, Metawave
Lesson: Radar Principles. Measure an object’s range using the physical properties of radar.
Lesson: Range-Doppler Estimation. Perform a fast Fourier transform (FFT) on a frequency modulation continuous wave (FMCW) radar signal to create a Doppler map for object detection and velocity measurement.
Lesson: Clutter, CFAR, AoA. Filter noisy radar data in order to reduce both false positives and false negatives.
Lesson: Clustering and Tracking. Track a vehicle with the Automated Driving System Toolbox in MATLAB.
Lesson: Radar Target Generation and Detection Project. Design a radar system using FMCW, signal processing, FFT, and CFAR!
Course 3: Camera
Instructor: Andreas Haja, HAJA Consulting
Lesson: Computer Vision. Learn how cameras capture light to form images.
Lesson: Collision Detection. Design a system to measure the time to collision (TTC) with both lidar and camera sensors.
Lesson: Tracking Image Features. Identify key points in an image and track those points across successive images, using BRISK and SIFT, in order to measure velocity.
Project: 2D Feature Tracking. Compare key point detectors to track objects across images!
Lesson: Combining Camera and Lidar. Project lidar points backward onto a camera image in order to fuse sensor modalities. Perform neural network inference on the fused data in order to track a vehicle.
Lesson: Track An Object in 3D. Combine point cloud data, computer vision, and deep learning to track a moving vehicle and estimate time to collision!
Course 4: Kalman Filters
Instructors: Dominic Nuss, Michael Maile, and Andrei Vatavu, Mercedes-Benz
Lesson: Sensors. Differentiate sensor modalities based on their strengths and weaknesses.
Lesson: Kalman Filters. Combine multiple sensor measurements using Kalman filters — a probabilistic tool for data fusion.
Lesson: Extended Kalman Filters. Build a Kalman filter pipeline that smoothes non-linear sensor measurements.
Lesson: Unscented Kalman Filters. Linearize data around multiple sigma points in order to fuse highly non-linear data.
Project: Tracking with an Unscented Kalman Filter. Track an object using both radar and lidar data, fused with an unscented Kalman filter!
Partners
One of the highlights of working at Udacity is partnering with world experts to teach complex skills to anybody in the world.
In this program we are fortunate to work especially closely with autonomous vehicle engineers from Mercedes-Benz. They appear throughout the Nanodegree Program, often as the primary instructors, and sometimes simply offering their expertise and context on any other topic.
MathWorks has also proven terrific partners by offering our students free educational licenses for MATLAB. The radar course in this program is taught primarily in MATLAB and leverages several of their newest and most advanced toolboxes.
Reflection
There is a quote, from a completely different context, “It took forever and then it took a night.”
That sums up how I felt building this Nanodegree Program. We spent over a year kicking around ideas for this program, starting work and stopping work, and there were times I thought it wasn’t going to happen. Then we got the right group of instructors together it came together faster than I ever imagined, and it’s beautiful.
