Udacity’s Sensor Fusion Nanodegree Program!

David Silver
May 23 · 4 min read

launched yesterday! I am so happy to get this one out to students 😁

Goal

The goal of this program is to offer a much deeper dive into perception and sensor fusion than we were able to do in our core . This is a great option for students who want to develop super-advanced, cutting-edge skills for working with lidar, camera, and radar data, and fusing that data together.

The first three months of the program are brand new content and projects that we’ve never taught before. The final month, on Kalman filters, comes from our core Self-Driving Car Nanodegree Program. The course is designed to last four months for new students. Students who have already graduated the core Self-Driving Car Engineer Nanodegree Program should be able to finish this specialized Sensor Fusion Nanodegree Program in about three months.

Curriculum

Course 1: Lidar
Instructor: ,
Lesson: Introduction. View lidar point clouds with .
Lesson: Point Cloud Segmentation. Program the RANSAC algorithm to segment and remove the ground plane from a lidar point cloud.
Lesson: Clustering. Draw bounding boxes around objects (e.g. vehicles and pedestrians) by grouping points with Euclidean clustering and .
Lesson: Real Point Cloud Data. Apply segmentation and clustering to data streaming from a lidar sensor on a real self-driving car.
Lesson: Lidar Obstacle Detection Project. Filter, segment, and cluster real lidar point cloud data to detect vehicles and other objects!

Course 2: Radar
Instructor: ,
Lesson: Radar Principles. Measure an object’s range using the physical properties of radar.
Lesson: Range-Doppler Estimation. Perform a on a radar signal to create a map for object detection and velocity measurement.
Lesson: Clutter, CFAR, AoA. Filter noisy radar data in order to reduce both false positives and false negatives.
Lesson: Clustering and Tracking. Track a vehicle with the in .
Lesson: Radar Target Generation and Detection Project. Design a radar system using FMCW, signal processing, FFT, and !

Course 3: Camera
Instructor: ,
Lesson: Computer Vision. Learn how cameras capture light to form images.
Lesson: Collision Detection. Design a system to measure the time to collision (TTC) with both lidar and camera sensors.
Lesson: Tracking Image Features. Identify key points in an image and track those points across successive images, using and , in order to measure velocity.
Project: 2D Feature Tracking. Compare key point detectors to track objects across images!
Lesson: Combining Camera and Lidar. Project lidar points backward onto a camera image in order to fuse sensor modalities. Perform neural network inference on the fused data in order to track a vehicle.
Lesson: Track An Object in 3D. Combine point cloud data, computer vision, and deep learning to track a moving vehicle and estimate time to collision!

Course 4: Kalman Filters
Instructors:
, , and ,
Lesson: Sensors. Differentiate sensor modalities based on their strengths and weaknesses.
Lesson: Kalman Filters. Combine multiple sensor measurements using Kalman filters — a probabilistic tool for data fusion.
Lesson: Extended Kalman Filters. Build a Kalman filter pipeline that smoothes non-linear sensor measurements.
Lesson: Unscented Kalman Filters. Linearize data around multiple sigma points in order to fuse highly non-linear data.
Project: Tracking with an Unscented Kalman Filter. Track an object using both radar and lidar data, fused with an unscented Kalman filter!

Partners

One of the highlights of working at Udacity is partnering with world experts to teach complex skills to anybody in the world.

In this program we are fortunate to work especially closely with autonomous vehicle engineers from . They appear throughout the Nanodegree Program, often as the primary instructors, and sometimes simply offering their expertise and context on any other topic.

has also proven terrific partners by offering our students free educational licenses for . The radar course in this program is taught primarily in MATLAB and leverages several of their newest and most advanced toolboxes.

Reflection

There is a quote, from a completely different context,

That sums up how I felt building this Nanodegree Program. We spent over a year kicking around ideas for this program, starting work and stopping work, and there were times I thought it wasn’t going to happen. Then we got the right group of instructors together it came together faster than I ever imagined, and it’s beautiful.

Self-Driving Cars

A publication covering news, predictions, and opinions about self-driving cars and other autonomous vehicles.

David Silver

Written by

I love self-driving cars and I work on them at @Udacity! Learn to build self-driving cars with us: https://udacity.com/drive

Self-Driving Cars

A publication covering news, predictions, and opinions about self-driving cars and other autonomous vehicles.