Geek Culture
Published in

Geek Culture

Advanced Techniques for Lane Finding (Self Driving Cars)

Advance Concepts like Gradient threshold, Color spaces and thresholding for the working of Self Driving Cars

Advanced Lane Detection Project of Self Driving Car Engineer Nanodegree which includes advanced image processing to detect lanes irrespective of the road texture, brightness, contrast, curves etc. Using Image warping and sliding window approach to find and plot the lane lines. Also determined the real curvature of the lane and vehicle position with respect to center.

Detected Lane Area alongwith Radius of Curvature and Central Offset (Image by author)

Introduction to Gradient Threshold

We can use Canny edge detection to find pixels that were likely to be part of a line in an image. Canny is great at finding all possible lines in an image, but for lane detection, this gave us a lot of edges on scenery, and cars, and other objects that we ended up discarding as shown in the image below:

Binary Combo Example Image (Image by author)

Realistically, with lane finding, we know ahead of time that the lines we are looking for are close to vertical.

So, how can we take advantage of that fact?

Well, we can use gradients in a smarter way to detect steep edges that are more likely to be lanes in the first place. With Canny, we actually taking a derivative with respect to X and Y in the process of finding edges.

Introduction to Color Spaces

We should convert our road images to grayscale, before detecting edges. But in making this conversion, we lose valuable color information. For example, in specific images, when we convert to grayscale, some regions almost disappears. In this article, we’ll investigate color spaces, which give us more information about an image than grayscale alone. And we’ll see, for example, that when we switched to another color space for a specific image, then we can get the disappeared section back from that image. Now, let see how that works.

Color Spaces and Thresholding

For the road images, we know that they’re all composed of red, green, and blue values or RGB. And in the previous medium article, we’ve used some combination of masking and color thresholds on these RGB values to pick out bright white lane pixels.

Binary Combo Image (Image by author)

And this lane detection can work well alongside gradient detection which relies on grayscale intensity measurements. However, RGB thresholding doesn’t work that well in images that include varying light conditions or when lanes are a different color like yellow. We can break any road image down into it’s separate RGB components which are often called channels.

Separate Images of R,G and B Channels (Image by author)

The brighter pixels indicate higher values of red, green, or blue, respectively. There are many other ways to represent the colors in an image besides just composed of red, green and blue values. These different color representations are often called color spaces. RGB is red, green, blue color space. We can think of this as a 3D space where any color can be represented by a 3D coordinate of R, G, and B values as shown in the image below:

3D RGB Color Space (Image by author)

There’s also HSV color space, for hue, saturation, and value. And there’s HLS, for hue, lightness, and saturation. These are some of the most commonly used color spaces in image analysis. For both of these, H has a range from 0 to 179 for degrees around the cylindrical color space.

HLS Binary Image (Image by author)
Separate Images of H,L and S Channels (Image by author)

The Steps Involved

  1. Computing the camera calibration matrix and distortion coefficients given a set of chessboard images. (9x6).
  2. Applying distortion correction to raw images.
  3. Using color transforms, gradients, etc., to create a thresholded binary image.
  4. Applying perspective transform to rectify binary image (“birds-eye view”) to get a warped image.
  5. Detecting lane pixels and fit to find the lane boundary.
  6. Determining the real curvature of the lane and vehicle position with respect to center.
  7. Warping the detected lane boundaries back onto the original image and output visual display of the lane boundaries and numerical estimation of lane curvature and vehicle position.

Camera Calibration

The first step in the pipeline is to undistort the camera. Some images of a 9x6 chessboard are given and are distorted. Our task is to find the Chessboard corners an plot them. For this, after loading the images we calibrate the camera. Open CV functions like findChessboardCorners(), drawChessboardCorners() and calibrateCamera() help us do this as shown in the image below:

Distortion Corrected Calibrated Image (Image by author)

Thresholded Binary Image

Detecting edges around trees or cars is okay because these lines can be mostly filtered out by applying a mask to the image and essentially cropping out the area outside of the lane lines. It’s most important that we reliably detect different colors of lane lines under varying degrees of daylight and shadow.

So, that our self driving car does not become blind in extreme daylight hours or under the shadow of a tree.

I performed gradient threshold and color threshold individually and then created a binary combination of these two images to map out where either the color or gradient thresholds were met called the combined_binary in the code.

Original Image (Image by author)
Thresholded Binary Image (Image by author)

Perspective Tranform

Perspective Transform is the Bird’s eye view for Lane images. We want to look at the lanes from the top and have a clear picture about their curves. Implementing Perspective Transform was the most interesting one for me. Also, made a function warper(img, src, dst) which takes in the Binary Warped Image and return the perspective transform using cv2.getPerspectiveTransform(src, dst) and cv2.warpPerspective(img, M, img_size, flags=cv2.INTER_NEAREST). The results are shown below:

Perspective Transform Example 1 (Image by author)
Perspective Transform Example 2 (Image by author)

Sliding Window — Fitting a Polynomial

Once I got the Perspective Transform of the binary warped images, I first used the sliding window method to plot the lane lines and fitted a polynomial using fit_polynomial(img) function.

Sliding Window Example 1 (Image by author)
Sliding Window Example 2 (Image by author)

Search from Prior Technique

Later on, I used the Search from prior technique and fitted a more accurate polynomial through my perspective transformed images using search_around_poly(image) funtion. Proper markings are there in the code to indicate each and every step.

Search from Prior Example 1 (Image by author)
Search from Prior Example 2 (Image by author)

Radius of Curvature and Central Offset

For calculating the radius of curvature and the position of the vehicle with respect to center, I made a function called radius_and_offset(warped_image) which returns curvature_string and offset. Used left_lane_inds and right_lane_inds for performing the task. Used function fit_poly(image.shape, leftx, lefty, rightx, righty) which returns left_fitx, right_fitx, ploty to calcuate the real radius of curvature and offset.

Final Results

After implementing all the steps, it’s time to create the pipeline for one image. Created a function process_image() as the main pipeline function. Also, I put the Radius of Curvature and Center Offset on the final image using cv2.putText() function. The result is shown below:

Final Image (Image by author)

References

  1. Computer Vision Fundamentals — Self Driving Cars (Finding Lane Lines)

2. Introduction to Neural Networks For Self Driving Cars (Foundational Concepts Part — 1)

3. Introduction to Neural Networks For Self Driving Cars (Foundational Concepts Part — 2)

4. Introduction to Deep Learning for Self Driving Cars (Part — 1)

5. Introduction to Deep Learning for Self Driving Cars (Part — 2)

6. Introduction to Convolutional Neural Networks for Self Driving Cars

7. Introduction to Keras and Transfer Learning for Self Driving Cars

8. Computer Vision and Camera Calibration for Self Driving Cars

--

--

--

A new tech publication by Start it up (https://medium.com/swlh).

Recommended from Medium

Word2vec in 2 minutes

Using Lime for Interpreting NLP

[Paper Reading] A CLOSER LOOK AT FEW-SHOT CLASSIFICATION

Deep NLP with Aerin Kim

How we solved multilabel classification and segmentation with very little data - PART 1

Automated Lip Reading : Simplified

Tensorflow 2.0 — from preprocessing to serving (Part 1)

APS component failure classification in Scania Trucks

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Prateek Sawhney

Prateek Sawhney

AI Engineer at DPS, Germany | 1 Day Intern @Lenovo | Explore ML Facilitator at Google | HackWithInfy Finalist’19 at Infosys | GCI Mentor @TensorFlow | MAIT, IPU

More from Medium

Getting started with ROS2 — Part 1

Monocular Depth Estimation in Python using Monodepth2 and Manydepth

Early Sensor Fusion and Object Detection for Self-Driving Cars

The Big Picture: Understanding What ROS Is For