Advanced Techniques for Lane Finding (Self Driving Cars)

Advance Concepts like Gradient threshold, Color spaces and thresholding for the working of Self Driving Cars

Prateek Sawhney
Sep 13 · 8 min read

Advanced Lane Detection Project of Self Driving Car Engineer Nanodegree which includes advanced image processing to detect lanes irrespective of the road texture, brightness, contrast, curves etc. Using Image warping and sliding window approach to find and plot the lane lines. Also determined the real curvature of the lane and vehicle position with respect to center.

Detected Lane Area alongwith Radius of Curvature and Central Offset (Image by author)

We can use Canny edge detection to find pixels that were likely to be part of a line in an image. Canny is great at finding all possible lines in an image, but for lane detection, this gave us a lot of edges on scenery, and cars, and other objects that we ended up discarding as shown in the image below:

Binary Combo Example Image (Image by author)

Realistically, with lane finding, we know ahead of time that the lines we are looking for are close to vertical.

So, how can we take advantage of that fact?

Well, we can use gradients in a smarter way to detect steep edges that are more likely to be lanes in the first place. With Canny, we actually taking a derivative with respect to X and Y in the process of finding edges.

We should convert our road images to grayscale, before detecting edges. But in making this conversion, we lose valuable color information. For example, in specific images, when we convert to grayscale, some regions almost disappears. In this article, we’ll investigate color spaces, which give us more information about an image than grayscale alone. And we’ll see, for example, that when we switched to another color space for a specific image, then we can get the disappeared section back from that image. Now, let see how that works.

For the road images, we know that they’re all composed of red, green, and blue values or RGB. And in the previous medium article, we’ve used some combination of masking and color thresholds on these RGB values to pick out bright white lane pixels.

Binary Combo Image (Image by author)

And this lane detection can work well alongside gradient detection which relies on grayscale intensity measurements. However, RGB thresholding doesn’t work that well in images that include varying light conditions or when lanes are a different color like yellow. We can break any road image down into it’s separate RGB components which are often called channels.

Separate Images of R,G and B Channels (Image by author)

The brighter pixels indicate higher values of red, green, or blue, respectively. There are many other ways to represent the colors in an image besides just composed of red, green and blue values. These different color representations are often called color spaces. RGB is red, green, blue color space. We can think of this as a 3D space where any color can be represented by a 3D coordinate of R, G, and B values as shown in the image below:

3D RGB Color Space (Image by author)

There’s also HSV color space, for hue, saturation, and value. And there’s HLS, for hue, lightness, and saturation. These are some of the most commonly used color spaces in image analysis. For both of these, H has a range from 0 to 179 for degrees around the cylindrical color space.

HLS Binary Image (Image by author)
Separate Images of H,L and S Channels (Image by author)
  1. Computing the camera calibration matrix and distortion coefficients given a set of chessboard images. (9x6).
  2. Applying distortion correction to raw images.
  3. Using color transforms, gradients, etc., to create a thresholded binary image.
  4. Applying perspective transform to rectify binary image (“birds-eye view”) to get a warped image.
  5. Detecting lane pixels and fit to find the lane boundary.
  6. Determining the real curvature of the lane and vehicle position with respect to center.
  7. Warping the detected lane boundaries back onto the original image and output visual display of the lane boundaries and numerical estimation of lane curvature and vehicle position.

The first step in the pipeline is to undistort the camera. Some images of a 9x6 chessboard are given and are distorted. Our task is to find the Chessboard corners an plot them. For this, after loading the images we calibrate the camera. Open CV functions like findChessboardCorners(), drawChessboardCorners() and calibrateCamera() help us do this as shown in the image below:

Distortion Corrected Calibrated Image (Image by author)

Detecting edges around trees or cars is okay because these lines can be mostly filtered out by applying a mask to the image and essentially cropping out the area outside of the lane lines. It’s most important that we reliably detect different colors of lane lines under varying degrees of daylight and shadow.

So, that our self driving car does not become blind in extreme daylight hours or under the shadow of a tree.

I performed gradient threshold and color threshold individually and then created a binary combination of these two images to map out where either the color or gradient thresholds were met called the combined_binary in the code.

Original Image (Image by author)
Thresholded Binary Image (Image by author)

Perspective Transform is the Bird’s eye view for Lane images. We want to look at the lanes from the top and have a clear picture about their curves. Implementing Perspective Transform was the most interesting one for me. Also, made a function warper(img, src, dst) which takes in the Binary Warped Image and return the perspective transform using cv2.getPerspectiveTransform(src, dst) and cv2.warpPerspective(img, M, img_size, flags=cv2.INTER_NEAREST). The results are shown below:

Perspective Transform Example 1 (Image by author)
Perspective Transform Example 2 (Image by author)

Once I got the Perspective Transform of the binary warped images, I first used the sliding window method to plot the lane lines and fitted a polynomial using fit_polynomial(img) function.

Sliding Window Example 1 (Image by author)
Sliding Window Example 2 (Image by author)

Later on, I used the Search from prior technique and fitted a more accurate polynomial through my perspective transformed images using search_around_poly(image) funtion. Proper markings are there in the code to indicate each and every step.

Search from Prior Example 1 (Image by author)
Search from Prior Example 2 (Image by author)

For calculating the radius of curvature and the position of the vehicle with respect to center, I made a function called radius_and_offset(warped_image) which returns curvature_string and offset. Used left_lane_inds and right_lane_inds for performing the task. Used function fit_poly(image.shape, leftx, lefty, rightx, righty) which returns left_fitx, right_fitx, ploty to calcuate the real radius of curvature and offset.

After implementing all the steps, it’s time to create the pipeline for one image. Created a function process_image() as the main pipeline function. Also, I put the Radius of Curvature and Center Offset on the final image using cv2.putText() function. The result is shown below:

Final Image (Image by author)
  1. Computer Vision Fundamentals — Self Driving Cars (Finding Lane Lines)

2. Introduction to Neural Networks For Self Driving Cars (Foundational Concepts Part — 1)

3. Introduction to Neural Networks For Self Driving Cars (Foundational Concepts Part — 2)

4. Introduction to Deep Learning for Self Driving Cars (Part — 1)

5. Introduction to Deep Learning for Self Driving Cars (Part — 2)

6. Introduction to Convolutional Neural Networks for Self Driving Cars

7. Introduction to Keras and Transfer Learning for Self Driving Cars

8. Computer Vision and Camera Calibration for Self Driving Cars

Geek Culture

Proud to geek out. Follow to join our +1.5M monthly readers.