Self-Driving Car: Finding Lane Lines
In this project, I built a pipeline to detect lane lines using Python and OpenCV. This pipeline contains the following steps:
- Camera calibration
- Perspective transformation
- Color thresholding and Region masking
- Finding lane pixels
- Measuring lane curves and curvatures
- Display the result back to original image
1. Camera Calibration
Image distortion occurs when a camera looks at 3D objects in the real world and transforms them into 2D image; this transformation isn’t perfect. Distortion actually changes what the shape and size of these 3D objects appear to be. So, the first step of analyzing camera images, is to undo this distortion so you can get useful information out of them.
There are 2 types of distortions we are interested in: radial distortion and tangential distortion.
To calibrate our camera, we can take pictures of known shapes and correct any distortion errors. The most common object for this task is a chessboard due to its high contrast pattern.
First we need to use our camera to take a lot of chessboard images and detect all the corners in those images. This can be done using function in OpenCV cv2.findChessboardCorners().
After that, we will use cv2.calibrateCamera()to find distortion coefficients. There are three coefficients needed to correct for radial distortion: k1, k2, k3 and two for tangential distortion: p1, p2. The formula to calculating coordinates of distorted points are shown below, where r is the known distance between a point in an undistorted image and the center of image distortion, which is often the center of that image (x_c, y_c)
2. Perspective transformation
In this step, we will transform our image to bird’s eye view. It will make later steps like measuring lane curvature easier. To transform the image, we first need 4 points from the source image and 4 points from the destination image and use the function cv2.getPerspectiveTransform(). This function computes a 3x3 transformation matrix that is needed when we want transform the image by thecv2.warpPerspective() function.
3. Color Thresholding and Region Masking
HLS (hue, lightness, saturation) is an alternative representation of RGB model. HLS is cylindrical geometry, with hue, the angular dimension, is the actual color. Lightness is how much white (or black) is mixed in the color while saturation value is how much gray in the color. A saturation value of 0 indicates mostly gray while 100% luminosity (L=255) is white.
The color of lane lines are white and yellow, and both have the saturation values in a specific range. Therefore, I only pick the pixels that have a saturation value between that range. Furthermore, white color can be detected with lightness channel by filtering out all pixels that have small lightness value.
Region masking is the process of eliminating parts of the image that are less likely to contain lane lines. As we can see, the lane lines mostly appear in the bottom half of the image, therefore masking out pixels on the top-half of the image will increase our accuracy in detecting lane lines since the image now only contain pixels that are more likely to be part of the lane lines.
4. Finding lane pixels
Next step, we need to categorize which pixels are in the left lane, right lane, or neither. After that, we will find the polynomial equation that best fits all the left lane pixels and another equation that best fits all the right lane pixels.
First, I took a histogram along all the columns in the lower half of the image. In our thresholded binary image, pixels are either 0 or 1, so the two most prominent peaks in this histogram will be good indicators of the x-position of the base of the lane lines.
We can use that as a starting point to search for the lines. From that point, we can use a sliding windows moving upward in the image (further along the road) to determine where the lane lines go.
5. Finding lane curves and measuring the curvature
We’ve estimated which pixels belong to the left and right lane lines (shown in blue and red, respectively) and we’ve fit a polynomial to those pixel positions. We can use this polynomial to compute the radius of the lane curvatures and how far the vehicle is from the center of the lane.
To recall, we’ve found the equation of the lane line:
The reason this is the function of y, rather than x because the lane lines in the warped image are near vertical and may have the same x value for more than one y value.
The unit of x and y here is in pixels, but we’d like to convert it to meters to calculate the lane curvature and position of vehicle. Let’s say the distance between 2 lane lines in pixels coordinates is 700 pixels. In real life, this distance is about 12 feet or 3.7 meters. Therefore, each pixels in the horizontal direction is equivalent to 3.7/700 meters in real life. Let’s call this value mx = 3.7/700. Doing the same in vertical direction we get my = 30/720, to indicate 30 meters vertically for every 720 pixels. The conversion happens like this:
The radius of the curvature at any point x of the function x=f(y) is given as:
This can be computed easily since we only need to computer first and second order derivative of the our function.
Next, we would like to compute how far our car is from the center of the lane. This can be achieved by calculating the horizontal distance between the center of the lane and the center of the image in pixels. After that, we can multiply with the constant mx to convert to real life distance.
6. Display result back to original image
The final step is to combine the result from the previous step with the original image.
To do that, we need to revert the perspective transformation we did earlier and place the output image on top of the original image.
Conclusion
This particular pipeline works well under normal and shadowy conditions where lane lines are harder to detect. However, since this pipeline has limited amount of parameters, it’s not robust enough to perform well under extreme conditions like raining and snow.
To sum it all up, with a few algorithms put together, we can create a pipeline that can detect lane lines. First, we correct for camera distortion. Then, we transform it to a bird’s-eye view, filter out irrelevant parts of the image, and find lane pixels using “sliding windows.” Finally, we calculate the equation of the lane lanes and measure the lane curvature. For the detailed implementation, check out my project on github.
