Diary of a Self-Driving Car Engineer #1: Finding Lane Lines

Entry #1 in my series “Diary of a Self-Driving Car Engineer”

7 min readApr 14, 2018

Thursday, April 12, 2018. Ready to begin project #1: Lane Line Detection.

Check out entry #2 of this series!

In this project, we build a lane line finder. Given a video stream, through Python3/Conda, the goal is to annotate the lane lines.

Submitting a Udacity project involves four steps: analyzing the rubric, writing the code, writing the writeup, and then making the submission. The rubric is given to us with specific criteria our code must meet. The code is written in a Python 3 Jupyter notebook. The writeup is just a text writeup explaining how the code meets each spec and the process of writing the code. The submission can either be a ZIP file or a link to a Github repo. For the purposes of this diary series, all my projects and submissions will be open-sourced on my Github profile and linked to at the bottom of every post.

Beginning the Project

The base repo for the project is located at https://github.com/udacity/CarND-LaneLines-P1. I recommend cloning it and following the instructions there on getting set up with the correct environment.

I forked it over to https://github.com/dhruvshah1214/CarND-LaneLines-P1. This is the public repository for my code for the first project.

I went to terminal, cloned the repo that I forked, set up the conda environment, activated it, opened the Jupyter notebook, and began writing code!

Writing the Pipeline

The goal of the project was to write the pipeline for a lane detector. The image processing functions were given to us in the notebook; we had to connect them together and turn a regular image from a camera into a processed image with lane lines annotated.

The functions given included: a grayscale converter, a canny edge detector, a gaussian blur, region of interest mask, a line-annotator function, a hough line function, and a function to overlay the line image onto the original image.

I began by writing out the pipeline. Here’s what I got:

Here’s the output of that code:

Phew! No bugs. Well that’s a first! :)

But after a closer look, there was something strange. There are a few extra, unwanted lines way out far. That’s probably because the lines are so small; because getting all the threshold tuning just right for such small detail in the 2D picture is difficult and rarely possible, lowering the top y boundary for the region of interest would filter out these unwanted lines. Also, on the left lane marker, we aren’t catching all the lane lines, so we’re going to have to adjust the mininum_line_length parameter that the hough_lines() function takes.

After changing the boundary, the img_roi variable now looks like this:

img_roi = region_of_interest(img_edges, np.array([[
     (125,imshape[0]),
     (imshape[1]/2–25, imshape[0]/2 + 50), 
     (imshape[1]/2 + 25, imshape[0]/2 + 50), 
     (imshape[1] - 70,imshape[0]) 
 ]], dtype=np.int32))

The hough_lines() parameters were changed to:

lines = hough_lines(img_roi, 1, np.pi/180, 1, 5, 1)

The new and improved output:

Much better.

The next task was to edit the draw_lines() method to draw only two lines on the image, the left lane line and the right.

I decided to distinguish lines by their slope and put the points for each line segment into four different lists, a left line x list, left line y list, and two for the right line, x’s & y’s. I’d also need to extrapolate the full line from the list of points. I wrote another method, draw_full_line() that took the image, x values, and y values, and used numpy’s polyfit() function to perform a “linear regression” of sorts and fit all the points to a 1st degree polynomial (known to some as a line!).

The code for the mothod:

def draw_full_line(img, x_list, y_list, color=[255, 0, 0], thickness=3):
    if len(x_list) == 0 or len(x_list) != len(y_list): 
        return
    
    line = np.polyfit(x_list, y_list, 1) 
    
    m = line[0]
    b = line[1]
    
    maxY = img.shape[0]
    maxX = img.shape[1]
    
    #take the lowest point on the image, and use the linear equation forumla to solve for its x
    y1 = maxY
    x1 = int((y1 - b)/m)
    
    # take an arbitrary point near the middle of the img
    y2 = int((maxY/2) + 60)
    # solve for x
    x2 = int((y2 - b)/m)
    
    cv2.line(img, (x1, y1), (x2, y2), color, thickness)

But it still didn’t look perfect. The lane lines were accurate for some frames, but were whipping around like crazy. I needed to do more parameter tuning. I looked up what each of the parameters meant and did so I could go about it a systematic way.

Parameter Tuning

Gaussian Blur

When there is an edge, the pixel intensity changes rapidly (from 0 to 255) which we want to detect. To make this easier, we should smooth over the edges. As you can see, the above images have many rough edges which causes many noisy edges to be detected.

The Gaussian Blur takes a positive, odd, integerkernel_size, which you'll need to play with to find one that works best. I tried 1, 3, 5, 9, 11, 15, 17 ( and check the edge detection (see the next section) result. The bigger the kernel_size value is, the more blurry the image becomes.

I started with 5, worked all the way up to 17, then back down to 13, and then switched over to editing other parameters, and after worked it back all the way down to 1.

The bigger kernel_size value requires more time to process. It is not noticeable with the test images but it is something to keep in mind when we process live video streams from cameras in a self-driving car. Try to minimize this and keep edge quality.

Canny Edge Detection

From Wikipedia:

it is essential to filter out the edge pixel with the weak gradient value and preserve the edge with the high gradient value. Thus two threshold values are set to clarify the different types of edge pixels, one is called high threshold value and the other is called the low threshold value. If the edge pixel’s gradient value is higher than the high threshold value, they are marked as strong edge pixels. If the edge pixel’s gradient value is smaller than the high threshold value and larger than the low threshold value, they are marked as weak edge pixels. If the pixel value is smaller than the low threshold value, they will be suppressed.

In summary:

If a pixel gradient (difference between itself and surrounding) is higher than the upper threshold, the pixel is accepted as an edge
If a pixel gradient value is below the lower threshold, then it is rejected.
If the pixel gradient is between the two thresholds, then it will be accepted only if it is connected to a pixel that is above the upper threshold.
Canny recommended a upper:lower ratio between 2:1 and 3:1.

A derivative of a function f(x) shows how fast f(x) is changing with respect to x.

A gradient is, briefly, a multivariable derivative; a gradient of a function f(x, y) is how fast f is changing with respect to x and y. For a black pixel (value 0) at (0,0) [so for the point f(0,0) = 0] right next to a white pixel (value 255) at the point (1, 1) [f(1,1) = 255] the gradient of f(x,y) near (1, 1) will be large, because the surrounding pixels are dark and the change in pixel value from its surroundings is large.

I recommend first setting the low_threshold to zero and then adjusting the high_threshold. If high_threshold is too high, you’ll find no edges. If high_threshold is too low, you’ll find too many edges. Once you find a good high_threshold, adjust the low_threshold to discard the weak edges (noises) connected to the strong edges.

Hough Lines

The HoughLinesP parameters are fairly straightforward:

rho and theta are the distance and angular resolution of our grid in Hough space. Read more on Hough Space here and Hough Transforms at OpenCV.
threshold: Only those lines are returned that get enough votes (number of intersections per grid cell).
minLineLength: Minimum line length (px). Line segments that are shorter are filtered out.
maxLineGap: Maximum allowed gap (px) between points on the same line to link them.

Video from the Final Product:

Conclusions and Further Thoughts

Lane lines aren’t really lines. Roads can swerve and curve, and along with them lane lines. Using polynomials to fit lane lines will provide much more accurate data and a better autonomous vehicle.

I also wonder if a deep learning approach can be taken to finding lane lines rather than a pure computer-vision/image-processing model. Right now, the parameter tuning was very specific to the given videos and I have a feeling it won’t hold together when more scenarios are given, like night driving, rain driving, etc.

This was an exciting and challenging first project that taught me a lot about image processing and filtering. I look forward to even more challenging projects that stretch my knowledge and lead me down the path to building a fully autonomous self-driving car that can provide a better future for all of us.

My code can be found at my Github repository here.