Road Lanes Recognition With OpenCV, Python, and iOS.
Computer Vision & Robotics A-Z Guide.
Give us a message if you’re interested in Blockchain and FinTech software development or just say Hi at Pharos Production Inc.
Or follow us on Youtube to know more about Software Architecture, Distributed Systems, Blockchain, High-load Systems, Microservices, and Enterprise Design Patterns.
This is my first assignment at Udacity Self-Driving Car Engineer Nanodegree. Udacity proposed to use Python, I like Python but I want something more practical rather than a toy project. So I decided to build the same functionality developed with Python but for iOS and with Swift and C++. And use it on a real road and probably it should recognize lanes in a realtime. We will check that later because this story is just about lanes on the image.
UPDATE: I’m extending this article already going through Term 1 assignments. We’re using Python extensively so article contains Python snippets too. Python version doesn’t have any performance improvements, it’s just a proof of concept.
You can find how to add OpenCV to your Objective-C / Swift project in my previous article here - How to install OpenCV into an iOS project
Also, we’re using next Python packages
It’s a good idea to calibrate a camera before using it. This is how we can get rid of barrel distortions. Actually, an iOS camera has a wide frustum but doesn’t have a huge distortion. That’s cool. You can read about camera calibration here How to calibrate the iOS camera using OpenCV
Lane detection pipeline looks like this:
Lane detection pipeline looks like this:
- ROI — Define ROI with crop function. It takes two arguments — image and bottom offset. All other parameters calculated based on image size and assuming that the camera is mounted in the center.
- Color correction — we can make source image better, with higher contrast and fancier histogram. We use bilateral blur, unsharpen mask and clache here.
- Color masking — We use two colors-shades of gray(from white to middle gray) and yellow. Well, for white lanes and for yellow lanes. The output is a binary mask.
- Edge detection — We use the Canny Edge Detector with automatic parameters calculated based on the grayscale version of the image.
- Detect candidate lines — We should find all possible lines which look like lanes or something similar to them, well, candidates. All detection is happening in Hough Space, then we filter out bad lines — median value with 2x standard deviation from each side. If there are no lines left, we repeat detection but with a lower value of gray.
- Linear regression and Extrapolation — We should calculate a single line for each side, extrapolate it to the bottom of the image(top in OpenCV space) and find intersection point.
DRAWING
A list of useful drawing functions with fancy Greek letters.
- Weighted image — simple alpha blending
- Stack images — stack two images side by side
- Show image — well, show this image!
- Show gray image — this is for one channel images
- Plot start — defines plot size
- Plot lines — to plot lines(x, y)
- Plot function — to plot function when you have xs and f. Y = f(x) if you don’t know :)
ROI
The first step is to define a Region Of Interest. We should mask a part of the road where all things happen and also cut it from the image to avoid additional computations. We should crop top and bottom of the image, we don’t care about what is too far away and what is just in 1–2 meters in front of the car.
We do not crop an image in python version because we don’t care about performance here, just applied a mask.
ROI for current test image has next coordinates defined relatively to reference image size. We assume that a camera is mounted in the center of the windshield.
- Region Of Interest — This is exactly what applies a mask.
- Crop ROI — Just a wrapper function
- Crop by Ref, Crop— we give a reference frame and reference lengths to crop out a region of interest. Horizontally from a center of the image. Vertically from top and bottom.
COLOR MASKING
There is an issue with yellow lanes. They are almost invisible in a luma channel. We should add a color threshold to mask out white and yellow lanes.
Notice, we plug white value from outside. We run color threshold recursively each time decreasing white value if there is no useful result.
- Binary HSV mask and Binary Gray mask — actually the same and it’s better to refactor these two functions into one. We apply lower and upper color bounds, cv2.inRange returns all values inside this bounds.
- Binary mask apply — returns binary mask, values are 0 or 255
- Binary mask applies color — same but not a binary, but RGB. Just badly named.
- Other two functions — wrappers around masking stuff.
COLOR CORRECTION
Then we need to color correct an image. Current snippet contains slightly more functions than we are using. In our pipeline, we pass an image through bilateral blur — to blur noise and preserve edges — better for us than a gaussian blur. Than unsharpen mask to make blurred edges sharper. And finally, clache to tune image histogram. We can use equalizeHist from OpenCV, but clache works better for us here.
- Equalize Histogram — Color correction based on the image histogram. Applied in YUV space.
- Clache — similar to equalize histogram, but better for us. You can find more here — http://bit.ly/2bSSPqF
- Canny — Canny Edge Detector wrapper
- Gaussian Blur — Don’t forget to use even kernel 3x3, 5x5 etc
- Biliteral — Biliteral blur is much better when you want to preserve edges. It smashes all noise but leaves edges almost untouched.
- Unsharpen Mask — It makes the image sharper. A name has been taken historically.
EDGE DETECTION
Next, let’s find features we should recognize. We can simply convert it to grayscale but better to say — to luma color space. Because actually, this is how lines on the road work. They are bright. You can perfectly see them on a red channel now. We will work with a one-channel image to avoid additional computations. Colorful images here is just for nice looking pictures. BGR to luma-chroma(YCrCb) color space. We don’t care about Cr and Cb. Our goal is Y = 0.299*R + 0.587*G + 0.114*B. This transformation is equal to CV_BGR2GRAY.
BTW, why BGR and not RGB? That’s a tradition. BGR order was popular when first digital cameras and OpenCV version 1 was born.
Next, let’s get rid of all parts with low intensity. Let’s rise gamma gain and threshold the image.
Note that horizontal line on top? That’s a car. White car. Cars are luma intensive too. We will not change gamma for now in python project.
Now let’s detect edges using Canny Edge Detector. It’s also known as Optimal detector. It satisfied 3 criteria:
- L0w error rate — good detection of only exist edges;
- Good localization — a distance between image pixels and an edge is minimized;
- Minimal response — one line per edge.
More theory is here: Canny Edge Detector
We leave aperture size(means kernel) equal to 3 for canny’s Sobel operator.
Notice, we use automatically generated values for canny. This is better than hardcoded values.
LANES DETECTION
Next, we run all available edges through Hough transform to find lines according to transformational parameters
We’re using probabilistic Hough transform here. You can find more theory here: Hough transform
- Grid-scale = 2
- Angle = 1 degree
- Number of votes = 50
- Min line length in px = 20
- Min gap between lines in px = 100
Also, we have criteria to remove outlier lines by angle. We define min and max slopes(in radians here). Lanes are in between. We can split them into 2 groups — one for the left and one for the right part.
Then we should find median lines for two group of points
And finally get a vanishing point
The result is a couple of lines from left and a couple of lines from the right. We should find an average line in both groups. The plot looks like this — in pixels space.
- Red lines — a right group of lines
- Blue lines — left a group of lines
- Green lines — lanes
PRETTY WELL!
To make detection more stable and remove jiggling we should cache N results and take an average on it. N = 3 for us.
What’s Next?
It’s enough theory for now and time to build an application for iOS. Once we will finish with algorithm implementation using C++, we can see all bottlenecks and way to improve.
