Lane Detection: Brief History and the Road Ahead
At KeepTruckin, Inc., we are developing artificial intelligence (AI)-based solutions to monitor drivers’ behavior, identify unsafe driving practices, and eventually convert these patterns into scores. Lane-related actions, such as tailgating, unsafe lane changes, and cut-off tailgating, constitute an integral part of a driving assessment. We at KeepTruckin are exploring the latest tools and techniques for lane detection as part of our research. To establish a context, this blog orients you to the prevailing trends in this area, and highlights their shortcomings.
As the number of self-driving cars increases, advanced driver assistance systems (ADAS) develop in parallel. One aspect of automation that we share with these autonomous systems is road lane detection.
For KeepTruckin purposes, lane detection is critical in identifying and ensuring safe driving practices. An on-board system with this capability can alert the driver in the event of an unsafe lane change. Lane detection is also instrumental in understanding road layout and surrounding scene geometry for proper route planning.
As we discussed in our distance estimation blog, accidents in the U.S. carry a hefty penalty for the driver at fault. One of the primary scenarios leading up to accidents is tailgating; that is, the act of following a vehicle at a distance shorter than the minimal required braking distance for the speed in question. In order to measure overall safety, in addition to knowing the distance from the lead vehicle (to measure tailgating), we also need the location of nearby lanes on the road. Drivers can be scored based on their position in their lane, their frequency of lane changes, and their distance from the lead vehicle.
For example, changing lanes without using turning signals is an unsafe practice. Also unsafe is the practice of changing lanes in such a way that the new distance from the lead vehicle falls below the safety threshold. Fleet managers can use such data to monitor and subsequently educate drivers with a view to reducing accidents. So if we agree that lane detection is important, let’s review which methods have been available so far.
Dashcam Images for Lane Detection
Lane markings have been widely used as the main identifiers of lane boundaries. A forward-facing dashcam mounted inside the cabin is typically used for image acquisition. From these images, lane lines are detected using either a model-based or feature-based approach. We’ll talk about each one.
Feature-Based Approach to Lane Boundary Detection
In the feature-based approach, low-level edge features are typically used for lane boundary detection. As you can see in the figure below, low-level feature based lane line detection works well with proper thresholding of the background region.
Some advanced computer vision-based feature detection algorithms have also been used in this context to make lane marking detection more robust. But with more complex scenery (for example, the appearance of buildings in an urban street view, or shadows cast by nearby structures or plant life), low-level features no longer stand the test.
To increase the robustness of line detection against complexities introduced by nearby structures, active sensing has been used, in the form of lidars. The idea of lidars is to use the difference in reflectance of black road as opposed to white or colored lane markings, in addition to the difference in the height of the road (to identify curbs). The inherent flaw in this approach is that it fails in cases where lane markings have been effaced. Its other downside is the cost of the sensor: the high-resolution lidar required is priced over $70,000.
Model-Based Approach to Lane Boundary Detection
In addition to simple feature-based lane detection, we have also relied on the common “rules” of lane configuration we have come to expect, such as curved or straight lines remaining in parallel. Barring a few exceptions, lane lines are generally parallel. However, in an image, parallel lines meet at a central point, called a vanishing point (VP).
This point lies on a virtual horizon, and this information can be used to impart a synthetic rotation to the view of the image. This is called an inverse perspective mapping, and it provides a bird’s-eye view when applied to an image. A bird’s-eye view is the top-down view in which a camera, parallel to the road plane, vertically surveys the road. This is helpful because it removes the perspective distortion. For a given camera setup, such a mapping stays constant once estimated. Second- and third- degree quadratic curves can then be easily fitted to get the lanes in parametric form. To filter straight lines from a noisy edge map, least squares or linear Hough transform has been widely used.
Although such methods are intuitive and efficient, they cannot distinguish between non-lane edges or lines in the images (such as road curbs, multiple lane lines, shadows), nor can they handle false negatives caused by erased or inconsistent lane boundary markings. Data-driven approaches can better handle these situations.
Data-Driven Approach to Lane Boundary and Region Detection
Low-level feature and model-based lane detection, as we have witnessed, have their shortcomings. A quick conclusion is that these methods can’t account for the vast variability of road scenarios. The approaches we discussed work under certain geometric constraints and fail when they are not met. Ideally, the inherent patterns would better be learnt directly from data. Deep learning (DL) is famous for this and can digest a multitude of patterns within the data. Good evidence of this is its success in ImageNet Object Detection Challenge, in which convolutional neural networks with deep architectures have surpassed the previous state-of-the-art solutions by a significant margin.
Deep learning has been explored for lane detection in recent years and has largely filled in the gap left by other approaches.
These samples demonstrate that lane lines can still be detected, even with minimal evidence. In contrast, the feature- and model-based approaches using edge maps are heavily dependent on actual evidence visible on the road.
What accounts for this difference in performance? Deep learning learns the patterns from a bulk of images with labelled or annotated ground truth in features that are cascaded in the hidden layers of the architecture. This is referred to as supervised classification of lane markings versus background. Learning the position of the vanishing point with or without ground truth helps to generate a semblance of lane lines, even with the absence of actual lines on the road or with lines obstructed by traffic.
Because learning from data requires a large number of annotated images, several annotated datasets are now available for training DL-based networks for lane detection; these include CULane, TuSimple, CalTech, and BDD. And new datasets continue to appear on the scene, such as FiveAI.
To further improve the results, DL can also combine information about the lane region and the lane boundaries. One recent work, depicted below, has employed this scheme.
Here, the synergy between the lane boundary and the lane area that is outlined by the lane boundary is being used. The error in area prediction is used as a feedback (loss) to correct the estimate of boundary, and vice versa. In short, road geometry is being exploited within a deep network architecture.
This brings up an interesting concept: roads are constructed with a geometric structure, and around the globe this geometry stays the same, or at least similar. To some extent, this factor was leveraged by the model-based approaches using low-level features. But with the advent of deep learning on the scene, the idea of using scene geometry has been pushed into the background. Recently, another research paper (Lee et al. [6]) used vanishing point-guided data to train deep networks for lane detection and has shown some promising results; refer to the video below:
This motivates us to combine a priori knowledge about other aspects of road geometry, such as parallel lane lines, their mutual distances, their count, and their types of curvatures. Using this knowledge, we can expect to further improve the performance of DL-based lane detection techniques. Our conclusion for now is that combining deep learning with road geometry will improve our results. To find out exactly why and how, make sure not to miss our next blog on this topic.
References
[1] Gupta, Tejus et al. “Robust Lane Detection Using Multiple Features.” 2018 IEEE Intelligent Vehicles Symposium (IV) (2018): 1470–1475.
[2] S. Kammel and B. Pitzer, “Lidar-based lane marker detection and mapping,” 2008 IEEE Intelligent Vehicles Symposium, Eindhoven, 2008, pp. 1137–1142. doi: 10.1109/IVS.2008.4621318
[3] Pan, X., Shi, J., Luo, P., Wang, X., & Tang, X. (2018). Spatial As Deep: Spatial CNN for Traffic Scene Understanding. In AAAI Conference on Artificial Intelligence (pp. 7276–7283). https://doi.org/10.2522/ptj.20090377
[4] Zhang, J., Xu, Y., Ni, B., & Duan, Z. (2018). Geometric Constrained Joint Lane Segmentation and Lane Boundary Detection. In European Conference on Computer Vision (pp. 37–47). https://doi.org/10.1007/978-3-030-01246-5_30
[5] M. Bertozzi and A. Broggi. Real-time lane and obstacle detection on the GOLD system. In Proceedings of Conference on Intelligent Vehicles, pages 213–218, 1996
[6] S. Lee, J. Kim, J. S. Yoon, S. Shin, O. Bailo, N. Kim, T. H. Lee, H. S. Hong, S. H. Han, and I. S. Kweon. VPGNet: Vanishing Point Guided Network for Lane and Road Marking Detection and Recognition. In International Conference on Computer Vision (ICCV), pages 1965–1973. IEEE, 2017
