Building a simple lane detection iOS app using OpenCV

Anurag Ajwani
Apr 29 · 12 min read

Getting started

Create SimpleLaneDetection project

Select Single View App template
Name the project SimpleLaneDetection

Process frames from the back camera

import AVFoundation
import UIKit
import AVFoundation
class ViewController: UIViewController { private var captureSession: AVCaptureSession = AVCaptureSession()
...
private func addCameraInput() {
guard let device = AVCaptureDevice.DiscoverySession(
deviceTypes: [.builtInWideAngleCamera, .builtInDualCamera, .builtInTrueDepthCamera],
mediaType: .video,
position: .back).devices.first else {
fatalError("No back camera device found, please make sure to run SimpleLaneDetection in an iOS device and not a simulator")
}
let cameraInput = try! AVCaptureDeviceInput(device: device)
self.captureSession.addInput(cameraInput)
}
override func viewDidLoad() {
super.viewDidLoad()
self.addCameraInput() // add this line
}
Info.plist with camera usage description
class ViewController: UIViewController, AVCaptureVideoDataOutputSampleBufferDelegate {
...
}
func captureOutput(
_ output: AVCaptureOutput,
didOutput sampleBuffer: CMSampleBuffer,
from connection: AVCaptureConnection) {
// here we can process the frame
print("did receive frame")
}
private let videoDataOutput = AVCaptureVideoDataOutput()
private func getFrames() {
videoDataOutput.videoSettings = [(kCVPixelBufferPixelFormatTypeKey as NSString) : NSNumber(value: kCVPixelFormatType_32BGRA)] as [String : Any]
videoDataOutput.alwaysDiscardsLateVideoFrames = true
videoDataOutput.setSampleBufferDelegate(self, queue: DispatchQueue(label: "camera.frame.processing.queue"))
self.captureSession.addOutput(videoDataOutput)
guard let connection = self.videoDataOutput.connection(with: AVMediaType.video),
connection.isVideoOrientationSupported else { return }
connection.videoOrientation = .portrait
}
override func viewDidLoad() {
super.viewDidLoad()
self.addCameraInput()
self.getFrames()
self.captureSession.startRunning()
}
Console output when processing frames

Import OpenCV to the project

Drag and drop opencv2.framework
Adding opencv2.framework to project options

Insert lane detection algorithm

Add new file to app using C++ File template
Name the file LaneDetector, check header file creation checkbox
LaneDetector.hpp
LaneDetector.cpp

Consume lane detection algorithm using Swift

Create header file
Create Objective-C file
Add an extra “m” to the file extension of LaneDetectorBridge.m
LaneDetectorBridge.mm
#import "LaneDetectorBridge.h"
guard let  imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else { return }CVPixelBufferLockBaseAddress(imageBuffer, CVPixelBufferLockFlags.readOnly)let baseAddress = CVPixelBufferGetBaseAddress(imageBuffer)
let bytesPerRow = CVPixelBufferGetBytesPerRow(imageBuffer)
let width = CVPixelBufferGetWidth(imageBuffer)
let height = CVPixelBufferGetHeight(imageBuffer)
let colorSpace = CGColorSpaceCreateDeviceRGB()var bitmapInfo: UInt32 = CGBitmapInfo.byteOrder32Little.rawValue
bitmapInfo |= CGImageAlphaInfo.premultipliedFirst.rawValue & CGBitmapInfo.alphaInfoMask.rawValue
let context = CGContext(data: baseAddress, width: width, height: height, bitsPerComponent: 8, bytesPerRow: bytesPerRow, space: colorSpace, bitmapInfo: bitmapInfo)guard let quartzImage = context?.makeImage() else { return }
CVPixelBufferUnlockBaseAddress(imageBuffer, CVPixelBufferLockFlags.readOnly)
let image = UIImage(cgImage: quartzImage)
let imageWithLaneOverlay = LaneDetectorBridge().detectLane(in: image)

Display lane detection results

Open object library
Search for UIImageView
Drag and drop Image View
Hold control key and drag the UIImageView onto the blank canvas
Layout options pop up menu
Control + drag and drop UIImageView into ViewController class
Reference configuration pop up
DispatchQueue.main.async {
self.imageView.image = imageWithLaneOverlay
}

Summary

Final notes

Onfido Tech

Stories from Design, Engineering, Product and Research at Onfido

Thanks to Ricardo Freitas and Kerem Gunduz.

Anurag Ajwani

Written by

iOS Engineer @ Onfido

Onfido Tech

Stories from Design, Engineering, Product and Research at Onfido