ID card Border detection using Emgu CV

Vinita Kasliwal
Jul 24, 2017 · 6 min read

Overview

The article talks about different ways of image processing and the technology and issues which we faced while developing this solution for a client. Consider a scenario of an end user application which processes different ID cards such as driving licenses, passports and other similar photo ID proof cards. There are many business scenario where this requirement fits such as online application for credit/debit cards, online bank forms etc. Normally in such cases, a user takes a photo of their ID cards and uploads it. More often than not, photo gets captured awkwardly resulting in a lot of white spaces on each side of the ID card. Sometimes the captured image is even skewed. Eventually, all these improper scans of ID proofs gets uploaded in the system and causes a hindrance at the time of processing of application as they don’t meet the pre-defined criterion. Therefore, in order to ensure that the images meets the pre-defined system criterion, certain level of image processing needs to be done on these images before they are uploaded.

So the requirement we had in hand was to:

  • ID card border detection automatically and the image should then be cropped
  • Remove if any skew found
  • Ensure it gets rotated or the orientation be corrected.

Image Processing

For any kind of Image processing requirement, OpenCV is one of the widely used library and is built on open source platform. Its mainly developed in C++ and in order to use it, a separate component is required to be implemented which should run independently. In order to use OpenCV in other languages there are several wrappers available.

EmguCV is one such cross platform wrapper for .NET compatible languages (C#, VB, VC++ etc). It’s an open source wrapper and can also be compiled for Xamarin and Unity applications along with usual .NET applications.

This blog will cover some of the APIs provided by EmguCV and how it can be used for border detection, image cropping and checking for skew. Emgu CV library is available as a nuget package and can be installed using following command:

Install-Package EmguCV -Version 3.1.0.1

Pre-processing

To start processing on image there are few pre-processing steps which are required to performed. The first step of pre-processing is to convert the image to greyscale and reduce the noice.

//Convert the image to grayscale and filter out the noise
UMat uimage = new UMat();
CvInvoke.CvtColor(img, uimage, ColorConversion.Bgr2Gray);

//use image pyr to remove noise
UMat pyrDown = new UMat();
CvInvoke.PyrDown(uimage, pyrDown);
CvInvoke.PyrUp(pyrDown, uimage);

Here PyrDown method downsamples the image and reduces the noise then by using PyrUp method performs up-sampling, thus enlarging the image. This improves the overall image quality as well as the size of image.

Next step for pre-processing is edge detection using canny algorithm.

UMat boundary = new UMat();
CvInvoke.Canny(uimage, boundary, cannyThreshold, cannyThresholdLinking);
CvInvoke.Canny(boundary, cannyEdges, cannyThreshold, cannyThresholdLinking);

Below is the output of this

Now that everything in image is marked with lines, it becomes easy to detect the most outer lines. Our image is now ready for edge detection.

Border detection

As we need to detect the borders of ID cards, we can assume that it’ll be rectangular. Going by this assumption there can be 2 ways of detecting borders and both can be used in conjunction.

1. Detect outer most lines

Since Canny algorithm already converted our image into collection of lines, if we can find the outermost lines then we may be able to find the border. Here we used ‘probabilistic Hough transform’ to filter out lines based on width & length. From ‘Hough transform’ we get a collection of lines, which we can iterate to find out points at extreme positions of the image

LineSegment2D[] lines = CvInvoke.HoughLinesP(cannyEdges,
1, //Distance resolution in pixel-related units
Math.PI / 180.0, //Angle resolution measured in radians.
80, //threshold
2, //min Line width
20);
if (null != lines && lines.Length > 0)
{
PointF[] pts = new PointF[]
{
new PointF(lines[0].P1.X, lines[0].P1.Y),
new PointF(lines[0].P1.X, lines[0].P1.Y),
new PointF(lines[0].P1.X, lines[0].P1.Y),
new PointF(lines[0].P1.X, lines[0].P1.Y)
};
foreach (LineSegment2D line1 in lines)
{
pts[0].X = pts[0].X < line1.P1.X ? pts[0].X : line1.P1.X;
pts[0].Y = pts[0].Y > line1.P1.Y ? pts[0].Y : line1.P1.Y;

pts[1].X = pts[1].X < line1.P1.X ? pts[1].X : line1.P1.X;
pts[1].Y = pts[1].Y < line1.P1.Y ? pts[1].Y : line1.P1.Y;

pts[2].X = pts[2].X > line1.P2.X ? pts[2].X : line1.P2.X;
pts[2].Y = pts[2].Y < line1.P2.Y ? pts[2].Y : line1.P2.Y;

pts[3].X = pts[3].X > line1.P2.X ? pts[3].X : line1.P2.X;
pts[3].Y = pts[3].Y > line1.P2.Y ? pts[3].Y : line1.P2.Y;
}
outline = CvInvoke.MinAreaRect(pts);
}

The above code snippet can give the coordinates of the lines which are at the extreme position in our image using which we can form a rectangle and that will be our border. But this technique won’t work in all the scenarios such as ID card having more prominent rectangle drawn on it or image with white background of a white ID card. These cases won’t give a correct collection of lines, thus may result in incorrect border detection. To overcome this problem we can go for another similar solution.

2. Detect outer most points

In this approach instead of finding lines, we will find objects at outer most location using ‘FindContours’. Contours are the curves joining continuous points with same color and intensity. As we have already applied Canny edge detection, all the objects are marked as lines, thus best suited for FindContours. It’s always recommended to use FindContours on binary image.

using (VectorOfVectorOfPoint contours = new VectorOfVectorOfPoint())
{
CvInvoke.FindContours(cannyEdges, contours, null, RetrType.List, ChainApproxMethod.ChainApproxSimple);
..}

Once we get the list of contours, we can filter the contours based on the area to further filter out noise.

for (int i = 0; i < count; i++)
{
using (VectorOfPoint contour = contours[i]){using (VectorOfPoint approxContour = new VectorOfPoint())
{
CvInvoke.ApproxPolyDP(contour, approxContour, CvInvoke.ArcLength(contour, true) * 0.05, true);

if (CvInvoke.ContourArea(approxContour, false) < MIN_CONTOUR_AREA) //only consider contours with area greater than 250
continue;
}var newRect = CvInvoke.MinAreaRect(contour);
var newRectVerices = newRect.GetVertices();
pts[0].X = pts[0].X < newRectVerices[0].X ? pts[0].X : newRectVerices[0].X;
pts[0].Y = pts[0].Y > newRectVerices[0].Y ? pts[0].Y : newRectVerices[0].Y;

pts[1].X = pts[1].X < newRectVerices[1].X ? pts[1].X : newRectVerices[1].X;
pts[1].Y = pts[1].Y < newRectVerices[1].Y ? pts[1].Y : newRectVerices[1].Y;

pts[2].X = pts[2].X > newRectVerices[2].X ? pts[2].X : newRectVerices[2].X;
pts[2].Y = pts[2].Y < newRectVerices[2].Y ? pts[2].Y : newRectVerices[2].Y;

pts[3].X = pts[3].X > newRectVerices[3].X ? pts[3].X : newRectVerices[3].X;
pts[3].Y = pts[3].Y > newRectVerices[3].Y ? pts[3].Y : newRectVerices[3].Y;
}}outline = CvInvoke.MinAreaRect(pts);

Using this approach, we can get coordinates of most outer points of the image using which we can create a border.

Post processing

Once border gets rightly detected, we can accordingly crop the image along the border and thus get a processed image containing only the ID card. If ID card is skewed i.e. rotated by some angle, border will also be rotated by the same angle and since we are creating object of RotatedRectangle it also contains the angle. So, when we crop the image using the rotatedRectangle it removes that angle from the image as well.

var processed = preProcessedImg.Copy(boundary);

Conclusion

Since EmguCV is compatible to be used in Xamarin, Border detection can be used in creating Apps like CamScanner which detects the border and thus allow user to crop the unnecessary part from the image.

Thanks for taking time to read the post. Please let us know if you have any comments or get in touch for any further details you need by dropping us a message here

Peritos Solutions

A startup focused on developing solutions for small - medium sized firms and solving complex business problems using technologies like Microsoft, SAP, IoT ,Cloud platform and also to create web and mobile based solutions. Partner with HR mind to help clients find the right talent

Vinita Kasliwal

Written by

ERP Consulting , Procure to Pay Specialist, A developer , Architect and everything in between

Peritos Solutions

A startup focused on developing solutions for small - medium sized firms and solving complex business problems using technologies like Microsoft, SAP, IoT ,Cloud platform and also to create web and mobile based solutions. Partner with HR mind to help clients find the right talent

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade