A Brief History of Edge Detection

9 min readDec 22, 2019

I think we all agree that Images are beautiful. They are a treat to click, manipulate and extract features from. My entire career as a designer and image processing engineer exists because I liked fooling around with images in MS Paint.

Speaking of features

If you are reading this blog must be either of two distinct scenarios that led you here.

I held you at gunpoint and forced you to read this.
You are interested in image processing even if it’s a tiny bit.

Well if you are the latter ones, then you must have come across the task of extracting features from an image, and one of the primary features everyone keeps talking about is Edges.

So what is an Edge?

An edge in an image is a boundary or contour at which a significant change occurs in some physical aspect of an image, such as the surface reflectance, illumination or the distances of the visible surfaces from the viewer.

Let us understand what that means, with an example:

Finding edges from a smooth and a regular image.

As we can see, the smooth image with a mostly uniform distribution of intensity finds no edges when passed through the edge detection algorithm. However, a regular image with more or less sharp discontinuities in its pixel values yields some discrete edges. The code that has been used to find the edges can be found here.

So, why do we need edges?

It is frequently the first step in recovering information from an image.
Edges typically occur at the boundary of two regions in an image.
They can be used to segment regions, identify objects and even detect faces.

Hence to sum it all up…

An edge in an image is a significant local change in the image intensity,
usually associated with a discontinuity in either the image intensity or the
first derivative of the image intensity.

Detecting Edges

The task of detecting an edge is essentially that of finding significant local changes in an image.

An image can be considered as an array where the values of the array have been sampled from a continuous function of image intensity.

A gradient is a measure of the change in a function. Hence significant local changes in an image can be detected by approximating the gradient of an image. It must also be noted that the gradient in a two-dimensional space is the equivalent of the first derivative and can be expressed as:

Where the magnitude of the edge can be written as:

And the direction of the gradient can be written as:

However, for digital images, we need not worry about calculating derivatives. A gradient of an image can be effectively expressed by finding differences.

Let’s have a look:

These differences too can be easily transformed into kernels that assist in convolution operations. If you are new to convolutions or kernel operations do visit this website for some uber-cool insights.

Nuff Said! Where ’Em Edges at?

Before we dive right into algorithms for detecting edges we must first briefly understand the fundamentals steps involved in these algorithms.

Filtering: Calculation of gradient does indeed give us significant local changes in the image intensity, however, it can be affected by latent noise in the image itself. To prevent this, the first step in any edge detection algorithm must be to remove the noise yet preserve the strength of the edges.
Edge Enhancement: Sometimes after filtering, it so happens that the edges and local regions lose their intensity by some measure. To counter this the edges can be enhanced or the local intensity changes, boosted, in order to regain some of the lost strength.
Detection: We need to be strict here! We’ll only be accepting those pixel orientations that qualify as an edge. Now to calculate this qualification we must set a bar, a threshold so to speak, depending on which the gradient may be classified as an edge.

A look at the Algorithms.

We will be looking at a few operators that are mostly used for edge detection before we go into the very famous Canny edge detection algorithm, and finally, a sneak peek into recent deep learning methods for computing edges from an image.

Sobel Operator

The Sobel operator is a discrete differentiation operator, which means it works the same way as we just demonstrated. Calculating the approximate of the gradients from the image. Instead of finding the simple difference between the image pixels it uses the following expression to calculate the horizontal and vertical gradients:

To try out the operator for yourself go ahead and use the following snippet

import cv2
ddepth = cv2.CV_16S
window_name = ('Nobel Sobel')# reading the smooth image and converting it into grayscale
gray = cv2.cvtColor(cv2.imread('<your_image>.jpg'),cv2.COLOR_BGR2GRAY)
# calculating edges from the image
# Gradient-X
grad_x = cv2.Sobel(gray, ddepth, 1, 0, ksize=3, scale=1, delta=0, borderType=cv2.BORDER_DEFAULT)
# Gradient-Y
grad_y = cv2.Sobel(gray, ddepth, 0, 1, ksize=3, scale=1, delta=0, borderType=cv2.BORDER_DEFAULT)
abs_grad_x = cv2.convertScaleAbs(grad_x)
abs_grad_y = cv2.convertScaleAbs(grad_y)
grad = cv2.addWeighted(abs_grad_x, 0.5, abs_grad_y, 0.5, 0)cv2.imshow(window_name,grad)
cv2.waitKey(0)

Scharr Operator

Scharr is also used to detect the second derivatives of an image in horizontal and vertical directions. They are more of derivative kernels rather than kernels focused on keeping symmetry.

For trying out the operator use the following code:

import cv2ddepth = cv2.CV_16S
window_name = ('A Scharr Affair')
# reading the smooth image and converting it into RGB color space
gray = cv2.cvtColor(cv2.imread('<your_image>.jpg'),cv2.COLOR_BGR2GRAY)
# calculating edges from the image
# Gradient-x
grad_x = cv2.Scharr(gray, ddepth, 0,1)
# Gradient-Y
grad_y = cv2.Scharr(gray, ddepth, 0, 1)
abs_grad_x = cv2.convertScaleAbs(grad_x)
abs_grad_y = cv2.convertScaleAbs(grad_y)
grad = cv2.addWeighted(abs_grad_x, 0.5, abs_grad_y, 0.5, 0)cv2.imshow(window_name,grad)
cv2.waitKey(0)

Now let’s bring out the Big Guns!

The Canny Edge detection algorithm employs the following steps to detect or compute edges from a given image.

Noise Reduction: A gaussian filter is used to remove the noise from the image.
Calculating Gradients: Finding image intensity from gradients.
Non-Maximum Suppression: For this, at every pixel, the pixel is checked if it is a local maximum in its neighborhood in the direction of the gradient, otherwise the edge is suppressed.
Hysteresis Thresholding: This is for classifying whether the computed edge is a sure-edge or not. For this a band of intensity gradients is taken, if the edge lies between the band and it’s upper value then it’s a sure-edge, otherwise, it is discarded.

The following is the code snippet for calculating the edge using the Canny edge detection algorithm. Go ahead, try it for yourself :)

import cv2window_name = ('Canny All The Way')
# reading the smooth image and converting it into RGB color space
gray = cv2.cvtColor(cv2.imread('<your_image>.jpg'),cv2.COLOR_BGR2GRAY)
edge = cv2.Canny(gray, 100, 250)cv2.imshow(window_name,edge)
cv2.waitKey(0)

However as you must have noticed, all is not well. There are two parameters (the upper and lower values for hysteresis thresholding a.k.a the band), tweaking these two parameters gives you better results on your image.

The effective and elegant solution to this problem has been implemented by Dr. Adrian Rosebrock of pyimagesearch. Do check out the algorithm here.

The importance of edge detection in image processing has led to the publication of many beautiful algorithms, some of which are listed here:

Let us go deeper now!

While mentioning these novel algorithms proposed in the past couple of decades for effective edge detection, I have purposefully left out one particular method.

Yes! You guessed it right, I am talking about algorithms that involve deep learning. They have been the buzzword of this era and rightfully so. Applications of deep learning algorithms to image processing have been quite famous lately. If you are new to deep learning I recommend you to go through this amazing playlist by 3 Blue 1 Brown. If you have some experience with deep learning algorithms and want to know exactly how they help us so much in image processing do go through this repository that I created with my friend Ayush Thakur.

Holistically nested Edge Detection

Before we see how a deep learning model is used for edge detection let us first understand the shortcomings of popular methods such as Canny.

Canny edge detection algorithm did a great job of finding local discontinuities, however, it could not detect edges semantically and that affects the accuracy of the detected edges.

Due to this shortcoming, a technique called Holisitically Nested Edge Detection (HED) was proposed. It uses a trimmed VGG like convolutional neural network for an image to image prediction task.

Network architecture from the original paper.

It uses the outputs of intermediate layers. The output of all 5 convolutional layers is fused to generate the final predictions. Thus the network is effectively hunting for features at different scales because the feature map generated at each layer is of different sizes.

Find the complete code for HED with Python and OpenCV over here.

Evaluating Performance

No good algorithm can be patted and congratulated before evaluating its performance. An edge computed image cannot be looked at subjectively and judged. There must be some agreed-upon objective metric of performance. It has been observed that there can be three types of error in edge detection algorithms:

Missing valid edges.
Errors in localizing edges.
Classification of noise as edges.

Let us take the example of Pratt’s figure of merit.

Some other evaluation metrics involve Peak Signal to Noise Ratio and Mean Squared Error that treat edge detection as a typical signal processing problem.

Summary

As we journeyed through the various phases of edge detection and in extension image processing itself, we discover that the shift in mathematics hasn’t been too wide. We realize that most architectures that we rely upon heavily for image processing tasks today have some common roots with fundamental building blocks of the topic. The methods have changed and are changing continuously. We don’t know our kernels in and out anymore. Yet the challenges still prevail. Edge detection has been one of the most popular research areas since the early days of computer vision. Though a number of algorithms have been published, there is no well-defined metric for selecting a particular algorithm for an application. Even in this age, where image processing has been revolutionized by Neural Nets, edge detection manages to remain as imperative and relevant a challenge as ever.

So, what will your solution be?