Measuring the size of objects in an image with OpenCV Python to assess the damage

Autor artykułu: Piotr Winkler

Today I would like to tell you how, using the OpenCV — Python library, I dealt with a task that consisted of measuring objects visible in photos, while taking into account the perspective of the photo. It was supposed to determine the size of damages such as scratches or flooding.

In short, using OpenCV, we can measure objects contained in photos. OpenCV, a comprehensive Python library for computer vision, has provided us with a rich set of image processing and analysis functions, including perspective correction. In the example shown below, our algorithm, in addition to measuring objects visible in photos, also corrects perspective.

These capabilities of the library can be used for various purposes, e.g. in the insurance industry to assess damage when claiming compensation. In this article, we focus on assessing things like scratches and other damage (caused by, for example, flooding) because OpenCV allows us to precisely determine the extent of the damage visible in the photo. These functions even work on photos taken from difficult angles. Reliable calculation of actual dimensions was possible by mapping pixels to millimeters. In the case shown below it worked on a test number of 40 different photos of various elements, and these photos were taken from different perspectives.

What is OpenCV Python?

OpenCV is a Python library for computer vision that allows you to perform image processing and computer vision tasks. It has an API in many different languages and it makes various functions available, functions used for working with images, such as object detection, face recognition, and tracking.

To show you exactly how it works, I created a simple scheme.

How to read an image in Python OpenCV

As you can see in the picture, the assumption was that the insurance agent would mark the application. Mark 4 points of the reference object (it could be a piece of paper or a credit card). Then they’d mark the damage (the scratch) with two points. Based on this information, the system will be able to show that the scratch is up to 130 mm. We created an algorithm that manages these operations, and added OpenCV to it in Python.

The algorithm first takes the points of the reference objects and organizes them in a specified order, because many algorithms for images require the points to be in order.

Then, the main thing the algorithm does is correct the perspective.

Image perspective correction — OpenCV

Let’s say the floor is flooded, it’s very extensive, and we can’t take a photo parallel to this damage. The insurance agent would take a photo at an angle. This angle could harm the real results of the algorithm. That’s why we conducted a correction of perspective, which is also available in the OpenCV library.

Then, when we had a corrected image, we defined the number of millilitres for each pixel of the image. And knowing how it looks, knowing what the length of the damage is in pixels, we were able to state the actual size of the damage. We prepared a simple demo for this presentation, to test it easily.

And it looks like this:

We mark the corners of the reference object and the size of the damage too. And here I wanted to show you how the perspective correction algorithm works, what it does in this case. Knowing that the object is a rectangle, we make OpenCV cram it into a flat rectangle.

Based on this, OpenCV generates an image transformation matrix which we apply to the whole image. Then we get something like this:

from typing import List, Optional, Tuple
import cv2
import imutils
import numpy as np
from consts import REFERENCE_HEIGHT_MAP
def order_points(pts):
# sort the points based on their x-coordinates
x_sorted = pts[np.argsort(pts[:, 0]), :]
# grab the left-most and right-most points from the sorted
# x-coordinate points
left_most = x_sorted[:2, :]
right_most = x_sorted[2:, :]
# now, sort the left-most coordinates according to their
# y-coordinates so we can grab the top-left and bottom-left
# points, respectively
left_most = left_most[np.argsort(left_most[:, 1]), :]
(tl, bl) = left_most
# now do the same for the right-most coordinates
right_most = right_most[np.argsort(right_most[:, 1]), :]
(tr, br) = right_most
# return the coordinates in top-left, top-right,
# bottom-right, and bottom-left order
return np.array([tl, tr, br, bl], dtype="float32")
def calculate_distance(p1, p2):
"""Calculate Euclidean distance - p1 and p2 in format (x1,y1) and (x2,y2) tuples"""
dis = ((p2[0] - p1[0]) ** 2 + (p2[1] - p1[1]) ** 2) ** 0.5
return dis
def calculate_real_size(points_to_calculate: List[Tuple[int, int]], reference_points: List[Tuple[int, int]], image,
reference_obj_type="A4") -> Optional[float]:
"""
Calculates real distance between 2 points on image, based on reference object coordinates
"""
if len(reference_points) != 4:
print("Reference object should be a rectangular. Please pass 4 reference points (vertices)")
return
if len(points_to_calculate) != 2:
print("Calculated size should consist of just 2 points!")
return
# first order the points correctly - it is required by OpenCV algorithm
reference_points = order_points(np.array(reference_points))
tl, tr, br, bl = reference_points
# compute the widths and heights of the reference object (in pixels)
width_a = calculate_distance(br, bl)
width_b = calculate_distance(tr, tl)
height_a = calculate_distance(tr, br)
height_b = calculate_distance(tl, bl)
# take the maximum of the width and height values to reach our final dimensions
max_width = max(int(width_a), int(width_b))
max_height = max(int(height_a), int(height_b))
# compute perspective transform matrix "m" based on assumption that reference object is A4 paper sheet or a
# credit card (and should be a rectangle in the image)
dst = np.array([
[0, 0],
[max_width - 1, 0],
[max_width - 1, max_height - 1],
[0, max_height - 1]], dtype="float32")
cv2.namedWindow('warped', cv2.WINDOW_NORMAL)
m = cv2.getPerspectiveTransform(reference_points, dst)
# this part is used only to show images - for presentation purposes
# ================================================================================================================
warp = cv2.warpPerspective(image, m, (max_width, max_height))
cv2.imshow('warped', warp)
cv2.waitKey(0)
shift = 900
dst = np.array([
[shift, shift],
[max_width + shift - 1, shift],
[max_width + shift - 1, max_height + shift - 1],
[shift, max_height + shift - 1]], dtype="float32")
m = cv2.getPerspectiveTransform(reference_points, dst)
whole_img = cv2.warpPerspective(image, m, (image.shape[0], image.shape[1]))
whole_img = imutils.resize(whole_img, width=600)
cv2.imshow('whole_img', whole_img)
cv2.waitKey(0)
# ================================================================================================================
# transform both the reference object points and the points to calculate - using the perspective transform matrix
# (we do this to avoid negative influence of perspective)
t_reference_points = cv2.perspectiveTransform(reference_points[None, :, :], m)
points_to_calculate_array = np.array([points_to_calculate], dtype="float32")
t_points_to_calculate = cv2.perspectiveTransform(points_to_calculate_array, m)
# calculate length in pixels of the longer side of reference object
t_bottom = t_reference_points[0][0]
t_top = t_reference_points[0][3]
t_right = t_reference_points[0][1]
reference_obj_length = max(calculate_distance(t_bottom, t_top), calculate_distance(t_bottom, t_right))
# calculate distance in pixels between points to calculate
obj_to_calculate_length = calculate_distance(t_points_to_calculate[0][0], t_points_to_calculate[0][1])
# knowing the real length of reference object and its length in pixels, we can calculate mm per pixel ratio
mm_per_pixel = REFERENCE_HEIGHT_MAP[reference_obj_type] / reference_obj_length
# knowing mm per pixel ratio and distance in pixels between points to calculate, we can calculate real distance
calculated_pen_length = mm_per_pixel * obj_to_calculate_length
print(f"{calculated_pen_length} mm")

So the whole image is flattened out in relation to a piece of paper or an ATM card, and we get the actual value as a result of this algorithm’s work. The client measured the actual damage to be 47 millimeters. The algorithm said 48. The requirement was for the calculation to be no less than 85 percent of the actual size for it to be within the margin of error. We tested it on 40 images, and it worked.

# A4–297 mm
# card - 85.6 mm
PAPER_SHEET_REFERENCE_HEIGHT = 297
CREDIT_CARD_REFERENCE_HEIGHT = 85.6
REFERENCE_HEIGHT_MAP = {
"A4": PAPER_SHEET_REFERENCE_HEIGHT,
"credit_card": CREDIT_CARD_REFERENCE_HEIGHT,
}

Regarding the potential possibilities of this project, we wanted to add the ability to measure the surface visible in the photo. It’s supposed to work in such a way that the person adding the photo (in our case the person reporting the damage) places a point cloud limiting a certain area of the image, and we then calculate the actual area (an operation which is also possible in OpenCV — we have already investigated it). We also planned to add automatic detection of the location of an object or the type of damage using AI algorithms.

If you want to explore more articles on the subject of software development, visit Profil Software’s blog. We have an array of articles on Python Software Development, including 10 things you need to know to effectively use Django Rest Framework or Make your code more Pythonic with Magic Methods.