Object detection lite : Template Matching

7 min readApr 27, 2022

Object detection is always been part of interest for computer vision enthusiast. World got new eye & new way of thinking and tracking object since emergence of detection algorithms . Starting from Region Based Convolutional Neural Networks [RCNN] to YOLO V5 , object detection changed drastically for higher efficiency of detection as well as less requirement of time.

There is an another niche in computer vision known as pattern recognition. What if we can use characteristics of Pattern recognition to detect and locate objects like object detection ?

Well, lets see how we can do it !

Pattern recognition

In image processing, pattern recognition is one of the important domain which has lots of real life applications. Many applications like license plate recognition, fingerprint analysis, face detection/verification and voice-based authentication use pattern recognition.

Template matching

Template matching is one of pattern recognition technique where we provide a single part or object from image as a template to mark multiple similar patterns from images. When we have very less dataset or our application is light one or when we have to just mark similar patterns despite of its variations then we can give try to template matching.

Template matching uses:

1) Search for the best match by minimizing mean-squared error

2) maximize area correlation

It uses phase correlation with Discrete Fourier transform. When we input image then it gets segregated into magnitude and phase transform to highlight features. From that features, it searches for peak and by finding peak, object localization is done.

Pixel level template matching:

It’s the simplest type of template matching where we don’t consider any kind of rotation, translational invariance, shifts in degrees. the template pattern is exactly similar to object patterns in test image. Due to its basicness, it is use for simpler applications where there is proper constraint over input window.

Advancing this simple method by adding little complexness, another method is partial template matching, where we allow partial matches with certain threshold.

Making this more useful, there is another method known as Piece template where the original template is broken down into its edges or more specifically its component segments and then matching is done.

It is very rare that we will get our patterns in image straight-forward as it is. Many times there are different augmentations of patterns like rotation, spatial invariance, blurs , translational invariance . For higher level template matching, we used feature-based matching. In this, we segment images into regions of interests. We can take different intensity terms , curves, distance measurement etc. This things are recorded and then mapped accordingly to test image . This technique is known as Feature based matching .

What if we don’t have strong features ?

For such a condition, we can use Template based matching technique. Here we take our template image and augment it into multiple augmentations. Then use this type of sampling to extract features and last matching of it with original image.

For identifying/Matching similarities, we can use :

1) Array representation of template and image

2) Euclidian distance

3) Measurement based on correlations

4) Root mean square distance

The best thing is we don’t want to do it from scratch. We can use inbuilt OpenCV's

res = cv2.matchTemplate(original_image,template,cv.TM_CCOEFF_NORMED)

Enough of theory now. we will dive into coding section. We will try one simple example with code . Lets take one image on which we will do template matching.

This will be our original image for matching template

################################
############# Imports ##################
################################import cv2
from PIL import Image
from skimage import io
import pandas as pd
import numpy as np
from PIL import Image
import imagehash
import cv2
import glob
from matplotlib import pyplot as plt

you will require some very common imports initially to work .

########################################
########  for manual cropping ###############
########################################cropping = True
x_start, y_start, x_end, y_end = 0, 0, 0, 0
image = cv2.imread(r'C:\Users\ChinmayB\Downloads\playing-cards-lay-in-a-messy-pile.jpg')
image = cv2.resize(image, (1960, 2000))oriImage = image.copy()def mouse_crop(event, x, y, flags, param):
    
    # grab references to the global variables
    global x_start, y_start, x_end, y_end, cropping
    # if the left mouse button was DOWN, start RECORDING
    # (x, y) coordinates and indicate that cropping is being
    if event == cv2.EVENT_LBUTTONDOWN:
        x_start, y_start, x_end, y_end = x, y, x, y
        cropping = True
    # Mouse is Moving
    elif event == cv2.EVENT_MOUSEMOVE:
        if cropping == True:
            x_end, y_end = x, y
    # if the left mouse button was released
    elif event == cv2.EVENT_LBUTTONUP:
        # record the ending (x, y) coordinates
        x_end, y_end = x, y
        cropping = False # cropping is finished
        refPoint = [(x_start, y_start), (x_end, y_end)]
        if len(refPoint) == 2: #when two points were found
            roi = oriImage[refPoint[0][1]:refPoint[1][1], refPoint[0][0]:refPoint[1][0]]
#             print(roi.shape)
#             h= roi.shape[0] 
#             w = roi.shape[1]cv2.imshow("Cropped", roi)
            #print(x, y)   
            cv2.imwrite("your template location",roi)
#             cropped_image = roi
#             cropped_image = np.asarray(cropped_image)cv2.namedWindow("image")
cv2.setMouseCallback("image", mouse_crop)#while True:
i = image.copy()
if not cropping:
    cv2.imshow("image", image)
elif cropping:
    cv2.rectangle(i, (x_start, y_start), (x_end, y_end), (255, 255, 0), 3) cv2.imshow("image", i) cv2.waitKey(0)
# close all open windows
 cv2.destroyAllWindows()

The above code chunk is not part of template matching but you can crop your template manually and can save it at desire location by this chunk. You can uncomment while loop to take multiple templates . After doing cropping, we will get our template .

This will be our cropped template by above code

###############################################
###### single Template matching ####################
###############################################
img_rgb = cv2.imread(Original image address)
img_rgb = cv2.resize(image, (1960, 2000))img_gray = cv2.cvtColor(img_rgb, cv2.COLOR_BGR2GRAY)
template = cv2.imread(template image address, 0)
h, w = template.shape[::]#methods available: ['cv2.TM_CCOEFF', 'cv2.TM_CCOEFF_NORMED', 'cv2.TM_CCORR',
#            'cv2.TM_CCORR_NORMED', 'cv2.TM_SQDIFF', 'cv2.TM_SQDIFF_NORMED']res = cv2.matchTemplate(img_gray, template, cv2.TM_SQDIFF)
# For TM_SQDIFF, Good match yields minimum value; bad match yields large values
# For all others it is exactly opposite, max value = good fit.
plt.imshow(res, cmap='gray')min_val, max_val, min_loc, max_loc = cv2.minMaxLoc(res)top_left = min_loc  #Change to max_loc for all except for TM_SQDIFF
bottom_right = (top_left[0] + w, top_left[1] + h)
cv2.rectangle(img_gray, top_left, bottom_right, 255, 2)  #White rectangle with thickness 2.cv2.imshow("Matched image", img_gray)
cv2.waitKey()
cv2.destroyAllWindows()

By above code, you can match single template in your image . It will be an exact match without any tolerance. But in reality, it is very hard to find exact matches. Lets do a single matching of template on original image

########################################################
########### Template matching for multiple objects #############
#######################################################import cv2
import numpy as npimg_rgb = cv2.imread(Original image address)
img_gray = cv2.cvtColor(img_rgb, cv2.COLOR_BGR2GRAY)
template = cv2.imread(template image address, 0)
h, w = template.shape[::]res = cv2.matchTemplate(img_gray, template, cv2.TM_CCOEFF_NORMED)
plt.imshow(res, cmap='gray')
threshold = 0.8
loc = np.where( res >= threshold)  
for pt in zip(*loc[::-1]):  
       cv2.rectangle(img_rgb, pt, (pt[0] + w, pt[1] + h), (0, 0, 255), 2)  
cv2.imshow("Matched image", img_rgb)
cv2.waitKey()
cv2.destroyAllWindows()

This is main part that we want. Here template will try to match with each and every pattern in image and whenever it will find similar pattern more than 80 percent of confidence, then it will mark a rectangle over there, which is same as we do in object detection .I tried to do matching of templates with 180 degree flip of original image i.e. I used two images as a template and its results are pretty nice .

You will ask me , “ what about other hearts that weren't detected ?” or “ what about that wrong detections ?”

Both answers are pretty simple. We should provide at least all 4 rotations of template then it will solve above problems.

for file in os.listdir(path_temp_sel):
fname,ext = os.path.splitext(file)
if ext == '.jpg':
print(file)
template = cv2.imread(path_temp_sel+'\\'+file)
imgrot = cv2.rotate(template,cv2.ROTATE_90_CLOCKWISE)
# cv2.imshow("rotated",imgrot)
# cv2.waitKey(0)
# cv2.destroyAllWindows()
cv2.imwrite(path_temp_sel+'\\'+'rotated_{0}.jpg'.format(fname),imgrot)

You can add above code in multiple template matching to get rotations of your templates inside same folder. This will help to increase performance of your recognition .

Despite of all, We have Python dedicated module for template matching .

pip install Multi-Template-Matching

you have to install it using pip command.

import MTM
from MTM import matchTemplateshits = matchTemplates(listTemplate,
                      input_img,
                      score_threshold=0.9,
                      searchBox=(0, 0, 3000, 750),
                      method=cv2.TM_CCOEFF_NORMED,
                      maxOverlap=0.1)

Where,

listTemplate = templates that we are using
input_img = original image
maxOverlap= amount by which two objects can overlap
score_threshold = confidence that we want for prediction
searchBox = coordinates of rectangle

Uses:

1) When we have lack of data then we can use target object itself from image for feature extraction and can match it on same.
2) When application has constrained input fixed place then template matching is better option.
3) Not a data intensive method hence can be used when we have lack of data for object detection.
4)Simpler applications where deployments of YOLO models are expensive task, then we can give one try for pattern recognition.

This is all what I think is important to know about this method. I tried to give you a broad perspective of template matching with theory and code with example. If you have any doubt or concern or if you want to connect, you can connect me on:

LinkedIn , Medium, & Mail

THANK YOU!!!

References :

Object detection lite : Template Matching

Template matching

Written by Chinmay Bhalerao