Complete guide to Similarity-based learning for Counterfeit detection Part-3
(Inference Pipeline for Counterfeit detection using Similarity Learning)
Harsh Parmar Khushboo Tak Trapti Kalra Yash Vardhan Singh
(IBM Consulting- AI and Advanced Analytics, CIC India)
In the last part of the Siamese Network blog series, we will go through how to create the inference pipeline for the Siamese Network.
To get a basic idea about Siamese Network, check...
Before diving into the inference pipeline, let’s quickly go through the training phase of the Siamese Network...
Training a machine learning model is just half the story. After a trained model is saved, we test the model’s performance and we use the model to make the predictions/decisions.
We can utilize the trained Siamese Network in three ways as follows:
- Get the predictions by passing pair of images
- Get the predictions by passing one reference image and one test image
- Get the predictions by passing a single test image to the Siamese network
Let’s go into the details of each of the above one by one.
1. Get the predictions using pair of images:
In this method, we load the trained Siamese Network as is. Pass the pairs of test images as shown in figure 3.1. The pairs can be auth-auth or auth-counterfeit. The model can generate output “0” to predict the Test image 2 as “Counterfeit” and generate output “1” to predict the Test image 2 as “Authentic” assuming the Test image 1 will always be from the “Authentic image set”.
Let’s discuss the implementation of the method discussed…
We will start the code by importing the requisite libraries.
After importing the libraries, our first step will be to load the trained Siamese Network.
Write the absolute or relative path of the trained Siamese network model directory in which the “.pb” file is stored in the load_model_path variable as shown in code snippet 3.2.
We will redefine the contrastive loss function or import the training script to access the loss function/method because we need to define the custom objects we used while training the network.
Then, we will load the Siamese Network as shown in the code snippet 3.2.
The next step will be to prepare a pair_test.csv using the test images for which we want to get the predictions.
The CSV will consist of three columns: “a”, “b”, and “label”. Each row of the CSV will contain the absolute or relative paths of the pairs of auth-auth or auth-counterfeit images in columns “a” and “b” and corresponding labels “1” or “0” respectively.
We will pass the absolute or relative path of the pair_test.csv in the csv_path variable and will provide the inp_shape as 224 because we have used a VGG16 model while training the Siamese Network. Also, provide the labels in dictionary format in the labels_dict variable for later use.
Now let’s quickly go through the helper functions to prepare the test data.
The process_image( ) function is to resize the image as per the model input size and convert the images as per the model need as shown in code snippet 3.4.
The generate_gt_pred_pair( ) function iterate through the X_test data frame, process both the test images of column “a” and “b” and pass them as a list to the model to predict the output and append the predictions to the y_pred list and append the corresponding ground truth label into y_true list and return both the lists as NumPy arrays.
The generate_result( ) function is to calculate accuracy, precision, recall, and store the confusion matrix as shown in code snippet 3.6.
Now let’s write down the main part…
The first line in Code snippet 3.7 will read the pair_test.csv into the X_test data frame.
The Second line will call the generate_gt_pred_pair( ) function using the trained siamese model to get the ground truth and predicted outputs for the test image pairs written in the X_test data frame.
In the last line of this code, we will take y_true and y_pred to calculate accuracy and generate the confusion matrix for the test image pairs.
2. Get the predictions using one reference image and one test image:
One can think about how to generate predictions using the Siamese Network when we don't have pair of images. What if we have one image and we want to test if this image is authentic or not.
Let's suppose we have test/query images for which we need to get the predictions as shown in fig 3.2. Each row has the absolute or relative path of the query images under column “image” and the corresponding label under column “label”.
How can we utilize the trained Siamese Network to get the predictions for the above query image set?
To do so, we can pass the images (query images) we want to get the predictions for test image 2 onto the trained Siamese Network as shown in fig 3.3.
We will keep a set of authentic images that we believe represent the authentic dataset as test image 1. These images are called reference images. Here we are taking only one image as a reference image to demonstrate. (see the visualization in fig 3.4)
To implement this method we will reuse all the code snippets except the code snippets 3.5 and 3.7.
We will pass the path of test.csv and the path of the reference image and will convert the reference image as per the model input using the process_image( ) function mentioned in code snippet 3.4.
We will need another helper function generate_gt_pred_ref_img( ) to extract the query images from the test.csv(visualization of the test.csv is in fig 3.3.).
This function will take the processed reference image and the trained model. It will iterate through the data frame of the query images, process the images, and pass the processed reference image as the first image and the processed query image as the second image as a list in the model.predict( ) to get the predictions for the query/test images. It will return the NumPy arrays of both the ground truth labels and predictions.
In the end, we will combine all the required pieces, we will first read the test.csv into a data frame and will pass this data frame of query images and processed reference image in generate_gt_pred_ref_img ( ) to get the ground truth labels and prediction arrays, which we will further pass to generate_result( ) function to calculate accuracy and confusion matrix.
3. Get the predictions using a single test image:
There is another way to get the prediction for single query images when we don’t have pairs. This method is a bit tricky.
We load the trained Siamese Network and extract the embedding model as visualized in fig 3.5. We will later pass the images through the embedding model to get the encoding/embedding of the images
We will start this method by loading the trained Siamese Network by reusing the code snippets 3.1 and 3.2.
We will extract the embedding/encoding model as shown in fig 3.5 by using the following code snippet:
The last line will show the summary of the embedding model. The embedding model will generate the embedding or 1-d visual representation of the image.
We will extract and store the embeddings for all train, val, and test datasets. The datasets will be in CSV containing the images and labels as shown in fig 3.2.
We will extract the images of each CSV with the help of the following helper function extract_single_images( ).
This function will return the arrays of images and corresponding labels.
Next, use the get_embedding( ) function to get the prediction for the embedding model which will be encodings/ embeddings of the input image list.
With the help of the above code snippet 3.13, we will get the embeddings for each train, val, and test CSVs and save them in a dictionary format as a separate pickle file. Do not forget to import the pickle library.
After saving the embeddings we will train an MLP or fully connected layer class set. In simple words, some dense layers classify authentic or counterfeit images.
We will write a small script for this. We will start it by loading all the pickle files of the train, val, and test set.
We will train and fine-tune a simple MLP model and save the trained model after testing it.
Once the MLP model is also saved we can combine all the pieces together and can generate the inference pipeline as shown in fig 3.5.
We will load the trained Siamese Network to extract the embedding model by re-using the code snippets 3.2 and 3.11.
We will load the trained MLP model using the classifier model path as shown below.
mlp_model = tf.keras.models.load_model(os.path.join(model_dir,"classifier"))
preds = mlp_model.predict(image_embedding)We will pass the query image to the embedding model and subsequently pass the generated embedding to the trained MLP model to predict if the query image is authentic or not.
Results
To demonstrate the predictions of the above methods, we passed the microscopic image pairs from the real-life datasets of fake and real products Fire resistant Fabric, and Ibuprofen pills to the siamese networks. The model outputs the similarity score that depicts if the pair of images are similar or dissimilar. The similarity scores for a sample of image pairs are shown below.
One can use any of the above methods to get the predictions using a trained Siamese Network and a query/test image set.
Siamese network, compared to the classification solution, is more robust to class imbalance. We will never have all the counterfeits available in the market and the fabric dataset we have used in this article also have a few images of counterfeit fabric. Similarity learning is robust and performs well with fewer data.
Hope after reading all the three articles of the series, you now have got a complete understanding of the similarity learning and its implementation.
There can be multiple applications of these methods. Apply to your project/work and do let us know if this helps! Thanks for checking our articles!
