Computer Vision for Complex Test Automation Challenges — Part II — Benchmarking of Algorithms via CLI
Finding similarities between images is quite a common and one of the important requirements in certain areas of the software development for ages. Despite the availability of plethora of tools, those tools employ pixel-to-pixel comparison technique as the basis. This technique is not without challenges and limitations. Please head over to my article Computer Vision For Complex Test Automation Challenges — Part I for further information.
Quick Intro
Out of necessity to build a versatile tool for finding similarities between large image data sets, I experimented around ten different image processing/computer vision algorithms and found three of them comparatively work well. These algorithms too may have limitations in certain cases, but convincingly better than pixel-to-pixel technique. Combining the three algorithms as the basis, built this standalone CLI tool — ImageBench.
Purpose
(a) Arrive at similarity scores for large sets of offline images in an easy and seamless manner without any dependencies on automated test runs
(b) Use the scores to determine the efficacy and veracity of various algorithms against images
(c) Find out the performance of each algorithm against various image data set
(d) Determine the order of the algorithms to be used in actual test runs in cases where it needs
(e) The scores generated with this tool can further be used to decide the efficacy, order and the combination of algorithms to be used with ImageVision.
Note: ImageVision is an independent library built for leveraging computer vision capabilities for image processing, image comparisons, complex visual interactions and visual anomaly detections to integrate with any dev/test tools.
Tech Stack
Python, Python dev ecosystem, OpenCV, Miniconda
Setup
It’s pretty simple and straightforward. Two packages need to be installed: a) Miniconda, b) ImageBench
Download the package from: https://github.com/yusuf-git/ImageBench.git
a) Miniconda Installation
After downloading, unzip the package. Run install.bat. This will install Miniconda in C:\py\miniconda
Installation Steps
1. Copy zip files to c:\py
2. Unzip miniconda.zip to C:\py\installer
3. Navigate to c:\py\installer\miniconda. Run Install.bat from the command prompt
4. Recommended to wait for maximum 3 minutes, but in ideal cases it’s not required. After waiting, you should be able to see c:\py\miniconda with all the necessary files
5. Restart the system
6. Open command prompt. Enter “conda activate”. You should be able to see the prompt changing “(base) c:\py”. This step confirms that the miniconda installation is successful.
7. Enter “conda deactivate”.
b) ImageBench Installation
Installation Steps
- Unzip ImageBench.zip to c:\py. The extracted files will be unzipped to c:\py\imagebench
- Open cmd prompt and navigate to c:\py\imagebench
- Enter “setup” and wait for a few minutes. This will create the necessary environments.
- Once setup is done, enter “run”. You should be able to see ImageBench sample images are being exercised by the algorithms with default configurations.
BRISK_FLANN_baselines and workspace folders are automatically created as a part of setup. By default, BRISK_FLANN_baselines folder contains baseline records for the sample images generated from the BRISK-FLANN algorithm operation. The workspace folder contains all artifacts generated from the run operations of algorithms. They include logs, results, baseline, runtime and diff images.
Given below some reference snapshots that you will see after running the algorithms:
Features and Configurations
For now, all configurations are passed through: img-comp-args.json
A. Feature Index-1
baseline_path, runtime_path: Specify the baseline and runtime image paths at the root level, regardless of the level of subdirectories. For example: D:\Automation\DW\SDV\src\test-inputs\images.
B. Feature Index-2
workspace_path: workspace_path is where the baseline and runtime images are copied to, but it will be a flattened copy — no subdirectories will be created. Additional subfolders will be auto-created for the organization of the artifacts based on operation sessions.
For example, the workspace_path at runtime will create: logs, diffs(for storing diff files between two images), flat_base and flat_runtime under image_ops subfolder.
Also, diffs and logs folders will have session specific subfolders to keep log files and diff imag8e files for the corresponding session.
During the copy, images that have conflicting names/invalid names are automatically renamed by suffixing the image paths.
C. Feature Index-3
mask_region, mask_region_excluding: Specify the coordinates of a region in the image, as a tuple, in scenarios where you would want to ignore certain portions of an image for comparison with another image. Ex: 30,25,60,60 stand for x1,y1,x2,y2.
mask_region_excluding, on the other hand, works opposite to mask_region i.e. it applies mask on the entire region in an image other than the specified coordinates.
This FI can be useful in scenarios where you would want to ignore the dynamic parts of an image.
D. Feature Index-4
aspect_ratio_required: Need to be “true” when source and runtime image dimensions are different, but need to be resized without sacrificing the image contents in accordance with the respective image’s width and height. However, if this field is “false”, image resizing operations will still be performed under the hood, but that’s raw form of resize i.e. image may be distorted in some cases.
E. Feature Index-5
intermediate_output: Set to “true” when you want to see additional console output. In some algo operations, this may show transformed/ processed / pre-processed images in separate windows for each image.
Conclusion for Part II
In this part of the article, we’ve gone through quick intro for ImageBench, downloading and setting it up, tech stack, and a part of the features and configurations.
Keeping the remaining features and the required amount of the relevant information into consideration, the remainder of this article about ImageBench will continue in the next part — Computer Vision For Complex Test Automation Challenges — Part III — Benchmarking Scores of Algorithms. The next part will comprise other features & configs., evaluation, sampling and performance scores of algorithms.
Thank you for taking the time to read this article :).
