Detecting Polka Dot Fabric Patterns With OpenCV
Authors: John Rei Enriquez | Gene Aguilar | Mica M | Dee | Jian James Astrero
Introduction
With nearly 200,000 product listings of formal wear dresses and attire currently on Queenly, our engineering team is continuously looking for ways to better categorize them. When it comes to providing our users with the best experience in finding their perfect dress, getting the details right is key. We’ve taken many conventional and unconventional approaches of blending text and image signals to categorize these products by color, type of occasion, and types of styles. Let’s talk about our unique way of looking at fabric design and pattern detection.
Here we’ll dive into how we implemented a polka dot detection algorithm using OpenCV, which, potentially, could be the basis for detecting other designs and patterns.
Existing Computer Vision APIs: Approach and Limitations
Before implementing our own approach, our team took a look at other Computer Vision APIs and platforms, including Google’s Cloud Vision API. These tools are great for broad image categorization, as well as asking the “buy or build” question for whether we would be able to use an existing tool instead of building a solution ourselves. To test these other CV tools, we ran some example images as shown below:
Shown here are baseline examples of a “valid” polka dot patterned dress and an “invalid” patterned dress, shown left and right respectively. What’s useful in the Cloud Vision tools are the confidence scores and rankings of their labels given to these images. From the Cloud Vision results, it looks like the challenge here is determining the right labelling threshold for what could be determined a valid versus invalid “polka dot” pattern.
We also thought about the computational processing needs of our algorithm. Labelling a dress as a “polka-dot” fabric is a strong and useful signal, but probably shouldn’t be too computationally intensive. OpenCV allows for more flexibility — you can tailor it to fit our specific and functionality of your project. It has a wide range of features and modules making it the better option for developing a customized computer vision project. OpenCV is also better at large batch processing because the images are processed locally on your machine. Unlike with Cloud Vision API the images don’t need to be sent over the network to a cloud-based service. This can save a significant amount of time. More specifically to our case OpenCV served us better for more specific tasks like detecting polka dots in comparison to Cloud Vision API, which was too general.
Neural Networks: Usual Approach and Limitations
Another popular approach to this type of problem might be in using a neural network to identify these features on the image. The steps to build this classifier would be preprocessing the dataset, integrating Tensorflow/Keras, and several standard convolutional layer operations to filter out and feature extract the polka dot fabric.
Using this model, we could technically still identified polka dots in a dress, but, the challenge of this type of approach might be needing a massive dataset to feed our model. Even if we had a large enough dataset, it would take too much time to train the model since we would need more layers, thus increasing the parameters and complexity of our model. Using a pre-trained model might be suited for our NN-based approach since the model could identify features already and even without sufficient data, it could still make a prediction based on our business needs. However, the training of this model would be more demanding since the model we will be using could still require a much larger number of parameters and complexity than would be necessary.
To put it simply, polka dots are really just a sequence of circles in a row. Neural nets might be overkill, right?
Prior Fabric Detection: Polygon-based instance segmentation
Detailed in the Applications of Computer Vision in Fashion and Textiles, researchers from Shenzhen University developed a polygon-based instance segmentation method for identifying different accessorized fabrics. For instance, they could determine different patterns of rhinestoned, studded, or bedazzled fabrics. Their feature extraction process involved identifying the shape and size of the “stones” on the fabric, taking into account image noise like blurred objects or rotated stones. This type of research demonstrated potential in the type of algorithm we ultimately ended up implementing.
The Polka Dot Algorithm
In looking for polka dot circles, we will be using the concept of “blobs” — a group of connected pixels that share the same value. We will try to make out blobs from dark connected regions in the image and setup parameters to choose which blobs will qualify as polka dots using OpenCV’s SimpleBlobDetector.
These can be broken down into three steps:
- Image preprocessing: resize image to uniform dimensions + remove background noise
Before performing blob detection, we applied a series of preprocessing methods to minimize image complexity in preparation for data analysis. The first step was resizing images to uniform dimensions to garner more consistent results considering that the blob detector uses pixels in some of its parameters. In resizing bigger images to smaller dimensions, we are also reducing the processing time for other resource intensive computations such as background removal. As the main goal of our algorithm is to detect polka dots in the fabric of a clothing, the background is an irrelevant piece of information that only restrains the accuracy of our analysis. Through rembg, an open-source tool that uses trained models in removing backgrounds, we are able to detect and focus on the subject of an image, which significantly reduces the probability of gathering false positives from blocks of circles that could be identified elsewhere.
Another image preprocessing technique we applied is thresholding, or simply converting RGB-based images into black and white images. Since the concept of blobs doesn’t require color identification, retaining colors will only contribute to the computational complexity of our algorithm. With a binary image, there would be less number of pixels that need to be processed and it would be easier to analyze the data and cluster pixels into different regions. Essentially, preprocessing images is an important measure to clean image data from unnecessary convoluted information before further processing and analysis, which not only reduces processing time, but it also enhances the accuracy of the results.
2. Find Potential Polka Dot Blobs using SimpleBlobDetector
Here we’ve considered several attributes of the polygons detected, including size/area, color-contrast thresholds, circularity, inertia (allow slightly elongated circles), and convexity.
3. Filter blobs further to match our criteria for Polka dots and tag the item with polka dots. We use these two main heuristics:
- Minimum threshold of x number of matched “blobs”, with x being a threshold for number of detected blobs to determine an intentional polka dot pattern
- Disregard blobs found on the leftmost and rightmost 20% of the image. In our image dataset, most of the time subject will mostly be centered or offset.
* if after step 3 and no polka dots were detected, we invert the blacks and whites of the image since white circles usually are ignored when using our blob detection parameters above and then we repeat steps 2 and 3. If still no polka dots detected, we can be confident there are no polka dots in the subject.
Results
We’ve discovered here that this shape-based detection algorithm is effective in being both computationally efficient enough as well as accurate enough for Queenly’s business needs. We decided on a decision threshold that leaned towards labelling more “false negatives” for polka dots. Our thinking behind this decision was that we’d want to optimize for the user experience: it’s better for our customers to have results that accurately fit their search criteria than be confused by seeing mislabeled products. Thus, Queenly now has a small, but growing and accurately labelled inventory of polka dot dresses!
Future Work
We chose “polka dots” as a starting point for testing whether this approach was worth expanding into other types of fabric detection (It might also be worth noting that polka dot fabrics are popular enough, but by no means the “trendiest” fabric style among “kids these days”). The effectiveness of our shape-based computer vision approach allows us to think about building similar quick and efficient detection methods to other of fabric trends.
Plaid and pin-striped fabrics are strong candidates for the next iteration of our approach, as they are fundamentally rectangles and lines respectively. One potentially more challenging pattern is detecting the ever-popular and ever-changing floral-print fabric. We’ll be sure to report our findings when we do!
The Engineering Culture Behind This Project
Our polka-dot detection project was done as a group programming exercise among all members of our engineering team! Though we’ve come from different engineering backgrounds in mobile, front-end and data engineering, and though some of us were new to working in computer vision, we were able to collaborate on the creative ideation and iteration of this project.
If you like this post, please consider working with us :)
