Search on online retailer sites is broken. Here is how to fix it.

Vineeth Veetil
Nov 7, 2016 · 4 min read
Image for post
Image for post

Search engine technology has advanced to a point where we have come to expect perfection from search results. However, not all search is keeping up with this pace. A glaring example is search on retailer sites. On many retailer sites, keyword search experience is inadequate and lacks product intelligence. Either there are very few results or they aren’t relevant. In the case of apparel for example, a search for ‘bohemian dress’ may end up in just a handful of results although the store may carry scores of products with a bohemian accent. In other cases, even simple search terms such as ‘sleeveless dress’ lead to disappointing results. The last resort for the user in many cases to browse the entire category, say ‘dress’, and hunt down the right match.

This is a general problem which exists because of a disconnect between image content and product intelligence. In fashion, for example, users use a variety of terms to find the right product. Some may be related to cut, sleeve and neck of the apparel, others related to pattern or even occasion and style. Unless the products are properly annotated with all these words used by the user, the results are not going to be relevant, leading to frustration. This annotation process in merchandising is completely manual. A human team takes a look at each product and writes down tags. They are assigned a list of guidelines and tags to cover in this process. First, this is not practical when there are thousands of new products added every week, since a person cannot accurately tag more than a few hundred images a day. More importantly, quality is a problem. In the case of fashion, how do you make sure that you cover every major type of style, pattern, neck and accent. How to adapt to the latest trending terms. How to make sure that tags are consistent across products ? How to cover trending style-related keywords such as bohemian which require a significant amount of creativity. In short, this process is error prone, ineffective and inefficient. From our studies, we see that less than 50% of relevant trending keywords are covered, even in the best retailer sites.

Thankfully, artificial intelligence and deep learning have now progressed to a point where image recognition is an achievable goal. However, there are significant challenges associated with applying this technology to fashion, especially with natural images and user generated content. These images may have background which can distract the recognition system. Bad lighting, different poses are other challenges. The clothing may be worn by user or in the wardrobe. Specialized algorithms and datasets are required for these systems to understand real world images. At, we have developed technology to address some of these challenges.

We recently indexed all apparel from a top fashion retailer using our system and compared the performance of search queries with and without auto image tagging. The results are staggering. We see on average 5x more relevant results from our proprietary technology, up to 20x in some cases. For example, a query for bohemian dress results in 4 results on the site, whereas 83 results with our annotation. Similarly, a query for tribal dress results in 4 results without auto tagging and 96 results with auto tagging. In addition, we are able to sort the results in a meaningful way using intelligence regarding how ‘bohemian’ or ‘tribal’ a dress is, which is impossible with a manual tagging approach. Since the top few results are most interesting in any search results, we help bring the best products to the top using our proprietary ranking system.

To conclude, we are now moving towards a world where the search systems in commerce don’t deal with products as mere SKUs with searchable meta data, but are capable of superior product intelligence. Intelligent systems which can understand in great detail about a product literally by ‘looking’ at it just like a human would. These intelligent systems can then match the user intent and query with the item they are seeking to create a better experience, instead of showing bad results because the human annotated information for the product was wrong or replicated from a different product.

At, we provide intelligent solutions for retailers and content publishers so they can take control of their visual assets — product images or social media images. If you want to learn more, send us a note at

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store