Mahesh Khatri
May 31, 2019 · 4 min read

The mystery of the misplaced shoe… Can AI help ?

Image for post
Image for post
The Shoe whose left counterpart was misplaced in the store

Last summer, while shopping in New York, my wife fell in love with a shoe at a clearance sale of a national retail chain. The problem was that only the right shoe was available in the store. The left shoe was missing.

So, the entire family got down to work. With the above photograph, she, my son and I spent many hours searching for the elusive missing left shoe. Along with the store’s sales staff including the manager.

Alas ! The shoe could not be found. “No problem !” exclaimed the manager. “I will search for it at the other stores”. Great, we thought. But, to locate the shoe details, he needed to create my son’s online customer profile in his computer systems. This would require his name, address, phone number & email. My son, who shares my passion for privacy, was paranoid about sharing his personal information and agreed reluctantly. But nothing came up in the search.

Disappointed, we returned to India. Weeks passed with no luck. During our weekly telephonic conversations, my wife would unfailingly ask if my son had heard from the store manager and he would reply in the negative. She also persuaded my son to once again visit the store to personally inquire about it. Still the same results. We finally gave up.

Time to ask AI for help ?

I tried to locate the shoe through the visual image search facilities of a few global AI & E-commerce majors, as well as the twin search engines with no success. Nothing found.

An AI giant’s website could not even recognize the shoe in the photograph above as the brown color it’s upper portion had easily blended with the brown color on the base where the shoe was kept. So much for smart AI. At times, it is really dumb !

Google Cloud Vision did report the following results on the photograph as shown below :

Image for post
Image for post
Some Google Cloud Vision Results from Uploading The Shoe Photograph

As seen above, the guesses were quite reliable. Google’s Cloud Vision was able to accurately recognize the shoe as an ‘Outdoor Shoe’. It also assigned reasonably good probabilities for the shoe’s descriptive labels. However, when it came to showing visually similar images, the results were completely off as is shown below for the top 3 closest images shown by it. Surprise, Surprise ! None of these look like our shoe above.

Image for post
Image for post
An image visually similar to the shoe as reported
Image for post
Image for post
The second visually similar image
Image for post
Image for post
And the 3rd Visually (dis) similar image

Do we see a pattern here ? The only common visual feature between the shoe photograph and the above 3 images is the brown color. So, Cloud Vision’s neural networks — which did the image processing of the photo — emphasized more on the shoe’s brown color as compared to the other visual features in it’s task of analyzing the shoe picture.

The US retail giant we were shopping at reported a 4% decrease in their quarterly adjusted Earnings Per Share (EPS), as compared to the previous year. I wonder how much of an impact to the above earnings decline was caused by lost employee hours due to inventory mismanagement problems like our misplaced shoe search .

Computer Vision & Image AI is far from perfection (at least as of last year, when the above image search was done). But it is making astoundingly exponential leaps annually. The Visual image recognition market size forecast is USD 25 Billion-plus market this year. If implemented successfully, AI will perhaps help offline retailers reduce their losses and minimize ceding of mind & market share to online e-commerce giants.

PS : Luckily, our story has a happy ending. We managed to find the missing twin.

Image for post
Image for post
Our (finally united) pair of shoes

Sigh ! A year later, the story repeated itself. But this time, for a limited edition pair of jeans. That’s a story for another day.

When can AI help ease my shopping pAIns ?

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch

Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore

Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store