eBay ShopBot in Pictureland

Imagine you’re walking past a shop on a busy city street when a black dress in the window catches your eye. It’s got character: it’s got a satin-finish with sequins with an amazing racerback, making it a unique piece. You must have it — but you know it’d be cheaper online. Would you actually take the time to type all those details, hoping to get good results? What if all you had to do was snap a photo?

We launched eBay ShopBot Beta in mid-October, a chatbot that works as your smart personal shopping assistant, which helps you find the best deals. Though it’s still in its early days, eBay Shopbot is already beginning to set itself apart from its peers. One of these key features is image search: the ability to find an item just by taking a photo and showing it to the bot, no words necessary.

When the beta went live, eBay ShopBot already understood hundreds of millions of items. Since then, we increasingly recognize more and more brands within eBay product listings just by feeding pictures to the eBay AI.

Challenges

Image recognition is especially challenging for an inventory like eBay’s, which is powered by our sellers — who could be everyday consumers, SMBs or brands. Not only are there a billion products to select from, but there are no photography standards to go by. The same product may have tons of different photos, from professional to amateur, well-lit to dim, crisp to blurry.

Users also shouldn’t be expected to take perfect pictures. Not every phone has a great camera, not every scene has great lighting.

Since we can’t all be professional photographers, it’s eBay ShopBot’s responsibility to be an even better image recognizer.

An item in the wild

Let’s start with a simple request. The strong colors, shapes, and patterns make it pretty easy for eBay ShopBot to find exact matches for toys like action figures and Lego sets.

But, what happens when the brand or strong colors vary, making it difficult for AI to interpret the image?

Take for instance the below image of this table lamp or wall mirror. Did you know that this style of spoked mirror is called a “Starburst Wall Mirror”?

Before, you may have needed an interior designer or decorator (or lots of time guessing in Google) to help find something that specific. Snapping a photo isn’t just faster, it’ll yield more results and show interesting variations.

Take a picture of a picture

Of course, seeing things we want isn’t just limited to store windows. We all know at least one Pinterest addict. So we wanted to make sure eBay ShopBot can understand your personal “vision board,” whether it’s comprised of photos you’ve taken, a magazine clipping or a professional photo from a website.

Searches like these are harder due to visual distortions like noise, glare, and reflections. We made sure eBay ShopBot factors these in. In the right-hand example above, we took a photo of our monitor, resulting in wavy lines. But by recognizing shapes and styles, eBay ShopBot still found a Yoda puppy costume– and even knew not to factor in the background color.

Exact match or similar?

eBay ShopBot doesn’t just look for exact matches, so there’s no problem if the exact item you showed isn’t on eBay — eBay ShopBot finds you items that are incredibly similar, too.

Image search for inspiration

If you see a pattern or design you like, eBay ShopBot can help find products inspired by that one. One of the eBay team members is a fan of butterflies, so naturally we searched with a picture of one. Results include bows and ribbons with the same motif — perfect for a butterfly-lover.

When you’re using image search for inspiration, your photo can be anything. We took a picture of rainbow pushpins scattered in a mesh jar, and since eBay has no listings for assorted pushpins in a cup, eBay ShopBot found vibrant art pieces instead.

What’s next for visual search?

There’s a lot more we want to teach eBay ShopBot when it comes to reading images. We’re already teaching it to read text in a picture, so when you show it a book or album, it can find other releases that may have different art. Of course, we also have lofty long-term goals, like multi-object recognition, video search, and augmented reality… but that’s a whole other article.

Want to learn more about the algorithms and engineers behind image search? Check out eBay’s Computer Vision site. This article is just an amuse bouche. We’ll keep updating you with more advances and cool new features as we evolve eBay ShopBot. Stay tuned!

Robinson