Improving Walmart Search to help our customers save time!

Walmart.com is one of the largest shopping sites in the US, with millions of visitors coming in to shop for food, essentials, toys and everything in between. Customers actively use our search engine to find what they need, and we continue to improve the searching and finding experience for our customers. In the last twelve months, we have focused our efforts in helping our customers discover the most relevant products for long, complex and less common queries (we call this segment “tail”).

Imagine the last time you searched for a very diet friendly cereal (from a specific brand), a very particular tv stand (one with a fireplace) or that new graphics card (with a specific configuration). Across multiple e-commerce platforms, these types of queries deliver less than ideal results. We want to solve that problem for our customers and help them save time.

About a year ago, we set an ambitious target for the improvement of our tail segment. Tail or less common queries suffer from limited traffic and user engagement, limiting our ability to effectively rely on those types of signals in our recall and ranking systems. As a result, we have had to look to market-leading strategies and technologies in the language processing and deep learning spaces.

To guide our process, we’ve used a combination of manual or editorial evaluations, market comparison, and customer research to identify and prioritize opportunities. In this case, a thorough analysis of our search systems helped us focus on two specific areas:

• Improving search recall through the use of the latest machine learning (ML) techniques — bringing up relevant items which were not shown before
• Enhancing search ranking systems using ML and AI (Artificial Intelligence) to bring the most relevant items into top search positions

In this post, we will talk about the above two opportunities, the specific projects and improvements we have launched, and the impact we have had on our customers. Overall, we have been able to improve search relevance for these queries by multiple percentage points — saving time for millions of Walmart customers.

Implementation and improvements

As mentioned above, one of the top areas of opportunity we identified was improving the quality and completeness of our search engine’s recall generation systems, with the goal of bringing new and/or highly relevant items into our final results sets. Specifically, we focused on two specific areas:

• Optimizing product type matching logic (both filtering and boosting logic)
• Improving how we understand the relative importance of tokens (words) in queries

In addition, we found opportunities in improving our reranking systems to lift search relevance. Due to our recognition of the fundamental structural differences between head and tail queries, we have developed new, separate ranking models for these query types.

Optimizing product type matching logic

When we built the Walmart search engine, we were focused on head or top queries that make up a significant portion of the shopping site sales. This meant that all search sub-systems were optimized for these queries. When we analyzed tail queries, we realized that our strict, head query logic systems were filtering many relevant products when applied to the tail. Product type or category is typically a cornerstone of a product search engine and getting that right is extremely important. So, we embarked on a journey to clean up existing product type matching logic and develop better boosting techniques for product types.

We launched the following changes recently and have seen search relevance improve by over 1.5%. In addition, customers are engaging much more strongly with our products within the tail space.

• Removing strict product type filtering and leveraging product type boosting to boost relevant products in the recall set
• Enhancing the list of query-item associations that are used to find more relevant items — we expanded the list and raised the bar on quality

Improving query terms and token weighting

Tail queries are often longer and more complex than head queries. Historically, we treated all tokens (unique words) in a query equally. Given the nature of tail, this approach was identified as a major barrier to getting the results quality we wanted. We now assign importance scores to each token in a query, allowing us to better extract and understand the customer’s underlying intent. We recently launched our first iteration of ML-driven models to assign these importance scores.

We spent time understanding which words or terms add less or more value to a query and intent, and how that varies by context. For example, in the cases of “birthday cake to order”, “Neutrogena products”, or “computer printers”, not all have tokens are equally important — ‘to order’ and ‘products’ and ‘computer’ are potentially redundant — particularly in the context of a product search engine. It is important to note that in other cases, these same tokens might be adding meaningful value to the query; for example “computer accessories”, or “products in my store”. With the launch of our first version, we:

• Introduced prediction of query term weights
• Started to leverage term weights in retrieval

We have seen search relevance improve by 4.8% and observed lifts in customer engagement and conversion.

Improved ranking systems to enhance experience

We have also invested in ranking systems to bring the most relevant products to the top positions of search results. We leverage two ranking layers in our search systems, (first and second round ranking), and have invested efforts into both layers to drive improvements in search relevance for our customers. Specifically, we launched the following enhancements recently:

• Strengthened the importance of product type matching in the first-round ranker
• Improved the re-ranker (or second-round ranker) to better balance relevance (item-query semantic matching) and engagement (item-query engagement) signals — the new version optimizes better for both signals and is more inclusive of the most relevant and engaged products

Together, these lifted search relevance by over 4.5% and showed improvements in user engagement.

The future and what’s coming

As we think about the future, we want to continue improving search experience for our customers, monitor and observe engagement on new and trending queries, and launch improvements that will make searching easier and save our customers time. We have many more opportunities ahead of us including:

• Leveraging deep learning techniques to enhance query understanding systems to understand explicit and implicit intent, and understand user context and needs better
• Improving the recall engine using new and advanced techniques to bring up even more relevant and long tail items
• Developing new features and signals in the reranking system using machine learning and tuning and optimizing existing ones to show the most relevant and popular products to customers

Walmart will continue to the place to shop for millions of customers and our goal is to continue to make search easier and help our customers save time. We hope you enjoyed reading this article. If you are interested in working with us, check our careers page or reach out to me, or my colleagues on LinkedIn.

Co-authored by Sarabjeet Singh and Ciya Liao

The work showcased in this article was led by a stellar team of data scientists and product managers including from Tony Lee, Mansurul Bhuiyan, Zhenrui Wang, and Mikhail Radik.