A/B testing search and discovery to optimize a marketplace’s fluidity

A powerful search, a more seamless marketplace

Videdressing connects individual buyers and sellers. Guiding these buyers to the items they want also means that the seller is satisfied as quickly as possible. Liquidity is guaranteed when products are being sold as quickly as possible: we, therefore, need to optimize interactions between users and items.

Visitors browse a catalog of products using Search and Discover: two complementary concepts that guide them. Our platform offers various interaction circuits (even if it is the same technical building block that answers queries):

• The catalog: browsing it is a bit like window shopping.
• Inspiration: merchandising and promoted content made by our teams.
• The search field, which responds to specific search queries and suggests related or popular searches.
• Personalization and recommendation: an algorithm which tailors the results depending on the user.

Creating a powerful search engine involves understanding the users’ intentions. An intention is defined by a number of dimensions, for example:

· Does the user have a vague idea of what he is looking for or do they know exactly what it is they need?

· How important is it to the user that he find the right result?

· How much time and effort is the user willing to invest to find the right item?

· To what extent does the user want to control his search using filtering and sorting tools?

· Is the user willing to discover items by receiving suggestions or does he want browse all of the results one by one?

· …

In this article, Shakhina Pulatova wrote about how search works at LinkedIn. She gave a remarkable explanation of the concept and the issues related to searching (see below).

Building Search Products With Machine Learning - Shakhina Pulatova - Mind the Product

Diversity of behaviors + deep diversified product catalog = complex analysis

When you go shopping, you will generally find yourself in one of the following two situations: you have a very precise idea of what you’re looking for and you head straight to the item that you want, or you browse the entire shop looking for a bargain or indulge in an impulse buy.

On a platform dedicated to second-hand fashion such as Videdressing, users can also find themselves in one of these two situations. We can observe that product discovery behaviors are incredibly diverse when product catalogs are diversified: our platform offers around 1 million unique items from over 10,000 brands.

On Videdressing, sellers are free to sell any fashion item and set the price. We don’t do any curation. Although we check and guarantee the authenticity of the items, we don’t pass judgment on any of the items put on sale.

Nevertheless, the platform must ensure that the best deals don’t get lost in this vast catalogue, despite the desire of many users to bargain-hunt. Videdressing can’t just be some sort of huge second-hand clothes shop where all the items are mixed up. As mentioned above, it’s all about our users’ satisfaction (sellers and buyers), but also the good health of the business.

Algolia as a technical platform

We have chosen to use Algolia for our search engine as well as for our catalog, and gradually Algolia will also become the technical platform for our merchandising.

This solution meets both our needs and limitations:

• Rapid roll-out to speed up time-to-market.
• The team needed to implement and maintain Algolia is much smaller compared to the team needed for a traditional search engine.
• Focus on the improvement of algorithms and data analysis.
• Quick response time for users and SEO.

It would be interesting to look at the reasons for this choice in more detail (most probably in a future article 🙏)

As with other search engines, a number of criteria would have to be optimized in order to achieve a completely seamless experience:

• Relevance: giving results that best match the search query.
• Ranking: boosting results to offer items that offer the best chance of conversions.
• Personalization: tailoring the results to the user profile.
• Case-by-case optimization (edge-cases): optimizing the research results that have low conversion rates.

We quickly found ourselves at the controls of a huge dashboard.

One of Algolia’s unique features is its Tie-breaking** ranking algorithm, which offers the opportunity to make changes to the ranking on the fly.

**Illustration of how Tie-Breaking works: inverting the two parameters in the formula means the results are different.

A/B test strategy to improve fluidity

For all of the above reasons, we believed we had to improve our search engine. However, we still needed to find the right KPIs to monitor in order to know if we are moving in the right direction. There is sometimes a strong desire to monitor macro indicators such as conversion (number of transactions/number of sessions), the AOV (Average Order Value), or the GMV (Gross Merchandise Value). Instead, we instinctively preferred to focus on indicators that are more closely related to the changes that we have made.

After a few weeks of experimentation, we concluded that this was indeed an excellent practice. We mustn’t forget that A/B testing is a science! While you won’t need to go back over your statistics courses, you will still need to master some basic concepts in order to be comfortable with the resulting decision-making.

https://abtestguide.com/calc/ is a tool that provides a good illustration of the statistical problems linked to decision-making as a result of an A/B Test.

We used two KPIs to analyze our tests for updating our ranking algorithms:

• CTR: the percentage of searches with at least one click on a product.
• The conversion (of searches): the percentage of searches which result in at least one addition to the wishlist or the basket.

While we were initially conducting A/B tests from an internal system, Algolia now offers a feature which makes it possible to centralize the configuration of A/B tests and facilitate the visualization of results (webinar on the subject: “How Videdressing optimized their search with A/B testing”)

The very first test carried out using the Algolia tool.

This is how we constantly test new parameter combinations in the Tie-Breaking algorithm, adding new parameters based on our analyses and by altering the order of these parameters.

We noticed that the test needed to last for at least one week in order to ensure that the traffic was representative enough (a very important parameter needed to be able to confidently draw percentage conclusions). We, therefore, record the conclusions and results of each A/B test. We’re not short on ideas! We already have an idea of what we want to test in the coming months.

However, we found that analyzing the performance of results personalization using A/B testing is a complex task. We can begin by comparing the conversion of users who are or aren’t presented with personalized results. However, users who have voluntarily customized their experience will certainly be more engaged. It’s therefore impossible to draw any real conclusions. We had to create an A/B testing system ourselves using “custom dimensions” in Google Analytics. Also, we consider that analyzing every small personalization parameters can lead to impossible decisions since we wouldn’t probably get enough traffic to ensure acceptable levels of confidence.

It’s very reassuring to see that the conversion rate of users who personalize their results is much higher than for others.

Continually improving our search engine is crucial for improving the fluidity of our marketplace. We perform continuous A/B tests in order to verify the strategic choices made about changes to the engine. We will certainly have to revisit changes made in the past, test them again and not forget the (still statistical) local maximum problem which could later prompt us to test much more radical changes, as opposed to the very iterative method adopted thus far.

An observation? An opinion? Feel free to send your comments!