Expo Experiment Stories 2
This is the second article in a series of articles around interesting experiments we’ve run on Expo at Walmart Labs and how we analyzed results. Note that actual metrics have been changed and do not represent actual site performance, except to indicate directionality.
Guest Cart for Online Grocery
By Arpit Kumar, Senior Manager, Product Analytics
Overview
Guest Cart is guided by a simple premise: the lower the friction a user encounters while building a basket, the higher their propensity to place an order. In other words, the further users are down the ordering funnel, the more committed they are, and the more likely they are to put up with the friction of having to sign in or create an account and less likely to drop-off.
Guest Cart Round 1 — an A/B Test
Before the Guest Cart feature was rolled out to entire population, the users who came to Online Grocery were forced to sign in whenever they tried to: a) Add something to their cart, b) Change their store. Backed by the above hypothesis, we launched the first iteration of Guest Cart in Q1 2018 to move the signing in requirement down to the checkout:
Control:
Need to sign in before they tried adding anything to their cart or changing their store
Variant:
Free to perform the above two actions without the friction of signing in, sign in only when the user initiated checkout
However, the Guest Cart Round 1 failed to deliver the expected results. The major reasons unearthed for the poor performance of the feature were:
- The landing page once the user initiated checkout: Instead of being funneled into the checkout flow, the users were landing on the homepage or any other page that they initiated checkout from after signing in, which led to high drop-offs.
- No clear messaging regarding what happens to items when users have both Signed-In Cart and Guest Cart
Imagine a scenario where you are adding items to your Guest Cart:
Guest Cart items (qty): Bananas (8), Apples (4), Orange Juice (2) Guest Cart store: Milpitas, CA
But what if you already had items in your Signed-In Cart?
Signed-In Cart items (qty): Bananas (2), Avocados (4), Orange Juice (2) Signed-In Cart store: Sunnyvale, CA
How should we handle the above scenario and what percentage of population was facing such a scenario?
Guest Cart Round 2 — Learn and Iterate
Learning from the above use cases, we launched Guest Cart Round 2 in Q2 with the following modifications for the variant:
- When a user initiates checkout in Guest Cart variant, he will be funneled into the checkout flow (Stock up Page) instead of him landing on the Homepage after signing in.
- In case of a user already having both Guest Cart and Signed-In cart (scenario explained above), we would merge the two carts and change the store to the Signed-In store (MERGE SCENARIO).
- Show clear messaging to the user that the items or store in his cart might have changed once he signs in from Guest Cart variant.
MERGED CART -> Add items from both Guest Cart and Signed Cart and keep Signed-In store as checkout store
Merged cart items (qty): Bananas (10), Apples (4), Orange Juice (4), Avocados (4)
Merged cart store: Sunnyvale, CA
The Merged Cart Scenario was faced by the qualified population on Desktop and the qualified population on Mobile.
During the course of this test we uncovered that the users who encountered merge scenario had lower probability of placing an order after initiating checkout compared to the users who did not see merged cart messaging.
On further analysis, we discovered that the most of users who faced Merged Cart scenario were removing items from their cart or changing their store after initiating checkout which added a lot of friction to placing an order thereby impacting our overall conversion metrics.
One positive thing that came out of this iteration was that the number of orders placed by users who created new accounts went up slightly.
Guest Cart Round 3 — Learn, Iterate, Succeed
Guest Cart Round 2 proved that merging the Guest Cart with the Signed-In Cart may not be the best possible solution. So, we performed one more iteration of Guest Cart in Q3 by introducing a third variant. This new variant will not merge the items in Guest Cart and Signed-In Cart. We called it the SWEEP SCENARIO.
In case of a user having items in both Guest Cart and Signed-In Cart, the Guest Cart will override the items in the Signed-In Cart and the user’s checkout store will remain the one that he had in Guest Cart.
Guest Cart items (qty): Bananas (8), Apples (4), Orange Juice (2)
Guest Cart store: Milpitas, CA
Signed-In Cart items (qty): Bananas (2), Avocados (4), Orange Juice (2) Signed-In Cart store: Sunnyvale, CA
Control:
Ask users to sign in before adding to cart or changing store
Variant 1 (Merge):
Merged cart items (qty): Bananas (10), Apples (4), Orange Juice (4), Avocados (4)
Merged cart store: Sunnyvale, CA
Variant 2 (Sweep):
Sweep cart items (qty): Bananas (8), Apples (4), Orange Juice (2)
Sweep cart store: Milpitas, CA
After 2 weeks of testing, Variant 2 proved to be a clear winner and resulted in an increase in both Add to Carts and Checkout Initiations per visitor.
The increased checkout initiations converted to a lift in Placed GMV, Bought Items and Placed Orders per visitor for the qualified population of the test.
Continue Iterating, Continue Testing
Though Guest Cart Round 3 positively impacted all of our business metrics, we believe that we can still improve the user experience. There might be some proportion of population that prefers the merged experience over the sweep experience.
To address their situation, we are hypothesizing a Guest Cart Round 4 geared at further improving the user experience in a way where users who sign in will not have to lose the items which were previously in their Signed-In Cart. Instead, the Signed-In Cart items will still be accessible to the user in the even the Guest Cart Sweep Scenario occurs. We are optimistic that this iteration will also bring a better customer experience for our Online Grocery users.
This iterative testing allows us to apply the learning from previous tests and make incremental improvements to the user experience. Without A/B testing, we would not be able to measure the impact of the changes as accurately.
