Expo Experiment Stories 1
This is the first in a series of articles around interesting experiments we’ve run on Expo at Walmart Labs and how we analyzed results. Note that actual metrics have been changed and do not represent actual site performance, except to indicate directionality.
Auth Retry for Soft Declines
By Ling Jing, Product Analytics Manager, Payments
Background
On Walmart.com, after people click on the “Place Order” Button, the backend starts sending payment attempts to the payment Gateways. During this process, some of the attempts fail due to Credit Card Authorization declines from the issuing banks. Of those failed attempts, most of the declines were classified as Soft Declines. Soft Declines are usually generic issuer errors such as server timeout or issuer unavailability, vs. hard declines which include items such as mismatched CVV, address, name or incorrect card number. Traditionally with soft declines, the site sends the customers back to the payment page and ask them to re-enter their payment information. It was observed that this flow caused some customers to drop off; Historical data showed some of the soft declines never resubmitted their payment information. With a big population drop off from Soft Declines, we knew we had a big opportunity here.
Features
To save the orders dropped off from Soft Declines, we implemented a feature that automatically resends the same card/name/address information the customer had previously entered after the payment receives a Soft Decline. The response of the automatic attempt overwrites the initial Soft Decline status. This version retries once, and the response on the auto retry is the final status of the attempt and was tested against the original version using Expo.
Procedures
The product analytics team suggested ramping plans and recommended on when to ramp up. They monitored both the operational and conversion metrics and sent readout the day before the ramp up day. The operational metrics used were “Attempt Level Auth Success Rate” and “Order Level Auth Success Rate” since this experiment is on the Auth attempt level and we want to make sure things looking good for both granularities.
Results
The final results showed that the Auth Retry feature resulted an incremental lift in GMV per visitor for credit card payments on Walmart.com. There were actually 2 factors that contributed to the GMV lift: Lift in Order counts and Lift in Average Order Size(AOS).
Historical data suggests that orders with high basket size are more likely to be Soft Declined. Therefore, this feature is helping on orders with higher basket sizes which were initially soft declined.
Conclusions
This feature was really successful in terms of the level of effort to achieve and the amount of revenue lifted. The great results of this also inspired us to apply the same feature to other business units like Photo, Vudu and Grocery. We look forward to seeing more successful stories from future experiments.While the results may have seemed obvious, running an AB test helped to validate and quantify the actual lift generated, which is a big reason for running tests to begin with. We look forward to seeing and sharing more stories from future experiments.
