Background
At The Washington Post, most of our site traffic continues to come from mobile. Traffic trends show continued growth across all platforms year-over-year, with growth on mobile web (mWeb) outpacing the rest. In October 2020, mWeb traffic grew 51% year-over-year, and we had more than 100 million unique visits(UVs) on our site just on mWeb.
Historically, we have seen better purchase funnel conversion rates on desktop. Mobile visitors are more likely to be coming in sideways from sources like Facebook and search than desktop visitors. That less intentional behavior means that mobile users are inherently less qualified (or likely to purchase). However, we believe that there is a significant opportunity to improve the mobile purchase conversion rate for users who visit the checkout page (27%), and we view our average checkout conversion rate on desktop (60%) as an aspirational goal.
I. Product page optimization: toggle and price display test
Earlier that year, we focused on making it easier for a user to choose a subscription plan and go to checkout. From our user research, we learned that potential subscribers had a challenging time understanding our subscription plans and pricing because we displayed the monthly retail rate, monthly intro offer, yearly retail rate, and yearly intro offer options all at once on our product page. Further, users were faced with another decision point (monthly vs annual billing cycle) when they landed on checkout. We hypothesized that we could reduce friction for users by making it easier to compare billing options on the product page rather than on checkout.
We designed and tested a billing toggle experience that allowed users to choose both their subscription plan and preferred billing cycle on the product page. We removed the billing cycle selection step from checkout, reducing the checkout page to just two steps. Additionally, we tested button labels on the subscription plan cards to help users understand the introductory rate savings by displaying the crossed-out full retail rate alongside the discounted price.
Results
The experiment reached statistical significance in one week. We found that the new experience lifted conversion rates on mobile by 8.1% and on desktop by 8.4% compared to control, resulting in a projected $724k in-year impact and a $1.2MM Year 1 impact.
II. Product page optimization: feature display test
The first test accomplished the goal of making it easier for users to compare their billing options. This next test was about helping users compare All-Access and Premium plans. We hypothesized that if we showcased the subscription benefits more prominently and made sure that users could see both plans at once on a small screen it would make plan selection easier. We expected to see a lift in overall conversion as well as an increase in Premium plan purchases. To validate this hypothesis, we designed a five-way split test to determine the best way to display plan features (list or sentence format), how prominent the retail price needed to be, and if showing both subscription plans at once helped users make a purchase decision.
Results
The winning mobile design prominently displayed the plan description as a checklist, removed the large retail price, and brought the Premium plan in to view on a smaller screen. It resulted in a 15% lift in product selection and a 9.49% lift in purchase conversion rate compared to control. It also increased the share of Premium purchases to 13%, as opposed to 5% in control. The financial impact of the winning design is projected at $100k in-year revenue and a $240k Year 1 impact.
We followed the mobile test with a similar test on desktop. The same design won, lifting product selection by 9.9% and purchase conversion rate by 10% compared to control. It also increased the share of Premium subscriptions purchased to 17.8% (versus 14.1% in control), leading to a projected $450k in-year revenue impact and a $2.5MM Year 1 impact.
III. Checkout optimization
Once we felt confident in the new product page design, we turned our attention to checkout page conversion. We noticed a downtrend in conversions on mobile while desktop remained steady.
We analyzed more than 50 mobile user session recordings on FullStory (a UX analytics tool that records user interaction data) to find opportunities and user pain points. From that analysis, we found key user challenges navigating the checkout form.
First, we updated our form field validations on checkout to improve usability and save vertical screen space on mobile. We moved field labels into the fields (instead of above them) and when a user clicks on a field the label text color now matches the field state color. These updates provide helpful in-the-moment cues to users and make validation messages more readable.
Second, we focused on re-imagining the mobile checkout experience. We hypothesized that if we could create a more streamlined checkout by removing unnecessary steps, reducing cognitive overhead, and elevating express payment options (PayPal, Apple Pay, Amazon Pay), we could improve purchase conversion on mobile.
Our hypothesis was informed by previous checkout testing; in early 2019, we ran a mobile checkout test that emphasized credit card payment over express payment options. At the time, we believed that since credit card was our most frequently used payment method, we could improve conversion by optimizing the experience for credit card payers. However, we learned that in optimizing for the credit card payment experience, we inadvertently marginalized express payment options, resulting in an overall lower conversion rate. Restoring prominence to express payment methods was a key part of our strategy for this new test.
Our team worked together to create a mobile-first checkout experience, which we validated through user testing prior to development. The new design was engineered to be responsive so that we could test it concurrently on both mobile and desktop. We incorporated a new shopping basket at the top of the form to make it easier for users to scan and understand the final purchase price. In addition, we eliminated the email address field for those paying by express checkout, and we also eliminated first and last name fields by collecting that data post-purchase. Finally, we revised the header to remove the hamburger menu and eliminate a potential drop-out point for users.
Results
The checkout optimization test ran for twelve days beginning October 8, and the results are statistically significant for both desktop and mobile. Our experiment shifted payment behavior toward express options and away from filling out the credit card form on both mobile and desktop. On mobile, we saw an 8% lift in purchase conversion compared to control, and we saw use of express payments increase from 49.99% of checkouts to 64.55%. This suggests that optimizing for express payment options leads to higher conversion, particularly on mobile. On desktop, we saw a 2% lift in purchase conversion compared to control, and we also saw use of express payments increase from 31.12% to 43.62%. These combined improvements translate to a projected $134k in-year revenue impact and a $1.7MM Year 1 impact.
Next steps:
1. Foreign currencies and express payment options
On October 21, 2020 we launched the new mobile-first checkout to 100% of traffic for U.S. digital and academic subscriptions, and on November 11 we released this experience for our international market. Our ability to improve conversion by shifting payment behavior toward express options has highlighted an opportunity with international users. Earlier in 2020, we launched PayPal as the first express payment method available for foreign currencies. A pre/post analysis of the 2-week period before and after the PayPal release found the addition of PayPal improved purchase conversion rates by 15.4%. In 2021, we plan to seize this opportunity and expand express payment options for foreign currencies.
2. Lead generation and cart abandonment
On November 19, 2020 we launched an experiment to determine whether adding an email registration step to our new checkout experience could drive net value in lead generation and outweigh any potential decrease in immediate purchase conversion. A previous test in August was inconclusive, but we found that the new leads generated from checkout converted at a rate of 16% versus 6% from emails generated elsewhere during the same 60-day period. Given that this showed promise we re-tested the original hypothesis in our new checkout experience.
The re-test ran for 13 days. On desktop, we found that the variant (including email capture) increased purchase conversion by 2.27% compared to control, while mobile saw a statistically insignificant decrease in conversion rate. For now, we have made email registration on checkout the default desktop experience, but we have not yet made that change on mobile. We are evaluating the immediate financial impact and will be monitoring lead conversion for the next 60 days.
We also plan to personalize the cart abandon email experience for these users who register but do not immediately purchase. For example, in email marketing to these users we will test whether including the headline of the story (from paywall) as well as the user’s selected subscription plan and price prior to abandoning improves conversion rates. This same personalization service could later be extended to paywalls and on-site messaging so we can help users pick up where they left off and easily complete their purchase in subsequent sessions.
3. Purchase funnel segmentation
In order to continue to optimize in 2021, we plan to segment our purchase data into more specific funnels that will give us a better understanding of how channel and the purchase journey impact conversion. For example, we want to understand how a user’s entry point (paywall, off-site marketing, newsletter promotions) influences their purchase behavior and how we might optimize for these different starting points.
4. Purchase funnel segmentation
Back in June 2020, we also shared updates on performance improvements for paywall and checkout. We’re doubling down on that and rebuilding our product pages in React. In addition, we are exploring how we might implement server-side rendering for our product pages to achieve the fastest possible page rendering time.
