A/B Testing for better health care
How small changes can make big differences
Our Digital Member Experience team (lovingly abbreviated to DMX) is charged with building out Oscar’s web and mobile apps. Over the years, we’ve introduced exciting new features to our app, from a doctor appointment booking flow to a messaging platform that members use to chat with their Concierge teams (to name a few features). But no matter what gets added or tweaked, our apps’ key purpose was, and always will be, to help our members navigate the confusing world of health care.
To evaluate how well we are serving member needs, we use a range of tools, including feedback from our Concierge teams, direct feedback from members via surveys and user research, and analytics. One of the tools we’ve built on top of analytics is our A/B testing platform. We use an internal tool named Belfry, which enables engineers to quickly configure and manage tests, and outputs all the resulting data in a centralized spot for analysts to access. And as we’ve found ourselves iterating on older features or evaluating the effectiveness of new ones, we’ve increasingly relied on A/B testing to determine how well we’re serving our member needs.
Experiment #1: Are icons clearer when accompanied by simple text?
Members have been able to message their Concierge teams for over a year now, but recently we asked ourselves whether our messaging icon might be easier accessed through a clearer call-to-action, that matched the language present in our header: “Get Help”. Previously we’d had a more minimalist design with an inbox icon, but no text. We wanted to increase awareness among our members of the option to message their Concierge team, and direct them towards their Concierge teams when they were in need of help, to cut down the time between when they searched for and obtained care.
As it turned out, members who saw the experimental design with the “Get Help” label were 15% more likely to send a message than members who saw the control design. The difference in adoption rate was even greater (23%) for members who had never used secure messaging before (which includes most of our new 2018 members). Another outcome was 9% lower call volume in the experimental group, likely because those members were able to type out a quick message instead.
Final Result: The new design is now live for all members, and we’re changing iconography on mobile to match as well.
Experiment #2: Are people more likely to book doctor’s appointments from a menu of options or a detailed doctor profile?
Oscar members can book doctor’s appointments with the click of a button through Oscar’s app. But the question of where to insert the booking flow was a hotly debated issue. Some people on DMX thought members would be more likely to book if they were launched directly into the appointment booking from the search results page, where they could get a sense of doctors’ availabilities.
Others thought that it would be useful to send users to a doctor’s profile page (where they’d have access to a complete set of information) after selecting a search result, and that they’d be equally likely to enter the appointment booking flow from there. Before moving the final step of the scheduling flow from search results to doctor profile pages, however, we needed to confirm that we weren’t inadvertently cannibalizing scheduling activity.
Ultimately, there was no significant difference in numbers of appointments booked between the experiment groups, though members who went through the experimental scheduling flow were 8% more likely to view a doctor profile than members who went through the control flow.
Final Result: Because it’s beneficial for members to know more about their doctors before visiting, we decided to locate appointment booking within doctor profile pages.
Experiment #3: Do banner photos of doctors provide any utility to our members?
Physicians who enroll in Oscar’s Provider Partner Program — a program that helps match members with trusted, high-value, trusted providers that get first access to new Oscar provider tools — receive large format banner photos on their profile. While these photos are distinctive, they represent a significant design constraint when iterating on provider profile pages. DMX believed the photos would be well-worth the time if they had a meaningful impact on scheduling or visit conversion, but that otherwise, developers’ time could be better spent working on other features.
To evaluate whether the banner photos made a difference, we tested changing partner program profiles to our standard photo display for 50% of traffic to these pages.
There was no significant difference in scheduling attempts or doctor visits when comparing members who viewed the large format photos vs. our normal format photos.
Final Result: We determined that large format photos weren’t a must-have, and 100% of traffic will now see the normal format photos, unblocking several design improvements for all provider pages.
In the world of health care, few decisions are simple or straightforward. Testing changes to key flows helps us validate that our product and design decisions are driving a better experience for members. There’s an infinite supply of new questions that arise each day, and A/B testing helps us at Oscar tackle them methodically, with our members’ best interest in mind.
