Experimenting: A Case Study from a BE Bootcamper

Josh Ly, guest author

This post is brought to you by BE Bootcamp attendee Josh Ly from Livongo. As part of BE Bootcamp, students learn how to design and implement experiments in the corporate context. Josh applied these learnings to an e-mail campaign experiment with the goal of helping more people manage their diabetes.

Background

According to a 2015 report from the CDC, more than 100 million adults in the U.S. are living with diabetes or prediabetes. Diabetes is a serious chronic condition that can lead to other serious health complications. However, when equipped with the proper lifestyle habits, such as a healthy diet, regular physical activity, and appropriate medication, people with diabetes can reduce many of these health risks. As such, behavior change to create long-term healthy habits has become an important answer to managing diabetes.

Livongo provides a whole person platform that empowers people with chronic conditions to live better and healthier lives. Many of the people with diabetes who are eligible for the Livongo program are covered through their employer’s health plan, giving new members a cellular-connected blood glucose meter, unlimited refills of strips and lancets, and access to personalized guidance from a Certified Diabetes Educator at no cost to them. One of the ways in which we communicate to individuals about their eligibility for the program is through email. After reviewing our messaging and communication strategy, we realized that there was an opportunity to amplify the immediate benefit of enrolling.

Livongo Device & Welcome Kit

Experiment Overview

We conducted an email experiment with our Spring seasonal campaign that we send out to all eligible individuals. New members who complete the registration process receive a Livongo Welcome Kit containing strips, lancets, and an advanced blood glucose meter that provides personalized insights with each check. We tested whether changing the thematic copy elements from “Join the Program” to “Claim Your Welcome Kit” in the subject line, headline, call-to-action button, and email body would impact click-through rates. From a behavioral perspective, “claim” leverages the following:

1. Social Proof — It highlights a sense of scarcity by implying other people are claiming this limited resource

2. Endowment Effect — It creates a sense of ownership over the limited resource

3. Halo Effect — It departs from any possible negative connotations associated with the word “program”

“Join the Program” email — Control

The “Join the Program” email served as our control condition and our “Claim Your Welcome Kit” email served as our experimental condition.

“Claim Your Welcome Kit” email — Test Condition

Results

We emailed over 15,000 people with diabetes eligible for the Livongo for Diabetes program. We saw statistically significant differences in engagement between our two conditions. Our “Claim Your Welcome Kit” condition increased open rates by 25% and click-through rates by 88%.

Impact

Our experiment increased our registration rate within 1 week of the email outreach by more than 120%. The results has strongly influenced how we measure the potential impact of applying behavioral principles to seemingly minor elements of our communication strategy. As a result, we will continue experimenting with the “Claim Your Welcome Kit” theme across other touchpoints with our potential members, including our direct mail campaigns, landing pages, and other emails.