Survival of the SaaS Customers

Sri Oddiraju
Aug 25, 2017 · 4 min read

An introduction to SaaS metrics and analysis through an epidemiological lens

When folks talk about SaaS products, they often focus on three key metrics: customer lifetime value (LTV), customer acquisition costs (CAC) and customer churn (attrition). Product and strategy teams obsess over how changes to their offering affect these core metrics that are the lifeblood of a SaaS company.

Yet, when it comes to analyzing customer behavior after a pricing change or assessing the impact of UX modifications, we find it difficult to design, analyze, and communicate our experiments. A/B testing requires scientific rigor and as a result, epidemiology provides an excellent toolbox for analyzing SaaS customers.

Epidemiology is, broadly speaking, the study of risk factors and patterns of disease in populations. While it may seem somewhat peculiar to consider a product as similar to a disease, consider the approach sales and marketing teams have towards increasing product adoption. For example, focusing on specific regions at a time or offering customer referral incentives are both tactics geared to spreading product adoption in disease-like ways.

Whether you buy into the analogy or not, epidemiology provides rigorous frameworks for communicating customer trends.

Customers are effectively “exposed” to new product updates or events and their longevity with a business can determine success or failure.

Walking through two examples:

1.) Cohort Analysis

Consider a SaaS company started in January that has 1000 customers by the end of October.

In November, 300 customers attrit and 100 are gained, resulting in 800 customers.

The problem we face here is: how do we contextualize the 300 customers lost? 300/1000 or 30% is one measure of attrition, but that does not take into account that some of these customers are “older” than other customers in terms of how long they have been using the product. Some of these customers joined in January, some in February, and so on.

We can split customers into cohorts based on when they adopted the product or based on another “event” that the customers were exposed to such as “service outage on May 31st” and analyze lost customers by cohort.

However, even customer cohorts based on month are error-prone since customers who join on March 31st are still considered as part of the March cohort, but are 30 days younger than customers who joined on March 1st.

This all leads to a more transparent metric, based on the concept of person-time from epidemiology. Each customer’s time with the product is tallied and grouped.

For instance, customers who joined October 10th and left October 20th would have ten days of observable person-time just as a person who joined on January 12th and left on January 22nd would also have ten days of observable person-time. These two customers would be placed into the same cohort of 10 days.

What you will likely see from this analysis is a cohort of customers leaving on the day when a free-trial expires (typically at the 30 day point). Other cohorts may stand out as particularly affected, which may warrant deeper investigation for solutions.

Similar approaches are used to study disease survival rates amongst groups observed over the course of different time periods.

2.) Attack Rates: Identifying Multiple Treatments

The challenge of analyzing customers is further complicated by the fact that customers have different experiences with a product. For example, some customers may subscribe due to a sales call, while others will sign up online. These customers may then have an introductory tutorial to the product through a video or through customer support.

These events are effectively, in epidemiology terms, different “exposures”. Identifying different exposures is crucial to assess causal relationships. Consider the following case on food-borne illness, from Leon Gordis, Epidemiology:

From Table 2–5, we can infer that eating resulted in sickness, but it is not clear whether it was the beverage or egg salad which caused the sickness.

From Table 2–6’s cross tabulation, we see that eating the egg salad has a significant impact on sickness while drinking the beverage does not. By identifying multiple exposures and analyzing the effects, we can propose more targeted solutions such as maintaining colder temperatures for egg recipes.

Relating this case back to our analysis of SaaS products, let us assume customers either join a product through direct sales calls or through online sign-up (“ate” vs. “did not eat”, respectively). After joining, a customer can either watch a video tutorial on how to use the product or choose to get a phone call tutorial from customer support (“Egg Salad” vs. “Beverage”, respectively).

If we focus on the direct sales vs. online sign-up view alone, we may be inclined to find fault with the sales calls. However, by identifying relevant exposures or customer “touches”, we see that the video tutorial is the culprit for customer attrition. As a result, the firm can focus on the underlying problem, and revise the video tutorial script with input from customer support.


Of course, these are only some basic applications of epidemiological views of SaaS customers. Diving deeper, we find many more tools such as odds ratios, case-control studies, cohort studies, and Kaplan-Meier survival plots. All this goes to say, we need not reinvent the wheel when it comes to analyzing our SaaS customers.

Just hire an epidemiologist and avoid the egg salad.

)
Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade