Why early stage B2B companies should not use churn as a leading metric

Working at a venture fund, an important part of my job is assessing start-ups. Within RIF we assess 500+ early-stage B2B companies on a yearly basis. While the structure and story lines are often very good, most companies can improve on how metrics are presented. For a starting company, it is important to steer towards product market fit from the very start. Next to qualitative feedback, metrics can give an indication of the progress so far and to what extent a company is ready to scale. However, it is important to remain critical. Too often, presented numbers bear little value, because an important statistical effect is neglected. Especially in a B2B context, the limited number of clients lead to problems when calculating unit metrics. In this post I will explain why and how to tackle this.

The problem

It is easiest to present the flaw using a hypothetical — yet realistic — example. Let’s say a founder of an early stage B2B company would present something like below to show the degree of product market fit achieved:

The person clearly uses SaaS unit economics to estimate product-market fit. In this example, he rightly uses churn as an important indicator and the yearly churn is calculated correctly (not annualizing an average monthly churn number). Churn denotes the percentage of customers (or $’s) cancelling the recurring/renewing billing contracts that they initially signed up for. So if Spotify has 100 customers and 1 person leaves every month, the monthly churn would be 1% (assuming that 1 person signs up simultaneously). Churn is generally perceived as a good indicator of customer satisfaction of the service delivered + related price point and if product market fit is obtained. However, presenting or steering an early-stage company using this metric can be misleading.

In statistics, an effect or correlation is never certain. It is somewhat intuitive that we are more confident about a correlation, if the amount of observations (samples) with the desired outcome is increased. With small sample sizes, it is important to acknowledge a phenomena called sampling variance. Sampling variance refers to variation of a particular statistic (e.g. the mean) calculated in a sample compared to its counterpart if the study would be repeated many times. A simple example of sampling variation can be imagined when tossing a coin. We know that the probability of throwing head with a fair coin is 50%. So if we would throw the coin 6 times, it is expected to hit head 3 times, right? Well yes, but that will only happen in 31% of the cases. In other experiments, one would throw either heads or tails more frequently. In those cases, one would calculate the heads/tails probabilities incorrectly if sampling variance was neglected. To prevent this, Confidence Intervals (CI’s) are used. These CI’s denote what the statistical minimum and maximum of the desired statistic is, given a certain degree of confidence (often 95% is used). So expanding on the coin flipping example, let’s say that we would have flipped exactly 2 times heads (the case in 23% of the experiments), the probability of throwing heads could be pinpointed at 0.33. However, CI’s* denote that the probability of throwing heads is likely between 0.08 and 0.71 (using a 95% confidence level). It can be concluded that a sample of 6 has limited explanatory power.

This sampling variance is also present when calculating client churn. Although the experiments are less ideal (every client and their probabilities are different, the product and churn probabilities develop over time), it still makes sense to have a look at its effect. Returning to our pitch deck, it is unlikely that the actual churn will end up at 10%. Using the numbers in the example, the sample size is 10 (in month 5 we signed up a new client), the mean of the churn is 10%. Based on these numbers, the actual churn (using 95% CIs), can fall anywhere between 1% and 38%*. The limited sample size leads to a lot of uncertainty. Moreover, if the variance would be higher, the uncertainty increases even more with the sample size. Let’s say the company has 2 churning customers in month 5, than the actual churn can be anywhere between 4% and 50%.

How to deal with it

Especially in SaaS B2B, the focus is often on blue-chip companies, implying a limited number of clients. It is therefore very difficult to say something meaningful about churn in the beginning. Investors might not have done the math, but feel that this data is unreliable. Analyzing reasons for churn can lead to valuable data and increase product market fit. However, doing calculations based on them is a bridge too far. Being aware of the limited significance and even including CI’s shows maturity.

You are an early-stage founder and confident you have achieved product market fit: how can you make that case? A good option is to focus on data with bigger sample sizes, like operational data from the product. Churn is a way to predict the stickiness of a product, however it normally follows healthy utilization and satisfaction around the SaaS product. Often activity on the platform (f.i. # logins, average time per week, number of modules used) can predict if people are happy with the product and if they are using it. However, the exact leading metric is different per company. Sure, economic factors (pricing, contract types) are neglected, but this is likely tweaked in the future anyway. Again, it is smart to think about the variance within your sample, but given that the amount of seats/users outnumbers the amount of B2B customers, it is probably less of a problem. Naturally, with an increased sample size, the uncertainty can decrease significantly.

This blog title is provocative — I still believe focusing and showing churn from the start is healthy. However, being aware of its limitations and including other data points is necessary to increase the empirical approach when running your company. Hopefully this will lead to companies being better able to assess their product-market fit in the beginning. And I can lift on that new knowledge as well!

• To calculate the CI a Binominal distribution is used, given the binary character of churn. It is assumed that the 10 customers could have churned this year once (12 months contracts) and that clients are independent. Jeffrey’s Confidence Intervals are used given their good behavior with small sample sizes and small unbalanced probabilities.

This story is published in The Startup, Medium’s largest entrepreneurship publication followed by 290,182+ people.

Subscribe to receive our top stories here.