We looked into why our subscribers churned–with the help of machine learning
This is the story of how we took an honest, analytical approach at how big of a problem churn really is and the likely suspects of causes.
We’ve written at length about how to attract more customers. Through robots writing articles, new ways to use our products and not the least working even harder and more methodical at making the best journalism in the local area.
But, we are letting some of our readers down. If we don’t address this, we will only burn through our potential market.
A little background
The Swedish media business has had a bumpy journey into digital distribution. Because of reductions in ad revenue we’ve had to leverage our readers’ willingness to pay for journalism directly. To find new subscribers is therefore essential, but to retain subscribers is even more important since there is only a finite base of subscribers for local news. Furthermore, it is five times more costly to re-acquire a user who left than to find a new user.
When a reader terminates their subscription, we call that “churn”. This is not only a loss of a revenue stream, we see it as the end of a relationship where we could’ve provided information to them and kept them engaged in their local community. From 2018 to 2019, Mittmedia managed to decrease its churn rate from 17% to 11.5%, but then it stagnated, went up and down again. Now, we feel it is time to have a deeper look at the reasons for churn in order to be able to combat the reasons for churn accordingly. What happened? Could we not keep them engaged? Did we deliver too little or too much of something? Were there signs we could’ve acted on? With these, and many other questions, we started our investigation into churn.
This was not the first time we’ve spoken about this subject at Mittmedia. And it’s not the first time we’ve tried to combat the problem. It has been an explicit part of our strategy to reduce churn, but the scope of the issue and the different parties involved meant that there was not one, uniform definition or strategy to tackle it. And, there was no uniform way to determine whether or not a measure was effective.
We had even done some work a long time ago when we looked at readers who had a long subscription. One of the things we found was that they read more articles in general, which led us down the path of pursuing more pageviews. But, the problem was that the measures we took then indirectly measured churn and did not take into account that correlation might not necessarily be causation. Did we keep people as readers longer or were we just tailoring our journalism and products to an already loyal reader base while neglecting others?
And how much of our churn rate was down to the customer experience and service? There could be some insights that reduce customer friction and makes the rest of the departments’ jobs easier.
It was clear we needed to work with churn in a way that aligns the entire company with one strategy, quantify the contributing factors and assess the impact of future churn work in one, holistic approach.
Overview of churn
To start, we needed a definition of churn that accurately described the problem. This sounds pretty straightforward, but the more you discuss it the more alternative viewpoints start to become more apparent. There is a probability that the reader didn’t really mean to churn, how do you make sure you don’t accidentally count those? And how do you make sure you don’t accidentally underestimate the problem by counting readers who cancelled their subscription, then rethought their decision after seeing great content and joined again?
A popular method we found was a time interval after cancelling where, if the reader re-joined within that interval, it was not counted as a churned reader. How long should that interval be? It comes down to if you would rather run the risk of over- or underestimate the problem.
We decided that underestimating the problem was worse than overestimating it. Thus, a customer is considered churned once they cancel their subscription, even if that means having access to the product. If there was a non-renewable subscription, the customer is considered churned once the subscription expires. The reason for this is that to convince a previous customer to start a new subscription is harder than to convince them to stay. By the time they’ve cancelled their subscription, the bar has been raised.
The needs
We started reflecting around the needs we have with churn: What would different departments use churn insights for? How would they apply knowledge about churn?
We gathered representatives from customer relations, the editorial staff, but also from our CTO for a brainstorming session.
First, and unsurprisingly, churn insights would be used for operative business development: How to target customers with a high risk to churn? What are these customers’ characteristics and how do we retain them? From the editorial side, the operative question was more from a content perspective: which content do we need to produce to satisfy high-churn risk readers so that the risk for churn diminishes?
Second, churn insights would be used on a more strategic level. Knowing why customers churn and which kind of customers stay with us, business management would be able to have more information on the features that our end customers appreciate in our products. It could also give a hint on which customer groups we should invest most in for the future. Our editorial could get more insights into what topics to cover to retain our current customer groups, but also what we need to write more of to gain new readers.
Third, the discussion took a turn to add churn predictions, so that the company can move from being reactive to proactive in the long run. Those predictions would be crude at the beginning and evolve in time as the churn model evolves and gets better.
The fourth and last point that came up when discussing needs with churn was that there are churn factors which we can directly act on vs churn factors which we can adapt to. For instance, the age of our readers is not something we have in our control, but knowing which age groups don’t read from us, will help us in forming strategies forward. We decided to take account of those four needs when starting the first iteration of churn models.
Defining the dataset
The dataset we chose was an aggregation of user behavior over their last month. If they are still an active customer that would mean the last month.
We chose the last month to counteract the fact that a user’s activity fluctuates. We were mostly interested in predicting whether a user would churn before the next payment, and we typically have monthly payment plans for our subscriptions. The data consisted of subscription data, such as the users’ age and gender, and their aggregated behavioral data such as how many articles they read during the last month. It was limited to customers who had an active “plus”-subscription for some time during a specific four-month period and a subscription longer than one month. This included users who were completely new during this period and users who had a subscription that had lasted for years.
Model choice
We choose to balance the requirements of our work, which was to make accurate predictions and explain the results. By explaining why a user is likely to leave, we know what to focus on to prevent more churn. Thus, we choose an array of three models with varying degrees of potential accuracy and explainability.
Using three models gave us more benefits. If the models would contradict each other, we would have to look into why that’s the case, whereas if the model agrees, we could be quite sure to trust the results. Furthermore, some models work better with one kind of data, while other data is better suited for other models. If one model would fail to reach a certain accuracy, we would have other models to step in.
All three models i.e the survival analysis, GBM, and neural network improve if fed more data. However, they all more or less delivered during our testing. To compare them in a way that made sense though, we used the same dataset for all three models.
In order to provide the Survival/Hazard and GBM with more relevant data, we did a factor analysis. This is a way to find correlations between the variables we use for the models to eliminate some of them. For example, pageviews and clicks are very related to each other, and both are estimated to have an influence on churn. It is then enough to just bring one of these variables, let’s say pageviews, into the churn model, since we know that clicks are related to pageviews and will not add any new information for the model. The factor analysis also provide us with insights on which variables correlate with churn.
Results
The above graph shows that the number of push notifications which a reader gets and opens correlates highly with churn, whereas reading more articles, being a paper subscriber before becoming a digital subscriber or subscribing via campaigns correlate with a low risk to churn. The following graphs will show some of the issues in more detail. Please note that since we excluded users who had only had a subscription for shorter than one month, the graphs will indicate that surving onemonth is 100%.
Push notifications’ relationship with subscription length
In our apps, users can subscribe to all kinds of topics such as ongoing stories (maybe a court case), geographic areas (maybe the reader’s home town) or specific subjects (maybe politics). Whenever a new article is produced that matches a user’s topic subscription, if the article is tagged with an appropriate news value, the user will receive a push notification. It’s used to reach our users at all times and keep them engaged, unfortunately we found evidence that it had other, undesirable effects.
We expect that the graph about pushes is going to be the most controversial amongst the editorial staff, since pushes has been a topic there since at least 2017. The main purpose with pushes has never really been defined, but now we know that users who open more pushes are also more likely to churn. It does not mean that pushes are bad, but maybe we could take this insight as an opportunity to review pushes and see if we can make them more valuable to our users.
It would also be interesting to dissect the push analysis further to see which kind of pushes correlate the most with churn. Maybe we see a possibility to adapt pushes to reader interests as well, as we do with articles?
Gender’s relationship with subscription length
The results of this graph fits well with an unrelated editorial initiative which is going on at the moment of writing: to find the topics which could attract the interest of women between 30 and 50. We know that we have a skewed reader group where women are underrepresented in texts and choice of topic in our articles. Our churn results show that this is could be a cause of our difficulty to retain women as an audience.
It is interesting to see that the age-groups 25 to 45 are churning the most amongst both genders for our readers. This is coincidentally also the age-group which has most money to spend in comparison to students who tend to be rather young or pensioners. To set this in relation: the average age of a Mittmedia user is rather high at 46 years of age for our plus subscribers.
Revenue from churned and still active users
It is not surprising that we gain more money from customers who stay with us a longer time. From all the users who subscribed during the time period of four months (April to August), we gained on average 1 345 kr from readers who were still an active customer during the last month of observation, but only 707 kr from users who had churned during the last month.
App activity’s relationship with subscription length
We can see that users who read 1 to 80 articles during the month of observation churned significantly more than readers who read at least 80 articles or more. That means that readers who consume about 2–3 articles per day or more are not as prone to churn.
A similiar relationship is also revealed for opening images in the mobile apps.
The more images a user opens, the less likely the user is to churn. That does of course not mean that it is a causal relationship — users who open images more often might just be interacting more with our products in general, and thus enjoy them.
Looking at a more rough picture of churn, we can tell that almost 10% of churn happens within a few months. Thus, this time is critical.
We can segment users into churn risk groups. The NN as well as the GBM can make predictions. The prediction shows that there is a 50% chance that a group of 17 000 of our users could churn within an undetermined timeframe in the future. But most importantly, we should concentrate on the user group which as a 80% — 60% chance to churn. The 80% group has 4618 users and the 60% group has about 12 000 users. We should concentrate on this group since 50% likelihood is not very informative, so we need to wait to see which way those users will swing, whereas users in the 90% likelihood are user who have basically decided to leave us, and it will take a larger effort to retain this smaller group.
We can also compare what separates users with a high risk of churning from those with a low risk of churning.
Finally we compared the users who are 90% likely to churn to the users who are only 10% likely to churn. We can see that users with a higher likelihood to churn seem to be in general less active in our products, since they almost don’t watch videos at all, they are not interacting with the disabling function for push notifications, they don’t use airplay so often and they experience more average payment errors.
Both groups have in common to come in equal amounts via campaigns to us. Campaigns seem to be good to get new users, and they don’t change the likelihood that they will inevitably churn but does change the length of a user’s subscription. It’s interesting that the number of devices used is twice as high with the group of users who have a low probability for churn. It probably indicates that several users are sharing one account, and thus the threshold for cancelling a subscription is higher, since it would cancel the subscription for all people sharing the account.
Conclusions
We believe firmly that correlation does not equal causation, thus we have been careful not to jump to conclusions. We will most likely use the findings to guide us as we spotlight different areas for thorough research and investigation.
However, some things should keep us worried enough to cause action. We are a local news publisher and thus have everyone in the local area as our target demographic. It’s worrying to see that women are subscribers a shorter period than men, and it’s also worrying to see that by far our largest category, sports, is being actively filtered by a large percentage of those who have a low risk of churning to such a degree. While sports is important, this does raise the question of how to produce content that caters to readers who are just not that into sports, too. We also have to be better at listening to what our female readers want.
Finally, we cannot conclude that push notifications are universally bad, but we will investigate it further.
Future work
In addition to our findings, what we are most excited about now it so find out how we can use the model to learn more. By doing tests and using the model for evaluations, we have an effective tool for finding more factors that correlate with churn and to what degree. We see potential in using this as a part of our customer service work to try different approaches at lowering the churn rate of at-risk customers, we see opportunities to produce more content about the topics we’ve neglected and we see interesting challenges in tailoring our products in ways that addresses our customers’ specific dissatisfactions with us.
The immediate way forward when it comes to research would be to look into the most obvious things we found that the editorial staff can use; how gender is correlated with churn and push notifications. Looking into how we can engage more women in how we produce and distribute content is something we’re currently looking into at Mittmedia. Push notifications will probably also most likely be another project where we review them and what we can do to make them contribute to our users’ engagement and retention.
At the very least, this work can give us new insights into our users … and ourselves.
Pontus Ekholm is a developer in Mittmedia’s data team.
is the product owner of Mittmedia’s data platform and the churn project is the CTO of Mittmedia