Churn at gobble
While every one loves food, I am super crazy about food. However, it is really hard for me to find great food to eat when I am busy, which is like 80% of my life. The reason is simple: (1). I do not really have time to drive to a restaurant, and then wait to be seated, and then order, and then wait, and then check; (2) I failed almost every time when I tried to cook for myself. I was excited to find out that the meal delivery service has been exploding. Gobble is such a company that delivers ready-to-cook dinners to your home every week so that you can make great food to eat at home within 10 mins! With this great idea implemented, gobble managed to attract a lot of users in the past three years; however, the company also found that many cutomers churned after several weeks. Consequently I was tasked by gobble to predict and analyze churn for gobble.
Here is some background information about how gobble works. When a user signs up on their website, gobble will send them menus for each week, and then they can either choose to purchase or skip orders for each week’s menu. If they get the order but they do not like it, they can ask for cancellation and refund. Users need to cancel the subscription to stop the service.
The company has two goals:
(1) To build a churn prediction model that can calculate churn scores for each user in order for marketing department to take action on likely-to-churn users.
(2) To identify the top predictive features of churn in order for the product department to conduct A/B test. These tests will help to find the most cost-effective interventions to resurrect customers.
II. Problem Scoping.
The first challenge I met in approaching this project was how to segment time along users’ life time. This is difficult for two reasons. First, churn prediction needs to be prognostic. Consequently, I need to decide the appropriate length of time to convert signals to features. Secondly, customers might churn at different times and the reasons for customers churning at different times might be different. While the segment of time can be arbitrary to some extent, I think there are actually three things to consider when choosing this segment:
(1) Internal cycle of the product: for gobble, it generates new sets of orders every week.
(2) The length of time the company needs to take action to resurrect churning customers ---- several days.
(3) The granuality of the data.
In this case, I plotted the lifetime of all users and then looked at the quantiles of the lifetime of all churned users. I found out that half of churned users quit in the first two weeks after sign-up. Moreover, a quarter of users churned between day 14 to day 60, and lastly, a quarter of them churn after day 60.
Thus I decided to categorize the churned users as such:
(1) light users (churned from day 0 to day 14)
(2) intermediate users (churned from day 14 to day 60)
(3) heavy users (churned after day 60)
Given that I only had 2–3 weeks to work on this project, I decided to focus on intermediate users. In other words, I focused on predicting and analyzing customer churn between days 14 and 60. Because the model needs to prognostic, the only features I can use are information available from day 0 to day 14.
III. Data and Pipeline
I first labeled the training set by creating a status variable as either:
(1) churned between day 14to day 60, or
(2) still active after day 60.
Then I fetched four sets of features from the company’s MySQL and Postgres databases:
(2). personal subscription settings;
(3). customer purchase behaviors;
(4). website activities.
For the first two sets of features, this information was static and available from day 0 once a user signs up.
For the second two sets of features, I had to filter all behaviors and activities to be before day 14 using the timestamp variable associated with each activity.
After data processing, I engineered around 40 features for the prediction model. More specifically, the features included the number of orders purchased, the number of orders cancelled, the number of orders skipped, the number of emails opened, the number of emails link clicked, the reviews, and so forth.
In the modeling step, I tried various classification models, including logistic regression and random forest. Because the random forest did not perform better than logistic regression, and after communicating with the company’s needs, it was clear that Gobble wanted an easily interpretable model, in the end, I chose logistic regression as the final model. I tuned the parameter of logistic regression using gridsearch. I also specified the regulization to be L1 because I wanted a parsimonious model that tells what top predictive features are. Finally, I obtained a model with the following performance on cross-validation:
AUC score: 70%
While the accuracy is only slight better than the random chance (Null accuracy is 58%), the model has a relatively high ROC-AUC score and in particular, it has a high recall of 80%. Recall is the metric to optimize because in this case, false positive is okay but false negative has bad consequences. It doesn’t hurt much if we mislabel some users who are not likely to churn as churn, but it would be really bad if we missed any users who are likely to churn. Given that the signals captures by the behaviors and activities only within the first 2 weeks are actually few (because a large proportion of the users did not purchase many orders right after signup), the performance of this model was highly applauded by the company.
Now that I have already obtained a prediction model to calculate the churn score for users, I went more deeply in analyzing the top predictive features to extract insights for the company to conduct A/B tests on. Intuitively, I found that number of orders cancelled and the number of negative ratings in review are highly predictive of churn.
More importantly, I found that the number of orders viewed by customers are negatively predicting churn. In other words, the more often a customer viewed the orders, the less likely that the customer would churn. This is reasonable in the sense that it is important to remind busy people that they have something great prepared as dinner for them beforehand, otherwise people get distracted by so many other food options.
Thus, based on this insight, I would recommend the company to either redesign the format of menus or change the way they push the orders to customers so that people get more attracted to the menus and incoming orders. Right now, the company sent out emails with information on newly coming menus in the format showed in the left.
As a user, I find this email can be redesigned to be more attractive, mainly because there is too much texts while the picture is relatively small. I find myself and most other people get particularly attracted to food (or people) when we see irresitabe beautiful pictures. Thus, I suggest that if we could design the email notifications in the format showed in the right (or something similar), people would feel more hooked up when they see these pictures. If each picture is linked to the page of order purchasing on gobble website, people would naturally follow the link to the gobble website.
The other possible change that can be made is to use alternative channels of marketing. For example, when users have not clicked the link through emails after 2-3 days, maybe we can try to send out the order information via texts.
Of course, because the coefficient from logistic regression model represents correlation instead of causation, the inferences and associated recommendations need to be approached with A/B tests. I would suggest the company to conduct experiment on the way they present newly coming food menus to resurrect more users.
Combining demographics, subscription settings, purchasing behaviors, and website activities in a logistic regression produced a predictive model that can be used to calculate churn score right after the first two weeks. One possible way the company can resurrect users is to get people view the menu much more often.
I want to say thank you to all fellow fellows and mentors at Insight, and data science team at gobble as well for their support in the process of accomplishing this project in two weeks. The process was challenging and inspiring! I really love the product at gobble and feel extremely lucky to help in understanding and improving it!
Thank you all for reading! If you have more questions regarding this project, please do not hesitate to contact me at email@example.com.