Donor Churn prediction for a not-for-profit organization

Devendra Chilwal
Nov 3 · 5 min read

Recently I had the chance to work for a not-for-profit organization and assist them to set up a data-driven analytical solution to predict donor churn and help retain regular giving donors. As the organization is 94% community funded, it is imperative to focus on donor retention because of the steep initial cost of recruiting first-time donors. The task for me was to identify donors who are prone to churn, through so that the organization can initiate pro-active action to retain them. As I had not worked on the churn prediction problem before, coupled with the fact that it was the organization’s first foray into data analytics, I decided to take on the project.

“Donor retention continues to present one of the greatest risks to future income stability, as most mid-value, high-value and bequest giving comes from long-term, single-gift donors.”
-Fiona McPhee, Head of Insights & Strategy, Pareto Fundraising

For context, donor churn or attrition is the percentage of donors who decide to cancel the association with the organization. According to The Fundraising 2018 Effectiveness Survey Report by Pareto, donor retention is trending downward and is at 45.5% in 2018 and it hurts nonprofits’ bottom lines, affects their ability to keep their programs funded, and leads to wasteful spending.

Source 2018 Fundraising Effectiveness Report

Churn prediction is hard

The first step in the process to identify a solution is to define the problem itself. How do you define a churn metric? Is there a standard definition for it? Isn’t it just a binary classification problem? Not entirely! Although it seems trivial, it is relatively challenging to nail down the definition of churn. The Netflix-class action lawsuit, where shareholders felt mislead over reported churn-rates, showed that all stakeholders seem to be confused about how to define churn. The court even dropped the case concluding there’s no official definition of it! Its definition can vary depending on business requirements and problem context. After many literature reviews and client discussions, we came to a consensus on how to define churn and calculate the target variable. Key takeaways from this exercise are to understand the problem in the context of the domain and when in a doubt get help from stakeholders.

Project workflow for donor churn prediction

Data exploration is key

Once the churn metric was defined, it was time to work on the solution. The next step was to explore the data, which we were cautioned was quite messy with lots of inaccuracies. Inconsistent data between features (e.g. deceased date is populated, but deceased status is false), columns with more than 90% missing values, columns with inaccurate data were merely some of the issues that we encountered with data. The process of cleaning data and creating a data dictionary cost us 20% time of the project but it improved the productivity and efficiency of the project.

Research papers are a great source of Information: Throughout the course of this project, I increasingly relied on research papers to comprehend how this problem can be approached in alternative ways and what insights have been found by researchers working on a similar problem. Multiple research studies found that demographic features like age, gender, income, religious beliefs, etc. tend to define donor behavior. Attributing to this information, it was decided to scour external data sources to include more demographic features in the dataset. ABS offers an extensive source for data that has been collected diligently and can be used confidently in the model. Income data from census 2016 was extracted, cleaned and merged with the present data. We also invested a considerable amount of time on engineering features from the available datasets. Multiple features were engineered to capture information like donor’s longevity, commitment to donations, amount donated, failed/successful donations, etc. The idea behind this task was to identify features from the donor’s demographic and/or payment details that can assist us in explaining their donation behavior. A valuable lesson learned here was that research papers are extremely important to explore new ideas as well as save time by not trying out things that clearly don’t work.

Models: Once the analytical dataset was prepared, it was time to develop some models. A binary classification linear model was used to identify whether the features were statistically significant and used accuracy measures like the area under the ROC curve and confusion matrices. The extremely high accuracy values caused us to realize that there is a class imbalance issue in the dataset, and looking at the accuracy and RoC would be incorrect. As we wanted to identify the donors who are likely to churn, we used precision and recall to measure how good the positive predictions are. It was critical for us to realize that standard accuracy measures are not suitable for every problem and that class imbalance remains an issue that we need to account for in a binary classification problem.

After using SMOTE to synthetically generate data to resolve class imbalance issues, we iterated over more models and used backward feature selection to find an optimal subset of features. Over multiple iterations, we were able to develop models with precision and recall around 0.70. The AUC values were quite high, but that could be attributed to SMOTE possessing the tendency to introduce bias in the dataset.

Variable importance for churn prediction

Clients were equally interested in interpreting how certain features affect the outcome, for which we used a random forest model to obtain partial dependence plots. Through this activity, we were able to identify some intriguing patterns in how a donor’s payment history and demographic features impact their probability of churning. We were able to conclude that features that explain the donor’s longevity and frequency were the most significant features, whereas some of the demographic features were rather less important but still significant.

Conclusion: The journey through this project was quite a learning experience for me. I had several learnings, not restricted to, the importance of a data science workflow, data exploration, literature review, continuous discussions with clients, the flexibility of moving between phases of development, etc. Although I wanted to try out a few more things, I am delighted that we're able to deliver a product that was usable for clients and make recommendations for future churn mitigation plans.

The codes for this project can be found here. Have any questions, connect.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade