Journey Towards Data Driven Customer Insights and Development of Next-Best-Action CRM Framework

By: Andrew Larsen with contributions by Kevin Cuxil, Clara Hernandez and Albert Na

At FOX Tech, we’re a community of builders, operators and innovators and each and every day we experiment, collaborate, and co-create to develop the next world of news, sports & entertainment streaming technology. With that, includes learning about you as a consumer and delivering content that we think you will enjoy.

The Data Science and Marketing Technology Teams at FOX Tech work collaboratively to build Data Products and generate Customer Intelligence Insights for stakeholders throughout the company. We’re always looking for innovative ways to utilize our first and third party data, whether it be working towards a better understanding of who our customers are and what they interact with in our products, or activating directly within our Marketing Technology Stack to optimize user behaviors. A great example of both sides of that equation is the build-out of our “Next-Best-Action” Customer Relationship Management (CRM) Framework. We thought it would be a great opportunity to share our approach and experience with it.

Next-Best-Action Modeling

Next-Best-Action modeling, sometimes referred to as ‘Best-Next-Action’ or ‘Recommended-Action’, is a machine learning approach to generating user personalized communication based on the behaviors and characteristics of the customer. The goal is to make communication more effective in obtaining a desired outcome from the user– such as: increased visits, time in app, user engagement, or likelihood of purchase. The idea is that if we can identify which behaviors are most likely to increase the odds a user does what we want them to do, we can in turn target specific users with specific tactics based on their existing activity, ultimately leading to more effective and efficient communication.

At FOX, we’ve built out an early version of a Next-Best-Action system using a combination of Machine Learning (ML), Predictive Analytics, Interpretative Data Science, and Experimentation. Below is a breakout of our journey to get there.

Steps To Building a Next Best Action System

Next-Best-Action systems are generally not out-of-the-box solutions: they typically need to be tailored to the nuances and goals of the specific product to make them effective. However, that doesn’t mean they need to be overly complicated. In fact, a specific goal of ours was to make each step of the process as simple as possible to ensure we could deliver results quickly and develop a system that could be deeply analyzed and understood. Interpretability of our system is of the utmost importance, as we need to be able to understand which behaviors (of many) are most important to focus on, design marketing campaigns that can effectively target them, and in turn, be able to measure the impact.

We took the following approach to building our system:

1. Define a Key Performance Indicator (KPI) to optimize (our Target Variable for Machine Learning)
2. Generate a user feature-set of behaviors (our Predictor Variables)
3. Build a Propensity Model to Identify Key Drivers of behaviors (Key-Driver Analysis)
4. Create CRM campaigns to target Key Drivers
5. Experiment, analyze, optimize

1. Defining A KPI To Optimize

We first needed to work with the business to define and operationalize a KPI that the Next-Best-Action system would be designed to optimize for. With a suite of products spanning both AVOD and SVOD services, there are an endless number of behaviors to track and KPIs to analyze. We settled on pursuing long-term user engagement as the primary KPI, defined as video starts in month 2 and 3 of a user’s lifetime. Our rationale? For our AVOD products, long-term engagement is directly associated with revenue, as increased viewership equals more ad impressions. For subscription services, revenue is perhaps more directly tied to user cancellation and churn. We treat engagement as a leading indicator of churn, and therefore focus on targeting engagement directly to keep users active and to mitigate users’ desire to cancel before they have an inkling to do so.

2. Generating a User Feature-Set of Behaviors

With the goal of simplification and interpretability in mind, generating the user feature set is perhaps the most crucial component to keep simple. We track thousands of events and user behaviors in our apps, many of which are highly correlated, or even entirely dependent, on others. It would be next to impossible to interpret which behaviors are most important if we extracted everything in our database. So instead, we worked with the business to select a subset of user features that we felt were:

1. Likely to be highly predictive of long-term engagement
2. Represented a diverse set of targetable behaviors that were relatively uncorrelated with each other.

Therefore, we kept our data set small so we could feasibly analyze it. Additionally, we made sure that we would be able to effectively tease apart the individual contribution of each behavior in terms of predicting our KPI.

The feature-set ended up including things like the following:

• Number of visits
• Number of episode starts
• Breadth of content library viewed
• Types of devices used
• Types of content viewed
• Consistency of visiting
• Recency of visiting

Among others.

3. Building a Propensity Model to Identify Key Drivers

There are several benefits to using Machine Learning for Next-Best-Action modeling.

First, we can get an idea of how well the feature-set we’ve chosen predicts the outcome variable, which tells us how strongly the behaviors are associated with the outcome. That is, the stronger the association, the more likely we’ll see a positive impact to long-term engagement if we can successfully target the specific behaviors in the set.

Secondly, the propensity model gives us a prediction for each user, given their set of behaviors, and how likely they’ll be to meet the long-term engagement criteria. This allows us to identify which users are more at risk of failing to meet the goal, and who we may need to target more aggressively (or who can potentially be avoided).

Lastly, while most people are aware that Machine Learning algorithms generate predictions, a lesser known feature of many algorithms is that they can produce statistics about the feature set that identify which behaviors are most important for predicting the outcome. We typically refer to this process as a “Key-Driver Analysis,” as the goal is to identify which user behaviors (the “Drivers”) are most strongly associated with the desired outcome. Variable Importances and Shapley values are common methods that help to quantify and rank-order the feature set in terms of importance. They work through a process of sampling the data repeatedly, calculating the impact of each feature in predicting the outcome, and aggregating the results across all samples. Several algorithms in common ML libraries offer variable importances out of the box, and we tend to use a combination of RandomForest and XGBoost to balance prediction accuracy and model interpretability.

4. Defining Campaign Strategies to Target Behaviors

Once Key Drivers of long-term engagement are identified, we can create campaigns specifically targeting them. This isn’t much different than typical campaign creation, other than we’ve dramatically reduced the set of behaviors we consider targeting so that we can:

1. Focus on only the most important
2. Launch highly specific campaigns
3. Personalize messaging to ensure we avoid sending campaigns that target something a user is already doing

In our case, we identified combinations of device usage, specific sets of content, and specific thresholds for number of episode starts as some of the most important features to target.

5. Experiment, Analyze, Optimize.

Ultimately, we need to be able to test and measure the success of our campaigns and iterate on both the system and campaigns to incrementally improve success. By logging the emails we send and the user interactions associated with them (opens, clicks, bounces), we can ingest that data back into our data warehouse (creating a feedback loop). We can then merge it with our user behavior, and evaluate whether we’re seeing an impact to the Key Drivers we’re targeting directly, as well as reaching our ultimate goal–increasing long-term engagement. At last, the long and endless process of optimizing the system begins by tweaking campaigns, the user feature set, the propensity model, and so on.

Moving Forward

It’s been a fun and challenging journey to build our Next-Best-Action framework, and we truly have an endless road of experimentation, refinements, and enhancements across every level of the system. Our biggest challenge moving forward will be continuing to balance the fine line between complexity and interpretability, as we look to add more user behaviors, utilize more advanced modeling techniques, incorporate automated Machine Learning optimizations, and continue integrating with more FOX Products. We look forward to the challenge and helping build the future of data at FOX.

About FOX Tech

Make Your Mark Here.
At FOX, we pride ourselves in shaking things up and making things happen. While being one of the most well-known brands in the world, we provide our employees with the culture of a start-up — fast paced, non-hierarchical, full of smart ideas & innovation and most importantly, the knowledge that each member of the team is making a difference in defining what’s next for FOX Tech. Simply put, we love to do great work, with great people.

Learn more and join our team: https://tech.fox.com