Solve Critical Business Problems with Data Science: A Glimpse into Dynamic FAQ
Sometimes, certain business-critical problems require new and innovative solutions. At Circles.Life, a technology company, our Data Science team builds AI-driven products to drive the company’s mission — disrupting digital services and giving back power to customers. We therefore attempt to solve business problems using the latest Machine Learning technologies.
Let’s dive straight into a business problem to show you what I mean.
Problem Statement
Currently, 10,000 tickets* are raised to our customer service every single week. With an average ticket resolution cost of $1/ticket*, this will cost Circles.Life $520,000/year*.
Can we cut this cost by 20%*? Is this possible?
*Numbers used here are not real figures and are purely for illustrative purposes.
How would you approach this problem? If we break down the statement in a structured and logical way, there are some different axes of improvements we could suggest.
A) Reduce the cost of solving a ticket:
- Decrease number of customer service agents
- Increase their productivity
B) Reduce the number of tickets:
- Prevent issues hence people raising of tickets
- Educate customers better so they solve the problem themselves (i.e. FAQ)
As a customer-obsessed company, we were not keen on threatening the quality of our Customer Service just to save on cost (A), and unfortunately, in a hyper-growth environment, we cannot guarantee a completely bug-free product (B1). Therefore, we opted for the last point: educating and equipping our customers to enhance their journey (B2).
We then narrowed down the project towards creating an algorithm that would provide a personalised and dynamic FAQ to each customer. This is feasible because there is a trend of customers preferring self-help as part of their initial support journey. A global survey conducted by Microsoft in 2017 revealed that 66% prefer self-help first before reaching out to customer support.
Approach
The first question you might ask: how do I identify the kind of problems customers are facing in order to address them through an FAQ?
Well, being in data science, we can make use of what we have. We know the best data available is the support ticket history, which has already been labelled and classified by our customer service agents. Once we know (and can therefore predict) what issues our customers have, it is easy to then map articles that would solve these issues.
From there, there are two components to handle in order to make the FAQ both dynamic and personalized.
1) Seasonality
- Show articles based on seasonality (i.e. customer receives its bill at the beginning of the month, so he/she is likely to need details on how to read his/her bill at this point in time)
2) User Profile
- Show articles based on users (i.e. if a customer travels frequently, he or she might be more interested in learning about toggling roaming on/off or different roaming offers)
The goal here is to design an algorithm that can combine both components in order to achieve the best performance possible.
Machine Learning Pipeline and AI Algorithms
Let’s dive into the technical part. The model aims to map a user vector x ∈ ℝⁿ to a list of FAQ articles p ∈ ℝᵐ where n is the number of available features to represent a user and m the number of FAQ articles. Typically, we use all the available information on the customer (socio-demographics information, telco usage patterns, past tickets history, roaming habits, etc) and m is the number or FAQ articles to show to the customer. In order to cover enough different possible issues without ruining the customer experience (info overload), we have decided to set m=5.
The first step consists of predicting “what issues does the customer have?”. Currently, issues are classified by customer service agents with 3 levels of granularity. For example, an issue could have these levels:
- level1: billing
- level2: ‘how can I pay my bill?’
- level3: payment methods
As one would point out, the number of different possible level1 (~10) is much less than the different possible level3 (~300). The challenge in prediction relies on figuring out the tradeoff between: the ability to predict correctly an issue (level1 will be easier because predicting top 5 out of 10 is much easier than top 5 out of 300), and the relevance of the prediction (to show a relevant article, level3 is much more informative than level1, which is more generalised).
Thus, we decided that the first block of the algorithm will be a mapping F such that F : x → i where i ∈ ℝˡ, l=42 being the number of different level2 issues. After some testing, picking up level2 as a prediction target appeared to be a satisfying solution as it presents relevant enough information while keeping an acceptable number of possible values.
In order to later select the top 5 FAQ articles, we had to build a classifier that can score each level2 independently. The classifier then becomes a set of independent classifiers that are mapping x → iⱼ where j ∈ [1,l] and iⱼ ∈ [0,1] being the probability of an issue j to occur.
Due to the strong class imbalance for each classifier (each classifier predicts 1 out of 42 level2) and the amount of data available, AdaBoost and Random Forest classifiers happened to be the best models for the task — with a 27% F1 score on average across all classifiers. While this might sound like a poor performance, it is actually largely acceptable due to the extremely strong class imbalance.
In fact, if you take the top 5 predicted level2, you have at least 1 which is correctly predicted in 70% of the cases.
Now that we have managed to build a predictor for “what issues does the customer have?”, we need to enrich the predicted scores i with some weighting W ∈ ℝˡ that account for seasonality. W is computed by taking the average distribution of level2 for a specific rolling time window in the last 4 months. For instance, if the time window is a day +/- 3 days and today is the 10th of July, we will take the average distribution of level2 on the period 7th-13th on the last 4 months (7–13th of April, May, June and 7th-9th of July). If it wasn’t apparent, W has to be recomputed every day to ensure the dynamicity of the FAQ.
After obtaining y=W⊙i and filtering the top 5 y with the highest weighted scores, level2 has to be mapped to specific FAQ articles: M:y → p . Currently, this is manually done by the product team which evaluates the most relevant articles for each level2 issue. The final mapping is then pushed to our database daily and is currently serving the Circles.Life app via an API.
Looking ahead in the long term, we can easily imagine one or more algorithms like collaborative filtering to personalise the FAQ articles mapping even further.
Impact and Conclusion
I hope that this article has shown you how you can (or at least, how we) solve business problems using a Data Science approach from breaking down the problem to building a full Machine Learning pipeline.
Using the process documented above, we managed to build and deploy this pipeline within a month. Since then, our ticket index (# of tickets / # of customers) has decreased by 15%* in the following months! It shows how data-driven thinking, AI and automation can impact the business in a sustainable manner and help find hidden patterns in a unique way.
*Numbers used here are not real figures and are purely for illustrative purposes.
