How Urban Company leverages Causal Inference to power Data-Driven decisions
By — Kshitiz Swaroop ( Data Scientist, Data)
In a world overflowing with data, finding correlations is easy. The real challenge is understanding causality.
This article presents 2 case studies where the methodology was used to improve the organization wide estimates on the revenue impact of various metrics.
Correlation vs Causation
Correlation is a quantified statistical measure that describes the size and direction of a relationship between two or more variables. Causation indicates that change in one variable is the result of the change of the other variable. This is also referred to as cause and effect.
A correlation between variables, however, does not automatically mean that the change in one variable is the cause of the change in the values of the other variable.
Causal Inference
Causal Learning and Inference allows us to precisely assess the impact of a variable on a given variable.
Causal inference is a robust framework in data science and machine learning that goes beyond identifying correlations to determining the effects of specific interventions, actions, or policies. It enables us to explore “What if…?” scenarios with confidence, even when conducting randomized controlled experiments is impractical due to costs, time constraints, ethical concerns, or feasibility. For example, Urban Company would never intentionally create poor customer experiences to measure their impact on revenue or customer lifetime value.
Using causal inference, Urban Company measures the impact of various interventions, such as discounts, on long-term outcomes like retention or Lifetime Value. By employing these methods, Urban Company can make informed decisions without the need for impractical or unethical experiments.
Personalized impact through Causal Machine Learning tailors interventions to individual needs, optimizing outcomes. Here, an intervention means a specific action done to obtain a desired outcome eg. a discount coupon, visibility of desired slots for booking, on-time arrival of a professional to a customers house etc. This approach allows us to understand the impact of an intervention on every individual, rather than the conventional approach of taking average observed impact across individuals in the population due to an intervention and assuming the same as the impact of the intervetion.
Causal machine learning methods aimed at personalized impact analysis try to address the variability in outcomes of an intervention by answering questions like:
- What is the effect of a treatment on different subgroups or individuals based on their characteristics or other predictive variables?
- How can we tailor interventions or treatments to maximize their effectiveness for individual patients or users?
The appendix lists out the meta-learner based methods used in Urban Company to drive Causal Learning.
Introduction to Urban Company Marketplace
A brief description of various terms referenced later is provided here. Urban Company is a marketplace that connects consumers needing home services (e.g., plumbers, carpenters, beauticians) with trained professionals offering those services. A consumer visits Urban Company, selects the required service, and is directed to a slot page where they can choose a preferred date and time for service delivery. Urban Company may block certain slots if it’s believed that no professional will be available; this is referred to as a Slot Block. Once the consumer selects an available slot, a request is generated with the service and slot details, which is then shared with professionals on the platform. Professionals can choose to accept or ignore the request. If no professional accepts the request, the event is termed a No-Response.
Case Study 1: Impact of No Response on Consumer Life-Time Value (LTV)
Urban Company is fundamentally a marketplace connecting customers with trained professionals. When a customer submits a request, they expect a professional to arrive promptly and fulfill the request to the highest standards. If no professional responds or agrees, it leads to a disrupted experience known as No-Response in UC. UC utilizes an ML model (details here) to ascertain if a request will receive a response.
A crucial input to the model is the impact of No-Response on the customer’s lifetime value. Previously, business and product managers estimated this impact by simply examining the customer’s lifetime value after a No-Response event, averaging it, and subtracting it from the average LTV of customers who did not encounter a No-Response when placing a request.
Naive Analysis
The impact of NO Response on LTV was 1.5 (5.9–4.4). This figure was used organization-wide to prioritize actions, including de-prioritizing revenue loss to reduce NR. Typically analysis stops at this level in most problem statements.
Causal ML Analysis
The Conditional Average Treatment Effect (CATE) determined by the X learner was 0.7. Upon deeper examination, we found that while the X learner method was correct, the naive analysis failed to account for correlated input variables, such as the number of services previously taken by a customer.
In most cases, identifying correlated variables from a large set to determine the causal impact of a specific variable is time-consuming, iterative, and requires significant domain knowledge. Causal ML provides an automated framework that quickly assesses the causal impact of a variable by isolating the influence of treatment variables from other factors affecting the outcome.
To validate the results of Causal ML, a cohort analysis by bucketing the users with similar past 6 month revenue in the past together was performed (since this variable is highly correlated to both future LTV and NR)
Cohort Analysis
We now observe that the impact of 0.55 is much closer to the CATE impact of 0.7.
The further gap is explained by deeper cohort analysis, which shows that certain cohorts of customers are more sensitive to no response than others. This insight was automatically determined by the Causal ML model, opening up the possibility of personalized prioritization of these cohorts to prevent NR.
Case Study 2: Impact of Request Loss
To address the issue of No Response, the marketplace team blocks time slots for which no available professionals are expected to accept the request. It’s crucial for Urban Company to estimate the impact of these slot blocks on future revenue, not only due to the immediate loss of revenue from unaccepted services but also the subsequent drop in customer Lifetime Value (LTV) caused by a poor experience.
We tackle this issue with a two-step approach:
1) Converting the slot block into Predicted Request Loss
2) Converting lost requests to lower customer LTV.
For the purpose of this section, we utilize the output of the request loss model to build a causal model. The initial challenge with the RL result is that it provides a continuous score between 0 and 1, analogous to the drop in customer conversion probability.
To estimate the impact, business managers previously grouped the RL scores and examined the LTV drop in each group, then calculated a weighted sum. The final impact was estimated to be 0.5 units. Using an S learner method yielded an impact of 0.7. Similar to the previous example, the gap was explained by intelligently segmenting customers based on past delivery numbers and other factors. Consequently, the ratio of NR to RL emerges as 1 versus 1, contrary to the previous incorrect estimation.
Conclusion
In a data-rich landscape, discerning causation amid correlations is paramount. Causal inference, within data science, empowers decision-making beyond mere associations. Through methodologies like Causal ML, Urban Company achieves nuanced, accurate estimates of various events/actions/products. Causal ML provides a simple out-of-the-box framework to automatically address these analyses and generate a more accurate estimate.
Appendix
Methods
This article won’t delve into the details of performing Causal ML, as there are numerous available resources and Python packages for that, which are listed in the references at the end.
A brief description of meta-learner based methods is given below since the same were used in actual analysis at UC for the given case study
The core problem being addressed in causal ML can be expressed as
Essentially train an ML model with samples with and with-out treatment, then for every sample take the difference in prediction setting the Treatment variable T as 0 and 1 to estimate τ(x) and take an average to get CATE.
A note of caution :- this method is highly sensitive to model regularization methods adopted in model training and frequently underestimates the treatment effect.
T learner
The T-learner tries to solve the problem of discarding the treatment entirely by forcing the learner to first split on it. Instead of using a single model, we will use one model per treatment variable.
Step 1: Train 2 models one with treatment effect samples M₁, the other model without treatment effect M₀
Step 2: Do a prediction with both models for every sample X
Step 3: Take the difference to estimate τ(x) and take an average to get CATE
A note of caution:- Typically for problems where the number of samples with treatment is low compared to not treated samples and the relationship between input variables X and output is non-linear, the effects are estimated incorrectly, especially across different cohorts.
X learner
To account for the deficiencies of the previous 2 models we have X learners which involves a 3 step process to estimate τ(x)
Step 1: This is exactly same as T learners where 2 models are created, after training on different datasets of treatment and no treatment M₁ and M₀
Step 2: We now estimate the treatment effect for the 2 different datasets using models trained on the other dataset ie.
Step 3: We now train 2 models to predicate these treatment effects
Step 4: The final treatment effect being
References
- Facure, M. (n.d.). Python Causality Handbook. Retrieved from link
- Uber Technologies, Inc. (n.d.). CausalML. Retrieved from link
- Chernozhukov, V., Hansen, C., Kallus, N., Spindler, M., & Syrgkanis, V. (Year). Applied Causal Inference Powered by ML and AI.. Retrieved from link
- Shevchenko, M., & Elisova, I. (2020). User Guide for Uplift Modeling and Causal Inference. GitHub repository. Retrieved from link
- Hernán, M. A., & Robins, J. M. (2024). What If: Causal Inference without Counterfactuals. Retrieved from Harvard T.H. Chan School of Public Health website: link
About the author
Kshitiz Swarop is a Data Scientist at Urban Company with a passion for solving scalable problems. In his spare time, he enjoys working on hobby projects with a Raspberry Pi or reading.
Sounds like fun?
If you enjoyed this blog post, please clap 👏(as many times as you like) and follow us (@UC Blogger). Help us build a community by sharing on your favourite social networks (Twitter, LinkedIn, Facebook, etc).
If you are interested in finding out about opportunities, visit us at http://careers.urbancompany.com
