For a business, progress is measured on the basis of a variety of metrics including top line growth, active users, unit economics etc. Theoretically, the principal objective of a business is to maximise profit. Now the business could be pursuing any of the above metrics depending on the stage of the business, risk appetite of the entrepreneurs as well as the ongoing sentiment within the investor community of that sector.
No matter what the firm chases, a bunch of initiatives will be carried out within the organisation to achieve the targets set out for the current quarter or year. This will usually be broken down to specific targets that each department will be chasing. For example the sales team will be chasing quarterly revenue targets and the marketing team will be chasing qualified leads.
Similarly for product teams there are many relevant product metrics that are usually measured. For an engineering team, product development velocity is also tracked to measure efficiency. We usually glorify ‘bias for action’ for managers and this translates to rolling out more features and getting more projects done.
One side of the argument is that action is better than inaction & we should avoid paralysis by analysis at any cost. This is a very sane, practical advice especially in business settings where there is a lot of uncertainty and you need to err on the side of momentum and optimism.
However, we still need to put in processes to define and track success metrics and loop back the learnings for continuous improvement. Going forward, this will help in identifying projects or activities that should be scaled up and investments that should be cut down.
It is also equally important to identify causal relationships between improvement in success metrics and investment in corresponding projects. This can get tricky unless we are really careful about not inferring false relationships.
Imagine that in a study you found a high correlation between university education and lifetime earnings. We should resist the temptation to infer that university education leads to higher income. Instead we can ask a counter question — perhaps students from higher income families are more likely to pursue university education, and further students from higher income families have better connections and more accessibility for job opportunities. ‘Being from a higher income family’ is a confounding variable that explains the correlation between education and lifetime earnings.
Points to keep in mind:
- Avoid making the mistake of inferring causation from correlation. However it can be a starting point to generate hypotheses.
- Controlled(and randomised) experiments are the gold standard for causal inference. A/B testing is a popular example
Now, imagine trying to do a controlled random experiment for identifying the relationship between university education and lifetime earnings. It is unethical to randomly send few participants to university and deny education for the other group. In a business setting, an example of the above would be inferring the relationship between lift in revenue and outbound sales effort. Management might not find it practical to have a control region devoid of sales personnel, just to do an experiment. There are other methods for causal inference.
- Propensity score matching can be used in situations where controlled(randomised) experiments are not practical or ethical to do.
I’d recommend reading this detailed post on causal inference by Edwin Chen for more interesting insights and examples of real life experimental set ups.
Originally published at IndiPM.
