Merging Intuition with Analytics

In the past few years, there has been a tremendous implosion of data into our lives. Everyone is expected to take sound decisions on the basis of data rather than just on what feels is right on basis of experience or hearsay from experienced people. Objectively speaking this is much better than dealing with decisions that people on what appears right without any logical backing, resulting in numerous bias creeping into the final decision.

During the course of my MBA, I laid my hands upon the book Predictably Irrational by Dan Ariely. Reading this book I slowly began to realise the various traps that everyone falls into due to unspoken bias within us (Luckily, reading about them helps you become more aware of them when dealing with similar situations). Slowly but steadily, I began to consume more books in this area. As I began to devour more material on these topics, I began to notice general business intuition that one develops due to his interactions is given lower priority than what insights data might throw at you. In stats, its often stated that “ Correlation does not mean causation”, which in simple terms indicates that every data trend or insight you identify needs a good theory to explain why. After all in the end data is just a representation of what has worked in the past, pointing you towards where one might have a higher probability of success (Trump did end up becoming the US president despite overwhelming odds against him early on his campaign). As I contemplated further on this topic, I ended up recalling a lecture during my MBA, wherein we were introduced to a concept called Decision Calculus. Professor hoped that this would help us in arriving at a optimum solution to a problem despite having limited data or just on basis of intuition or gut instinct.

Decision calculus was a concept introduced by John D.C. Little, a professor in Sloan School at MIT in 1969, with hopes to allow managers make more data driven decisions rather than norm those days. As I read through the paper published in 1969, it got me thinking that this has potential for applications even in today. Most of us when faced with a uncertain task or lack clear data revert to our instincts on basis of our experiences at ground level to arrive at a decision for way forward. Now, instead of letting go of the idea, you can actually construct it into a more analytical model and make a prediction. To explain this idea, I had recently written a learning note for my team with an example of a simple real life business problem. The model that I described in the note was developed by me drawing heavy inspiration from the example explained by John Little in his note and merging it with a typical situation that I face on a regular basis in my line of work.

One impact that this line of thinking will have is that it will force one to explicitly articulate your assumptions. As you also work on developing a mathematical model of your ideas, you focus on what kind of data will be needed, helping you make better predictions. Secondly, as you start utilising the model, you need to add/subtract input parameters from the model, developing a unique model which meets your needs rather than trying to adopt any standard model. As a concept, this is still a nascent idea that I hope to improve with passage of time, helping me draw better ideas or insights.

Those keen to learn more on improving their knowledge on Forecasting, Decision Making and Data analytics can certainly try their hand at these below books, which I read over the past 2 years and found them pretty valuable

Superforecasting: The Art and Science of Prediction

How Not to be Wrong: The Hidden Maths of Everyday Life

The Signal and the Noise: The Art and Science of Prediction

How to Lie with Statistics

A Field guide to Lies and Statistics

Thinking fast and Slow

I keep exploring newer books to read in this areas, hoping to update this list with more interesting reads with passage of time