Opinion
Data-Driven Decision Making is a Chimera
Data does not necessarily lead to better decisions
What is Data-Driven Decision Making?
Data-driven Decision Making (DDDM) is the process of making decisions based on actual data rather than intuition or anecdotal evidence. Data-driven decision making seems like an obvious choice. Why would you rely on gut feelings? Whether they are strategic or tactical, data helps you identify patterns, anticipate trends and make more confident decisions [source]
What are the risks in Data-Driven Decision Making?
Are there any downsides to DDDM? A strict adherence to DDDM might bias the decision-making process.
First of all, decision-makers might be tempted to narrow down the focus only on factors for which there is data.
Secondly, it may be tempting to further reduce the scope to deciding factors for which not only there is data but for which it has been possible to develop a prediction model that it is believed to be accurate enough for the use-case at hand. The ability to produce a forecast might lead to over-confidence and to underestimate the impact of rare-events. On top of that, the ability to produce forecast might bias the decision-making process in an additional, perhaps subtle way: it is often the case that the plans that are based on forecasts receive a different scrutiny with respect to plans that are not based on forecasts: opponents of a plan can cast doubt on forecasts.
A third possible risk to the adoption of DDDM is that decision makers need evidence in order to justify a plan. If information is not readily available, the decision making process might be stalled until such information is gathered.
To teach how to live without certainty, and yet without being paralyzed by hesitation, is perhaps the chief thing that philosophy, in our age, can still do for those who study it — Bertrand Russell, A history of western philosophy
Gathering data might turn into a menial exercise done for the sake of making progress and move forward with a plan. The danger is confirmation bias. It is therefore necessary to design decision making processes that norm how decision makers access data and generate evidence. It is fundamental to frame the decision-making in a way that prevents you from moving the goalpost after you’ve seen where the ball landed [source]. In other words, the decision criteria should be set in advance, before having access to data.
Least but not last, DDDM relies on data but data quality might be poor. According to HBR, only 3% of companies’ data meets basic quality standard [source]. Data can be inaccurate and biased. Is bad data better than no data at all? Not necessarily.
It is also important to note that a decision might be ultimately based on a forecast. And forecasts might be wrong even if based on perfectly accurate data. Bertrand Russell (1872–1970) warned us about the dangers of inductivism. In his parable, a turkey inferred by induction, collecting several days of observations, that it would be fed every day in the morning. The turkey grew strong confidence in this assumption as more data was accumulated day after day. The animal expected to be fed, like any other day, but on Christmas Eve, the turkey had his throat cut instead. As in Bertrand Russell’s parable, forecasts based on historical data can be accurate only as long as we can expect the economic and competitive environment to remain in line with its representation in the data. This is a very strong assumption that very often is proved wrong [source].
Is Data-Driven Decision Making truly possible?
No. There is no such thing as Data-Driven Decision Making. There is always a human in the loop, whether the human is the ultimate decision maker or he is the designer of the system that will automatically take decisions. And even if the human in the loop were a perfectly informed and rational individual, she/he would still have to deal with a number of stakeholders. The human-in-the-loop would have to gain the trust and the endorsement of the internal and external stakeholders. In other words, it is often impossible to exclude the involvement of a boundedly rational agent in the decision making process or its design.
Data-Informed Decision Making (DIDM) has more and more frequently replaced DDDM in business literature. By replacing DDDM with DIDM, decision scientists aim at acknowledging that lack of data and data quality issues are the rule rather than the exception; that decision makers need to constantly make decisions under some degree of uncertainty and cannot assume that the future is predictable from historical data. DIDM also accounts for the fact that it is not just about data: decision makers need to actively cope with the risk of biases in the data as well as in the decision-making processes.
There are numbers near our decision somewhere, but those numbers don’t drive it [source].
