Hard-Boiled Wonderland and the Accidental Infomaniacs

Making decisions in an uncertain age of information deficit and overload.

Article by Dina Zyadeh, Associate Innovation Officer — Operations.

Illustrations by Hans Park, Strategic Design and Research Manager.

Among all of the quirks humanitarians have, our relentless and compulsive need for more information is probably the most excusable. Information, after all, is our ultimate currency — the preferred medium of exchange for the decisions we must make amidst crises. And there’s good reason to obsess over how many relief items to stockpile in case of emergency or which shelters to urgently retrofit in order to ease overcrowding and prevent people from falling sick with Covid-19. It’s decisions like these that present profound implications for crisis-affected people for generations to come.

It’s no surprise then that of all the limitations that plague our decision making perhaps the most agonized about is information deficit. Making the ‘right decision’, we’re told, requires that you have complete information about all of the options open to choice, in addition to the foresight to compute the consequences of every single one of those options.

Yet it’s exactly this idealised approach to decision making that Herbert Simon argued isn’t realistic in practice and doesn’t often result in satisfactory decisions. In his 1978 Nobel Prize speech, he advocated that this approach be replaced by a model of “bounded rationality” because “rationality is bounded when it falls short of omniscience; and the failures of omniscience are largely failures of knowing all alternatives, uncertainty about relevant exogenous events, and inability to calculate consequences”. In other words, while we might idealise a linear and rational decision making process, the reality is that we’re always constrained by biases, limited resources like time and money, and the availability of information when making decisions.

The observations of Simon and many others since haven’t stopped our transformation into accidental infomaniacs. In an effort to bridge the chasm between our ideal decision-making selves and our mortal decision-making selves, we’ve concocted the decision-making equivalent of “hair of the dog”: trying to eliminate the uncertainty and information constraints we face when making decisions through more information seeking and processing.

Decision making in the age of uncertainty

Both Covid-19 and the climate crisis have tragically brought the complexity of making decisions under uncertainty into sharp focus. Uncontested facts remain largely elusive, yet critical decisions about which families to prioritize for emergency cash assistance, for example, must be made with sparse and flawed information. When confronted with deeper levels of missing knowledge, uncertainty cannot be reduced by simply gathering more information. This is primarily because the uncertainties are “unknowable at the present time, but will be reduced over time”.

There are two important factors that compound the difficulty of making decisions in uncertain times. First, the interactions within and between the complex systems we live in produce important outcomes that do not mirror the past. Therefore, information about the past is not always a useful predictor of the future. Second, the decision to gather more information in order to manage uncertainty depends on the level of the decision makers’ satisfaction with existing knowledge, as well as their level of risk aversion.

When faced with uncertainty, decision makers can either: a) make a decision with currently available, imperfect information; b) defer the decision and gather more information, sometimes at the cost of delaying action; or c) entirely ignore the uncertainty (not recommended).

Irrespective of which option is chosen, it’s clear that decision makers’ satisfaction with existing knowledge is coloured by their underlying values, goals and perspectives. Our beliefs intricately shape every decision we make by informing the options we think are available to choose from and by influencing our predictions of how our decisions will turn out. Even in best case scenarios where an evidence base exists for specific interventions, different people may reach different conclusions based on the same evidence.

As we wade through the murky reality of making difficult choices, our ability to successfully navigate uncertainty depends on our willingness to use imperfect information, despite its limitations. Even more importantly, we must be “encouraged to admit ignorance, explore paradoxes and reflect collectively”.

This begins with a willingness to identify the decision making problems we face and how they affect our actions. Many of these still need to be explored sufficiently in our day-to-day work and outside of decision theory textbooks. Some of the most common ones we face are loosely categorized below.

Every organization is a decision factory

Given the weight of a decision’s outcomes on people’s lives, the value of the decision making process is often eclipsed by our fixation on the outcome (i.e. the result of the decision). Looking at whether the result was good or bad to determine whether the decision was good or bad is ineffective because decision quality and outcome quality are not aligned perfectly. This is especially the case when the relationship between the two takes a long time to materialize.

Annie Duke describes ‘resulting’ (also known as outcome bias) like this: “You can run a red light and get through the intersection unscathed. You can go through a green light and get in an accident. This means that working backward from the quality of a single outcome to figure out whether a decision was good or bad is going to lead to some poor conclusions”.

She goes on to say that “there are two ways uncertainty intervenes in the decision process…imperfect information intervenes before the decision. Luck intervenes after the decision but before the outcome. Luck, by definition, is something you can’t do anything about”.

Even if a decision process doesn’t alone determine the set of outcomes that will actually happen, a sound process does determine which set of outcomes are possible and which set of outcomes are impossible. And under uncertainty, narrowing the range of possible outcomes is in itself, a monumental outcome.

Speaking to the importance of the decision making process, the psychologist Daniel Kahneman, wrote that “whatever else it produces, an organization is a factory that manufactures judgments and decisions. Every factory must have ways to ensure the quality of its products in the initial design, in fabrication, and in final inspections”. Yet in some organizations, fixating on outcomes is often easier than understanding, challenging and changing the process that led to them.

You can’t panic-buy crystal balls

What if our ability to navigate uncertainty relied less on our ability to predict the future, and more on our ability to take short-term, adaptive actions to achieve our long-term goals, irrespective of how the future unfolds? And what approaches do we have at our disposal to implement improved decision making rituals?

These are some of the questions we started exploring in 2018 when we launched an Innovation Fund at the UN Refugee Agency to encourage innovations in areas that can lead to improved decision making. These areas of work are by no means exhaustive, but they were the areas we felt were underexplored and underutilized in our day-to-day work.

Modelling and Simulation

Models and simulations are approximate representations of our world. Some decision makers confuse them for ‘crystal balls’, but their strength lies in their ability to present a range of predictions rather than a singular truth. The utility of models is less about predicting the future or optimising toward a specific outcome, and more about minimising regret and risk by identifying a handful of actions that perform reasonably well across large ranges of uncertain futures.

From low-fidelity options like tabletop scenario games to more complex social and behavioural modelling methods, they can be useful tools to help us move from “predict-then-act” approaches to ones that enable us to assess the risk of different interventions before deciding which of them to pursue. This makes them especially suited to address uncertainty, including the black elephants in the room like climate change and emerging infectious diseases.

You’ve probably heard the aphorism that “all models are wrong, but some are useful”. This is true, given that a model is a representation of a system, built on assumptions we have about that system and its interrelationships with other systems. With climate change-induced displacement, for example, there is a wide range of plausible scenarios about the frequency, magnitude and other characteristics of displacement that will continue to unfold.

The climate is a complex system being pushed past familiar bounds, and with many intricately interlaced dependencies that produce important, and sometimes unprecedented, weather outcomes. These major weather events can trigger other unique events, like new waves of displacement and are particularly difficult to represent accurately in a model. It’s this simplification of reality, in addition to the large data requirements and resource constraints that have, until today, limited the applicability of modelling and simulation in certain contexts.

Despite some of these shortcomings, the fundamental strength of modelling is the ability to bring a plurality of possibilities to the table, some of which were never in view. They can enable us to facilitate critical discussions, particularly at local levels, about risk tolerance toward certain interventions. It’s these relationships and discussions that models foster that make them potentially useful decision making tools when it comes to designing for resilience, before the next emergency begins.

Inclusive intelligence

We define this as novel ways of engaging people in decision-making, particularly in framing the problems that affect them, defining priorities, and informing and overseeing our interventions, including the services we offer.

Involving refugees in decision-making is not a novel suggestion. It’s a foundational practice of ours to understand the needs and preferences of the communities we serve. When dealing with high uncertainty, however, an over-reliance on focus groups and surveys as the primary means to identify risks and build trust around proposed interventions is not always sufficient. It’s worth testing emerging methods to help us formulate actions with communities in the present to adequately deal with future uncertainties.

Participatory risk assessments and collaborative modelling are two examples of promising methods that incorporate diverse wisdom about key sources of risk and options to mitigate them. Similar to modelling and simulation interventions, these approaches are not an easy fix to managing uncertainty and require a certain level of trust, in addition to expert facilitation that can harness the diversity of opinions and maintain interest throughout the process. But they are worth exploring since they engage communities in a “purposeful learning process that elicits and formalizes the implicit and explicit knowledge of participants to support decision-making and action”.

Storytelling

Thanks to the collective efforts of many people in recent years, the primacy of using data as opposed to anecdotes or intuition in decision making is accepted as a norm. While there is more work to be done in this area, the application of computer science techniques to help us transform large volumes of data into information, and to turn this information into action is proving critical to our ability to manage uncertainty.

To complement this, there are also many ongoing efforts to understand the effects of emotion on decision-making, since it’s well-recognised that “emotions constitute potent, pervasive, predictable, sometimes harmful and sometimes beneficial drivers of decision making.” Given this, it’s unrealistic, and potentially undesirable, to construct a hard-boiled wonderland where emotions play a limited role in decision-making. It also begs the question: in what ways should emotions be harnessed to support decision makers in situations of crisis and uncertainty?

Storytelling is one area we’ve been exploring that has the potential to harness emotions to improve engagement around complex issues. We know that stories transmit culture and can transform organizational norms around decision making. Values around inclusive decision making and good decision practices (e.g. ‘two-way door’ decisions) are already embedded in the aphorisms and metaphors we use daily.

Storytelling has a negative reputation among some people who view it as an exercise unmoored from reality. While stories shouldn’t be substituted for evidence, the use of stories and aphorisms is promising to frame complexity and to build better decision environments where people feel empowered to make decisions.

“We knew this was coming all along — we predicted it”

Now more than ever, the humanitarian community is under tremendous pressure to anticipate the course of multiple crises, allocate scarce resources, and prioritise and implement a host of interventions with imperfect information. It’s easy to feel a loss of agency when faced with such a daunting reality.

While we’d like every decision we make to turn out well, that’s unlikely to happen since we’re working with imperfect information and luck. What we can strive towards, however, is assembling a successful ‘decision portfolio’ that advances us toward our goals and minimises regret and risk. With such a portfolio, any individual decision may turn out well or not, but the overall portfolio should contain enough good decisions that ensure we continue to save lives, protect rights and build a better future for those we serve.

Having invested in areas of work that encourage better use of information through the Innovation Fund, we learned a great deal but nothing more fundamental than this: improvements in information gathering and processing alone don’t guarantee good decision processes, nor can they alleviate deep uncertainty. An independent evaluation of our work highlighted the remarkable achievements of projects, but found that project teams didn’t fully understand how their projects contributed to and improved decision making in the organization.

There are some that think a good decision process is more trouble than it’s worth in an age of deep uncertainty. They’d prefer to rely on intuition instead, but “you can’t go back and examine with any fidelity how your gut arrived at a decision. You can’t peer into your gut to know how it’s operating…You can’t ‘teach’ your gut to somebody else, such that they could use your gut to make decisions. You can’t be sure that you are using your gut the same way each time”.

While it’s not possible to rely on intuition or control the outcomes of our decisions, it’s imperative that we first invest in understanding the decision making problems that plague our work. And second, we owe it to ourselves to develop better decision-making rituals and nurture positive decision environments for the uncertain times of today and tomorrow.