Core Concepts #5: Uncertainty

The “when and where” (among other things) of tradespace exploration

This is part five of a seven-part series of posts on the core concepts of tradespace exploration, designed to help beginners become familiar with terminology and the general structure of tradespace data. Click here to find the other posts in this series, as well as other Tradespace 101 posts.

We’ve previously described the basic skeleton of a decision: stakeholders spend resources to acquire alternatives that generate benefits. Simple, right? All we need to do is find the alternative with the best combination of low resource requirements and high benefits to maximize our value. Sometimes it may be that easy, but here’s some bad news: for most problems uncertainty makes it unlikely or even impossible for us to know for sure which alternative that is.

Uncertainty is sometimes described as the “when and where” of tradespace exploration because the environment in which the alternative will be used (now or in the future) is frequently unknown. Consider reserving a rental vehicle in advance of a day-trip vacation: the weather on that day will impact which vehicle is the best. If it is sunny, I would want to have a motorcycle to enjoy the rays, but I would prefer a car if it rains. If it snows, I might need an SUV! Since the weather during my vacation is out of my control, it is impossible to guarantee that the vehicle I reserve will be the best choice when I pick it up.

That said, the “when and where” description actually undersells the breadth of potential uncertainty. Uncertainties come in many forms — ranging from inexact measurements, to large-scale forces like supply and demand, to the deliberate but unknown decisions of another friendly or hostile person. Any variable that can’t be controlled by the stakeholders or is unknown at the time of the decision is a potential source of uncertainty. Uncertainty plays a part when analyzing nearly all decisions, by affecting either the stakeholder’s needs or the resources required / benefits provided by the alternatives. Actively considering potential sources of uncertainty (and their effects) is a necessity when tackling a new problem.

It is also important to keep in mind that uncertainty can have both negative (risk) and positive (opportunity) outcomes. Many stakeholders choose to balance expected and worst-case performance, by considering both normal and negative outcomes of uncertainty. This is a safe, risk-averse decision strategy, but notably ignores the potential strength of an alternative that is capable of seizing opportunity. “Future proofing” — spending more money up front for the sole benefit of taking advantage of yet-unreleased technology — is one common example of an opportunity-focused choice: remember when you could buy a 5G-capable phone before any networks actually offered 5G service?

Future proofing may cost more with no benefit now, but it will pay off in the long run — if tech and infrastructure (like cell towers) evolve the way we expect

So how can tradespace exploration help people make better decisions while contending with uncertainty? There are many different techniques with their own pros and cons, but we recommend augmenting a MATE tradespace with Epoch-Era Analysis (EEA). EEA is a framework for describing uncertainty in a way that promotes exploration — building up your intuition for how (and how much) the uncertainties impact the relative desirability of different alternatives.

An epoch can be thought of as one possible outcome of all the uncertainties: a snapshot in which all the variables outside of stakeholder control (epoch variables) are not uncertain, but rather known and constant. As you might expect then, we generate a list of epochs in much the same way that we generate alternatives, by determining the possible levels of those variables and then enumerating different combinations of those levels. Using the vacation rental example again, I could use a “weather” epoch variable with levels of sun, rain, and snow. Each epoch effectively says “assume my vacation is sunny/rainy/snowy”: we take the uncertainty and convert it into however many not-uncertain epochs are necessary to capture the different outcomes.

Considering multiple epochs while analyzing a decision is often described as a form of “what if” analysis: “What if this happens? What if that happens instead?” The shape of the tradespace scatterplot can vary significantly between epochs — the alternatives may rearrange because the resources required, benefits produced, or stakeholder needs are impacted by the epoch variables. Ideally, we can find an alternative that is excellent in every epoch: we would call this alternative robust to the uncertainty or value robust. The more robust an alternative is, the harder it is for uncertainty to make the decision backfire.

The tradespace changes shape across the different epochs due to the impact of uncertainty — different vehicles are most desirable in each epoch. In addition to benefits, resources can also change: the motorcycle is so dangerous in the snow that the possibility of injury is “priced in” to the cost. Here, the regular car appears to be the most robust due to its consistently low resource cost.

EEA is primarily a possibilistic form of uncertainty analysis, in contrast to probabilistic methods. Probabilistic analysis quantifies uncertainty statistically with probability distributions and usually results in various figures and tables with distributions, error bars, or clouds. However, these probabilities are unknown for many complex, “big picture” uncertainties! Returning again to the vehicle rental example, I might have a lot of difficulty estimating the relative likelihood of sun, rain, and snow — especially if I am reserving my rental months in advance. I could guess, but that would make the statistical insights unreliable.

On the other hand, possibilistic analysis does not require probabilities or likelihoods, limiting the subsequent mathematical analysis techniques to those that do not take average or expected values but rather those that consider only the breadth of possible outcomes. The goal is not to forecast what will happen (or even what is likely to happen), but rather to understand what could happen: focusing our attention on how uncertainty can change the shape of the tradespace and our search for robust alternatives. This makes possibilistic analysis like EEA particularly well-suited to complex uncertainties (e.g. technology development) that are difficult to project.

That said, if you do have access to reliable probability distributions for your uncertainties, EEA is also flexible enough to incorporate them! Typically, the distributions will be defined for each epoch variable, and then the different epochs can have their own relative likelihood calculated from there and the alternatives can be assessed for distributions of value. This also gives EEA the unique ability to improve over time: you can start with possibilistic analysis when you have less data at the beginning of the process and switch to probabilistic analysis if more data about the uncertainty becomes available later.

Stakeholders are not able to choose their preferred outcome for uncertainty, in contrast to how they choose an alternative. However, in some cases it is possible for stakeholders to influence uncertainty to resolve in a beneficial way. For example, a car manufacturer notices that government policy concerning emissions regulations is a significant uncertainty for their profit: epochs with high regulation result in lower profits and force them to change their production (the alternative) to more electric vehicles. They may choose to spend lobbying money and political clout to make low-regulation epochs a more likely outcome and avoid the worse scenario.

If you are looking for help brainstorming uncertainties / epoch variables, it may help to use the following prompts:

• Context and Preference variables. Context and preference variables are subtypes of epoch variables that indicate what is affected by the uncertainty: the resources/benefits associated with the alternatives or the stakeholders’ needs, respectively. Make sure to consider ways that both aspects of the decision might vary. It is possible for an epoch variable to be both a context variable and a preference variable if it affects both parts.
• Endogenous vs. Exogenous. Uncertainty can occur both inside (endogenous) and outside (exogenous) of the alternative. Most of our example uncertainties in this post have been exogenous, because stakeholders are more likely to lack control of things that happen outside of the alternative. However, it is also useful to consider if there are any sources of endogenous uncertainty in your problem. Using the vehicle rental example, perhaps you know that the rental company sometimes rents out vehicles without cleaning the seats thoroughly. Receiving a dirty vehicle will impact your enjoyment of your vacation, with “dirty” epochs featuring worse benefit regardless of the weather. However, the motorcycle may be less affected by this uncertainty because the amount of effort you need to clean it yourself is less — just brushing off the seat, no vacuum required.
• Correlated variables. It can be easy to assume that all epoch variables are independent, but sometimes uncertainties are correlated. Note that this applies even to possibilistic uncertainties: we may not know the likelihood of a government shutdown or a single-party supermajority in Congress 20 years from now, but undoubtedly those two things won’t occur together. Consider if any of your epoch variables have other, related uncertainties that should be captured — and then use this knowledge when performing your “what if” analysis to construct believable epochs.