The Flaws of Best & Worst Case Analyses
A standard technique to account for uncertainty with project planning, revenue forecasts, and other kinds of estimates, is to use best case & worst case scenarios to understand a range of possible outcomes.
A model is created in some program, copied twice, then filled in with all of the expected best case and worst case inputs. The idea is that this will provide the best case and worst case results. It won’t.
Best & Worst Cases are Poorly Defined
First off, the ‘best’ and ‘worst’ cases used in predictions are almost never the actual best and worst cases. Calling them such is one of those lies that’s only accepted because it’s said so often.
Most people understand that best and worst cases aren’t literally the best and worst cases. But that leaves the question of what they do mean, other than what the terminology would indicate.
Some would say that ‘worst case’ really means a reasonable worst case. But then absurdity is only rejected in favor of ambiguity.
What does reasonable mean? If you heard on the news that an analyst made an important projection of global warming with upper and lower bounds that seemed “reasonable to the analyst”, would you have any idea what that meant?
Solving the Ambiguity Problem
There is a mathematical way of having a very precise definition for a lower bound or upper bound. That is to choose a specific percentile to aim for.
If you think that a project could take at most twenty hours, perhaps you can assign a percentage to that. You could say, “I am 95% confident that this will take twenty hours or less.” There’s always something that could go wrong and break your most confident forecast, but you still may have a lot of predictive power.
In comparison, the reasonable worst case will mean very different levels of extremity to different people. The worst 5th percentile is concrete. It’s difficult enough discussing predictions directly; it’s much harder when there’s a second disagreement about what exactly is being discussed.
Best & Worst Case Analyses Aren’t Mathematically Accurate
Even if you used precise definitions of best and worst cases, the math would likely be incorrect if you were to apply them.
Imagine you are taking the sum of five different expenses, and you have a worst case expected value for each. Some people would attempt to add each expense’s worst case scenario to get the worst case total. This won’t work.
If you were precise, you might use the lowest 5th percentile definition of worst case. The problem is that the sum of the 5th percentile inputs is not equal to the 5th percentile of the output.
To show this, I’ve created a simple simulation. Five independent expenses are each expected to cost between $10 to $50 (the 5th and 95th percentiles.) I’ve summed them using Guesstimate, which uses Monte Carlo sampling to run thousands of simulations.
If one were to simply add together the worst case and best case forecasts, they may expect that the sum would range (5th to 95th percentiles) from $50 to $250. Instead, it’s $86 to $180.
From the simulation, the $50 point actually is less than the 0.0025th percentile, rather than the 5th percentile. The $250 point is at the 99.89th percentile, not the 95th.
While one originally meant for the result to represent 90% of possible outcomes, it instead represents 99.9% of possible outcomes. And that’s just with a sum of five numbers, imagine how much more extreme it gets for larger simulations.
Here, you may be thinking,
Does that difference matter?
Yes! For a few reasons. First, the incorrect range ($50 to 250) may be considered too wide to be useful, and mostly disregarded. There was a lot of information in the inputs, but a poor analysis would make the result far more uncertain than it actually was.
Second, the wide range of the output may cause the original modeler to change the input values to make the output seem more reasonable. They may adjust the worst case to be better, and the best case to be worse until the output looks more in line with their expectations.
“The worst case marketing projections were fiddled with until our worst case revenue numbers seemed good to us.” — Hypothetical honest disclaimer
This also would mean that their worst case numbers would have to change depending on the size of the model. It’s a mess.
Simulate Instead
In many cases a simulation is both more precise and easier to make than a best & worst case scenario model. The simulation described above took thirty seconds. You don’t need to replicate the model multiple times when you write in your uncertainties directly, you just enter your ranges for each parameter, using two numbers instead of one.
