How to spot bad forecasts using these 6 rules of effective forecasting
Recently, I read an interesting paper by one of the big futurologists of our time, Paul Saffo. If you want to know more about this topic, read some of his work! What I found interesting about his work, is that it gives an honest assessment of what can be predicted and what can not. In a world full of fake-experts and people that believe they know what the future brings, I feel we need to take a step back and look at what effective forecasting really is. Let’s dive into it!
The goal of forecasting is not to predict the future but to tell you what you need to know to take meaningful action in the present. — Paul Saffo
What is forecasting?
The primary objective of forecasting is to identify the full range of possibilities, not a limited set of illusory certainties. Unlike a prediction, a forecast must have a logic to it. That’s what lifts forecasting out of the dark realm of superstition. In order to give you a complete picture, here is a widely used definition for forecasting:
Forecasting is a technique that uses historical data as inputs to make informed estimates that are predictive in determining the direction of future trends.
The Dunning-Kruger effect
First, let’s quickly dive into the Dunning-Kruger effect. As the graph displays, people with little knowledge are, most of the time, the most confident in their forecasts. However, as people learn more about a field or subject, they tend to grasp the difficulty of a subject and notice that what they believed to be simple is not that simple at all. This is also true in the world of forecasts, making it even more important to know if and how forecasts can be made, to prevent taking on the word of fake-experts. The forecasting rules in this article will help you to evaluate forecasts yourself, to make sure you are not listening to wrong forecasts from people on the peak of “Mt. Stupid”.
I hope the following 6 rules can give you the tools to evaluate forecasts yourself!
Rule 1: Define a Cone of Uncertainty
A cone of uncertainty defines and maps the borders of possibilities that extend out from a particular moment or event. The cone broadens your understanding by revealing overlooked possibilities and exposing unexamined assumptions regarding hoped-for outcomes. The most important factor in mapping a cone is defining its breadth, which is a measure of overall uncertainty. In other words, the forecaster determines what range of events or products the cone should encompass. The cone of uncertainty is a model that is also used a lot in other fields, like to describe software projects.
As you can see in the figure above, there are 4 types of futures included in the cone of uncertainty. Making a cone of uncertainty helps you realise that there are these types of forecasts. The best thing we can do in forecasting is to map every type of future, maybe even multiple types of scenario per type of future, and to keep in mind that each future scenario has a different likelihood of happening. The 4 types of future are:
- Possible futures: includes all the kinds of futures we can possibly imagine — those which “might happen” — no matter how far-fetched, unlikely or “way out”.
- Plausible futures: a smaller subset encompassing futures which “could happen” according to our current knowledge (as opposed to future knowledge) of how things work.
- Probable futures: contains futures which are considered “likely to happen”, and stem in part from the continuance of current trends. Some probably futures are considered more likely than others; the one considered most likely is often called “business-as-usual”. This future is a simple linear extension of the present.
- Preferable futures: Contains a future scenario that, by contrast, is concerned with what we “want to” happen; in other words, these futures are largely emotional rather than cognitive. This does not mean that this future is less important, as we all know setting a goal to strive for is important to give direction.
Rule 2: Look for the S Curve
Change rarely unfolds in a straight line. Evidence of this can be found in models all around you, for example, the Gartner Hype Cycle or Moore’s law. In futurology, the S-curve is very important, as most important developments typically follow the S-curve shape of a power law.
Very large, broadly defined curves are composed of small, precisely defined and linked S curves. This can also be seen in the picture above. For a forecaster, the discovery of an emergent S curve should lead you to suspect a larger, more important curve lurking in the background.
The four phases of such an S curve (Initiation/Birth, Acceleration/Growth, Deceleration/Maturing, Saturation) can be seen in the logistic growth curve. GP (growth point), IP (inflection point) and SP (saturation point) are points on the curve that indicate a change in the curve. The art of forecasting is to identify an S curve pattern as it begins to emerge, well ahead of the inflection point. The inflection point here is the point that development or technology starts to take off and start to impact the economy of society. Prediction of the inflection point is hard. However, wise forecasters are able to predict it based on the identification of early indicators of the inflection point.
Rule 2 encompasses the search for the trends within the defined cone of uncertainty. Looking for an upcoming inflection point is hard, but important in forecasting. In order to find the indicators of inflection points, let us look at rule 3.
Rule 3: Embrace the Things That Don’t Fit
The entire length of the S curve to the left of the inflection point is full with indicators — subtle hints that when aggregated become powerful indicators of the upcoming future. The trick is to spot these indicators, which can only be done by noticing and understanding things that don’t fit, things people can not classify or that they will even reject.
Because of our dislike of uncertainty and our preoccupation with the present, we tend to ignore indicators that don’t fit into familiar boxes. But by definition, anything that is truly new won’t fit into a category that already exists.
— Paul Saffo
The key is that the indicators of the inflection point are not standardized or the same for different trends. For each different forecast, other indicators can be spotted. The key is spotting them by embracing the things that don’t fit.
Rule 4: Hold Strong Opinions Weakly
One of the biggest mistakes a forecaster can make is to over-rely on one piece of apparent strong information because it happens to reinforce the conclusion he or she has already reached. This will result in a close mind, which is very dangerous for a forecaster.
Be open to evidence that might suggest something else than your current beliefs. If you must forecast, then forecast often — and be the first one to prove yourself wrong.
Rule 5: Look Back Twice as Far as You Look Forward
As earlier definition already explained, forecasting is a technique that uses historical data as inputs to make informed estimates. History can be important to search for trends and extrapolate them into the future for good estimates. However, you should watch out with looking back, as the recent past is rarely a reliable indicator of the future. So when you look back for parallels, always look back at least twice as far as you are looking forward. Search for similar patterns, keeping in mind that history — especially recent history — rarely repeats itself directly. But be warned, the hardest part of looking back like this is to know when history doesn’t fit.
Rule 6: Know When Not to Make a Forecast
As you may have noticed by now, it is very hard to make a forecast. it is important to note that there are moments when forecasting is comparatively easy — and other moments when it is impossible. Be skeptical about changes, and avoid making an immediate forecast — or at least don’t take any one forecast too seriously. There are some other factors that might also impact your decision for making a forecast:
1. What is the forecasting time horizon?
2. How frequently are forecasts required?
3. What data is already available?
4. How volatile is the future?
One thing is clear. When it is better not to make a forecast yet, the incoming future will bring plenty more indicators that will help with forecasting, sooner than you think… Stay alert!
Rounding up
Let’s end with a quote from the man that helped us with these rules for forecasting. To keep in mind when hearing your next so-called expert predict the future.
At the end of the day, forecasting is nothing more (nor less) than the systematic and disciplined application of common sense. It is the exercise of your own common sense that will allow you to assess the quality of the forecasts given to you — and to properly identify the opportunities and risks they present. But don’t stop there. The best way to make sense of what lies ahead is to forecast for yourself.
— Paul Saffo
Thank you for your time reading!
