TL;DR. Chapter Summaries
Short, condensed summaries of each chapter for those in a hurry.
Preface Summary
Forecasting combines experience, group consensus and data into information that informs a decision. Even a guess is data of sorts.
The current techniques for making decisions in product development are flawed for the following reasons –
1. Forecasts are overly simplified and don’t capture the assumptions that were made when the forecast was modeled. Often the forecasts assume complete information about the problem and stability of the delivery system which is almost unheard of.
2. Scant use of historical data to re-enforce and inform decisions. There is a preference to ignore everything in the past, and start estimating anew every time.
3. Preference of pre-defining a “planned due date” and then shoe-horning whatever into that time. And then being surprised when that doesn’t work.
4. Striving for perfect predictable outcomes when so much needs to be learnt as the product unfolds and more is understood about the actual problems being solved.
5. Belief that all metrics are evil and NO MEASURE is superior to even simple measures. Often belief is that metrics will be used to manage individuals disregarding that will occur even less informed without measures.
Chapter 1 — Forecasting
Forecasts are used to assist human intuition when making decisions.
Forecasts have –
1. A statement about some future outcome or event outcome yet unknown (an answer to a question about the future)
2. A statement about the level of uncertainty such that the forecast has a chance of coming true, but not guaranteed (an answer to an agreed level of uncertainty)
3. A way of eventually testing the actual outcome against the forecast (otherwise its conjecture, and my guess is as good as yours)
Forecasting is about answering the right question, to an agreed level of uncertainty, with as little effort as possible.
Good forecasts -
- Show multiple options, not just one. Leave the human to decide “best.”
- Use duration rather than date when start date (commitment) is unknown.
- Update the forecast with the latest information, rather than sticking to the first forecast value given.
Main barometer for whether a forecast is “good” is that it communicates information well. If a forecast helps “see” the options available, and judge the impact of one option over another that I wouldn’t have seen through gut-instinct alone, then it’s a good forecast!
A valuable forecast is one that changes behaviors in advance of an undesirable outcome. Forecasts should highlight results that are unexpected, not just confirm the expected.
Good forecasters -
- Answer the right question — triaging and splitting
- Balance general and contextual views — start general then adjust by context
- Incorporate New Information — But don’t over-react
- Use Feedback loops — Have a constant desire to understand why their forecasts are right and wrong
- When data conflicts, decide wisely which side to err.
Chapter 2 — Forecasting, Strategy
Any new forecasting method just has to be better than the current method. Avoid the “it’s not perfect” trap.
Assumption based forecasting
Forecasts aren’t just numerical or date values. A forecast is often a numerical value in addition to embedded assumptions (stated and un-stated)
Forecast the assumptions to hit values, dates or dollars. The value, cost or date will happen if our modeled assumptions ALL come true, and we have modeled the right assumptions.
Forecasting is a discussion, not a negotiation. Avoid trying to force assumptions into estimates that hit a date you didn’t predict. Turn the conversation into a discussion about what investment level is needed to succeed.
Get the (often pre-conceived) date people expect and forecast the assumptions to hit that date. Stakeholders have a date in their head. Pry it out. Present what’s required to come true or be managed to hit that date. Don’t negotiate to do more in less time, negotiate what’s needed to deliver more in less time (or doing less).
If nobody will give you a “desired date,” make one up. Discuss the options of build cost versus cost of delay. Model the assumptions required to hit a series of dates. Negotiate what the right investment level.
Use hit/missed assumptions as the forecast scorecard and status report. Any assumption that fails means the forecast is out of date and likely invalid.
Assumption based forecasting recap
The process of assumption based forecasting comes down to six steps –
1. Ask stakeholders when they “expect” delivery. If they don’t know, make up your own targets.
2. Determine what is necessary to hit that date. These are assumptions. Get rigorous feedback on any missing assumptions.
3. Determine what value makes each assumption untenable. These are tipping-points that make a forecast viable or comical.
4. Get rough estimates for each assumption. You can stop the moment one assumption fails by falling past the tipping-point. This forecast is a no-go.
5. Spend time getting data and doing more analysis (firming up the estimates) for assumption estimates that are closest to their tipping-point values. You can stop the moment one assumption fails.
6. Always share the assumptions when giving the forecast. Use the assumptions as the status report of the forecast validity going forward.
Definitions
Forecast: To estimate or calculate in advance; predict or seek to predict.[1]
Estimate: to form an opinion or judgment about to judge or determine generally but carefully (size, value, cost, requirements, etc.); calculate approximately[2]
Guess: to form a judgment or estimate of (something) without actual knowledge or enough facts for certainty; conjecture; surmise.[3]
Forecasting is carefully answering a question about the future, to a transparent degree of certainty, with as little effort as possible.
Is my forecast model reliable?
Back-testing (before work has started) — See if your model actual would have forecast a recent outcome. Go back three months and see how the actual known today compares to what the model said.
Assumption-testing — assumptions coming true (after work has started) — Confirm the assumption (estimate ranges and the basic must-haves) are actually matching reality. If they miss, so will your forecast!
Baloney Detection
Ask better questions! Be skeptical. Insist on understanding why. Look for simple ways to double-check what you are told.
Chapter 3
There are necessary condition assumptions and measurement assumptions in forecast models
Measurement assumptions are the numerical building blocks of forecasting. The three pieces of measurement information needed for forecasts are -
1. How much initial work (size)
2. How much work gets added over time as work is done (growth)
3. How fast we build and deliver that work (pace)
The basic model for time based forecast is -
Resources: Throughput Forecasting spreadsheet (downloadable here).
Chapter 4
“Sampling: 1. (Statistics) the process of selecting a random sample”
(Harper Collins Publishers, 2014)
We use sampling when it’s too intrusive, costly or difficult to gather every piece of data.
The three rules for reliable sampling are –
1. Take samples at random (avoid cherry-picking on purpose or accidently).
2. Avoid sampling that may exclude or include some part of the population (censoring).
3. Understand when the results of sampling are likely more correct than other methods (for example, knowing when expert range estimates might be better).
Always average the answer using multiple methods for estimating. Each one may be strong (or weak) in a certain way.
Tip: Look for the missing, not just the available. Absence of expected data, is data in its own right.
Chapter 5
“Uncertainty indicates we have limited knowledge about the future and can only represent our understanding with possibilities, and the probability of those possibilities” (Spetzler, Winter, & Meyer, 2016)
Key points and tips discussed in this chapter:
- Uncertainty is limited knowledge about how a future event may play out.
- Probability is the number of possibilities that match what we need divided by the total number of all possibilities.
- Just because you haven’t observed it yet doesn’t mean you can get away without counting it as a possibility.
- Nine to eleven samples give a good indication of the total likely range when sequential uniform numbers are involved.
- Although black swans exist, there are a lot more white ones.
- We should read Nassim Taleb’s work on the limits of traditional probability and statistics in the face of massive impact, low probability, unforeseeable events.
Coming Soon
Chapter 6
Chapter 7
Chapter 8
