Software Forecasting Capabilities

Many decisions go into running software and IT companies. These decisions have some element of uncertainty about the future, and this is where some form of forecasting and estimating future values is necessary (I include the good old gut instinct as a crude forecast). In order to make informed decisions, the capability to forecast certain future values needs to be developed, and this article looks to list those capabilities, and a plan for developing and improving them further.

Many of the questions we set out to answer when planning what work should and could be delivered by an organization or team are bad questions. They may be well meaning, but the question itself isn’t the forecast that should be used in making better decisions. The ubiquitous, “When will it be done?” is the classic dysfunctional question. The answer to this question assumes a fixed scope and a fixed delivery rate, both of which are control levers for optimization. It also assumes that nothing else is more important to do, that this “one” thing will absorb all of the available resources and the goal is to do it as quick as possible. If it is that urgent, then more investment in team size, skills and knowledge may be a better decision. But this never gets asked, because the team is answering the “when” question. If team size is truly fixed, and every bit of scope needs to be delivered (or nothing works), then perhaps “when will it be done?” has some merit as a question, but i think in those cases you should just leave the team to do it rather than bothering about knowing when (and making it later by absorbing some resources to do it).

For forecasting, I believe there are only two questions:

1. How much, a question of size
2. How fast, a question of delivery pace

From these two forecasts,

how long = how much / how fast.

delivery date = start date + how long.

Both how much and how fast are controllable. Any forecast needs to spell out the rationale for fixing these to certain values, and the assumptions that were used to make that case.

To answer what should be started next requires an answer to a third key question, How urgent. The answer to how urgent is needed to know what investment is needed and if its too high or low for any given option compared to others.

Understanding the capabilities required to answer these three questions allows organizations to know the weak spots in analysis and forecasting. Any weak spots will mean sub-optimal decisions are being made. It should be the goal to consistently look for ways to increase these capabilities or to find easier and less expensive ways to perform these capabilities that yield a similar result.

Capabilities versus Maturity Models

What is proposed here isn’t a maturity model. This article aims to list the core capabilities necessary for forecasting and avoid being involved with how or how well. Each capability probably does have a maturity model, and I do have opinions, but local context will need to be know to know what will work and what level of detail is needed for each capability.

Capabilities are the “What” not the “How” or “How Good.”

If you insist on needing a maturity model. The first step along that maturity model is having a way to do each and every one of these core capabilities. From that point, keep looking for improvements and keep investing in better ways until the cost of improvement is too much for the value that new level of detail brings to any decision.

I start by asking how each of these capabilities are performed now, and if there is some analysis in place, i give it a Yes, otherwise I propose the easiest first step of analysis. Doing something is better than having no answer at all.

The “No” answers are a list of forecasting risks. Decisions where these forecasting inputs would be used are based on no analysis, and you need to assess the risk of making an incorrect decision in that realm. Some might not make sense in your context. For example, if team size is truly fixed, having “No” way to analyse staff skill and changes is a low risk. If you are legally compelled to do some features next no matter what, then being able to quantify priority is less important. You can see that any fixed maturity model might lead you to invest in non-consequential capabilities. Don’t do that, look to improve and develop the capabilities that support the decisions you can make and the outcomes you can change. Ignore the rest for now.

How much (sizing) Capabilities

The key capabilities required to answer how much are -

• Determine segmented groups of potential deliverable options. Story mapping is my preferred mechanism for deriving deliverable groups.
• Determine the size of these groups (in units that pace is measured or convertible to). Reference class forecasting is my preferred mechanism for relative size estimation against prior completed work. Keep he size of prior delivered features to compare proposed features against to quickly assess relative size.
• Determine how much work size grows over time from defects, new ideas, lessons learned, etc. Analyzing historical data and sampling the sources of growth is my preferred mechanism. Pick five sample features and rigorously record what got added over time to deliver these five.
• Determine how much work splits from this “size” to the size the team delivers and pace is measured. Splitting for the team to deliver should be encouraged, as it helps the team maintain flow and quality. Just measure it and remember to account for it when forecasting (1 to 3 times is common).
• List what work might be needed (risks), but its too soon to tell (e.g. performance might be slow and need more optimization, the website might fail on certain browsers and need to be fixed).

How fast (pacing) Capabilities

The key capabilities in answering haw fast are -

• List of equipment, capabilities and skills required to deliver an option. My preferred way to capture and analyse this is a simple spreadsheet: Capability Matrix.
• Determine the constraint (point of minimal average flow rate) of the delivery system. My preferred method is to value stream map, or look at the capability matrix spreadsheet. If the team is delivering, look for where work sits idle, and the constraint is the process following that point.
• Determine the amount of work delivered in a period of time through the constraint in the delivery system. System pace will be the pace through this constraint, making this pace the right value for forecasting. My preferred method is to look at throughput at multiple points in the delivery process and find the lowest. This is the constraint, and the flow at the constraint.
• Determine periods of impaired or increased flow. Often events and seasonal vacation periods have a non-negligible impact on delivery rate. Have a way of collecting when these periods will occur and a way of estimating the impact on throughput during those periods.

How urgent (prioritization) Capabilities

The key capabilities in answering how urgent are -

• Determine the impact if an option isn’t delivered. This is the ability to understand the near term impact of value from items not being delivered.
• Determine when the impact of not delivering an option begins. Some options don’t incur impact until future dates. This is the ability to understand the longer term impact on future value.
• Determine how the impact of not delivering an option impact other future options. This is the ability to understand dependencies between options.

Process Capabilities

These are a set of capabilities that help understand process and expectations. These differ from the forecasting capabilities, but they are necessary as inputs to enable those capabilities.

• Determine what types of work is delivered. The ability to know what different types of work are flowing through a system and their specific stresses on the delivery system (skills, effort, etc.
• The ability to know what work is in progress. Seems obvious, but a system must be able to show what work is waiting to be started, has started and has delivered.
• Time in process. The ability to measure how long it takes from start to delivered for different types of work.
• Delivery rate (or throughput). The ability to measure (and predict) how fast work is completed by type in a system
• Arrival rate. The ability to measure (and predict) how fast new work arrives into the system.
• Age of in-progress work. The ability to know the current time in process total for all in-progress work trended over time to see if it is growing or shrinking.

Summary

This article outlines the capabilities that need to be present in order for software and I.T. organizations to make informed decisions. I’m sure there are more than I've listed here, but this is the minimal set of capabilities delivery systems need to be optimized over time.