Hypothesis-driven development — including a checklist for dangers of estimation
This post is part of 101 ideas for agile teams.
Context
Your team estimates the effort needed to build a particular scope. Either someone outside of your team requires the estimation. Or the estimates are used by your team to plan or coordinate its work.
Probably, your team struggles with the estimates because they tend to be too optimistic. But even investing more time and energy into the estimation process does not result in better accuracy of the estimates.
Action
Building a software product is a complex problem done by a complex system (development team and its environment). You can find an excellent introduction to complexity theory here.
Here is a list of reasons why your team’s estimates are probably wrong:
Diseconomy of Scale
Estimation usually bases on two concepts: First, split up the work into smaller pieces (e.g. User Stories). Second, sum up all the estimates.
But these concepts would only apply if the individual parts did not interact with each other. However, we build a single system so the parts will be coupled in some way, and they cannot be looked at entirely independently.
If you build a system with two features, and if we somehow know that feature A can be built in 5 days and feature B in 3 days then how long does it take to build A and B? Teams often think that the answer is 8 — the sum of 5 and 3. But that is very unlikely because if features A and B are coupled, then they will have an impact on each other. Maybe you can reuse something for the other feature; maybe the features do not play very well along with each other, resulting in more time needed for implementation. The total sum of the time required is the time for each feature plus the time required for integration of the two features.
Therefore, a simple sum is at best an estimate on the minimum effort needed.
Quality attributes of the built (software) product are emergent and not predictable
In modern (software) products based in the cloud with lots of systems in the product’s environment, predicting quality attributes like performance or load behaviour is not accurately possible. There are too many parts that are not entirely under the team’s control. As a result, teams are often confronted with different system behaviour than expected and must react to those emergent behaviours. Sometimes, even small changes in the system topology can result in severe performance issues. Minor effects in different parts of the system can build up to significant problems on the scale of the whole system due to their interaction.
Team synchronisation and knowledge management
If you don’t know The Mythical Man-Month, please read it.
Even if you knew that a feature could be built by a single developer called Dave in two months, it would tell you rather little about how long your team would have. Does your team have all the knowledge that Dave has? How much time do the team members spend for synchronising? Can the work be split up in mostly independent parts so that the team can work on these parts in parallel? Can everybody work on everything, or will there be times when team members are blocked and cannot work at all?
A team does not scale perfectly. The synchronisation of the team members (often called meetings :-) ), work that cannot be perfectly parallelised and learning affect the effort needed to build the product substantially.
Changing technology
Estimates are based on current knowledge of current technology. But technology changes rather quickly nowadays.
If you build a software product running in the cloud, you get a lot of functionality out of the box and can quickly build fascinating products. But cloud technology changes all the time. Services change their behaviour or get deprecated. Even worse is web development. No day passes without a changed npm package. And we cannot just stay on the technology as it was on the day we estimated the effort needed to build the product. We have to keep up with the cloud platforms so that our software can run. We have to keep up with the packages in our web client or sooner or later the toolchain we use to build the system is not supported anymore.
If the team does not invest continuously into being (quite) up-to-date, the result is even more effort is needed once the system breaks — probably under a lot of time pressure when it happens.
How much effort is needed to keep up with technology changes is impossible to estimate. You do not know how the technologies you use will change — for example, whether the technologies introduce breaking changes or not.
Defects
You probably heard the saying that the later you find a bug, the more expensive it is to be fixed. If not, google it by clicking here.
But more importantly, when talking about estimation is the fact that fixing defects result in task switching. We need to mentally switch back to the feature that is buggy. We need to build up the mental context from when we developed the feature. And this takes time. Time that is not included in the original estimate.
Blockers
Things that keep us from doing what we want to do:
- missing information from stakeholders
- infrastructure problems like crashing computers, services that don’t work as expected
- we are not ready to start when the project begins (missing people or tools)
- absences of team members (vacation, sickness)
- team building
Unknown unknowns
There are things we know for sure and are, therefore, quite easy to plan and estimate. Then there are things that we know that we don’t know them, but maybe we can estimate how long it will take to build up the knowledge. And then there are things that we don’t even know of — the unknown unknowns.
Because you don’t even know that they exist, how do you want to make an estimation? It is simply not possible.
Maybe you think now that of course there are such things, but they are infrequent and therefore we just set aside some time for them — no significant risk.
Unfortunately, there are quite a few unknown unknowns when building a software product:
- The users and their behaviour. The first time you see a real user use your software, you are probably going to be surprised.
- An aspect of the software architecture gets in your way, and you need to re-architect parts of your system. Likely candidates are archiving, security, scaling or performance (e.g. latency in a distributed system).
- How all the technology the product is based on will evolve.
Variability and overutilisation
In a system with no variability and complete knowledge, estimation is easy. However, our real production systems are far from this ideal. Variability is all over the place:
- Team capacity
Team members get sick, are on vacation, or join or leave the team. - Product backlog
The items in the product backlog change all the time (hopefully) based on things learned through feedback. - Technology
Technology today changes almost daily. - Variability
Even the amount of variability itself changes all the time from phases with a lot of stability to phases with lots of instability and back again.
Most plans and estimations neglect effects of variability on efficiency and are therefore overly optimistic.
What makes the situation even worse is that once the negative effect of variability on efficiency kicks in, people get pressured to work harder. The system becomes over utilised. Overutilisation has bad effects on how quickly you can react to some spontaneous event. Heavily over utilised systems get thrashed (Thrashing in the sense of computer science). The result is that efficiency goes down, reinforcing the pressure leading to a downward spiral of efficiency death.
How to deal with complexity
Instead of making a detailed plan based on estimates, your team is better off by adopting a process that addresses the inherent complexity in software product development:
- Define a hypothesis
If we build this, that will happen. - Verify with an experiment
Execute the fastest and cheapest way to prove or disprove the hypothesis. Get real feedback from real users doing real work. The only thing not real is your prototype. - Adapt and repeat
Incorporate the insights of the experiment into your next hypothesis and repeat.
Estimates are not commitments. Estimates are always wrong, but sometimes helpful.
Estimates are no longer used to make detailed plans, but as a rough guess for a lower boundary on how much effort will be needed.
What you gain
When working with hypothesises using estimates only as a rough guess, you get a process that matches the kind of problem you try to solve. A complex problem needs a problem-solving tool that can deal with the aspects of complexity like non-predictability, non-linearity and adaptability.
How to strengthen
Hypothesis tree
Instead of defining just one hypothesis, you can define a tree of hypothesises. Start with your first hypothesis as the root. Then define the next hypothesis to verify for both when the first hypothesis was proved and when it was disproved. And so on.
How to deal with customers who want to know what it will cost
You can lie to them, or better define a hypothesis like “We think that it should be possible to build all you want in x months. This hypothesis is only likely to hold when we can get these couple of features done in 4 weeks.” Build a tree of hypothesises and start working.
Risks
Knowledge of complex systems and its non-linear behaviour is somewhat limited. So you will probably be met with disbelief and sentences like: “We just need to put more effort into better requirements engineering and then make good estimates.” Try to educate people about complex systems and their inherent properties like non-linearity, emergence and adaptive behaviour. You can find an excellent introduction to complexity theory here.
Please help me improve this idea by providing feedback.
More ideas at 101 ideas for agile teams
Many thanks to bbv Software Services for making this blog post possible.
