Software estimates: wrong and necessary
TL;DR
Software estimates are necessary unless you have infinite resources or are working for fun. Monte Carlo forecasting is one of the best estimation techniques because it removes the need for guessing. Give it a try by using the free Monte Carlo simulation spreadsheet by Adrian Fittolani
Software estimates are notoriously known to software developers as a big time-waster and serve as a plentiful source of jokes. In addition to a typical dry description one can expect from a wiki, the Wikipedia article on Software development effort estimation hosts a decent humor section with quotes like:
Hofstadter’s Law: It always takes longer than you expect, even when you take into account Hofstadter’s Law.
This blog post is exploring the fun part in greater details by identifying the 5 laws of Software Estimates:
1. Estimates are Waste
2. Estimates are Non-Transferable
3. Estimates are Wrong
4. Estimates are Temporary
5. Estimates are Necessary
So, the general agreement among software engineers these days is to accept that estimates are always wrong. Let’s try to see if it’s possible to make estimates less wrong.
Asking for estimates is OK
Requesting estimates is not an offence. When ordering pizza we need to know when the order will be completed. Customers don’t accept the answer: “Your pizza is ready when it’s ready”. Passengers wouldn’t understand a train timetable with a clause: “ETA is pointless because we never get on time”. Home owners wouldn’t sign a contract stating “Constructing a house is a complex process so we’re unable to provide any estimates until before the completion phase”. Obviously, predicting small repeatable tasks like pizza delivery is easier than house construction. Even then, the construction industry is pretty good at hitting the deadlines, which can be impacted by suppliers, weather, and other unforeseen circumstances. However, estimating development of something unique like software, is much harder.
It’s commonly known that each software project is exclusive because no one has ever built the same exact thing before. That’s true, but it doesn’t mean that software development teams should avoid any estimation. After all, products and services they produce are funded by someone and impact plans people make based on the estimations provided. A marketing department may need to prepare an advertising campaign towards a new product release; the board may need to confirm funding for the project that comes next in the pipeline; the users want to know when they can get new features in their hands, etc.
While software estimates are inherently hard, there are some proven techniques that make them possible.
Confidence level estimates based on gut feeling
This is when the team is trying to answer a question “Are we confident the project will be delivered in x weeks?” If the answer is NO, repeat the question: “Are we confident the project will be delivered in x+1 weeks?” and so on until the agreed amount of weeks is determined.
Another variation of this technique is to ask “How confident are we in delivering this project in 8 weeks?” If the answer is less than 80%, keep adding weeks until the confidence level grows. Either way, this method is probably the least accurate because subconsciously people tend to underestimate.
In summary, confidence levels can be off target by 200% on either side. Sometimes, however, it is good enough to start the project with. In rare cases it may even be more accurate than any other guesstimate, but this would rather be a pure coincidence than a rule.
Estimates based on comparative sizing and prior experiences
By analyzing the requirements, an experienced developer can fairly accurately predict the level of complexity for a project. Sometimes that’s all you need. It would provide a guesstimate good enough to make a decision. For example, if Service A took 1 month to complete, and Service B looks similar to Service A, the overall estimation to deliver Service B would also be 1 month:
If the services need to communicate, you need to add a buffer time for the integration between them. Sometimes building this type of service integration can be more expensive than building the actual service. Let’s assume building a service and connecting it to another one are similarly complex tasks (which is not far from the truth in the microservices world). The total estimation would be 3 months: 2x services and 1x connection between them:
What happens if we need to develop another similarly sized service that talks to the other two? Can we assume that the estimation will increase by 1 month? Probably not. The overall complexity of the solution would most likely increase two-fold because the increased number of connections should also be accounted for:
Overall, comparative sizing can be quite accurate for projects of a similar nature done by the same team but is not applicable for completely new undertakings with many unknowns.
Story sizing estimates
Decomposing a project into user stories allows better estimations on the story level. The biggest downside of this approach is the inability to size all necessary stories upfront. Until the team is ready to take on the next story, its definition can change quite significantly from the original estimation session.
While story sizing is not a bad exercise, the teams should always remember not to spend too much time on it. Another issue is when the focus during the design/grooming/refinement session is shifted towards answering the question “How big is this story?” rather than addressing the more relevant questions: “Do we understand WHY this story is needed and IS the acceptance criteria clear?” Focusing on the time/effort estimation leaves the team less chance to challenge the story itself and look at it from a critical perspective.
In the end, decomposing a big problem into a set of smaller sized tasks is commonly used by most Agile teams. The subjective bias should be minimized by inviting all team members to the exercise. This method is pretty good and gives adequate results especially for mature agile teams where members are experienced enough in story slicing and everybody understands well the complexity of individual stories.
Common estimation pitfalls
- Saying that making it right is impossible and we’d better not waste our time at all.
- Ignoring estimation where time and budget are critical factors. Using any of the methods above is better than nothing.
- Being put off by an estimation higher than anticipated. If you have to have this project done you’d better start it sooner.
- Wasting too much time on story sizing.
- Assuming the new project/team is going to produce similar results to what has happened in the past on another project and to a different team.
- Attempt to make the estimates perfect and not allowing for extra scope being added later due to “unknown unknowns”
- Treating the eventual estimation as a commitment or a hard deadline — it’s just a prediction and everybody in the business should understand that.
Estimation biases
Humans are pretty bad at estimating things and the main reason for it is a whole set of cognitive biases developed over the past few thousand years. Wikipedia article lists over 20 cognitive biases that cause under- , or over-estimation, for example:
Dunning–Kruger effect: The tendency for unskilled individuals to overestimate their own ability and the tendency for experts to underestimate their own ability.
Illusion of control: The tendency to overestimate one’s degree of influence over other external events.
Optimism bias: The tendency to be over-optimistic, overestimating favorable and pleasing outcomes (see also wishful thinking, valence effect, positive outcome bias).
Planning fallacy: The tendency to underestimate task-completion times.
Avoiding biases is almost impossible so the best people can do is to understand them and minimize their impact. Majority of software projects are underestimated. The main reason for that is the impact of technology, dependencies, unknowns, and other factors projected through the biases mentioned above. By knowing that, project managers sometimes double the numbers produced by the estimation sessions. This doesn’t sound good, does it? Is there an alternative to estimates that would negate the influence of the many biases out there?
Monte Carlo forecasting
The biggest difference between estimation and forecasting is that the former involves guessing while the latter is based purely on historical observations and some mathematics.
If all the stories in the project are similarly sized, predicting becomes much easier. For example, if the team has completed 10 stories in the last iteration, it should be fair to expect another 10 stories to be delivered in the next iteration. In reality, however, team velocity depends on things like story uniformity, availability of team members, length of the sprint, team commitment to this particular project, and so on. As a result, some iterations may be shorter by a couple of days and some may see only 6 stories completed rather than the average 10.
With historical data available we can use the free Monte Carlo simulation spreadsheet by Adrian Fittolani to determine a probable range of delivery dates for the entire project. For example, in the following picture, we have 34 stories completed over the last 5 sprints (with 100% focus on this project):
There are 30 more stories to do for the project, so we enter 30 into the Work Items column:
Finally, the Project Distribution graph shows the probabilities of the project completion time:
The probabilities of the project being completed are: 6 sprints with 91% confidence, or 8 sprints with 100% confidence.
The chart enables a meaningful conversation with the business about the anticipated delivery timeframes. It can be added directly to the team dashboard or distributed to the stakeholders in a status update report.
The Monte Carlo forecasts are not based on estimates and give high level of accuracy if enough historical data is accumulated. It works best in teams practicing granular story slicing where all cards are of a similar size. Obviously, there will be smaller and bigger cards, but overall, during a period of several iterations, those cards will balance each other off and won’t distort the results.
The following tips were learned while delivering an 8-month project using the Monte Carlo simulation:
- Forecasting should be based on accurate historical data, so start measuring team velocity to have at least 5 sprints of data.
- Allow for more stories to be added to the backlog. Re-run the simulation when the backlog changes.
- Rather than forecasting the whole project, define milestones/checkpoints and track your progress in achieving them.
- Celebrate success when the next milestone/checkpoint is reached.
Summary
Software estimates are necessary, and if done correctly, can setup realistic expectations for the team and the stakeholders. Estimations based on historical data tend to produce more accurate results because they remove cognitive biases from the equation. One of those estimation techniques is the Monte Carlo simulation, which makes a prognosis of the project completion dates based on historical team velocity. The resulting forecasts shift the focus from subjective opinions to objective calculations to help the team and the business stay in control of the project delivery.
