Why software engineers struggle to estimate effort

lessons for leaders of software engineers

The sprint was just 10 working days. At sprint planning 10 days ago, optimism was high. The team played planning poker and ended up with what seemed like a reasonable sprint plan of 63 story points. Day 3 of the sprint and the burn-down chart was not reflecting expected progress. Not enough points had been burned down.

Yet optimism remained high — the lack of visible progress was easily explained away by the team. “That story is almost complete…some unexpected surprises, but I should be able to catch up.” Common statements to explain away the lack of objective progress.

Fast forward to day 10 of the sprint and some stories are still not done. These are the very stories that, just 10 days ago, engineers confidently stated they could get done in the sprint. Of the stories that are done, a few are not quite what was expected. Not the expected level of completeness — possibly missing some aspect of the feature entirely. Or the performance sucks! Definitely not ready to put in front of customers, without a bunch of additional work.

How did the estimates go so wrong?

The intoxicating power of creation

Creativity is intelligence having fun — Albert Einstein

The software medium is unique in that it enables rapid construction. You can go quickly from an idea in your head to “working” software. Software is a unique “soft” medium that allows its users to quickly shape a new form. And if the form is not right, to quickly reshape it.

If the result is not what is desired, it is just as easy to modify what was built. Also without developing a detailed plan. Worst case, you can scrap the entire thing and start all over again. And probably create something even better than when you attempted it first.

All with little cost, and with a high degree of independence of others. No material costs! All you need is an editor to code, and an environment to let your code manifest into your creation. It’s just as easy to observe your creation in action.

This power is intoxicating, habit inducing. It breeds a sense of optimism that anything can be done. At a pace that is not realistic. With an incorrect sense of completion. All but the most seasoned software engineers suffer from this.

In this context, estimating effort is difficult. It may be possible to estimate the first iteration of creation and observation. But how many iterations will it require to get the desired result?

Generally, estimates tend to be overly optimistic.

The desired result

Teams suffer from lack of clarity in this area. Agile methodologies like Scrum have introduced the concept of done-done, one done when the user story/feature meets its acceptance criteria, and another when it passes all the quality tests. For this to work, one needs to have clear and objective acceptance criteria.

It can be painful to list detailed acceptance criteria. Making them objective and binary is also difficult. This typically requires a collaborative session between the product owner, engineers, and quality. A session in which acceptance criteria are written, re-written, combined, decomposed, until all parties agree they are right. This collaborative engagement by engineers prevents them from rushing off and implementing something they “think” is required. It forces them to focus on developing code that meets each of the acceptance criteria.

It is not sufficient to just define acceptance criteria. The development process must enforce them. If you’re using Scrum, don’t allow engineers to burn down points until all acceptance criteria have been met. The product owner has signed off. The quality owner has signed off. Only then do engineers earn points.

One question that typically comes up is about the environment to use to validate acceptance criteria? Dev? Integration? Staging? Production? Choose what makes sense for your business. It does not have to be Production.

Another typical question — who goes first? product owner or quality? My recommendation is for the product owner to view progress as frequently as possible in the engineer’s Dev environment. To “course correct” frequently. If you adopt this practice, the product owner would likely be first to sign off, and to do it in the Dev environment. Quality sign off should never be in the Dev environment. At a minimum, it must be the integration environment.

You may not see immediate results. However, a few sprints of this practice and enforcement, and you will start to see results.

Complexity, unknowns, risks

The highly flexible medium in which we work is also highly complex. Especially when you consider that what we build never lives in isolation. It is typically built upon many other pieces of technology. And it must integrate with other systems. The ecosystem of other technologies and system is also rapidly evolving. What was true a few months ago, may no longer be true.

All of this introduces many unknowns into the software development process. The more the unknowns, the higher the risk that a quality deliverable will not materialize.

The natural tendency with many junior engineers is to deal with these as they happen. Seasoned and hardened engineers know this can be an expensive strategy, potentially requiring a great deal of re-work later in the project. At a time when the customer and market pressure is high.

These unknowns and the complexity of the living in the rapidly evolving software ecosystem introduce a high degree of variability into estimates.

As leaders we can do some things to counter this. Introduce some long term thinking and planning. Technical design documents and reviews are a practice that helps with this. Looking ahead a couple of sprints to uncover known unknowns is another. Investigative spikes can be scheduled in the current sprint to reduce the risk of these unknowns.

These risks must be factored into the delivery dates provided to business. Set yourself up for success, by targeting a date range and leaving yourself some leeway on scope and quality. Identifying upfront the “range” of scope that can be delivered and the “range” of quality that might be acceptable, can give you options to ensure you deliver in the target date range.

Non-functional requirements

What security environment does the system have to operate in? What industry standards does the system have to comply with? PII? HIPAA? etc. Does GDPR apply? What’s GDPR? The compliance landscape is also continuously changing. What data loss is acceptable? What down time is acceptable? How will data migrations be done? Other maintenance? Performance requirements can also fit in this bucket, assuming they aren’t already covered in the acceptance criteria.

Thinking these through systematically will reduce the variance of estimates. You will be able to focus on those non-functional requirements that are absolutely required, and defer the rest.

A good way to do this is to have a technical design review document that has all the system or non-functional requirements. Each item must either get a “not applicable” designation with a clear explanation of why. Or it must explain how the new piece of code will meet this requirement. In the process, you will have triggered the necessary thought process to include this effort in the estimate.

The role of product and engineering leaders

Be aware of these phenomena. Build balanced teams with adequate representation by seasoned engineers. Ask lots of thought-provoking questions. Be sure all aspects have been thought through. Introduce engineering practices that balance but don’t stifle the creativity of software engineers.

References

[1] GDPR. The EU General Data Protection Regulation. http://www.eugdpr.org/

[2] Brooks, Frederick P. The Mythical Man-Month: essays on software engineering. Adison-Wesley, 1995.