Why Estimation in Software Development is Usually Terrible

Estimating how long it will take to implement software is hard. Deceptively hard.

Those folks that get it probably only do so because they’ve been wrong many times before. Wrong to the point where they’ve been forced to work late hours, consume unhealthy amounts of caffeine (who knew that was possible?), or miss deadlines resulting in broken contracts and likely financial pain. I most certainly have been there.

In the early days of running my own agency, projects were constantly behind. Looking back on it, I was incredibly naive. It took almost losing my business before I truly began to understand how hard it is to estimate the time to write a complex piece of software.

Psychologists call this the Dunning-Kruger Effect. It is the phenomenon where by less experienced individuals overestimate their abilities while the more masterful tend to do the opposite.

Typically, after a few failed attempts at estimating how long a piece of software will take to write, a software developer will lose virtually all confidence in their ability to estimate. At least, if they haven’t the people who depend on their ability to deliver, will have.

So why do developers tend to be terrible estimators (this author included!)?

Imagine you are sitting in Times Square, New York (it doesn’t matter if you’ve never been there, indulge me). You can see through the crowds to a McDonalds restaurant across the other side of the square. Your task is to estimate how long it will take you to get there (on foot) as accurately as possible.

Take some time to assess the situation and make some assumptions. Lets say its a weekday and its winter. There should be comparatively few people walking through the square. You can see the McDonalds so it probably is not a long distance to walk.

Whatever other considerations you make its probably quite reasonable to expect a short walk to the McDonalds from your current location. For the sake of the exercise we’ll estimate 5 minutes. How confident are you about that estimate?

Now imagine you start in the same place but you are asked to estimate how long it would take you to get to Grand Central Station. If you happen to be a New Yorker then you’d know this is straightforward: jump on the shuttle and you’re there. Your estimate would be 10 minutes or so.

But what if you’ve never been to New York before and you don’t have a smartphone with you? What if instead of Grand Central Station, you were asked to estimate your travel time to some far-off farm in Virginia? One you’ve never even heard of. Even the New Yorker might struggle now.

The problem here is uncertainty. The less certain we are about how exactly to do something the poorer our estimates of how long that thing will take to achieve.

The problem here is uncertainty. The less certain we are about how exactly to do something the poorer are our estimates of how long that thing will take to achieve.

And the reverse is also true. If we are particularly certain about how to do something then our estimates tend to be poor too: we underestimate how long it will take to travel down a familiar road (see “Well travelled road effect”).

In software teams this problem is compounded. Each person on the team may now have a different level of understanding, a different opinion on how something might be achieved and have a different level of confidence (perhaps they are on a different section of the Dunning-Kruger curve).

Agreeing on how long something will take to develop now requires that all members of the team reach a reasonable level of certainty about how the feature or product will be implemented.

In my experience that almost never happens. Even if the team thinks they have reached a consensus about how something will be done there is usually at least one member of the team who has a different idea. And very often this person doesn’t realise that their idea is different to the others.

And so we come to the second reason why software estimation is usually terrible. This time it is a direct consequence of the fact that most software projects are developed by teams of people and not by individuals.

Information Asymmetry, commonly discussed in economic circles is the effect where by two parties believe that they have a shared understanding but, because of a mismatch of interpretations of an idea or of the context in which that idea is made they actually have two very different perspectives.

I’m reminded of a story of a Sioux man, Cincala who was brought to dine with an Englishman, Colt after being captured.

The fork was a clumsy thing. He went to wipe his mouth on his sleeve but he held up his hand to stop him:

“Stop. Use your napkin”

“What’s a napkin?”

Colt held up a small, white square of cloth.

“But it’s clean!” he protested “I’ll get it dirty”

To Colt, the idea that a person would spoil their clothing to spare a napkin was disgusting but such a simple disagreement has behind it layers of complex reasoning, context and upbringing.

While estimating a problem in software development and thusly seeking to reduce the uncertainty about a problem, a pair of developers may present ideas on implementation.

It so often is the case that a seemingly simple idea is presented but quickly dismissed. Its detractors not having the faintest idea of the myriad complexities and constraints that resulted in its forming. Complexities and constraints that take precious time to present and explain. Time that is rarely afforded in a high-pressure startup or consultancy.

Take the developer who has spent the past month or so working closely on a piece of software and has come to understand its nuances and foibles. She understands the shortcuts that were made as well as the reasons the corners had to be cut. Not just the decisions but the delicate details of the discussions that were had to make them.

This developer may have the best ideas. Or, her closeness to the problem could result in her failing to see the forest for the trees be unable to present a solution that would be afforded by thinking “outside the box”.

Regardless, it is decidedly clear that she has a vastly different perspective than a developer who has only recently joined the project.

How we think about how long something will take rarely in fact has anything to do with time. Rather, it has everything to do with complexity, uncertainty and the inherently flawed human cognitive process.

And this is exactly why estimation in software development is so terrible.

Enthusiastic readers will find strategies to improve accuracy in software estimation in How to estimate in software teams — Part 1.