why is software effort estimation so difficult?
If you ask a software business sponsor today what their biggest challenge is when dealing with IT, many would respond that software engineers cannot reliably deliver working software to an agreed deadline within known existing parameters (team size, days in the week). In the main, IT is often perceived as an unreliable partner to its sponsors.
Certainly there is no shortage of both anecdotal and evidential examples of “failed” software projects: delivered late, under quality, without necessary features, and/or over budget. And if these metrics are baselined against the estimate, we either have a serious problem with competency or with estimation. So are we quite as bad at estimating effort to create software as it seems?
In the 1990's I worked with a client who was running a “two year” project which, one year in, was almost three years behind schedule.
Lest we forget, “modern” computing (from the 1940's “turing-complete” machines such as Colossus and ENIAC onwards) were funded and developed not to solve the problem of conducting calculations that a human could not do, but to execute these calculations at a speed which were impossible, or at least inefficient to be carried out by humans. Functions such as creating ballistic trajectory tables during the second world war could be done by human operatives but would take over a month each to produce. What we now call “time to market”, the speed of delivery of the outcome of software has always been a feature of software engineering.
Granted, software engineering is not like other engineering disciplines; there are far more opportunities for unexpected outcomes caused by unplanned-for interaction between components, and the behavioural attributes of software components are far less well understood than say, in structural engineering. An increase in scale of components means that there are many more permutations to “break the plan”. Unknowns caused by novelty (of language, architecture, business scenario), optimism bias and inexperience are factors that indeterminately and (to users) opaquely affect the accuracy of estimates. Estimating something that’s never been done before is hard; where exactly do you go for guidance?
IT’s historical response to such complexity is to lock down the requirements. NO changes permitted, or the “plan” (that which we all bow down to) must be redrafted. This approach has historically broken the trust-relationship between business sponsors and technology. Often users became frustrated and increasingly assertive with this bizarre constraint; needs are neither fixed nor fully expressed at the beginning of a project. Technology responded by holding users to ransom: “we will not start the project until the needs are fully understood”. And any software engineer who attempted to put forward an effort estimate in this culture was forever anchored by downward pressure on their estimates.
Insufficient time and resources are often invested to understanding the problem, let alone the solution. And one thing we know is that software design is full of undiscovered rabbit holes waiting for us to fall in to; Der Teufel steckt im Detail.
Adoption of both behaviour driven and agile engineering has started to solve some of the fundamental dysfunctions within software teams but are often still challenged by the question: “how long will it take to get this value?”
The #noestimates movement seeks to sidestep the question insofar as “software is essentially a “creative, variable pursuit, [where] solutions are often revealed as we go along” and imposition of estimates “does not allow for (or, at least, does not embrace) emergent design, requirements, change and innovation” (via @neil_killick) so fixing scope is essentially broken thinking. All true. It’s also true that (from the perspective of development and test at least) estimates are fundamentally irrelevant to shipping working code (which after all, is the point, right?) And worse, estimation potentially threatens the holy grail of software engineering: estimates have the potential to disrupt flow. So #noestimates is a pretty enticing concept, especially for a developer.
But is that it? Are estimates really conceptually indistinguishable from a fixed scope? Should estimates be avoided because they contribute nothing to the code, are essentially difficult and disruptive?
#noestimates is on the money when it seeks for “teams to commit at the outset to building and delivering the best possible product by a given date and/or for a given amount of money”. And yes, “release planning using velocity is contrary to an iterative approach (holistic, evolutionary improvement of the product) and is more in line with a purely incremental approach (delivering a pre-defined Product Backlog feature by feature)”.
But perhaps this somewhat misses the point. technology is a service industry; IT exists to meet the needs of users, often by creating working software. Iterative software development to improve features, yes of course, but unmet need is unmet need, whether that is for code or for estimates.
Software engineers expect their users to give them direction in BDD / story writing and fund software development activity. Software engineers seek commitment, guidance and a partnership (of sorts) from business users, and vice versa. This is not a transaction that takes place at code delivery; it happens before, during and after the sprint. It happens continuously, across projects, products and suites. And the estimation game is a joint activity not exclusively an IT task. By refusing to engage in estimation, IT is failing to fulfil its commitment to developing the trust-relationship with its business stakeholders.
The most dangerous strategy is to jump a chasm in two leaps.” ~Benjamin Disraeli
From a business sponsor’s perspective, software engineers asking for a degree of trust where estimates are not required straight off the bat is a non-starter. How do you decide if you want to invest in something unless you have a qualified estimate as to how much it’ll cost or when you’ll get it? Moreover, in banking, just as for the first modern computers, value is often a direct function of the delivery date. Being able to go long on turkeys doesn’t have much value if the system ain’t working by thanksgiving. And being able to get a missile to hit its target is no good if the war is over and lost, which, after all, is what drove the creation of the modern computing industry (vs. human processors) in the first place. Software engineers must go to their users to establish their users’ needs; a qualified, accurate estimate is a need.
Without a doubt, estimation is an over-valued activity in many organisations. Every organisation has ‘clipboard project managers’ whose sole purpose is to maintain a gantt chart as if they were in some kind of wilful purgatory. Yet, perhaps the great difference between a coder and a professional software engineer is that professionals code against constraints. One of the major constraints is the reality of time being finite. So whilst (in its purest form) estimation is somewhat irrelevant to making working code, in the real world, time is a real constraint that professionals have to learn to work within.
In highly regulated environments, “the best possible product by a given date and/or for a given amount of money” may not realise any business value at all if that “best possible product” is insufficient to meet regulatory requirements. There is a mandatory fixed scope and it is unavoidable. If then there is a fixed scope to be met and perhaps the date is fixed, then (assuming quality and skills are constant) the only remaining variable is money. Any investment must return value by increasing the ability of the team to execute.
Perhaps estimates are less important for start-ups building an MVP? But wait. By definition, there are a minimum set of features required to realise value. And usually a fixed runway of funding. Without estimates to guide such decision making how do we know that the product vision is in any way realistic? Without this knowledge, do we know if we shouldn’t consider pivoting now? Good quality estimates are necessary to fail fast.
What about environments where any minimum scope has been reached and value has been baselined? Perhaps estimates are less valuable here, but we rarely (but not always) find ourselves in this space after technology and business stakeholders have reached a mature, stable and trust-based relationship.
BDD / SBE / ATDD does enable a team to scale by front-loading the effort (more on this in the next blog post), but it’s neither a linear, perfectly scalable investment (“nine-women, one-month”), nor have we ever seen investment made without some estimate of the work to be completed.
Estimation is also a value path. In order to estimate, development (a coder and tester) needs to review, consider, and produce a high level plan of execution for the feature. So in many ways the discipline of “forcing” estimation is very important: it ensures communication within the team, and that both the developer, tester and user have a common understanding of what is to be done, how it will be done (to a degree), the universe this solution exists in, what the risks and unknowns are.
Management. Somewhere in any organisation, there’s a person who decides whether we hire that extra developer, or tester, or splits the teams. This person has needs too. Usually to act to avoid the crisis, not after. For projects (yes, we used the “P” word), qualified estimates enable management to help avoid failure and sponsors to gain confidence that the team understands it’s WIP.
We have to square the circle here. Yes it’s absolutely necessary to educate all parties involved about what an estimate actually is. Estimates are not guarantees of delivery. And estimates without confidence intervals are near-valueless. But if software engineers refuse to offer such qualified estimates at all they shouldn’t expect to be given the opportunity to start most software activities.
Technology and business stakeholders look to each other to build a mutual partnership, trust-based relationship of equals. Estimates are part of that equation; without high-quality estimates (that are jointly understood as being no more than estimates), business stakeholders can (and will) avoid starting most development initiatives, and impose six-sigma on to the rest.
Farid Tejani is a Partner at Ampersand, a boutique strategy and management consultancy specialising in sustainability, emerging technology and product innovation.
