Why estimations are always wrong and how to deal with it
I read a fantastic book by Vasco Duarte called NoEstimates: How To Measure Project Progress Without Estimating. Estimating time & cost is always one of the main problems in software development, and I’d like to share some ideas I’ve learned from the book. If you like it, I highly suggest that you read it yourself.
What is an estimate?
How much it will cost to build this application and how much time it will take? It’s an obvious question you would expect in a vendor-buyer relationship. For vendors, an estimate is a tool to compete and win the bid, for buyers, it’s a tool to select the fastest implementation. When an estimate is provided, and the project starts, it quickly becomes a deadline that should be met. That means that a project with a given scope must be delivered within a set amount of time with the predefined budget.
Unfortunately, this almost never happens — according to CHAOS report that Steve McConnel is referencing in his book “Software Estimation: Demystifying the black art”. A software project has about 25% chance of being delivered on time, according to chaos report surveys. If you’re using an agile approach and you have excellent managers you can increase your chances to 42% but is this risk of not delivering on time or over budget is low enough?
Why estimates are always wrong
The main problem with estimation of a software project is high uncertainty. You can easily estimate work that consists of independent repetitive tasks with known complexity. That’s not the case in software development.
Walking on water and developing software against a written specification is easy if both are frozen.
Right estimations
Project managers are often asked to provide right estimations. Nobody wants to hear realistic estimation because they are not competitive. Because of that managers often provide estimates that sound right and then try to deliver something within a defined deadline. It has tremendous consequences, and this is the main reason customers, and support teams struggle so much.
Right estimation might provide a short-term gain but always hurts the project. In the end, according to the iron triangle, you have to sacrifice scope to keep time and budget intact. That’s the only way to make the project work, but it should be done in the right way, by prioritizing features that bring the most value.
Unfortunately, most of the projects are ‘all or nothing.’ It’s a very stressful way of working for a developer — you work hard every day to do the best you can, but instead of feeling confident and inspired you to feel desperate. Because of the high level of uncertainty, it becomes a battle with everyday change requests. The amount of work is growing as deadline approaches, and it feels like there’s no hope.
Accidental complication
According to J.B. Rainsberger’s talk he gave in 2013 there are two types of complications in a software project — essential and accidental. Essential complications describe the complexity of a problem or a feature on its own. It’s a known complication that can be analyzed and understood before starting the work.
An accidental complication, on the other hand, is something that creeps into the work because no feature or problem exists independently of the system it works within. There are a lot of complications that come from the program systems and how they are written and from the processes and organizational structures.
Because of accidental complications you can’t compare features implemented in different projects or different teams. You can’t even compare them within the same project and team really because codebase was different.
The cost of accidental complication is usually higher than the cost of the feature itself.
Change requests
Change is something that software projects should embrace, especially in the beginning of the project. Unfortunately, most of the time changes are not accounted for when the estimate is provided. First, there are a few of them. As the first iteration is finished and testing starts the floodgates open and flow of changes, both functional and conceptual, becomes an every-day problem for the team.
There’s nothing wrong with changes. It’s just that they affect the scope and implementation time dramatically. Can you predict all the changes that will occur and their impact? Sure, you can expect a certain amount of functional bugs and even account for them, but what about total blind spots, the functionality that wasn’t thought of in the beginning and would require a significant change (that will also include accidental complexity)? The answer is no, no matter how hard you try.
Accepting that we’re late from the start
Every project starts with an estimation. According to Wikipedia, an estimation is is the process of finding an estimate, or approximation, which is a value that is usable for some purpose even if input data may be incomplete, uncertain, or unstable. (https://en.wikipedia.org/wiki/Estimation). Since the data is always incomplete and uncertainty is high in the beginning, an estimation is almost always wrong. And at some point, you must accept that you’re close to the estimated deadline and you still have a lot of work to do.
I’ve been working on many projects, and this happens all the time. The only way to recover a project that is late is to cut the scope. You can’t throw in more people even if you have the budget, because more people will increase the overall complexity, they’ll need time to get up to speed and communication costs will be extremely high. The only thing you can do is reduce the amount of features your product will have or simplify their implementation.
Flexible scope management
Enter flexible scope management. Often this term scares people off because the change of the scope is associated with scope creep and feature bloat. But it works not only in the direction of adding requirements, it also works in the opposite direction.
When we understand that the project can’t be delivered on time, we should prioritize the features and simplify them, reducing the scope and reducing the complexity and time needed for implementation. But we don’t need to wait for the deadline to be entirely or almost missed to do that. If you approach the project from the flexible scope from the very beginning you can’t miss the deadline, even if the deadline moves to an earlier date — you will always have most critical features which could be implemented within the time that passed already done.
