Project Bias and the Agile Solution

Anyone that has managed projects for any length of time knows the feeling of promising a launch date only to have that project miss, sometimes by a mile. It happens all the time and it messes with the entire cohesive fabric within a team and occasionally within the company itself. The Project Manager is frustrated, angry and worried about their job. The product owner is mad at the project team and is expressing that fact to the Project Manager. The customers and executives were expecting this new product, feature or enhancement so they are left wanting. Project delays can send a ripple throughout a company.

Thanks to the work of two behavioral economists, Nobel Laurette Daniel Kahneman and his research partner Amos Tversky we have some insights into why projects are regularly late: The Planning Fallacy. I was first introduced to the work of Kahneman in his book Thinking, Fast and Slow. Since that time, I have delved into the works by these two gentlemen and have learned much. But I digress.

The key to the Planning Fallacy is that as humans we tend to estimate our tasks as taking far less time that said tasks will actually take. For the most part, people tend to be more optimistic in their estimates in everything they do, not just projects. Whether it is cooking that familiar recipe to building that deck out back, it is part of the human brain to underestimate.

This tendency is highlighted when we specifically look at projects. Our estimates for a project or task are much shorter than experience would dictate. When you combine the optimism from each project team member for their own tasks to all other estimates for all of the tasks, we compound the problem exponentially. The result becomes a wildly over-optimistic timeline.

Project Estimates Go Astray

Ask any experienced project manager and they will tell you that this is one of the aspects of a project that they fight continually. Providing good estimates and delivering on time, especially in a waterfall environment, is one of the hardest things to do. The bigger the project, the longer timespan for the project and the more team members involved the worse and worse the estimates get.

But the bias does not stop at just the timeline. Kahneman did further work with Dan Lovallo and in 2003 they published a paper that expanded on the Planning Fallacy. Based on their research, our tendency to be very optimistic in our estimates extends not only to timeline, but to costs and risks associated with the project.

That means not only are we bad at estimating the amount of work and when that work will be complete, but we will also underestimate the amount the project will cost and the risks that are associated with the project. These are two of the three legs of the project management triangle: Scope, Schedule and Budget.

Inaccuracies in just one leg of the triangle put the whole project at risk, let alone two legs. If a project team is clearly underestimating the project for both costs and risks how successful can we expect to be on any given project?

According to a 2017 KPMG study only 31% of projects are likely to deliver on time and only 29% are likely to deliver on budget. But, what Kahneman and Lovallo is telling us is that we tend to estimate both of those factors incorrectly. If we underestimate both of those metrics, how successful will we really be? With numbers like those should we just accept that projects will be late? If two of our three legs are affected by our own biases, how then do we effectively forecast our projects?

Traditionally, project databases are kept by individual companies. Sometimes industries might have a similar, but larger scale data and today these can also be found on the internet. These databases collect information on projects, both successful and failures, to assist other project teams in their estimating. The database identifies all the key factors in the project and a team would be able to access it, search for similar projects and base their estimates on these previously attempted and completed projects.

Solving the Planning Fallacy Through Agile

This database approach is one method, but there really needs to be a better means to combat the effect of the optimism bias on our projects. The key is to approach projects in a non-linear fashion. Throw away the traditional waterfall methodology that is still common in some software companies and many non-software businesses by looking to Agile.

Agile opens the process of project planning in such a way that it can, if done correctly, counteract many of the biases we see in traditional project planning. Agile encourages and promotes openness to all aspects of the project. This allows the group as a whole to better manage estimates on time, within budget and reduce risk.

The key to Agile’s estimates is that they are open to the group. This allows for feedback, questions and doubt. The more open a project is with how its estimates are come by, the better for the overall project.

Scrum addresses these issues by having the team do user story estimating which allows each member of the team to provide their own estimate. If there are discrepancies between estimates, the team discusses why they estimated this user story a particular way. Debate and discussion around estimates is a healthy tool to counter the inherent bias that we all bring to the table.

Addressing these issues with Extreme Programming (XP) is also an option. Built into XP is a set of principles, some of which address these biases. XP emphasizes Economics ensuring that team members keep the budget in mind, whereas traditional project management separates the budget from the individual team members. XP also emphasizes the need for pair programming to alleviate the cost in time and money of rework, code refactoring and general quality.

Lean brings Seeing the Whole to a project. Seeing the whole means understanding what is happening on the project, seeing the project clearly and measuring the project using the appropriate tools. But lean does not stop there it asks the project team to deliver as fast as possible to minimize the cost of delays, while eliminating waste.

Basically, the Agile processes provide a counter-balance to the Planning Fallacy. The fallacy is not gone, but when larger projects are broken down into individual user stories the estimating is made easier. Each user story will suffer from the fallacy, but through the openness of the process to questioning and debate, the estimate is made more accurate. In addition, even if the user stories are susceptible to the fallacy the estimate errors are smaller and less frequent than would be seen if it was an individual project manager trying to estimate an entire project alone.

XP and Lean add another level to the project making all project team members accountable and part of the solution. The team works together to limit the Planning Fallacy both before the project starts and every day during the project’s execution.

Optimism is a key ingredient in a happy, successful work / life balance, but in projects it injects bias that we cannot allow. It is our job as project managers and team members to limit the bias of optimism. We need to anchor our estimates and this reality must permeate throughout the company. If we do it wrong, it can be very costly. Do everything possible to get it correct. The best way, as I see it is through iterative and incremental estimations where the entire team is involved every day; in other words: Agile.

Recommended Reading:

Michael Lewis, “The Undoing Project: A Friendship That Changed Our Minds” about Kahneman and Tversky.

Daniel Kahneman, “Thinking. Fast and Slow”

Dan Lovallo and Daniel Kahneman, “Delusions of Success: How Optimism Undermines Executives’ Decisions”, Harvard Business Review (July 2003).

For more blog post similar to this, please subscribe at

Originally published at on January 3, 2019.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store