Agile Estimating and Planning — Rationale
I was recently asked to put together a presentation on Agile Estimation and Planning tailored for Director and VP level management. I think estimation and planning is one of the hardest agile concepts to grasp. It requires a shift in thinking and eventually a shift in the role of management in software development. I’ll try and summarize the presentation here, in 4 minutes or less.
Understanding Agile estimation requires accepting that, for most products, software and software development is complex. I am referring to the Complex Context or domain in the Cynefin framework for decision making. Even seemingly simple apps we interact with today are far more complicated than we think. They’re usually hosted on cloud platforms, employ a stack of languages, are dependent on 3rd party libraries and technologies, integrate with external systems, collect data from distributed systems, function on multiple operating systems… the list goes on. In addition to this, the team or teams working on them are made up of people with different skills using different tools who communicate with each other with differing levels of success. Add to this mix the speed at which business and market needs change today. You can see the problem.
Most of us in the software industry know this, but we simply don’t know how to work effectively in the Complex Context. We may think that if we put the right processes and controls in place we can manage this like managing an assembly line or a small construction project. Or we think that if we just hire enough experts and break the problems into smaller parts, we can predict the outcomes of our work with an acceptable margin of error.
These strategies work in the Cynefin Simple and Complicated Contexts, but not in the Complex Context. When they don’t work, we want to double down, applying more processes or getting more experts, which results in analysis paralysis or a development process with ever more stage gates, documents and approvals. And the Rational Unified Process was born.
Experienced software developers instinctively know how to work in the Complex Context. They don’t look at a feature, design the solution end-to-end, write the code all at once then compile and go home. They write a small piece of code and see if it works. Then they write a bit more. Rinse and repeat, refactoring as they go. Most software development organizations today also practice continuous integration and run unit and integration tests at least daily, because they know they can’t reliably predict what the outcome will be, even with a team of senior developers.
This is also the agile approach to planning software development. We still have processes, we still make plans and we still need experts. But we recognize that we can’t reliably predict outcomes. So we build in small increments, conduct small experiments and adjust our plans. We still estimate the work, but we recognize that those estimates are wrong, so we minimize the time and effort invested in those estimates and re-assess the plan frequently.
So before we can talk about how to estimate and plan in the Complex Context, we as managers and developers have to accept the following truths:
- Estimates are always wrong and our old strategies (padding, getting more expertise, spending more time analyzing) will result in longer timelines not better estimates or outcomes.
- “Plans are worthless, but planning is everything.” - Dwight D. Eisenhower. The key outcome of planning and estimating is not the plan itself. It’s the exchange of information and the establishment of communication channels between the people doing the work.
- The plan has to be flexible enough to change as more information becomes available. This means either adjusting scope or moving dates.
Good scrum masters/agile coaches need to know the above well, as they should be explaining this frequently. When Sales or Marketing is asking for concrete dates for delivery of large features weeks in advance, it’s up to the scrum masters or product owners to explain why that can’t be done and build trust by providing frequent updates on progress and decisions.
So how do we plan and estimate? Iteratively and collaboratively, while minimizing our upfront investment. This post has been longer than I intended, so I’ll cover the “how” in more detail in a follow-up post.