Can I have those red project plans, please?
A heuristic-based approach to frictionless agile software development planning
Plans. Detailed. With all the activities and their durations and dependencies. Hard cast dates. Can we produce them, please? And if the estimated dates don’t fit the expected ones, let’s mark the plan red — we will crank up the project management activities to turn these dates into green ones. We will escalate, pull in favours, and apply the “right level of pressure”.
This approach to planning is often the norm in software development, and I find myself repeatedly at its receiving end. I am sympathetic to these requests, but I don’t subscribe to the delivery philosophy of “let’s work hard on the dates”. It results in burnout and stress. It lowers the quality. We can do so much better than that.
We need to use alternative approaches that recognise the unpredictable nature of software development. Let’s explore some options.
Why the dissonance?
But first, why do we have this dissonance between project management and delivery teams?
Project managers are placed in tricky situations. Tasked with creating project plans, they are caught between the pressure of senior stakeholders needing results and certainty of plans and the teams and the intrinsically unpredictable nature of software development. Creating plans with hard-to-achieve dates, something that I call “red dates plans”, seems to reassure those senior stakeholders that the situation, while challenging, is somewhat in control. These plans seem to offer the (false in my view) confidence levels that we can still do something about those dates. We can still work harder, can’t we?
This dissonance often results in a perceived lack of planning, but often it is not the case.
Software development is an act of discovery. Teams often operate in complicated contexts, where dependencies can be deep and many. Where we don’t know many answers a prior (technology or domain ones), and we need to employ a process of discovery and exploration. After all, we will know if the solution worked after we’ve built it and the users interacted with it.
An approach to planning — the use of planning heuristics
The act of software development is a complex one and requires a different approach to planning.
So, what can you do about it? One approach is to utilise planning heuristics, such as the one I often deploy:
- (1) reduce the granularity of the problem by decomposing (chunking) it into smaller releasable problems;
- (2) deploy a mechanism of using multiple work item types (functional items, spikes or experiments, tasks), each with its specific set of activities and workflow;
- (3) estimate the smallest releasable chunk but adopt a probabilistic outlook;
- (4) visualise the work and track how work items flow through the system, not the plan;
- (5) re-forecast when you have new information.
Is it a recipe? No, it’s not. It’s a heuristic. Dave Snowden, the creator of the Cynefin framework, defines heuristics as “measurable guidance that can adapt to unknowable unknowns”. By its nature, it is an approach to a problem that is incomplete given the available knowledge but can help guide thinking in the appropriate direction  and adapt to uncertainty.
Software development requires an approach to planning that considers its nature and embraces uncertainty.
1. Reduce the granularity by decomposing (chunking) the problem into smaller problems
Reduce the problem space to multiple releases, taking an iterative approach to delivering the functionality:
- aim for each iteration/release to be functional, even if the solution is not yet complete;
- aim to make each release as small as possible; give them relevant names so that team members would understand at a glance what they will deliver;
- accept that ‘iteration’ means that sometimes you will need to redo the work done in previous iterations.
Using this approach, you help the team to focus on a limited number of problems at any given time — an excellent technique for limiting work in progress.
2. Deploy a mechanism of using multiple work item types, each with its set of activities and workflows
The time spent defining various types of work items is well spent, distinguishing between functional items, tasks and spikes.
- express functional items as behaviours of the product you build and not as activities the team needs to carry out to implement the feature or behaviour. Express the functional items as User stories;
- use Tasks for activities required to complete project management-related activities (e.g., organise weekly architectural review sessions);
- use Spikes for situations when you cannot define the activity or the desired behaviour upfront. Time box them as experiments with a clearly defined goal.
Define a workflow for each work type. Map out what work needs to be done to take an item from an idea stage until it is done (as in properly done, functionality deployed in a target environment). In the workflow, define all the activities required to complete the work, such as solution design, architecture, implementation, and verification. Highlight activities that typically require external approval.
3. Offer an estimate for the current release
To provide an early estimation, continue the previous steps by asking yourself several questions:
- do you know everything you need to know to complete the release? If not, who and how could help you identify these unknowns?
- what are the dependencies on your work items? Are those who need to meet your dependencies aware of them? Could they commit to completing them in a reasonable time? Could they publish an SLA?
- how confident are you that you would not uncover more work or that the work would be harder than you think?
With all these questions answered, continue estimating your work items' duration. I recommend using at least two approaches:
- expert blink estimation — ask multiple experts to offer a view on how long they reckon it would take to complete those work items. Ask for a likelihood or confidence level (see also https://peterpito.medium.com/confidence-levels-96796ec1be97), ensuring they don’t influence each other in their answers; time-box this activity;
- run some Monte Carlo (MC) simulations using Focused Objective’s forecasting spreadsheets .
On the back of these steps, you get a range of durations. You will need to choose which output to use, the expert view or the MC simulation one. Always use ranges with correspondent confidence levels regardless of which estimate you consider. Why? Because you want to convey the message that your estimations are not deterministic and that it’s hard to be accurate by providing a single date. Call it a forecast, and same as the weather forecasts, be prepared to show that you don’t know for sure when you’re going to complete the release.
Once you have a range of durations, ask yourself if you could set a start date — as in, do you know a point in time when all the conditions to start the work are met? When the answer is yes, use that date and add it to the top of the estimated durations, coming up with a range of end dates. If the answer is no, stick with durations until you derive a start date.
4. Visualise the work and track the work items, not the plan.
Take all the items through the workflow, working on a subset of problems at any time. To preserve the flow and not artificially increase the work in progress, start work on new items at the same rate you complete them.
Worry less about tracking a Gantt-style plan. Pay attention to items that are likely to age. Visualise blockers and ask for help for items that are blocked or are likely to become blocked. Be aware that you might not always know ahead of time if something will take longer until you are working on it.
5. Re-forecast when you have new information.
Measure the Throughput and re-forecast. You can start using Focused Objectives spreadsheets , transitioning to more advanced tools such as Actionable Agile .
As soon as you acquire new information, re-forecast, always showing our latest view.
Planning is important. It is important even in software development. Project management and project managers are not the enemies of software delivery. Quite the opposite.
But to achieve harmony in the teams, we must acknowledge that software development needs an approach suited to its nature. An approach that deals with work with high variability and typically low flow efficiency — one where discovery and design play a more prominent role than in many other fields.
There are no silver bullets, but there is knowledge. And there are techniques that work better than others. The one from this write-up is based on a heuristic. As warned, it might not always work, but I used it successfully in large-scale engagements multiple times. It is influenced by complexity, probability and queuing theory. Give it a try, and let me know how it works.
Here to a harmonious future …
- Simple Heuristics that Make Us Smart — Gerd Gigerenzer; Peter M. Todd; ABC Research Group
*inspired by Focused Objectives forecasting spreadsheets