Planning and Estimation
From the “Software Engineering Cookbook” Series — How good planning and estimation can help you deliver robust software on time
One more article from the “Software Engineering Cookbook” Series, targeted at growing engineering teams. Today let’s talk about planning and estimation, but first, let’s go back to Unikorn Inc. and see how they are doing…
Another day at Unikorn Inc.
It’s the start of the sprint. We have already sat down with the Product team in the past two weeks to flesh out the features we want to work on, and now it’s down to us to come up with a sensible plan that can deliver as many of these features as possible.
Features at Unikorn fall into two camps: big and small. Normally, we get to work on two or three big features per sprint, and we fill the remaining time with small features and bug fixes.
Sometimes, we need to drop one of the big features from a sprint delivery because we are not ready. Neither the VP of Engineering nor the VP of Product likes this. In fact, they pretty much hate it, as they have to go and reset expectations with the rest of the executive team, and also because it slows down product velocity and makes us fall behind the competition.
There’s nothing much we can do though, we seem to run into unknowns quite often, and that slows us down. We tried to add contingency in the estimates for each feature, but that just makes us less predictable and stretches big features over multiple sprints, making us slower and less motivated.
What are we doing wrong?
The development plan
Regardless of specific sprint process, there are normally three activities that the engineering team needs to perform before implementation can begin:
- Task breakdown : The deconstruction of the feature into smaller components / tasks that can be assigned to different team members for implementation
- Estimation : How long it takes to do each of the tasks
- Work assignment : Who is going to do each of the tasks
These activities are closely related and affecting each other: task breakdown depends on the team’s skills set and work preference, while estimation is highly dependent on who does the work (i.e. the work assignment).
In the following sections, we will look at them in more detail in order to come up with a more accurate plan for the sprint, hopefully reducing the amount of “surprises” found during development and therefore make it more likely to complete on time without sacrificing quality.
Task breakdown
The purpose of task breakdown is to divide the work into smaller tasks that can be assigned to different engineers so that the feature can be implemented “in parallel” as much as possible. It also helps the team to analyze the feature a bit more before coding begins, uncovering potential gaps in the technical specs.
There are some guidelines that we can set up when doing task breakdown to maximize the effectiveness of the exercise. Here are three basic ones that I found useful.
Try to make each task self-contained and executable by a single engineer
What does this mean?
“Self-contained” is about the definition of “done”. A task can be considered self-contained if you can easily determine whether it is completed or not after execution.
“Executable by a single engineer” is necessary to make the task schedulable during the “work assignment” phase, and to track the work itself. If you find that a task requires two engineers to do the work, you can likely split it into two interdependent tasks. If your team uses pair programming, it is OK to consider the pair as a “single engineer” in this context.
Keep the task size sensible
Try to make each task small enough to be digestible and big enough to be significant.
If the breakdown creates too many small tasks, integrating them might take a considerable amount of time.
On the other hand, if the tasks are too big they might end up taking a long time to be reviewed, as well as delaying other work which depends on them.
The “sweet spot” for task size is somewhere in between these two extremes. You can arbitrarily set a guideline for this at 1 day / 1 point and adjust accordingly by observing the commits that come into your code review system.
Be clear and concise in your task definition
When defining a task, you want to be clear on what you are trying to achieve (i.e. the expected result of the change) and how it should be done, maybe explicitly referencing or linking to the technical specs.
Additionally, you should think about the interdependencies between tasks and clearly mark them in your task-tracking software. This will help you figure out how to better arrange the work to avoid waiting on others (i.e. determining the critical path).
If time permits, it is worth to get the task breakdown peer reviewed within the team to ensure the tasks are written in a way that is understandable by the readers, rather than by the writer alone.
Estimation
Estimation of a task is highly dependent on who will do the task. On the other hand, you cannot assign resources without knowing how long it will take to do the work.
For this reason, it might be worth to create an initial estimate based on an “ideal mid-level engineer” profile, and then adjust this estimate during Work Assignment according to the engineers allocated.
Estimation is hard due to the amount of variables that come into play during implementation, such as changes in the codebase, code dependencies and the engineer’s knowledge of the system.
However we can adopt some recommendations to gradually make the estimates more accurate. Let’s take a look at three such tips.
Consider worst- and best-case scenarios when estimating
When considering how long it would take to perform a task, we normally make a number of conscious and unconscious assumptions on what is available and how we are going to use it, for instance the state of the codebase, any third party library involved, API availability, etc.
Sometimes, things do not go as planned and we need to either implement missing functionality (e.g. a missing helper function), refactor existing code or downright change the implementation plan for the feature.
Other times (more rarely), we have “lucky finds” during implementation that shorten the completion time.
When we estimate, it is worth considering worst-case scenarios as well as best-case scenarios in order to provide a more accurate estimate. A formula I often use in estimation templates is the following (derived from a book I read a long time ago):
[Best + (Normal * 2) + Worst] / 4
Where “Best” represents the estimate for the best-case scenario, “Worst” the opposite case and “Normal” would be the engineer’s initial “gut feel” on the time it would take to complete the task.
The variation between “Best” and “Worst” can also be used to infer the level of confidence for the estimate. Little variation indicates a higher level of confidence, while a wider range indicates the opposite.
Keep a risk register alongside the estimates
Once worst case scenarios have been considered, it is worth recording them to have a better idea of what is likely to go wrong during implementation.
The risk register can be just an additional tab in the estimation spreadsheet, with a sentence or two explaining the risk alongside an indicator of likelihood (Low, Medium, High, or Unknown).
Thinking about risk and reviewing the risk register might uncover knowledge gaps in the team, as well as code debt in the system.
Review the estimates and the risk register at the end of the sprint
This is an important step to improve your estimation. Although no two tasks are the same, reviewing the estimates and the risk register at the end of the sprint can help the team build up a better “gut feeling” for future tasks and possible risks.
It can also help mitigate individual bias on estimation and risk analysis, as different people are naturally inclined to be more aggressive or more conservative based on their personalities.
Work Assignment
Once you have a good task breakdown and initial estimation, it is time to pick the team and finalize the implementation plan.
Unlike the previous two activities which were more concerned with technology and numbers, work assignment deals mostly with people and their feelings, and it can greatly influence your team’s spirit and velocity. Let’s look at how to make the most out of this activity.
Balance skill requirements with personal interests
It is much more effective to work on something that you like than working on something that you hate. We should therefore try as much as possible to match personal interests and training opportunities with the work at hand, rather than just looking at the current skill set of a given engineer and matching that to the requirements for the task.
Following this advice will most certainly increase your team’s morale, at very little (if any) velocity cost.
Of course in order to do this, we must first identify what are the personal interests (in terms of work) of each engineer. This requires both talking to the person (e.g. in regular one-to-one meetings) as well as observing their work quality and mood when performing specific activities. I will go into further detail on personal growth in another article.
Manage the risk
As we have seen previously, the estimates come with an indication of confidence and a risk register.
We can start managing the risk during the work assignment phase by picking “experts” or more senior members of the team to oversee work in delicate areas, or to actually perform this work themselves.
At minimum, we should ensure the attention of these experts is directed to those tasks during code review.
Be fluid
I am an advocate of the 10–15 minutes daily sync-up with the team, usually at the start of the day to check on progress and working through any hiccups.
This is a good time to adjust work assignments accordingly and course correct. The task allocation plan should be a living document that represents today’s work and the projection for sprint completion.
In order to improve on planning, it might be worth to keep a history of the changes to analyze at the end of the sprint. Doing so might uncover (across several sprints) some reallocation patterns that might be extracted and turned into a process refinement.
Estimating work is never easy. No two tasks are the same, and no two people are the same. Even if they were, the environment (code base) would most certainly not be the same. Getting to a precise estimation of work is almost impossible without doing the work itself.
However, we can improve estimation accuracy by setting up and following some common guidelines like the ones we just looked at.
Most importantly, like any other activity planning is a continuous process of learning and optimization. You should not be afraid of tweaking your process and guidelines, trying new techniques and dropping what is not working for your team.
I hope you enjoyed this article and found it useful. Please feel free to share your experience and leave your thoughts below on planning and estimation and I will be more than happy to start (or continue) a conversation.
Meanwhile, good luck on your journey and keep on writing good software!
