So you have the functional requirements for your project in front of you (right? if not, please read Project Management for Developers 101 and then come back here), all of your questions have been answered, edge cases have been defined, and you are ready to get to work. And then (if you are lucky) your boss asks you for a reasonable estimate of when you will be able to deliver. Or (if you are not so lucky) you are informed that the project needs to be done by some fixed date X number of months (or weeks) from now. How do you answer? Is it even possible to give an honest answer that has a shot at being on target? With the right preparation, I think that this can be achieved.
The general plan for our estimation is:
- Be sure that you know what you have to build (and estimate)
- Translate functional requirements into a technical plan for implementation
- Break the technical plan down into smaller chunks and give gut estimates. Repeat Recursively.
- Ensure breakdown completeness, and add up those hours
Now let’s go through it step by step:
1. Know what you have to build and estimate
Simply put: if you don’t know what you need to build and don’t have a pretty good degree of confidence that your understanding of the required functionality matches up with the client/stakeholder who you are building it for, then any technical breakdown or estimate you produce is going to be garbage.
This is not to say that you have to know every last functional requirement for the entire project before starting. Often this is impossible or impractical (or just a waste of time, if the end requirements are based on the first stages of the project — see my article on PM for Devs for more on this). However, in order to be able to estimate any given section of work, you do have to know what that individual section is supposed to do.
Note: if you are working on a project where you can only firmly define the functionality that is required on the next section (or sprint) of work or two, then it would be dishonest and misleading to provide anything beyond a very general estimate (with lots of warnings and provisions) for any section beyond this. If you know the first 10% of functionality required and your boss tells you that he needs an estimate for finishing 100% of the project you have a few options. You can tell the truth and say that you can only attempt an accurate estimate on the section where you know the functional requirements and maybe make an attempt to give a very rough estimate for the rest (while being sure to cover your ass in the process). Or you can give in to the pressure, confidently give an estimate for the whole thing (or agree to the date that is presented to you) and then try your best (but be ready to take the blame if you don’t you don’t make the deadline, which is the most likely outcome).
2. Translate functional requirements into a technical plan for implementation
This is your first step of actual technical planning for the implementation of actual work. You may have done some technical work during the requirements stage that was needed for a proof of concept, for validating assumptions, etc. Any of that work may be thrown out or reused here. But here is where you can look at the overall big picture of what must be done and start planning on how you will do this.
At this point, you just want a high-level breakdown. For example, if the overall project is a new page in a CRM tool that will allow for recording information about upcoming sales, your high level breakdown might contain the following high level areas:
- Set up DB Architecture for new tables, changes to existing tables
- Modify Data Access Layer to work with new DB Architecture
- Add new functions and modify existing functions on service layer that are needed for retrieving/modifying/manipulating/caching/etc the new data
- Add new API functions/modify existing API Functions
- Front-end work to allow for display of new data, update/insertion of data, etc.
Many projects will share these general types of high-level groupings to breakdowns: DB-DAL-Service-API-FrontEnd. However, these are not ubiquitous and are not present for all features/projects/solutions. And in the case of a project that is focused on just changing things on one layer of the app (unlike this full-stack oriented solution), your top-level breakdown will be one level down, breaking down the overall solution into major sections of work in the given application layer where you are working.
Each step in this breakdown will not have an hour estimate associated with it (yet). The important thing here is to identify what are the main sections that you will be working on.
3. Break the technical plan down into smaller chunks and give gut estimates. Repeat recursively.
Now we get to the meat of it.
The operational idea here is that every big section of work can be broken down into a smaller section of work. And those smaller sections of work can be further broken down. In theory, you could break down a large section of work into thousands of steps detailing which letters to type. It would of course be absurd and counter-productive to break down your tasks to this level. There is definitely a law of diminishing returns in effect here.
Our overall goal is to get the breakdown level small enough (by the end of the next step) that you can easily estimate how much time it will take to complete the task with a high degree of confidence and a low degree of risk. It will take some experience and trial and error to get a feel for exactly where this threshold lies in terms of the amount of functionality and the amount of time. I have personally found the sweet spot to lie around two hours per granular item. Lower than that and you are wasting your time with breaking it down more than you need to in order to achieve your target estimate accuracy. Higher than that and it is quite likely that you are going to forget things that you need to do in order to complete the task (things that you would have “discovered” if you had broken it down further) — this will result in going over estimate.
(That said, there are definitely times when I will estimate a task all the way down to 15 minutes. There are some tasks that are really that small, and there is no reason to over-estimate them in order to conform to some artificial minimum time requirement per task.)
So take one section of work and begin to break it down. A section might start at a gut estimate of 3 weeks (or even just “a whole lot of time”). Break it down into smaller pieces. If you can make a gut estimate for a section that you broke down, record it. And keep on breaking it down into smaller, more granular chunks of functionality until your estimates for those items are below your estimation threshold (I recommend above that this be two hours, but see what works for you). Then repeat for all sections.
While breaking down an individual section, it is important to have the spec open on your other screen to the corresponding section in the spec (you have multiple monitors, right?). The breakdown and estimate is worthless if it doesn’t actually implement the functionality that is needed. So be sure to dive deep into the spec for each section, and make sure that every requirement is being addressed. And often in the course of doing this, you come up with questions that are not addressed in the spec: edge cases, details that were not discussed but will be significant, potential questions of scope, etc. Now is the time to bring them up. Email/talk to the relevant stakeholder, make a note to follow up or flesh out the details of the spec. But whatever you do, don’t do nothing. Catching these items at this early stage of the process can often be the difference between high quality work delivered on time, and buggy output delivered late.
4. Ensure breakdown completeness, and add up those hours
While making your breakdown, it is important to include everything that you will need to do. This includes things like: creating the actual new pages and files, new controllers (server or client side), creating and updating unit tests, testing the actual functionality of a complicated item before declaring it complete, importing graphics, fixing regression issues, and making the actual deployment. Also, be sure to estimate the time that will be needed for developer ramp-up, when appropriate (this can sometimes be a somewhat significant time investment, especially in scenarios when developers will start to work on new complicated sections of logic to which they have not yet had exposure).
Try to think of all of the things that happened on previous projects that you didn’t think of until the end and then had to do, or that were the cause of simple-to-fix bugs (regression or otherwise) immediately after release. And then be sure to include these in your breakdown and estimate as well. They may not seem to be so significant, but they can really add up in terms of time and distraction.
Now add your hours. If you are using software designed to help with estimates like Microsoft Project or any of the myriad of online tools available for this, this may be done automatically. If you are using Excel, a doc or some other mechanism for recording hours, then add them up manually. You should now have hour totals from each small granular section up to each large section.
This is the point where most people will add on some padding. Ten, twenty or even thirty percent. Give that extra margin to help ensure that you deliver on time. While this is OK to do when you are starting, and are not yet in a position to be super confident about the completeness of your breakdown and the accuracy of your estimates, as you break down more projects this is something that you should work to avoid. Large amounts of padding is a crutch that will prevent you from totally relying on your breakdown, and by extension, will always give you an excuse for not being as complete and accurate as you can be. Complete and accurate breakdowns and estimates are possible. They take some work, but when achieved, can pay dividends down the road.
The one allowance you should take in this area is in translating hours into days. Your boss would rather hear that you will have the component ready for QA by May 6 than that it will take you 129 hours to complete. But when making this translation, don’t make the mistake of dividing your hours by eight just calling it a day. For almost all of us, an eight hour workday does not equal eight hours of productive work. We live in a world full of distractions, and even a few interruptions over the course of a day can result in hours of productive time lost. We have meetings, need to go to the bathroom, eat lunch, talk to your coworkers or take a break. A good rule of thumb is to assume that you will only be productive for 80–85% of the time you spend at work, at a maximum. This translates to somewhere between 6–6.5 hours in an eight hour day. Divide your total hours by this number. (And if you know that you have other duties or meetings that will consistently take up more of your time, then take these into account as well). This will allow you to come up with a much more realistic number of days for your estimate. And when using these days to calculate an actual release date, be sure to skip weekends or any known vacation or holiday days (nothing is more annoying than completely nailing your estimate, but having people expect the work to be done earlier because you didn’t take into account your vacation or the company holiday).
What If I Screw Up the Estimate?
You are probably telling yourself: “Yes, if I could break down everything completely, and estimate everything accurately ahead of time, then of course I will always deliver on time. And if I were that kind of a prophet, then I could just buy tomorrow’s winning lottery ticket and never need to work again. But estimating is freaking hard!”.
So what do you do when you have tried your best to break down the project ahead of time, estimated the tasks as best as you can. And then one day you come to the “Automate DB Backups” task estimated for 4 hours, and you suddenly realize that you forgot to take into account the eight new DB servers that were installed in remote facilities last week. You don’t know what the estimate is, but it will almost certainly be an order of magnitude higher than the original one. What do you do?
- Don’t Panic. Everything will be OK!
- If this is going to affect someone else in the short term (ex: someone is expecting a deliverable in one day that will now take a week), let them know right away that there will be a delay and you will give them more details shortly
- Reestimate the task, drawing on your new-found knowledge. If new questions need to be asked from others, ask away
- Update the estimate wherever you keep the numbers, and communicate what happened to anyone who needs to know
These things happen on every single project. No matter how hard you try. You can try to mitigate them by being as thorough as you can be up front, but when something like this happens, just go with the flow.
Estimations, especially at a granular level, can be very annoying to do. Breaking down your project the right way means taking things that you would ordinarily procrastinate over (for example: figuring out every thing that you need to do on the front and back end in order to make that new component work) doing them up front. There can be a tendency and inclination to just half-ass these types of things, to leave your estimates on the high end, to gloss over detail and to just add some percentage of time on the end for “padding”, and hope that this will make up for any corners that you cut earlier in the process. Although this can sometimes end up working out (if you pad things enough), doing so will prevent you from receiving one of the main benefits of a up-front thorough estimate: being able to plan out just about every detail of the work up front, eliminating surprises and avoiding surprises. When you break down a section of work into small chunks, you force yourself to think everything through. And you can be much more confident in the resulting breakdown, both from a time and a scope perspective.