“Can this be done by Thanksgiving?”–Using Math to Estimate Task Durations

Published in

SSENSE-TECH

8 min readMay 16, 2019

When it comes to estimating tasks and deadlines, clients tend to treat date and cost estimates as a promise. Surely, I’m not alone in having found myself trapped in projects where the theoretically soonest possible date was treated as a promise.

“Can this be done by Thanksgiving?”
“Well, maybe if everything goes right it could be done by Christmas.”
“Great! Christmas it is! I’ll tell the CEO.”

Given the fact that SSENSE is uniquely positioned at the intersection of fashion and technology, this can also be a reality at our place of work. In order to account for seasonality and trends in the fashion industry, along with the infamous pace of the tech industry, we are forced to move fast.

Having witnessed countless attempts at timeline and cost estimation, I have heard every rule of thumb imaginable–“Multiply your first guess by Pi”, “What you think the client thinks is reasonable, plus a couple of weeks”, and by far my favorite “Your estimate times 1.25 times the number of non-technical people in the room”.

While these techniques might buy you enough time to complete your project, they won’t help improve the estimation’s precision, and could leave you in a sticky situation when you have to explain where a high estimation came from.

Why are we so bad at estimating?

Say your kids start practicing soccer. You have to pick them up from school and bring them to practice. On the first day, let’s say you leave three hours before the practice to make sure you arrive in time. Despite ample time, you arrive an hour late. Maybe there was an accident on the road, maybe you got stuck behind a tractor or perhaps found yourself on a slow-moving highway.

Fast forward to next week, when it’s time for your kids’ soccer practice again. Based on last week’s experience, you leave four hours in advance. This time, you find an unusually smooth journey, and make it in just under two hours, leaving you with an uncomfortably long wait before practice begins.

We rely on cognitive shortcuts

Our brains are wired to quickly provide ballpark estimates on just about everything. Long story short, it is a mechanism that ensures survivability for the human race.

When put on the spot for a quick estimation, it is normal to feel attacked and instinctively give the best guess we can. These estimates are usually not that bad considering the speed at which we produce them.

“It took me 4 hours last time.”
“It’ll surely take 4 hours this time.”

The tradeoff for this shortcut is that it overlooks important information that could help us make better decisions. It would be more efficient to take your time and go through a real estimation process, especially in a non-life-threatening situation like project planning. Thinking fast and slow by Daniel Kahneman is an excellent read if you want to learn more about it.

Ignoring the basics of probability

Let’s apply the soccer practice scenario to a project. Let’s assume that you are estimating the timeline for a project that is very similar to one that previously took you three hours. You could stop here and say that the project will take three hours. That wouldn’t be your best option since it would ignore a lot of relevant information.

According to the theories of probability, and good old common sense, the odds for the project to take exactly three hours aren’t 100%. There is also a lower probability that it will take 2.75 hours and again, another equal probability it’ll take 3.25 hours.

If we plot the estimations and their probabilities, we would get something like this:

I’m using normal distribution to simplify the example but a project could follow a completely different distribution.

We could also reduce uncertainty for our project to narrow the bell. Going back to our soccer practice analogy, it would be the equivalent of checking for the weather and traffic information.

Reducing uncertainty would have this effect on our plot:

This probabilistic representation allows us to understand all the possibilities before us. Our most probable estimate is still three hours but we are asserting that the chances of it taking exactly three hours are fairly low. Relying on cognitive shortcuts and ignoring the basics of probability is a dangerous mix. It leaves us with a ballpark estimation and a poor understanding of the situation.

Math to the rescue

I prefer to avoid the term ‘accurate estimate’ as it serves more or less as an oxymoron. We can alternatively use math to get a more accurate estimate based on the probability distribution of the time our project will take.

As a Data Scientist, I explain complex models on a daily basis. I understand that math isn’t for everyone and I won’t take it personally if you skip the following section. I explain everything in words in the next section.

Let’s assume that the tasks within a project are independent variables; in other words, the time to complete task X won’t affect the time to complete task Y.

Let’s also assume that all the tasks are discrete variables to simplify the math. We can now use this quote from Sums of Independent Random Variables — Chapter 7 from “Introduction to Probability” by Grinstead et Snell. This work is redistributable under GNU license and can be downloaded here.

The notions we are interested in go as follows:

Suppose X and Y are two independent discrete random variables with distribution
functions m1(x) and m2(x). Let Z=X+Y.
We would like to determine the distribution function m3(x) of Z.
To do this, it is enough to determine the probability that Z takes on the value z, where z is an arbitrary integer. Suppose that X = k, where k is some integer.
Then Z=z if and only if Y=z−k. So the event Z=z is the union of the pairwise disjoint events X=k and Y=z-k, where k runs over the integers.
Since these events are pairwise disjoint, we have

Thus, we have found the distribution function of the random variable Z. This leads to the following definition.
Definition 7.1 Let X and Y be two independent integer-valued random variables, with distribution functions m₁(x) and m₂(x) respectively. Then the convolution of m₁(x) and m₂(x) is the distribution function m₃=m₁*m₂ given by:

for j=…,-2,-1,0,1,2,… The function m₃(x) is the distribution function
of the random variable Z=X+Y.
It is easy to see that the convolution operation is commutative, and it is straightforward to show that it is also associative.

Real life application

Say we have a task X, “Get home from work”, the function m₁(x) would be the probability that the task will take x hours.

We also have a task Y, “Get to the soccer field from home”. The probability that the task will take x hours is described by the function m₂(x).

We can plot the functions as follows:

For given project Z, composed of tasks X and Y (Z=X+Y), we want to know the function m₃(x) that would return the probability of the project taking x hours.

To compute the function m₃(x), we can use the convolution of m₁(x) and m₂(x). The convolution formula — the discrete version — is also available on Wikipedia.

We can plot the resulting function and get a pretty good idea of the probability of how long the project will take:

Now our most probable scenario for project Z is that it will take three hours. However, the probability that the project will take either 2.75 or 3.25 hours is not negligible. Uncertainty is now easier to account for.

But wait, there’s more

The distribution functions of a project with additional tasks could also be produced fairly simply. The convolution of the distribution functions of the first two tasks would have to be “convoluted” with the distribution function of the following one. This could be repeated indefinitely to accommodate any additional tasks.

Since convolution is commutative, the order in which we work wouldn’t matter. We could also do the same exercise to estimate costs, function points, working-hours, or with any other set of independent variables.

My favorite feature of this method is that it allows you to go as deep as you want in tasks, sub-tasks, and nested sub-tasks, as they are all described by distribution functions.

A more realistic case

Software development projects are notorious for their low success rates. With multiple constraints, key-players, and stakeholders, projects can quickly become too complicated to be estimated trivially. Unlike a round-trip to your local soccer field, there is often no ‘most probable scenario’:

Distribution of the probability that a realistic “random” project will take x hours

Understanding how and why a project has an odd distribution is a crucial skill in project management. Moreover, in this era of data and information, that company culture of taking time to assert those quantitative methods over instinct will ensure a healthy, competitive, and manageable future.

To me, the fact that most of the executive team here at SSENSE have STEM backgrounds is one of its biggest assets. Knowing that your peers understand and can challenge the reasoning behind your decisions is part of what keeps SSENSE ahead of the curve.

Conclusion

Chances are you won’t go through a full estimation process the next time you have to take your kids to soccer practice. However, you now understand how thinking about the estimate as multiple possibilities will help you minimize risks and make better decisions.

Remember that:

Giving a single estimation relies more on luck than skills. It is easy to minimize risk by assessing all the possibilities and their probability.
Breaking down projects into tasks will help you be more accurate. During the decomposition, pay close attention to the details of what you plan to do and how.
Estimation should be a slow and rigorous process, not a quick and intuitive one. Rely on numbers, not on instinct.