Shake It Up — Adding Interruptions to Project Burn Down Simulations

In the “Shake It Up” series I am taking my readers with me on a journey towards understanding Monte Carlo simulations, and then how I turned spreadsheets into forecasting tools.

TL;DR: A good Monte Carlo simulator for project forecasting uses throughput as a variable for burn-down. A great Monte Carlo simulator takes into consideration other realities that can impact the progress of a project (the good-great format was inspired by Geoff Watts). In this article we will:

1. Understand how to simulate Technical Debt and Bugs
2. Understand how to simulate Scope Creeps
3. 🚨🚨🚨 A challenge for the readers on finer work breakdown🚨🚨🚨

Previously in this series…

In my last article I have shared a first attempt at building a Monte Carlo simulator using Google Sheets. The outcome had its successes and windows for further improvements and opportunities.

One of these was the realization that the simulated burn downs were mostly based on throughput, and were independent of other factors that are typically faced during a project’s lifetime. If we look at our experience when working on projects, our progress is usually hampered by several other variables such as:

1. Technical debt
2. Scope Creeps
3. Finer Break Down of Work

The challenge ahead was to try to find a simple and fair way to simulate the above variables. Let’s explore them together.

Interruption Source #1: Technical Debt

Technical debt is a tricky things to quantify when working on a project or product.

What is technical debt?

In software development, technical debt is the result of prioritizing speedy delivery over perfect code. The result would be brittle functionality, open to surprises in the future. The difficulty in measuring technical debt is that you do not always know what you do not know!

Technical debt is a reminder that shortcuts come at a cost: a cost that will impact our forecast.

How can we simulate technical debt as a variable?

A typical cost of technical debt is the introduction of bugs. Whenever a bug is discovered and fixed, this is not done for free but at the cost of a team’s delivery capability. We can, therefore, deduct “bug points” from the randomly generated throughput score to simulate the slow down impact.

Simulating Bugs

I would consider two approaches to simulate the impact of bugs on a forecast.

1. Historical data — Bugs are typically tagged separately in the work management systems and therefore a team can look into its history and feed the data to the simulator. The assumption in this case would be that the bug creation pattern is systemic, and not endemic to the project. If this data is handy, it is a simple solution to simulating bugs.
2. Mathematical Distributions — The other option could be to simulate the generic behavior of bug creation, rather than the team’s trend of creating and fixing bugs. This requires us to explore into the world of using mathematical distributions and while it is a more complicated approach, it can have its advantages, and so it might be worth exploring.

The first option is clear if you have the data at your finger tips. In the spirit of discovery, I am interested in exploring the second option.

There is a wide variety of mathematical functions around that produce different shapes of numeric relationships, and choosing one will depend on the behavior that we want to replicate.

In my case, I imagined bug discovery as a relationship between the work done and the bugs fixed. Basically, my assumption was that the more work developers managed to finish in a week, the higher the probability of them discovering and fixing bugs.

My function, therefore, was to be a relationship between throughput and bugs. When generating a random throughput number, I would map it to the function which would give me a number of hypothetical bugs. In plain English, in any given week a team can discover and fix 0 bugs, or any other number of bugs proportionate to their throughput. By randomly selecting a number within this range, I would be maintaining a level of unpredictability that crucial for Monte Carlo simulators.

For further flexibility, I wanted to be able to configure the perception of risk on the code-bases too, as different code-bases carry different technical debt baggage. Once I identified the function to be used, the risk level was a small adjustment in a parameter that would sharpen or flatten the curve for higher or lower risk levels. Sharper curves would produce more dramatic ranges, reflecting higher risk of finding bugs.

Interruption Source #2: Scope Creeps

Scope creeps are as unpredictable as technical debt. The idea of scope creeps is that at some point in the project, new ideas and features start creeping in as important scope. Good product management might control the impact, but scope creeps remain another fact of life with regards to project forecasting.

Modelling scope creeps from historical data can be trickier as it is typically harder to distinguish original scope from scope creeps in work management systems. Even if rules like date brackets for ticket creation can provide a good indication, we all know that tickets are created all the time and so it is not a robust rule.

The good news is that we have already explored the way of mathematical functions. Now we know that if we can picture our assumption about scope creep behavior, then we can probably find a mathematical function that mimics it.

In my case I imagined scope creeps as a function of new tickets over time. Basically, the longer a project takes, the more impatient stakeholders get and therefore the higher the probability of new scope to be squeezed in as “required features" without removing anything from the original set of features. The end result will look something like “moving goal posts".

The approach I chose was similar to that of bug simulation, this time the function would relate project week to the new scope.

A Technical Challenge

As I tried to model the scope creep variable in my Google Sheets, all of a sudden my sheet started flickering like it’s Christmas time 🎄🎄🎄.

“Those [spreadsheet] lights…” 🎄

I soon realized that I had just implemented a cyclic dependency. My week counter was relying on the remaining backlog, that was relying on the throughput calculation, which included the scope creep variable, which relied on…the week counter!

The best way around this problem was to work out the scope creep variable independently of the rest of the formulae. Instead of relying on the project age, which was necessary to keep due to way the simulator was implemented, I decided to ask the user to input a cycle duration. Row after row, a number would be increasing until it reaches the cycle duration entered by the user, after which it would refresh. Ideally the input is as close to the simulation’s trend as possible, and so I also decided to give the user some guidance while entering this value.

Interruption Source #3: Finer Work Break Down

Finer work break down is another activity that can move a project’s “goal posts”, in that the original idea is understood to be better tackled if done so in smaller chunks. Such decisions are typically taken during implementation and are really hard to predict.

There is a range of certainty also on this variable. Well known scope is less susceptible to being broken down further than less known domains. This is because the developers would be more confident in their initial representation of work.

Another pattern of behavior would also be that finer break down is higher in the initial stages, and settles down as the project approaches its end.

Now that you have know some of my ideas in approaching simulations, let’s try a learning exercise:

How would you model finer break down in a simulator?

🚨Let me know in the comments on this post, or by any other means of communication. I will be looking forward to learning from your point of view.🚨

Conclusion —What we Learned and our Next Frontier

In this post we have built on our experience and knowledge of Monte Carlo simulators by introducing other variables in our spreadsheet simulator. We have discussed how we could simulate bugs, scope creeps and also finer work break down.

The resulting simulator is a more realistic one. We still have one big challenge. To this date, the simulator is restricted to using rows to simulate its burn downs. When testing it out on a 150 work item backlog, I had to use 1000 rows to generate roughly 50 simulations. 50 simulations, in the world of Monte Carlo, is too low to be reliable. We need to be able to get hundreds, if not thousands of outcomes to reliably use a simulator. With the current approach, this would bloat our spreadsheets to thousands of rows.

In the next article, therefore, I will look into ways how to improve this situation and be able to get more simulations without creating monster-spreadsheets.

Matthew Croker is a Team Process and Data Coach, specialized in the software development industry. Through his work he helps teams within companies focus on learning how to work best together by analyzing their data, simplifying the setup of processes and creating habits that boost their productivity.