An Introduction and Step-by-Step Guide to Monte Carlo Simulations
Since I started using Monte Carlo Simulations for forecasting instead of using estimations with our teams, I’ve gotten several questions about how they work and more importantly what data is needed to run them.
In this post, I’ll explain to you what a Monte Carlo Simulation is, why this might be interesting for you, and will walk you through the different steps of how it works.
What is a Monte Carlo Simulation?
Wikipedia describes the Monte Carlo Method as follows
Monte Carlo methods, or Monte Carlo experiments, are a broad class of computational algorithms that rely on repeated random sampling to obtain numerical results. The underlying concept is to use randomness to solve problems that might be deterministic in principle — Wikipedia
To forecast, we try to “simulate” the past and apply it to the future. We run many of those simulations and see what is “most likely” to happen. It’s a “probabilistic forecast”, meaning we don’t get a single result, but many options that are associated with a probability.
How Many & When
When applied to forecasting in software development, we can use the Monte Carlo Simulation to answer two questions:
- How many items do we manage to close till a specific target date?
- When will a specific amount of items be done?
Having answers to those questions is very powerful when we’re planning, and Monte Carlo Simulations allow us to answer this easily without estimations in just a few seconds. More information can be found in Daniel Vacanti’s books “Actionable Agile Metrics for Predictability” and “When Will It Be Done?”.
Monte Carlo Algorithm
The goal of this post is to make it more clear on how a Monte Carlo Simulation works. I’ll first explain the algorithm on a high level and then go more into the details. Some terms, like Throughput, might not be clear, please bear along with me as I’ll elaborate more on them during the post.
A simplified version of the algorithm to make forecasts with Monte Carlo Simulations looks like this:
- Create a run chart of your Throughput
- Randomly select values on this chart and sum up the Throughput. This is the simulated Throughput for a single day. Do this either for the number of days you want to simulate (“How Many”) or till you have 0 items remaining (“When”).
- Write down the results (either the sum of the Throughput or the amount of days).
- Execute steps two and three many times (like 10'000+ times)
- Plot the results on a histogram. Identify the “risk profiles” that you are interested in, for example, what you got in 85% of the simulations.
Before we start with step 1, the Throughput, let me elaborate on our sample data.
Sample Data
In my opinion, it will help you follow along a bit more actively. To make it a bit more engaging, I’ve created a spreadsheet that you can use to follow along. You can find a copy of it on Google Docs or on OneDrive if you prefer MS Excel. Ideally, you have it open on the side while reading. I‘ve marked the parts relevant to the spreadsheet with this icon: 📝
In our example, we’ll be using the Throughput of a team from December 2023 and we’ll forecast:
- How many items will be done in the next 30 days?
- When will 28 items be done?
Throughput
The only thing we need to run a Monte Carlo Simulation is to have data on the team’s Throughput.
Throughput: The number of work items finished per unit of time. Note the measurement of throughput is the exact count of work items. — The Kanban Guide
The unit of time can be anything you want it to be: days, weeks, Sprints. For the sake of this post, we’ll be working with days.
Note: If you would use another unit of time, say weeks, the Monte Carlo Simulation would also work with this unit. We would forecast how many items we manage in a certain amount of weeks (instead of days) and how many weeks it takes to complete several items.
To calculate the Throughput, we simply need to know on which date your items were closed.
If you are using a digital tool, the chance is very high that you already have that available. If you are using a physical board, just note down the date when an item was closed.
What about the Other Flow Metrics?
You might be thinking, wait a minute, what about the other measures of flow? Are WIP, Cycle Time, and Work Item Age completely irrelevant for Monte Carlo Simulations?
The short answer is yes. You don’t need them to run a Monte Carlo Simulation.
The longer explanation is that, while you don’t need them for the Monte Carlo Simulation, they still are an indicator of how “good” your forecast will be. Controlling and taking action on those measures, helps us get more predictable Throughput. And the more predictable our Throughput is, the “better” we can forecast.
Creating a Throughput Run Chart
Our Monte Carlo Simulation needs the Throughput ordered by days.
📝In our spreadsheet, you can check out the “Throughput” sheet. In column A, we see the dates when items were closed.
You can see that some dates appear multiple times, this means several items were closed on that day.
We now create a run chart with this data. As mentioned above, we use the data from all of December, so we create a table that shows how many items were closed each day in December:
We can also show this in a chart:
This will be the base for our simulations. You can see there were days where the team managed to close 4 items, while on other days nothing was closed.
What about Holidays and Weekends?
The attentive reader has probably noticed: We include weekends (like the 2nd and 3rd of December) and the whole holiday period around the 25th of December. We use the Throughput per unit of time, independent of holidays, weekends, and people being sick or on vacation.
When using Monte Carlo Simulations, we can ignore such things that often make “capacity planning” tricky.
Simulate a Future that looks like the Past
What the simulation helps us to do, is that we use the “past” in the form of our historical Throughput, to simulate the future. People will have weekends in the future (I hope), and they will also be occasionally sick. This is a strength of Monte Carlo Simulations over traditional capacity planning, as such events are factored in.
Now you should put some thought into which Throughput period you select as a base for your simulation. The month of December was randomly chosen, and you might not want to use two weeks over Christmas when everyone was off.
As a rule of thumb: use a “history” that you believe will look similar to the future you’re trying to forecast. If you’re team just formed a month ago, don’t try to use a 3-month history.
Also, if you wonder if you could use more special sampling models (for example use Mondays to simulate Mondays), I recommend this in-depth post by Prateek Singh:
Now that we have our history in the form of an ordered Throughput run chart, we’re ready to run our Monte Carlo Simulation!
Monte Carlo — How Many
If we do a “How Many” simulation, we have a specific “target date”. We want to forecast how many items we can finish in this period.
📝 If you are following along in our spreadsheet, move to the “How Many” sheet. You can read along and then run a couple of runs of the simulation yourself — follow the instructions in the sheet.
Let’s assume we want to forecast what we manage in the next 30 days.
For a single run of our Monte Carlo Forecast, we will simulate those 30 days. For this, we need 30 random values (one for each day) between 1 and 31 (because our history from December has 31 days).
The random number we get is used to “look up” the Throughput value from this day in our history. For example, if we get a 20, we’ll use the Throughput value from day 20 in our history (which would be 1 in our case):
If you do this for all 30 days, you’ll end up with a number. This is our simulated throughput through 30 days.
📝 I encourage you to try it out, play with different values and check the results.
We should now take note of the result, where we count how many simulations had the specific result. For my example, we would now increase the count for 27 by 1.
As we want probability, we should run this more times. Often about 10'000 times is used to get meaningful results. Feel free to run some runs on your own, for example, 10, and create the result tables for yourself.
After you run this several times, you’ll end up with a table that contains the results:
Tables are not ideal to visualize this, so you can also plot this in a chart:
You can see that the very high and very low numbers happened less frequently. This is because you are very “unlucky” if you hit often days that have 0 or a very high Throughput (in our example 4).
To make this useful for our forecast, we can see how many runs managed to close “x items or more”. In this example, we have 100'000 runs. This means, 1% of the runs are at 1'000. So we sum up the runs, ordered by most items closed. When we get to 1'000, we know that “There is a 1% chance to close x items or more”. In our example below, the items closed at the 1% mark is 47. So in other words, there is a 1% chance that we manage to close 47 items or more in the next 30 days.
While 1% might be a bit too risky for most teams, you can apply the same strategy to look for more meaningful “risk profiles”, like 50%, 70%, 85%, or 95%.
📝 If you are following along in the spreadsheet, identify those risks and how many items we manage to close.
And that’s it, that’s how you can run a “how many” forecast using Monte Carlo Simulations.
Monte Carlo — When
While we had a “target date” or a specific amount of days when doing the “How Many” simulation, we have a different situation for the “When” simulation. This simulation answers the question: How long will it take to close x items?
📝 If you are following along in our spreadsheet, move to the “When” sheet. You can read along and then run a couple of runs of the simulation yourself — follow the instructions in the sheet.
Let’s assume we want to forecast when we manage to close 28 items.
Before, for a single run of our Monte Carlo Forecast, we simulated 30 days. As we don’t know the days, our simulation will now run until we close 28 items or more. We again need a random value between 1 and 31 (as our history is the same as before). We look up the Throughput value from this day and subtract the number from the remaining items.
We do this until we are at 0 (or below). That “day” is the result of this simulation:
In the above screenshot, you can see that it took us 16 days to “close” 28 items.
We should again take note of the result. Before we had a specific number of items in a fixed amount of days. Now we want to count how often each day was the one where we got to 0. For my example, we would now increase the count for 16 by 1.
📝 If you follow along, feel free to play around with the numbers again and see the different results you’re getting.
Again, we should run this more often. Then, we’ll create a table with the results just as with the “How Many” simulation:
We can now apply the same logic as before, and identify the different risk profiles. As an example, there is a 52% chance to close 28 items in 30 days or less:
Congratulations, you now ran both a How Many and a When forecast using Monte Carlo Simulation.
How Stable Throughput Affects Monte Carlo Simulations
In the beginning, I mentioned that we only need Throughput for the Monte Carlo Simulation. However, the other flow metrics might be an indicator of how well the Monte Carlo Simulation will work. As you now should have a solid understanding of how it works, this might make more sense.
If we were perfectly predictable, we would have a “stable” Throughput. This means, that every single day (or whatever unit of time you are using), we would have the same value. If that were the case, every single run of the Monte Carlo Simulation would result in the same result.
📝 The original Throughput sums up to 32 items for December. You can go back to the Throughput sheet, and change all entries in Throughput/Day to 1. That adds up to almost the same total, but what happens with the simulations if you run a couple of them? What do you observe?
On the other hand, if you have a long period where there is nothing closed, and then suddenly a large amount, you would have more “extremes” in your simulations. As most of the time you will select days with 0 (as the chance is higher), but for some runs you might get many instances where you pick the “high Throughput days”. This will increase the “spread of the results”.
📝 Again, what if we change how the total Throughput of 32 is distributed over the whole month? Change the Throughput/Day so that the team would have had many days with zero, while there are just a few days where there was a lot of stuff completed.
This means, a focus on flow, making sure your items don’t age too long, your WIP is not too high and you are working in small batches will affect the Monte Carlo Simulation.
There are also more advanced methods to analyze how predictable your Throughput is: The usage of Process Behaviour Charts with so-called XmR charts. However, that would go way beyond the scope of this post. You can find more information about this in Daniel Vacanti’s new book “Actionable Agile Metrics Volume II”:
Tools for Monte Carlo Simulations
Before we wrap up, I wanted to share some tooling for running Monte Carlo Simulations. It’s nice to run it manually, but not very practical. In the end, you want to have it automated to run it frequently, or ideally even automated, so that you can move towards continuous forecasting.
Write your Own
As you went through the algorithm now, I hope that you saw that it’s not complex. You should have a computer to run many simulations, but if you can write code, you should be able to write a script that does the calculation for you.
The advantage of this approach is that you can customize it fully to your needs. It allows you to integrate with your systems, and your pipelines, and take all the special cases you have into account.
MonteCarloCSV
If writing your own is not an option, you might look at the GitHub repository MonteCarloCSV.
It’s a Python script that works with any CSV file as input. Just export the “closed dates” of your items into a CSV, feed the script with it, and your simulations can be run.
This might be a good option if you don’t want or don’t know how to build your tool. You can also use it as an easy way to try Monte Carlo Simulations out, without investing too much time. If you like it, you can even start building your tool based on the existing code.
Last but not least, the code is open source and you can run it on your system. Your data does not need to leave any of your systems. If you or your company is concerned about where their data ends up, this might be the tool for you.
Actionable Agile
Actionable Agile (AA) is “the tool” for all things flow. It’s very powerful and integrates with Azure DevOps and Jira. It’s an SaaS offering, and you can find more information on their website:
FlowViz
Last but not least, if you are using Azure DevOps or Jira and rely on Power BI to visualize your data, FlowViz might be a great alternative for you. It’s free to use and very powerful. It can run Monte Carlo Simulations as well as display other useful information for analyzing the flow of your teams:
Conclusion
In this post, you went through the steps that are needed to create a forecast using a Monte Carlo Simulation. You created both a “When” as well as a “How Many” forecast. Furthermore you saw that the only thing you need to do this is to have the data on the Throughput of your team. And that’s something you most likely have already!
I hope the post helped you to demystify Monte Carlo forecasts and gave you a good insight into the method itself.
As you now know that you don’t need too much to try it, I’d encourage you to give it a try yourself. If you do, let me know how it went!
Additional Resources
Following are some additional resources that I recommend to check out to learn more:
