JIRA’s Control Chart is unfit for Kanban
Let’s assume you do Kanban and you are using JIRA. Most probably you found that the charts JIRA offers you are not very satisfying. You just get a Cumulative Flow Diagram (CFD) and a Control Chart. Also the control chart is quite misleading in some aspects, which will be pointed out in this article.
What’s the problem with the JIRA Control Chart?
First of all the use of the standard deviation instead of percentiles is common but not wise. Lead time data is hardly ever normally nor symmetrically distributed. That’s why the standard deviation corridor is not very meaningful. You should use percentiles.
Second of all the control chart shows elapsed time per column, not lead times. The description at the top of the control chart page states that it shows “cycle times” (let’s for the sake of simplicity call cycle time and lead time the same thing in this article). But that is only true if you look at the complete workflow — not if you select a part of the workflow.
That basically makes the summary in the upper left corner of the chart which looks something like this…
1w 1d 22h average 2d median < 1m Min Time 44w 5d 2h Max Time 107 Issues
… is only useful if you look at the complete workflow, i.e. all columns. That is not always what you want. Let’s have a look at an example.
Lead times for a part of the end2end workflow
Let’s say the workflow for your issues looks like this:
BACKLOG > ANALYSIS > CONCEPT > IMPLEMENTATION > REVIEW > DEPLOY > DONE
Let’s also say that REVIEW is out of your hand, it’s done outside of the team. Of course you want to know what flow within your team looks like. So you want to look at lead times only from ANALYSIS to IMPLEMENTATION.
Below the control chart you find a drop down menu “Refine report” and intuitively you only check the columns accordingly: ANALYSIS, CONCEPT and IMPLEMENTATION.
What does “Refine report” for columns really mean?
What you get is, however, most probably not what you think. It is important to distinguish between “elapsed time” in a range of columns and “lead time” for the workflow represented by that range of columns.
Those two are in fact only the same if the rightmost column in that range is Done (which is is not in our case) or if all issues are in a column downstream of the rightmost column in that range (which you don’t see in the control chart).
Example: If you filter the Control Chart for columns 1, 2, and 3 (ANALYSIS, CONCEPT and IMPLEMENTATION) it will show also issues that are in status IMPLEMENTATION at the moment plus issues that are passed this status.
That might make sense from a Control Chart point of view (it says elapsed time for the y-axes), but to summarize lead times for statuses up to IMPLEMENTATION I only want to consider issues that are passed status IMPLEMENTATION. Because only then implementation is done. So you have to be aware that the numbers above the control chart, the data summary mentioned above, do not show you lead times for ANALYSIS, CONCEPT and IMPLEMENTATION.
That’s why JIRA’s Control Chart is misleading concerning Flow metrics
If you look at the end to end workflow, as I said that is not a problem, because the last column is usually done/closed. It is, however, tricky for teams using the Control Chart covering only a section of the workflow for metrics like in our example.
I know teams looking at a certain section of the workflow, so they only pick a subset of columns in the control chart like in our example above. In such a case mean/median as shown in the control chart are not a very good indicator for actual lead times. And the error is unsystematic because it depends on the number of issues in the last column which still are in that very column and not downstream of it.
Another problem with the control chart is that JIRA only looks at elapsed time per column, which it then just adds up to compute “cycle time” for the issue. For issues going back and forth in the workflow this is not what you would expect as “cycle time”. At least I wouldn’t.
I would expect the clock to start ticking the very first time the issue goes into the first column we look at and not the last time it does that.
Example: An issue cycles between ACCEPTANCE TEST and IMPLEMENTATION for 7 days. After it is finally done we look at the “cycle time” from ACCEPTANCE TEST to DONE. In that case the entire time the issue spent in IMPLEMENTATION after going through ACCEPTANCE TEST for the first time will not be part of that cycle time. It depends on policies whether this is what you want. But I think it’s misleading in most circumstances.
And yet another problem is that the control chart does not give you the option to exclude issues with obvious special case outlier characteristics like issues under 1 minute lead time. Those will further bias your mean/median metric.
And finally: Mean/median of a lead time distribution are not a very good indicators for the Kanban system characteristics anyway.
A real data test makes you shiver
To test this I looked at a data set from a real Kanban team. I took one month of data. JIRA showed 97 issues in the control chart whereas my report showed 68, because it had excluded 16 issues with a “cycle time” < 5 minutes and another 13 because they were still (or again) in a state within the target set of columns. JIRA showed 1.5 days as median, my report showed 5 days. Which is a rather dramatic difference.
Both, issues under 5 minutes and issues still in a column of our target workflow reduce mean/median of lead times. Meaning: The JIRA control chart does not show you lead times. Not even close.
Bottom line: Use JIRA’s control chart with extreme care for Kanban metrics
Don’t be mistaken, the control chart is still your only friend if you want to produce metrics for Kanban system in JIRA without additional plugin or API access. And as a matter of fact there are very good plugins that give you Kanban metrics with JIRA, namely the ActionableAgile Metrics plugin. But in many cases it is not that easy to get a plugin installed because of budget, compliance, security policies and whatnot.
The JIRA control charts let’s you access elapsed time data for every column for every issue, which gives you the opportunity to compute real lead times. We will look at how that works in a future article.