Nowadays, many IT companies use Jira for managing work items. It’s very easy to install this tool, create a new project, create some board, add work items (tasks, user stories, bugs, whatever we want), tune the configuration and observe how card flow from “To do” column to “Done”.
Yay, is it really so simple? Urmmm… not really. There are many possible pitfalls and traps when you use Jira. There are also some serious limitations of this tool. But today we will not talk about the pros and cons of Jira. Today, we will try to use Jira as a data source for metrics during creating software product.
Why and what to measure?
Let’s start with “why”: Why do we want to use metrics in software development?
There are several possible answers. We want to use metrics to take decision basing on data, not on opinions. To find areas for improvements. To see the trends. To avoid local optimizations. To see real picture of the situation. It sound’s reasonable, doesn’t it?
But what should we measure? The answer depends on your needs. Let’s start with basic Kanban metrics. We want to see a cycle time graph, our throughput and cumulative flow diagram. Let’s generate the reports and let’s see if the data is useful.
Why to use Jira as source of the metrics?
Jira does not excel in creating reports — but this tool has some reporting features. It’s quite easy to generate some reports. Most of them are available as gadgets that you can add to dashboards.
So you created a dashboard, added the report and got some results. And now you need to ask the question: is the data in my report correct?
The best way I know is checking data together with people who work with this tool. They have the knowledge on how they use the tool and they can check if your way of thinking is correct. You should build shared understanding how the metrics are created.
It can look like this:
- Hello team. I made some Lead Time report. It shows that the average Lead Time for your stories is 40 days.
- Hm. Looks quite long. How did you calculate it?
- Here’s list of stories, here are creation dates, here are resolution dates. We calculate the duration between these two dates and then we calculate average.
- Let me look at this… OK, I see some stories that were closed as duplicates. There was no work at all, we just closed them. You shouldn’t count them.
- I see. Do we have any way to recognize duplicate stories?
- Yes, we set “Resolution” field to “Duplicate”.
- OK, so I will remove them from my report.
Every time when we present data — we need to be able to show the method of calculation and source data.
We can also make one step more. We can ask the team about the best way of creating the metrics: “I would like to know how many days it takes from creating the task to closing it. What is the best way to gather this information?”
Case study — let’s start with measuring throughput
Let me present some metrics created in real IT project. When we started gathering metrics in the project, we decided to start with measuring the throughput. Throughput in Kanban refers to the amount of work delivered over a certain period.
At the beginning, we faced the first question: what should be the unit of work to measure? In this project we used two-levels backlog: we had Epics that were decomposed to Stories and Bugfixes. In the first attempt we decided to measure the number of delivered Stories and Bugfixes.
The first idea was measuring the number of Stories in “Closed” status. But wait… This idea was completely wrong! It occurred that Story got “Closed” status in several different cases: when it was delivered to Production, when it was discovered as a duplicate or when somebody decided “it’s not needed”. And there was no differentiator between “story closed as delivered” and “story closed as not needed”. Dead end.
We decided to use another approach. All changes in the product were deployed in releases. We used “Fix Versions” field for attaching Stories and Bug to correct release. When we merged dates of releases with number of changes, we got the following picture:
You can notice almost no changes in November and higher bar in December. It’s because all releases in November were delayed — and finally all changes from November were deployed to production environment in December.
There are some interesting features of this diagram:
- We count only the changes delivered to production — there is no bonus for “completed but not deployed to production” tasks.
- We can see if we deliver more or less changes to production over time.
- Unfortunately, we don’t measure the size of the stories neither their value. We count every story as 1 point on the chart. So big stories and small stories make the same increase of the bar.
What about Lead Time?
Throughput was easy. Now, it’s time for something more challenging — Lead Time. Lead Time is the period between a new task’s appearance in your workflow and its final departure from the system.
We decided to measure Lead Time only for tasks delivered to production. Jira has fields for two dates: Creation Date (this field is filled automatically during task creation) and Resolution Date (filled during filling “Resolution” date). So we listed our Stories in Jira, exported them to Excel and drew the next chart:
Each dot represents one Story deployed to Production system. X-Axis is the date of closing the Story (date of deployment to Production system). Y-Axis is the age of the Story in days.
First thing you can notice are vertical bars. It’s because we deployed most of our Stories in releases.
Basing on these data we calculated the average time of delivering a Story to Production system. It was 67 calendar days. 80% of stories were delivered within 93 days. So if you add some Story to the backlog — you have 80% probability that it will be delivered within 3 months.
But the most important information for me is the trend line represented as red dotted line. It’s horizontal — so it means that average age of deployed stories is more or less the same. It’s good news — it means that Stories don’t pile up in our backlogs.
Now let’s generate the same chart for Bugfixes:
We have visible bars again, but there are also Bugfixes deployed outside of the releases. Values on Y-Axis is much smaller — and it’s good, because it means that the average age of our bug is shorter. But there’s one alarming thing in this chart — the trendline. It goes up, so it means that our Bugfixes spend more and more time waiting for the solution. It definitely needs our attention…
Cumulative Flow Diagrams
We also generated Cumulative Flow Diagrams (CFD) for all Backlog Items. Good news: you can create them directly in Jira. You need to create Kanban board with a proper set of issues and CFD is available in “Reports” section. Bad news: Jira cannot draw this diagram if we have too many items. Our report stopped working at the level of 7000 items…
While preparing these diagrams we mapped all the statues into three columns: “To do” (all statuses before a team takes an item into sprint), “In progress” (all statuses used when item is in sprint) and “Done” (item is deployed to production).
Let’s take a look at CFD diagram for Bugs:
Oh, the number of active Bugs (orange and blue area) is growing. To make the problem more visible, we also created the custom report that displays number of active bugs. Here it is:
It’s clearly visible that our technical debt piles up.
In some cases, exporting data to Excel is not sufficient. Some organization used quite complex Jira setup. Here is one of the examples:
- There were Jira projects with Features (project was available for end customers)
- Features were connected with Epics via issue links
- Epics were decomposed to Tasks and other Backlog Items
- Tasks were assigned to team members
- There were time logs for every Tasks — so there we knew how much time was spent on each and every tasks
Standard Jira reports don’t cover so many connections. So we wrote our custom scripts that get data through Jira API and to crawl through all connections.
Using the scripts we calculated that average Lead Time for Features. It was about one year. So if customer creates some Feature, he waits — on average — 12 months to get this Feature in new system release.
We also calculated that average time spent on making all tasks assigned to a Feature is 160 hours. It’s more less one month of work for one person.
And this was a surprising moment — to have something done we need one month of work, but the customer needs to wait for this value for another 11 months… Isn’t that a clear area for improvements?
Charts above show that you can take get data directly from Jira and create different metrics for your project. Sometimes it’s not easy — you need to verify the method with the team, sometimes you also need to take care of data quality.
But in the end, you have the impartial picture of your project.
You can take this data and present the data to teams and managers, take improvement actions and check how these actions impact the metrics.
Why not use this opportunity?