Setting the grounds for evolving Kanban using metrics at Clarity AI
Proto-Kanban
When I joined Clarity AI in March 2019, the product development team was moving from Scrum to Kanban and was growing fast, with new team members joining every week.
As with most teams moving into Kanban, we were implementing what is known as proto-kanban system.
Basically, we had a JIRA Kanban board with several columns, two classes of service defined (expedite and normal), different issue types defined (epic, story, bug, task, subtask) and we were running Standup, Replenishment and Retrospective meetings daily, weekly and bi-weekly respectively. We had the commitment of releasing new functionalities into production every week.
This model already had several advantages for us over Scrum: we were visualising our work better and starting to change our mentality focusing on stop starting, start finishing. In addition, it provided us with more flexibility to react to changes in priorities, something that is crucial for a fast growing company like Clarity AI.
Following the Kanban principles start with what you do now and agree to pursue improvement through evolutionary change, we wanted a smooth transition into Kanban, step by step, making small changes.
Evolving Kanban: implement, measure and evaluate.
Taking into account the principles and practices defined by Kanban methodology, we want to evolve our Kanban implementation step by step, applying one small change at a time. By taking advantage of the different feedback loops we have, the idea is to identify possible improvements. The approach we follow is similar to the lean product development techniques.
We define an experiment, test the change during a period of time and, then, evaluate the impact.
We like to take decisions supported not only by our experience but also supported by data. That’s one of the reasons we like Kanban. We can measure our performance and flow, helping us moving in the right direction.
When we define an experiment, we establish what the goal of the experiment is and also how we are going to measure it.
Measuring Kanban
Metrics: what to measure
Possibilities regarding metrics are almost unlimited. There are, literally, hundreds of things we can measure.
The set of metrics to measure depends on the experiments we want to evaluate.
As a consequence, our metrics system might evolve over time, including new metrics as needed to measure new experiments while trying to keep complexity low. As we increase the number of metrics, understanding data becomes harder.
Therefore,
we defined an initial set of metrics that can support most of the experiments we are planning to run in the future as part of the Kanban transformation. This set of metrics is extracted by type of issue in JIRA: Story, bug, task, …
Cycle times: this metric might be considered as the cornerstone of metrics. It measures the time that a ticket spends moving from a column to another column. In our case, we defined two different cycle times, having our commitment point at in progress column. One will measure the time from commitment point until it’s merged into our pre-production environment. This allows us to be aware of the time needed to develop a functionality. The other cycle time will measure the time from commitment point to production. This one is similar to the previous one but it takes into account how the release process affects our delivery. With only those two cycle times we have a lot of information to understand our performance. We use the 85 percentile to compute our data and provide estimations.
Throughput: we measure the number of tickets as the number of story points we push into production by month and release.
Average time in column: we check how much time on average a ticket stays at every column. We find this metric very useful as it allows us to identify stages where time is wasted, like ready for / waiting for columns.
Blocked time: this one is pretty important for us. Our teams are autonomous and are integrated by different talent profiles from data science to designers or developers. This put us in a position of, ideally, solving any blocker internally. We measure how long tickets are blocked during our cycle times and how many tickets we block by month and release.
Releases: we measure how many release we do by month, how long they are, how many tickets and points have been delivered and how much time we spend on testing and releasing our code.
Common charts
Regarding metrics, there are some useful and common charts that the community is using. Having metrics and data is cool but we need to be able to read, digest and understand that data in an easy way. At Clarity, we are mainly using two of the charts that ActionableAgile tool provides.
Cycle time scatterplot chart: it represents cycle time for every issue type by date. With that, we can check what tickets are outliers (cycle time is higher than 85 percentile), throughput of tickets during a period, how many have been blocked… As I said before, metrics become a new feedback loop that warns you about problems.
Cumulative flow diagram (CFD): this one helps us understand the stability of our system as well as visualise in just one chart 3 of the most important metrics: cycle time, work in progress and throughput. In addition, we can check the input rate vs output rate of tickets here.
Configuring JIRA for extracting accurate data
If we are going to use metrics and data to take decisions and improve our flow, we need to be sure the data is as accurate as possible and that it really models how we work. In any case, we are always cautious when reading our data. With the amount of tickets we manage in our system, the uncertainty might be high.
In order to set a proper metrics system, we need to configure our JIRA project accordingly so we can categorise our work. With that in mind, adding labels representing the team who is in charge of the work; adding components representing the type of work needed(data, styling, frontend, backend); adding story points as a complexity estimation by using a custom field; and using fix version field to group our tickets by release.
We also have several policies defined. For example, a ticket can’t be moved into ready for testing column if Fix Version field hasn’t been filled or a ticket can’t be moved into merge request column if subtasks are still under development. Too many things to take into account for a human being. How can we not forget all of them and keep a healthy metrics system?
We implement our policies as part of JIRA’s workflow, allowing transitions only when policies are met.
Following Kanban recommended practices, we make our policies explicit. We consider this as a very important practice for us given we are a remote friendly company, with most of the team working remotely one or more days per week and some of them working 100% remotely. Therefore, over the past few months we have spent some time modifying our JIRA configuration so, right now, all of the policies or agreements we define as a team are integrated, if possible, as part of our JIRA workflow.
Applying all policies as restrictions and conditions at JIRA workflows and automating tasks have strongly helped us on having a healthier Kanban board, preventing errors on the visualisation of our work. No more tickets without fix version, no more tickets moved by mistake. Moreover, having a rigid workflow improves metrics accuracy. Last but not least, this also helps us integrate new members into the team as they don’t need to know all of our rules/policies. They will just learn them while working on a daily basis.
Clarity Agile Dashboard
As we use metrics to validate our experiments, we are interested in the evolution of our metrics over time, not in the value of the metrics itself. Let’s say we introduce an improvement in our release process that we expect to contribute to delivering functionalities into production faster. To validate the change we decide to check lead times. If our hypothesis is correct, lead times might be decreased. Then, our use case will be mainly comparing data between periods. Before and after the change. In this example, we don’t mind about how long lead times are (well, we mind but that’s not the point when validating an experiment), what we care about is that lead times are shorter after the change.
With the tools we had, comparing metrics over time was possible but difficult and time consuming. Also, checking the data by release was not possible.
With those requirements in mind,
we have developed our own Clarity Agile Dashboard application, which represents the evolution of all our metrics by month and by release.
The tool has two sections: an overview which displays a summary of the evolution of metrics taking into account all teams and detailed information for every issue type that can be filtered by team and size.
When do we use it? We check monthly data by team before doing our retrospective meetings. We check metrics in advance to identify possible issues in our flow. Then, we identify possible solutions and define an experiment.
We use our metrics data as a new source of feedback at retrospective meetings
The tool is highly configurable and is currently under development and it will be, hopefully, open sourced soon.
Next steps
Having all the tools in place we are now able to run experiments and measure if the change is affecting positively in our work or not. We can start evolving our system and boost our performance.
From now on, every change we introduce in our system will be validated by data, not only by our experience and/or feelings. In any case, we might treat data with caution and we need to really understand data. We can’t take into account data if amount of samples is not enough. Also, we need to be aware that improving some metrics might deteriorate other metrics. For example, it’s not worthy decreasing lead times if the quality of our product decreases.
So, the first experiment we will run is: change the way we do our Standup daily meeting from a what I did and what I’ll do approach to a Kanban approach. Will it improve our flow?A new entry in medium will be written with the results, stay tuned!
