Kanban v. Scrum — By the Numbers

How we determined what worked better for us


As I’ve written before, we ended 2013 with some enviable development statistics. We could churn out new product capabilities in ~ 10 days from start to finish, and this gave our business a much needed boost of responsiveness. Thanks to our development decisions and product priority sequencing, we were delivering increasing value to the customer with the same work mass, & this was having significant top line impact on the growth of our business. Throughout 2013 and in January 2014, our 10 member development team followed Scrum as the development methodology.

How did Scrum fail us?

While Scrum produced great outcomes, it led to some consistent & undesirable artifacts

  • `We would under-deliver on our Sprint Commitment by 10-20%, which included unfinished work. At the same time, we would work on backlog items anywhere from 5-10%, and inevitably deliver some of it.
  • Most of our releases would cluster towards the end of the 2-week Sprint cycle causing some stress on our systems which exacerbated the stress on our team.
  • Most of our scoping decisions were made under time constraint of the Sprint deadline.
  • We were unable to understand risk to delivery in any meaningful manner. This meant that outcomes could not be changed while the events unfolded.
  • Product backlog was prioritized only every 2 weeks, though invariably there was some thrash as we deliberately traded commitments with backlog items.

The most important undesirable consequence was the toll this methodology took on our team that was proud and ambitious. Each sprint left us with a feeling that we were under-performing. No amount of long term data could overcome the feeling of not hitting Commitment.

It’s not about Commitment, it’s about commitment

After much contemplation & reading, we decided that we needed to change our development process. We wanted a process that reflected our commitment to deliver value to our customers, not to Story Points, which is a surrogate for value. We needed a development methodology that would deliver the following

  • Prioritization in real time to constantly drive engagement & revenue, not in 2 week cycles
  • Scoping decisions made under collective judgment, not time pressure
  • High quality code delivered systematically
  • Responding to risk in a timely manner such that outcomes could be changed

We decided to give Kanban a try starting February. We created some swimlanes, some prescriptive limits max tickets allowed in each lane, and we were off to the races.

Evaluating Kanban

After a month of Kanban, all of us were feeling good about our development process, but was it working for us? I decided to inspect our development data.

Before cracking open Excel, I stated the following null hypotheses which if borne out by data would prove that Kanban was no better than Scrum, and in fact worse in some ways

  • Development cycle times remained as variable as they were during Scrum
  • We were delivering fewer story points under Kanban
  • Coding was no more parallelized as it was during Scrum

The resulting data is clear. In every category that matters, there was meaningful improvement.

Kanban vs. Scrum — by the numbers

In fact, the very shape of our development data shifted, evidenced best by these two charts which compared elapsed days or cycle time (a measure of our responsiveness) and parallelization (a measure of our teamwork and code quality), for a given work mass (denoted by size of the circle)

Cycle time vs. work parallelization under Scrum

Under Scrum, cycle time for higher mass items was higher than lower mass items. We were unable to parallelize effectively to reduce cycle time and increase our responsiveness.

Cycle time vs. work parallelization under Kanban

Under Kanban, greater number of tickets were parallelized to a greater extent while cycle time was contained systematically. You can see this by studying the spread of work across developers, while cycle time is contained.

Of course, there were some other important wins, which were a natural consequence of adopting Kanban

  • Work was prioritized in in real time., reflecting real business needs. This led to some great decisions. To cite an example, our product engagement doubled in 1 month, thanks to a new capability we had delivered. Our team was able to focus attention on making that product more reliable to support an increasing number of users — exactly what the business needed.
  • Due to prescriptive limits on work in flight, we were able to limit our work in flight. This resulted in greater focus, and greater involvement of our product and development leadership in scoping decisions
  • Our ability to estimate risk became objectively tied to cycle time, and we were able to respond by devoting extra brains to the problem at hand
  • Most importantly, the team was a significantly less stressed as everything simply flowed — releases, priorities, and risk to deliver was everyone’s problem.

So which is better — Scrum or Kanban

It is clear that for us, Kanban is a better development methodology — closer to a state of Zen compared to the start — stop world of Sprint. It is working, for us so far because of some key reasons.

Most important reason is our team cohesion, and a true sense of commitment our development team has for the business. In return, the business trusts and empowers the product team. This means when I tell our sales team or customer success team or our senior leadership team that our priorities can change any day, they react with celebration — not mistrust.