Measuring the Success of an Agile Implementation?

Recently, a colleague posted the following question in Slack: “What types of metrics are used to indicate the success of an Agile implementation?”
It got me thinking: What does measuring the success of an Agile implementation achieve? The agile software manifesto describes a set of values and principles. To my estimation, those are not goals, but means to achieve goals. Would the more appropriate question be, “how well did agile principles help the organization affect the changes it was looking for?” How well could you answer that question in isolation, sans context about the goals of that specific organization? Doesn’t that potentially devolve into a navel gazing exercise?
I wonder how often organizations implement agile practices (hopefully informed by its values & principles) without understanding what specific and measurable changes they are attempting to drive. Doesn’t that give the impression of decisions being made in the reverse order? Shouldn’t an organization decide it desires to improve in say, becoming more responsive to the accelerating pace of change, reducing waste by applying the art of simplicity, increasing innovation, etc., before deciding to apply agile principles? If the organization’s goals are to increase predictability, improve process efficiency, ensuring compliance and limiting decision making to only the right level of the org chart, is applying agile principles even the right fit? If a real change (not just chasing the latest fad) in company philosophy does not proceed the transition to agility, doesn’t it run the risk of creating expectation mismatches throughout?
How often are organizations transition to agile methods without effectively communicating the corresponding changes in goals and values. If you had previously succeeded at your job by being laser focused on the “on time and on budget” principles*, couldn’t you see these “new” agile practices as a potential hindrance? Do those managers understand that speed to market, ability to pivot quickly, and a focus on ROI vs. conforming-to-plan are now the new targets? Is there are a clear understanding that the measures of success have changed? Without clear communication about changes in direction and what impacts they are attempting to affect, these agile methods will be likely be judged against previous goals.
If your organization still funds “projects” instead of investing in solving problems, your organization might be sending mixed messages about agility. If failing on an experiment is punished but failing to experiment is not, your organization might be sending mixed messages about agility. If stopping a project/product based on new information learned while building it is seen as a disastrous career move, your organization might be sending mixed messages about agility. And if your organization is sending mixed messages about agility it’s likely that there isn’t a broad and shared understanding of why and for what purpose the principles of agility are being adopted.
As I pondered this question over the last week, I’ve come to appreciate some value in measuring “the success of an agile implementation.” But these are indirect measures**. Indirect metrics can provide localized information in terms of immediate direction. This can be valuable as it can be harder to know what to do, specifically, when big picture metrics are headed south. That being said, I’ve witnessed plenty of bad behavior when indirect measures are treated as the end goal, either because the true goals are not measured or because they are unknown.
Arguing over whether something is a bug or a missed requirement is pointless to your customer, but I’ve seen having indirect metrics like bug counts cause feuds between groups that are pitted against each other due to the mismatch in these targets. Your customers want software that helps them achieve their goals. Whether something is a bug or missed requirement results in the same outcome for them; the software is less capable of helping them achieve their goals. Arguing whether your company is doing Agile Thing A better or worse than Agile Thing B is similarly pointless to your customers, as they primarily care about how/if software help them achieve their goals, whatever their specific goals are. And isn’t understanding the big picture goals of your customers the biggest part of developing fantastic software in the first place?
Notes:
* I’m not saying that on time and on budget are not important in agile environments, only that when conflicts arise (and they always do) between those goals vs. responsiveness/innovation/simplicity, that values of agility are given a greater weight.
** That same slack post included a discussion about indirect metrics. I was tempted to included my questions/musings about those topics in this article, but I didn’t want to confuse the issue. Maybe in my next one.
If the person who posted the original question is reading this, please don’t this article as an attack for asking the question. I believe it is an important question to ask, as the question sparks even more important discussion on what answering that question really means.
