Mitigating Metric Misuse: Preventing the Misuse of Metrics and Prioritizing Outcomes Over Outputs
The business needs feedback on technology investments. Teams need insights into flow efficiency and potential bottlenecks.
Part 3 of a continuing conversation regarding today’s delivery system metrics: Flow Metrics, DORA, and the traditional concerns regarding the Gamification of numbers.
Links to the previous posts:
Part 2: Developer Experience: The Power of Sentiment Metrics in Building a TeamX Culture.
What problem are we trying to solve?
Identifying the specific problem you are trying to solve with metrics is essential. Could you suggest other solutions apart from using these metrics? How can we determine where to invest and track progress if we don’t use them?
The problem we are trying to solve is the improved efficiency of software delivery and employee engagement. The focus is on continuous improvement of flow. Using metrics, we can illuminate insights into bottlenecks and obstacles that reduce the team’s ability to deliver software. Our goal is to continuously improve the flow of work, which ultimately leads to better outcomes. Improvements in outcomes reflect efficiency improvements.
Business interest in metrics (investing in technology, investing in work)
- Are we improving our business by investing in technology? Are we getting better?
- Return on investment, return on outcomes
- Delivering faster with high quality
Teams (delivering work, removing friction, feeling successful)
- Improve efficiency by reducing waste, shortening lead and cycle times, optimizing workflow, and promoting employee engagement.
- We do this by providing teams with data, insights, and optics into bottlenecks and areas of friction that generate conversations around why these bottlenecks exist and brainstorm experiments to resolve them.
Is there an elephant still in the room? What about the Gamification of metrics?
Concern for system metrics like Flow and DORA is still the team’s focus, as they try to gamify the numbers instead of focusing on the data and looking for patterns that highlight bottlenecks and friction, otherwise known as areas of improvement.
Stakeholders need system metrics, and using them effectively within the organization is essential. Some tools can be expensive. There is also a risk of gaming the system to achieve a desired metric, and these tools’ values decrease when teams focus solely on the numbers.
How can we avoid becoming hyper-focused on these metrics this time around? How can we encourage teams to use them? We should separate the business view from the team’s perspective. The team should focus on the insights and illuminated areas of improvement, not just the numbers.
Some leaders adopting these newer metrics and dashboards measuring flow and DORA still warn that Gamification wins, and teams fall back to focusing solely on improving the number.
Yet, numerous teams have succeeded by fostering a positive culture and adopting the right mindset. These teams analyze the patterns and identify the areas that pose obstacles. Doing so enhanced the flow, mitigated friction, and boosted engagement, activity, and overall satisfaction.
The key is leadership.
Bad managers or incompetent managers will diminish efforts.
If you still fear team gamification and misuse of metrics that defeat the value of modern ways to measure and motivate efficiency improvement, consider improving your leadership instead of blaming the tools or teams.
The increasing pressure on engineering leaders to be “more data-driven” has pros and cons depending on the managers leading the effort, even with today’s metrics and the “why,” bad managers can erode the value of these modern team data insights.
Depending on the competencies of the managers leading the effort, the push for engineering leadership to be more data-driven can have positive and negative effects, despite the availability of metrics and understanding of the “why.” In the case of bad managers, the value of these team insights can be quickly diminished.
Although metrics like Flow and DORA can offer valuable insights into team efficiency and process bottlenecks, it is crucial to remember their purpose. These metrics serve as tools for understanding and improving the system, not micromanaging, unfairly critiquing the team, or ranking performance across teams.
These are “team” metrics. Misusing these metrics to measure individual performance is an unfortunate managerial anti-pattern. As with comparing teams, managers focusing on individual performance can lead to a toxic culture and create an environment where team members might manipulate the metrics rather than focus on delivering value.
If your teams prioritize numbers instead of identifying improvement areas and working together to overcome challenges, consider examining the person guiding the team and reporting the team’s metrics.
Competent and influential managers:
Leadership needs to create a clear cultural imperative, acknowledging that, while sometimes it may be unavoidable, it is human nature to want to focus on the numbers. However, intentionally doing so will not be accepted. It is important to reinforce a culture of improvement and help teams understand that metrics are not the ultimate goal. Instead, metrics result from efforts to enhance different processes, such as removing bottlenecks, improving flow, automating processes, and enhancing practices. With the focus on improving rather than the numbers, each improvement will increase metrics over time.
- Foster psychological safety for teams to make all work and impediments visible.
- Don’t use metrics to compare or punish teams. Each team has a unique set of customers, complexity, and challenges.
- Use metrics in retrospectives to drive discussion and ideas on improvements.
- Celebrate experiments and improved trends.
The Benefits
Teams should be encouraged to view and use the metrics differently than how the business views them. Teams finally have data to advocate for investments in other work besides features.
There are ways in which teams can benefit once they have data to back up the evidence of their bottlenecks and show the business and stakeholders the value of investing in and addressing these bottlenecks. Teams can use this data to demonstrate the necessity for investing in technical debt and efficiency improvements rather than just investing in feature work. The benefits include:
- More data to act upon: Give your team more data and insights to talk about, and if required, act upon it before things start to fall off the rails.
- Exposing Bottlenecks: Flow Metrics and DORA Metrics can help teams identify bottlenecks in their development process. Bottlenecks include areas where work is consistently getting held up, causing delivery delays. By identifying these bottlenecks, teams can focus on improving these specific areas through automation or other solutions leading to overall improvements in efficiency and delivery time.
- Promoting Proactive Improvement: Using these metrics encourages a proactive approach to improvement, as teams can use the data to identify potential issues before they become significant problems. Early detection can lead to a more efficient and effective development process.
- Demonstrating Value Beyond Features: Often, stakeholders focus on feature delivery as the primary measure of a development team’s value. However, these metrics can help prove that a team’s value extends beyond delivering features. They can show how improvements in technical debt reduction, process efficiency, and team collaboration can also provide significant value.
- Facilitating Conversations with Stakeholders: These metrics provide teams with the data they need to have meaningful conversations with stakeholders about where investment is required. They allow teams to move beyond subjective arguments to data-driven discussions about the state of the development process and what is needed to improve it.
By adopting these newer system metrics, with the support from exemplary leadership, and a great culture, teams can avoid focusing solely on the metric numbers to please the business and shift instead towards an improved flow, higher team member engagement, and a more balanced and sustainable approach to software development.
The article was originally published on June 21, 2023.
