Why Traditional Productivity Metrics Fail to Show True Development Progress

Why we built an engineering management product that will help your run your teams, not just grade individual contributors.‍

Table of Contents

1. Harmful Cultures: Lessons Learned from Individual Metrics
2. The Promise of Team-Centric Productivity
3. A Vision for a Generative Culture
4. The Impact of Positive Teams
5. A Frontline View: Introducing Middle Metrics
6. Empathetic Managers & Mission Driven Team Members
7. Conclusion‍

TLDR;

Traditional metrics don’t provide an intuitive and fair way to assess your teams, push work forward, or gain insight and visibility into what’s going on during sprints (or work cycles, if you’re using Kanban). Measuring modern engineering teams and their productivity requires a more intentional approach including a whole new set of metrics, principles, and tooling. The cornerstone of this approach is, what we call, middle metrics. Middle metrics help engineering managers see what’s going on in the day-to-day, and enable them to improve long-term progress by getting ahead of immediate issues. Staat centralizes disparate data found on separate project management and development tools and visualizes these middle metrics and makes them accessible and actionable.‍

Harmful Cultures: Lessons Learned from Individual Metrics

Have you ever tried to introduce a potential analytics tool that you were thinking of using with the individual contributors on your team? If your experience is anything like ours, you will have found that they shut you down even before you can even finish your sentence.

Individual contributors are skeptical of analytics (and rightly so) because analytics have regularly been weaponized to inaccurately hold them accountable for errors and failures that are outside their control.

Engineering teams and individual contributors have historically been represented by metrics that, at best, suggest activity and at worst, single our individual contributors out without full context — both failing to capture the full picture of productivity.

Metrics like lines committed, time spent on code, or even velocity only capture a narrow picture of a team’s productivity. We’d go as far as to say that they are vanity metrics encouraging inadequate management and an incomplete view of development teams’ progress. The resulting misinformed and inaccurate characterizations of individual contributors potentially lead to the promotion and maintenance of harmful cultures. In fact, we believe that these harmful cultures are what primarily stall progress and hinder productivity rather than any one individual error.‍

Forsgren, Humble, and Kim substantiate this perspective in their industry-leading book, Accelerate.

In it, they outline three kinds of organizational cultures:

1. Pathological (power-oriented) organizations are characterized by large amounts of fear and threat. People often hoard information or withhold it for political reasons, even distorting it to make themselves look better.
2. Bureaucratic (rule-oriented) organizations protect departments. Those in the department want to maintain their “turf”, insisting on their own rules, and generally doing things by the book — their book.
3. Generative (performance-oriented) organizations focus on the mission. How do we accomplish our goal? Everything is subordinate to good performance, to doing what we are supposed to do.‍

Pathological organizational cultures are most likely to employ these kinds of individual-centric, harmful productivity measurements. They go as far as to say:

Pathological organizations look for a “throat to choke”: Investigations aim to find the person or persons “responsible” for the problem, and then punish or blame them.‍

This approach is ineffective and harmful because progress and failure are never in the hands of a single contributor or individual developer.

They conclude that:

In complex adaptive systems, accidents are almost never the fault of a single person who saw clearly what was going to happen and then ran toward it or failed to act to prevent it. Rather, accidents typically emerge from a complex interplay of contributing factors. Failure in complex systems is, like other types of behavior in such systems, emergent (Perrow 2011). Thus, accident investigations that stop at “human error” are not just bad but dangerous. Human error should, instead, be the start of the investigation.‍

Both progress and failure are cumulative, complex, and contextual. Although it is possible to find the source of a single error, why that error was created, missed, or overlooked requires a holistic investigation at a system-level view to truly understand the technical and cultural mechanisms that enabled those errors to pass by unnoticed. Identifying the individual or individuals at fault without also diagnosing these fundamental mechanisms is akin to treating a symptom of a cut while leaving the wound open and untreated.‍

That being said, we are not, however, suggesting that individual contributors are blameless or that there shouldn’t be accountability. Instead, we believe greater and more effective individual accountability is a product of generative team cultures that hold space for open communication, trial & error, and quality. An alternate paradigm.‍

The Promise of Team-Centric Productivity

Team-Centric Productivity places teams at the center, reimagines productivity, and fosters healthier, more positive, and therefore more effective teams. Team-centric productivity is not a novel idea. In fact, much of this article probably sounds intuitive to you. The paradox is how rarely team-centric productivity is being measured in modern organizations. We believe that it’s not so much a matter of novel methodology or data-driven principles, but rather intentional and consistent behavioral change. Change that starts with leadership.

So how should leaders approach Team-Centric Productivity and engage their teams to begin the work of behavioral change? We recommend starting with the principles.

Team-centric productivity is rooted in the following fundamental principles and assumptions:

1. A vision for a generative culture (as outlined by Forsgren, Humble, and Kim and mentioned earlier in this article)
2. The impact of positive (happy) teams
3. Team-level metrics and monitoring
4. Empathetic managers & mission-driven team members‍

A Vision for a Generative Culture

Forsgren, Humble, and Kim have found the attributes of an organizational culture that are predictive of generative cultures:

Figure 1.1

This paradigm is not simply about the surface-level, casual benefits. Generative cultures are predictive of both higher software delivery performance and better organizational performance. (Accelerate, pg 75)‍

The Impact of Positive Teams

Taking it a step further, we mapped the research done by Forsgren and team against the work by Kim Cameron and his colleagues at the University of Michigan regarding the effect of positive teams and uncovered some surprising insights. Cameron’s research, Effects of Positive Practices on Organizational Effectiveness, shows that improving performance has little to do with benefits or processes, and more to do with the establishment of “positive and virtuous practices”.‍

Figure 1.2

‍As you can see in Figure 1.2, there is an extraordinary amount of overlap between the attributes of Generative Cultures and the Positive & Virtuous Practices present in Positive (and therefore higher performing) teams. Although this is a straight-forward premise, its application is complex because of organizations’ culture. We’ve observed that the difficultly lies in changing and/or reimagining team culture and how teams are measured.‍

A Frontline View: Introducing Middle Metrics

Project, business, and technical leadership have historically had two views of productivity: project-level and organizational-level. Tooling and products available help manage or visualize tasks on a granular level or reveal high-level team and project health. As important as these two views are, they do not help the frontline engineering manager. As involved as engineering managers are in the planning of projects and the evaluation of teams, their day-to-day work mostly requires them to clearly see what is going on in the short-term in order to improve long-term outcomes. Unfortunately, the tooling available to them does not give them this view. The project management tools offer too much granularity creating an overload of data, and the health trackers are too high-level. The gap is in the middle. Front-line engineering managers seek the centralization of disparate data found on separate project management and development tools or middle metrics.

Middle Metrics strike the balance of granularity and oversight to help managers make decisions every day that improve the long-term health and metrics that the industry tracks. These middle metrics are notoriously difficult to gather, organize, and synthesize — perhaps why there are so few efficient solutions to this problem.

We have identified the major middle metrics as follows:

• Stale Issues: Checking change in progress to today’s date
• Spike in Comments: # of comments in 24 hours
• Spike in Progress: # of in progress issues increased substantially in the last 24 hours
• Unblocked Issue: if a blocking issue is completed, the issue that was blocked becomes unblocked
• Added Scope: work added during a sprint or work cycle that was unplanned
• Actual Work Time: time available for individual contributors to code
• Recycled Issue: Issue that was carried over from the last sprint or work cycle

These middle metrics bring nuance and context to higher-level metrics such as delivery lead time, deployment frequency, time to restore service, and change fail rate. By enabling engineering managers to have a better grasp of the frontline day-to-day, it is now possible to improve these higher-level metrics by addressing and troubleshooting issues in real-time.

Middle Metrics pave the way for increased team productivity by serving as the bridge between granular project management and high-level project health.‍

Empathetic Managers & Mission-Driven Team Members

The final, and arguably most important principle in Team-centric Productivity is about people. We believe there are two sides to this coin: empathetic managers and mission-driven individual contributors. Empathetic managers foster generative team cultures by creating psychological safety, collaboration, and meaning within their teams. They take the time to understand their team members, carve out time for them, and hold the organization accountable for their success. Mission-Driven Team Members are those that believe in the “why” of their work, and work towards that “why” regardless of the tasks and pain that might be required to achieve the greater purpose. They know that by putting the mission and team before themselves, they accomplish their individual objectives.‍

Conclusion

Team-centric productivity combines the proven and predictive benefits of researched insights with the intuition and instincts of the industry’s top managers. As a result, managers gain the ability to ask the right questions, more easily remove blockers, facilitate better communication, and foster more positive, higher-performing teams. They’ll be able to clearly see what’s going on and because of that visibility, plan ahead more effectively. As software development continues to evolve, our systems of measuring will as well. We believe that the combination of generative cultures, positive teams, and middle metrics will help pave the way towards more equitable and effective engineering.