Software Engineering Metrics: Part 2 — Identify Your Audiences
A six-part series on the why how and what of software engineering metrics.
This six-part series of articles looks at a metric strategy you can apply to a software engineering function throughout the entire delivery lifecycle, from the backlog through to production:
- Part 1: Start with why
- Part 2: Define your audiences
- Part 3: Understanding stocks and flows
- Part 4: Define your metrics
- Part 5: Monitor and observe
- Part 6: What to do with your data
Part 2: Identify your audiences
Once you understand your categories, you can begin to think about the audiences that will consume your data and why they need it, naturally leading to an intuitive set of metrics.
Cascading details
You will want to cascade your metrics up and down the business across different audiences. Each audience will be interested in all or a subset of your metrics and to varying levels of detail.
In general, the requirement for metrics becomes less detailed the higher up you go. Conversely, in some areas (perhaps finance and objectives) the metrics become less detailed as you go down the chain.
Teams and leadership
Most likely, engineering, product and delivery functions make up your teams, all of whom play a role in delivering software and are overseen by people with a primary interest in portfolio and budgets.
You’ll also have a leadership team who want oversight of what’s going on across teams and at the very top of the firm an executive panel which only needs a summary.
Potential audiences
Referring to your various RACIs can help with identifying audience groups and understand their varying needs for data. Here’s how your audience groups might be structured.
Executives
Executives are responsible for how the company is performing as a whole, and technology is just a part of that. They will look for a summary rather than detail but should be able to drill down if needed.
They will also communicate a summary of progress against company targets back down the business to help contributors, managers and leaders with their decision making and prioritisation.
Leaders
You will likely have a technology leadership team that will want a broad overview of all teams. They will want to make decisions about priorities, headcount and budgets, and to be sure that all activity is contributing to the company’s objectives.
Managers
Managers are people who lead one or more teams. They will need data that helps them understand how their teams are performing, help with decision making and prioritisation and to identify positive changes.
They will be accountable for software quality and performance, so will need some detail around that.
They will likely be responsible for budgets and headcount, so will need to see where they stand.
Having data about how the company and their team are doing against objectives helps motivate and inspire the teams.
Individual contributors
These are your engineers, product owners and delivery leads. Their primary concern will be data to help them understand and improve their team performance. They will want to keep an eye on code and product quality, as well as real-time system performance. It’s also super useful if they know how the work they are doing is contributing to the company’s objectives.
The matrix
You’ll end up with a model a bit like this. You’ll want to report at a higher level of detail where there are more ticks.
What metric categories will interest your audiences?
Each audience will have a varying degree of interest in each matrix category. Some will have no interest in some data (individual contributors and finance, perhaps) whereas some will thrive on detail (maybe executives and objectives). The categories will be very personal to your own company set up.
Here’s a suggestion of the different levels of interest for each matrix category.
Velocity
Velocity is the bread and butter of agile teams, using it measure the speed and efficiency at which the work flows in and out of their backlogs. There’s also a lot more you can tell about the type of work they are doing (see innovation, optimisation and maintenance in Part 4: Define your metrics).
You can’t (and shouldn’t) compare velocities across teams as they have different agendas, estimation methods, team sizes, purposes, products and more. However, you should strive for cross-team consistency where you can by collecting the same sort of data in the same ways. Each team must track its own velocity consistently over time to help identify trends.
Velocity means little to leaders, but it can be useful when communicating up about cross-team dependencies and constraints. It means nothing to executives, so shouldn’t be rolled up to them.
Quality
Quality means different things to different audiences:
- Teams writing and deploying software will view quality in terms of test depth and breadth, bugs released into production and issues arising from them.
- Operational teams will measure how often systems alert due to problems.
- Product owners will want to know how often the user experience is affected by incidents.
- Managers will want to identify process improvements to raise quality standards.
- Leaders will want an overview to be sure the organisation is delivering excellent value for the customer.
- Executives will probably need quality issues escalated to them, but won’t want regular metrics.
Performance
Much like quality, performance needs more detail for engineers and teams as they are the people who can respond and make improvements.
Of course, performance is vital across the organisation as it affects running costs, SEO and user experience too. So you’ll likely want to roll some summarised performance data up to the execs. Organisations often want to benchmark performance against competitors.
Finance
Technology is typically a cost centre, so keeping track of budgets is critical. Understanding performance against budget is vital for headcount, licensing, and cost attribution, reducing waste and leading to the total cost of ownership.
Objectives
The company exists to serve its users and deliver a return for its shareholders. The executive team will be accountable for a set of objectives, missions or goals. These are cascaded down the organisation to help with prioritisation and decision making.
The leadership team will be responsible for delivery against objectives, so needs to report progress up to the executive team, who will need to report to the board and shareholders.
Next — Part 3: Understanding stocks and flows
Metrics typically track the value of something over time. There is a way to extract more value and to understand better how to influence and change things; it’s called Systems Thinking.
