Leveraging Quality Metrics: Inspect & Adapt, Measure & Learn

Written by Stephen Evans

The Quality Assurance Team at Tommy exists to inspect, measure and challenge the quality of our e-commerce platforms. We can attest that any product team’s ability to focus on quality is influenced by and dependent on many factors, both internal and external. However, it is generally accepted that the most significant influencing factor is mindset. That is to say, encouraging team members to prioritize, take interest in and ownership of the quality of their product. But how can instilling this mindset be achieved?

One of the foundational tenets of the Agile Manifesto is to Inspect & Adapt. By introducing qualitative and quantitative insights, the team will be empowered to identify positive and negative trends (inspection). Next, the team can use this data to implement corrective actions (adaptation). Finally, they should evaluate the effectiveness of those improvement actions (further inspection), thus continuously improving the effectiveness of the team…and the quality of the product.

But it’s important to keep this measuring mindset where it belongs. Telling a team: “Start measuring X, Y and Z and we’ll see how you’re really performing,” quickly risks taking away control and ownership, likely leading to resentment, disillusionment and a begrudging attitude towards data. In order to improve the team’s effectiveness, allow the data they generate to belong to them. Let the team collect and interpret it.

If this advice has peaked your interest, here are a few additional tips for nurturing a culture of inspection, reflection and adaptation.

1. Start By Asking What Hurts The Most

Even if the answer to the question, “What is hurting the team and/or the product the most right now?” isn’t something that can easily be measured, it starts the conversation from the correct team and product-centric place. Your team’s process of inspection should always begin by identifying a purpose and ‘north star’ to work towards.

2. Dig Deeper Into What Hurts Most

Usually this comes naturally after answering the previous question, but it’s worth making this step explicit. Rather than choosing a north star on behalf of the team, maintain team ownership of the pain point by questioning and agreeing on the impact together.

This process might not be so simple. Is the impact anecdotal, a gut feeling or are there already metrics that support the issue’s impact? For example, would you say there are too many defects found in the upper (UAT, Production-Staging) environments? If so, let’s measure this pain and see if data can provide further insights…

3. Make The Most Of Best Practice Metrics

It’s likely that your team’s pain points are quite common and, as such, can be identified and tracked using existing, standard measurables.

Are there too many defects found in the upper environments? Is the team saying that bugs are being found too late in the process? If so, maybe measuring MTTD (Mean Time To Detect) can offer solid metrics to work from and consistently track.

MTTD is just one example, but there are plenty of others: MTTF, velocity predictability, defect count (by story count), defect root cause, etc. are all measurements designed to track the maturity of quality and practice. They also all naturally relate to common issues a team may site as pain points, or something that affects the quality and healthy development of your product.

By applying these common measurables to an issue or improvement opportunity identified by the team, you’re securing buy-in and greater appreciation for the value of measuring. It’s far better than saying to a team, “Start measuring X, Y and Z and we’ll see how you’re really doing.”

4. Start Small

It’s tempting to get carried away with all the things that can be measured, and the promise of having tons of rich information with which to steer the team. However, too much too soon can devalue the data, create frustration and over-burden the team with follow-up actions. (Speaking of which, don’t forget that measuring only provides insight, it doesn’t mean things will get better on their own. It’s what you do with this new stream of data that will determine improvements).

Indeed, having tons of corrective actions coming from data-driven conclusions can be disruptive to the cadence and balance of the team, ultimately having a negative impact on the development of the product. As such, it’s wise to start small and see how you progress.

5. Work With What’s Already There, And Don’t Over-Complicate

Applying metics to your pain points may feel daunting, but the great news is that you can start with tools you already have. If you use JIRA, tweak it to provide additional fields that will enable collection of the required data. For example: In which environment was the defect found? What kind of defect is it? What was the root cause of the defect? How long does it take to fix? All of this can be pulled from JIRA pretty easily and at minimal cost to the users of the tool.

The bottom line: try not to create a significant distinction between the current way of working and what is needed to measure.

6. Be Consistent

Metrics are only useful if they’re collected consistently and correctly, otherwise they provide only partial glimpses of a bigger picture, become anecdotal, or worse still: can be false and misleading. This is another reason to start small, introduce additional metrics slowly and give yourself the bandwidth to…

7. Discuss And Adapt

When it’s time to discuss the metrics you’ve consistently collected, it’s important not to look at them in isolation. There is a new stream of data, coupled with qualitative interpretations of what the data could mean, plus other mitigating or supporting factors that you may or may not be formally measuring. There are many threads to consider, and you shouldn’t limit the conversation to the numbers.

Instead, discuss what you’ve collected in the sprint retrospectives alongside opinions, feelings and any other source of potential input. Use this amalgamation to agree on reasonable interpretations and follow-up improvements that can be implemented.

8. Don’t Stop!

Keep going! It’s a continuous path of improving, adding to, and perhaps even discarding measurements. Continue doing what works and has value, stop doing (or adjust) the things where the value doesn’t warrant the effort needed to collect it. Reflect on how these measurements are helping you to understand your team and product better, and identify the external influences that might be impacting you.

Happy measuring!