Fundamental Quality Metrics

As the leader of a Quality organization in a tech company, I frequently get asked “How good is the product?”, “How buggy is it?” and similarly broad questions. These questions most frequently arise when reporting upwards to executives or laterally across organizations.

It is easy to get lost in nuance and generate an answer that provides no value, but often that granularity isn’t what the person asking the question is looking for. What they’re asking for is a way to tell if the company is getting better or worse, and by how much — the most fundamental of quality metrics.

An easy way of arriving at a figure for this is by using some generic math. I’ll walk you through how that works for my company, here.

We use three severity levels: S0 (Urgent), S1 (High) and S2 (Low).

Of course these levels shouldn’t be taken as equals, so how do you deal with the varied importance of bugs? With multipliers like this:

(# of Open S0s*5)+(# of Open S1s*3)+(# of Open S2s*1)=n

Then, just track that number over time. If your severity ecosystem has more than 3 levels, simply bump everything up. S0*7, S1*5, etc.

These multipliers are semi-arbitrary. They work only to increase the weight of the raw numbers of different severity bugs, and so are applied in a largest-to-most-severe relationship. Feel free to adjust those at will, as every organization views its system differently.

Note there’s no upper limit to the figure, and the number itself isn’t terribly relevant as a single data point, but the first part of our solution is at-hand.

When collected at a frequency that makes sense for your product (daily, weekly, monthly) and tracked over time you can see if the number is getting larger or smaller, and by what percentage.

If on Week 1 n=100, and on Week 2 n=105, you’ve had a 5% increase in bug gravity. Now you can answer the questions “How buggy are we today?”, “Are we gaining or losing ground?” and “By how much?

What are the systems you use to keep track of bugs? How do you report on the quality of your product? Leave me a comment; I’d love to hear your thoughts on the subject.