Become a Product doctor! Learn how to check your product’s health

Introduction:

Have you or your team ever tried to measure the health of your software product? It may sound weird, but if we can measure the health of an Individual with different tests, why can’t we measure the health of a software product?

For many of us, the first thing that comes to mind when we think about health of a software product is the number of bugs or defects it has. However, the number of open bugs or closed bugs doesn’t actually represent anything about health of the product.

Let’s see why it’s important to measure the health or quality of a product with a real time situation.

Imagine that there is a product called “ABC” with 50 red bugs and 120 moderate bugs with an increasing trend over the past few months. This raises the following questions:

• Why are the number of bugs increasing?
• What are the crucial factors causing it?
• How to measure the overall quality of the product?
• What are the necessary steps or precautions that can improve the situation?

With our practical experience, we were able to derive an algorithm that regularly examines the health of a product to help find which areas or portfolios need to improve for a stable / bug free product.

What are the key parameters that impact a product’s health?

Any software that is being developed or has already been released, tends to have a number of bugs or defects. Basically, Bugs are unwanted behavior from a software that does not adhere to the business’ needs. They could be anything that differs from the “Expected” results during any user interactions.

Before diving into the Topic, Lets understand and co-relate this whole concept with an example — let’s say that one day “Paul” feels his temperature rising and he wants to know the reason. To find out, Paul goes to a doctor who does a number of tests to check his body temperature, weight, blood pressure, etc. and asks him about any potential symptoms he may have experienced. Based on Paul’s answers, the doctor can draw a conclusion and decide on what could be done to improve the situation.

In our example above, different test results helped to analyze Paul’s condition. Similarly, there are multiple factors that could impact the health of a software product.

The three major crucial pillars that serve as data to measure a product’s health are:

• 1st pillar — Total number of unresolved Bugs in the Bug backlog.
• 2nd pillar — Total number of bugs reported in a specific period (eg: Month)
• 3rd pillar — Total number of bugs resolved in a specific period (eg: Month)

Let’s see why these three parameters are important.

• 1st Pillar — Total number of unresolved bugs in the backlog — This pillar plays a pivotal role in determining the bug backlog’s status, which allows us to estimate a product’s quality and stability since its released to the end users.
• 2nd Pillar — Total number of bugs reported in a specific period (eg: month) — This pillar helps us monitor and understand the new releases or regression bugs found in a specific period.
• 3rd Pillar — Total number of bugs resolved in a specific period (eg: month) — This pillar helps us to understand how the team is prioritizing and resolving bugs in a specific period.

How we measure a product’s health:

Decoding the Algorithm:

The primary element that must be identified in Product Health is to set the expected frequency.

Depending on the expectations from the stakeholders, Product Health could be measured on Monthly, Bi-weekly, Quarterly etc. Once we decide on the frequency, we must give each pillar a percentage.
As discussed above, we identified 3 main pillars that impact the quality of the product. If your team identified other parameters that could potentially impact the product’s quality, you should adjust the percentages accordingly.

We have 3 pillars that affect the product’s health, so we decided to split and give each pillar 30%. The remaining 10% will be the base score for the team’s performance. Next, we separate the bug tickets based on their priority — blocker, red (very high, High), and moderate (including trivial and low). This way, each priority list would be assigned 10% within its pillars, i.e.: blockers would get 10%, red bugs would get 10%, and moderate bugs would also get 10%.

When calculating the product’s quality score, the key parameters we must take into consideration are each pillar’s multiplier, and we can determine the optimal one for each pillar to ensure a balance between them.

In our experience, the most optimal pillar multipliers are:

• Blocker bugs — 0.5
• Red bugs — 0.2
• Moderate bugs — 0.1

The reason for giving the above multipliers are:

• Blocker bugs are showstoppers and the software product’s reputation and member base will be hugely impacted, we allocated 0.5 multiplier.
• Red Bugs are critical for the Software product, we allocated 0.2 multiplier.
• Moderate bugs are not critical but still they impact the site and members, we gave 0.1 multiplier.

However, we encourage you to adjust these based on your team’s respective situation.

Now, we need to calculate the quality score for each pillar. The logic we used for each was:

• 1st pillar — Total number of unresolved bugs.
  10 — (total number of blocker bugs X blocker multiplier) + 10 — (total number of red bugs X red multiplier) + 10 — (total number of moderate bugs X moderate multiplier).
• 2nd pillar — Total number of bugs reported in a period
  10 — (total number of blocker bugs X blocker multiplier) + 10 — (total number of red bugs X red multiplier) + 10 — (total number of moderate bugs X moderate multiplier).
• 3rd pillar — Total number of bugs resolved in a specific period
  Exec ((total number of blocker bugs closed) + (total number of red bugs closed) + (total number of moderate bugs closed)).

Below, you will find a flow chart that details the logic behind our algorithm’s high-level calculations.

Practical Example:

Let’s review an example to fully understand the algorithm:

• Step 1: Lets assume that there is a product with the below statistics

• Step 2: Lets calculate the multiplier for Pillar 1 and Pillar 2
• Blocker Bugs: We would multiply every blocker bug in Pillar 1 and 2 by 0.5 Based on this, blocker bugs have a value of 6 and 9 respectively, and after adding the multiplier, the score would be 6 X 0.5 = 3 | 9 X 0.5 = 4.5
• Red Bugs: We would multiply every red bug in Pillar 1 and 2 by 0.2. Based on this, red bugs have a value of 36 and 16 respectively, and after adding the multiplier, the score would be 36 X 0.2 = 7.2 | 16 X 0.2 = 3.2
• Moderate Bugs: We would multiply every moderate bug in Pillar 1 and 2 by 0.1. Based on this, moderate bugs have a value of 53 and 21 respectively, and after adding the multiplier, the score would be 53 X 0.1 = 5.3 | 21 X 0.1 = 2.1

Step 3: Normalization for Pillar 1 and Pillar 2

• As every pillar is allocated a score of 30, each priority would have a score of 10. Normalization would be applied to the scores we got after calculating step 2
• Pillar 1: After normalization, we must calculate the new score — (10–3)+(10–7.2)+(10–5.3) = 14.5
• Pillar 2: After normalization, we must calculate the new score — (10–4.5)+(10–3.2)+(10–2.1) = 20.2

Below, you will find a chart that details the complete logic behind our calculations.

Step 4: Normalization for Pillar 3

This pillar represents the bugs that the business team aims to close. So, before proceeding with this pillar, we should discuss with both business and engineering about the target goals that needs to be achieved.

Let see how this would work with an example:

• Blockers — The business decision is to close 15 bugs, of course, the main agenda of any product team is not to have any blocker bugs in Open status.
  If the team is able to meet the goal, the team’s score would be 10 / 10. Anything less would result in a negative multiplier based on the number of missing resolved bugs. If the team managed to close 12 bugs out of targeted 15 (15–12), the score would be (10–0.5X3) = 8.5
• Red — The business decision is to close 20 bugs
  If the team is able to meet the goal, the team’s score would be 10 / 10. Anything less would result in a negative multiplier based on the number of missing resolved bugs. If the team managed to close 15 bugs out of targeted 20 (20–15), the score would be (10–0.2X 5) = 9
• Moderate — The business decision is to close 10 bugs
  If the team is able to meet the goal, the team’s score would be 10 / 10. Anything less would result in a 0.1 multiplier based on same logic as above.

Below, you will find yet another chart that details the complete logic behind our calculations for Pillar 3.

Step 5: Calculating the product’s quality score

We can calculate the total quality score of a product based on the score of its 3 pillars — Pillar 1 score + Pillar 2 score + Pillar 3 score.

The total from all 3 pillars in the above example: 14.5 + 20.2 + 27.5 = 62.2 out of 90. Teams would also get 10 points based on their performance, which can be linked to sprint delivery rates or any other factors you decide on based on your needs.

In our example, we gave 10 / 10 for the team’s performance. Thus, the final product quality score, based on the result above, would be 62.2+10=72.2. This score may be further simplified as 7.2 / 10.

How can this algorithm helps to measure and improve product quality score?

We hear you asking, “How can my team benefit from the quality score?” and to answer that, first we must understand what the key statistics are and what they can tell us.

After calculating the quality score of each pillar, it becomes easy to identify the difference between the expected and actual results for each pillar.

In the above example — the 1st pillar’s score is 14.5 out of 30. This should raise a red flag for the team because there are too many backlog bugs which needs to be prioritized and fixed in the upcoming days or sprints.

Similarly for Pillar 2 — it’s score is 20.2 out of 30. This shows us that there are too many bugs in new deployments or feature releases. The team should investigate how they can improve the deployment process.

Lastly, Pillar 3 has a score of 27.5 out of 30. As this is completely reliant on meeting the business’ expectations, the team just needs to understand why they failed to deliver and correct it next time.

Is it possible to automate the product quality report?

Definitely! Manually creating and maintaining this report would be a tedious and time-consuming process.

We can easily automate 80% of the report with our product management tools and an Excel workflow. The columns which are in grey in the picture above are fully automated and the ones in white must be added manually.

With help of our Agile team, we were able to generate dashboards with statistics for each pillar and the respective queries.

All we need to do is add the stats from the dashboard to the Excel sheet to get the final product quality score. This took us all of 10 mins, including the 20% manual work updating the stats and sending the reports to the various stakeholders.

Conclusion:

We calculated the product’s quality score based on what we believe is the optimal combination of multipliers and pillars, which helps us get a concrete score about the product’s quality.

We believe that the results will encourage discussion within the engineering team to solve any quality issues of a product.

Written by:

1. Hari Janapareddy (Product Quality Manager)
2. Tibor Farkas (QA Lead)

Thank you Gary Kovács for your writing advice and editing skills.

Legends:

Blockers — Bug which falls into Blocker Priority.
Red Bug — Bug which falls into Very High, High Priority.
Moderate Bugs — Bug which falls into the Moderate, Trival.