Quality Score @ MiQ

Introduction

At MiQ, we have close to 70 different applications and microservices that cater to various organization’s needs.

The quality of an application or microservice has many dimensions, starting with the functionality described in the API contract, but also including many other qualities such as performance, and security. This is where we were finding it difficult to understand the quality of each of our application/microservice as these dimensional data points are fragmented in the ecosystem (as shown in the below diagram).

Testing-related information is scattered across different platforms.

For example: If a person wants to know the Unit Test coverage for an application, they would have to log in to the sonar portal and get that information. Similarly, they would have to log in to JIRA to get the number of bugs found in an application.

To address this issue, the Quality Crafters team at MiQ came up with the idea of having The Quality Score for each application.

What is Quality Score?

Quality Score is a unified points system that addresses the issue discussed earlier. The quality score is a tangible measure of how an application is doing in terms of standard practices listed down by the Quality Crafters Team. A quality score will give us a single number that resembles the ‘quality’ of an application instead of different metrics across a plethora of different tools.

Quality score provides a quantitative measure of an application from a quality standpoint.

As shown in the above diagram, we are calculating the quality score of an application/microservice by fetching its UT coverage data from SonarQube, Defect count from JIRA and Component Test coverage/UI automation coverage from S3.

How is the Quality Score calculated?

The quality score is calculated based on the following parameters:

1. Unit Test (UT) — Unit testing is a software testing method by which individual units of source code — sets of one or more computer program modules together with associated control data, usage procedures, and operating procedures — are tested.
2. Component Tests (CT) — We test those objects that can be tested independently as a component without integrating with other components
3. Integration Tests (IT) — Integration Testing is defined as a type of testing where software modules are integrated logically and tested as a group
4. Critical Defects — Critical defects usually block an entire system or module’s functionality.
5. Non Critical Defects — Any other defect which is not critical falls under this category.
6. Security — Security Testing is a type of Software Testing that uncovers vulnerabilities of the system and determines that the data and resources of the system are protected from possible intruders.

Each parameter is given a Weight (High/Medium/Low) and the criteria for RAG (Red/Amber/Green). The score is calculated on the basis of weight and criteria as explained below.

1. Unit Test (UT)

Weightage = High

At MiQ, we have set a benchmark for our Tier 1 applications to have at least 80% UT coverage to score maximum points for UT. Below is the criteria for all other Tiers.

2. Component Tests (CT)

Weightage = High

At MiQ, CT coverage is measured using two things:

a. Code Coverage

b. Count Coverage

Code coverage is self-explanatory.

We calculate count coverage using the following formula:

Count Coverage = (No. of test cases automated/Total no. of test cases) * 100

We have set the benchmark for our Tier 1 applications to have at least 80% code coverage as well as count coverage to get a Green Rating.)

3. Integration Tests (IT)

Weightage = High

ITs are similar to CT except that the dependent services/API calls are not mocked.

We have set the benchmark for our tier 1 applications to have at least 50% code coverage to get a Green Rating.

4. Critical Defects

Weightage = High

In our opinion, for an application to get a high-quality score, it must have 0 critical defects found in production after a release.

5. Non Critical Defects

Weightage = Medium

These include defects that do not interrupt the BAU. These could be cosmetic issues and hence have been given a medium weight.

Example

Assume we are calculating the quality score on the basis of Unit Tests.

Criteria

Weight Points

RAG Points

Let’s say an application ABC has 85% UT code coverage. Then the score calculation will be done as follows:

Max Score = (Points for the weight ‘High’) * (RAG points for Green)

Max Score = 4 * 5 = 20

Since ABC has 85% UT coverage it falls under Green and hence gets 5 RAG points.

Now, Unit Tests have High weightage, so the multiplier is 4 for UT.

Points scored by ABC = 4 * 5 = 20

So ABC scored 20/20 points.

Benefits

1. Quality Score helps in establishing the standards that need to be met for all applications within an organization.
2. Gives a consolidated score view to the management about the quality of applications.
3. Makes it easier to spot areas where attention is needed to improve the quality of applications.

Sample Quality Score report @MiQ

As shown above, if an application scores more than 80% rating points they are marked in green. Similarly, if they score less than 50% are coded as amber and those scoring less than 30% are marked red. These numbers can change from organization to organization.