Quality Automation Metrics in DevOps

DevOps Today -

With digital transformation impacting businesses, continuous change has become an inherent part of many organizations. For such organizations, Agile and DevOps is not an upcoming trend but a norm.

According to a Forrester Consulting survey of North American and European IT decision-makers, 78% of respondents cited improved quality as a chief demand of business groups. DevOps builds quality into the entire software delivery chain by laying emphasis on communication, collaboration, and integration among various stakeholders in the software development process, i.e. development, QA, and operations.

Need for Metrics -

Two age-old statements that aptly describes the need for Reporting and Metrics

  • We can’t control things which we can’t measure
  • What gets measured gets done

The above statements culminate to the fact that regular measurement and reporting helps you make decisions to improve your results. The most critical measurements are called Key Performance Indicators.

With increased Shift left Testing adoption in Software Development, traditional QA Metrics like Defect Density and Defect Removal Efficiency is just not enough to calculate the efficiency of QA and DevOps as well.

DevOps Metrics -

Any enterprise dashboard following Agile/DevOps is having these 6 Metrics to better track the Quality and Success of the projects.

• Frequency of Deployment

• Change Volume

• Lead Time (from Development to Deployment)

• % of Deployments Failed

• Mean time to Recovery

• Performance Metrics

The Increased demands on what appear to be opposing a Faster, Better and Cheaper QA requires Organizations to rethink their quality validation and testing activities. Standard automation of test cases is no longer enough. Instead, Quality Assurance and Testing teams must adopt integrated quality management models, a business outcome focus, and the use of intelligent data analysis to form test strategies and created automated optimized validation checks and test sets. This is where DoX comes in, an IT automation platform by Suneratech.

QA and Testing teams in DevOps world should focus on business risk and provide insight on whether the software can be released. To achieve this shift, QA teams need to stop asking “are we done testing” and instead concentrate on “does the release candidate have an acceptable level of business risk?”

Quality Automation Metrics in DevOps -

Here is the list of metrics that should help measure Project or Department or Organization level QA Automation efficiency in DevOps world.

1. Requirement coverage through Automation Testing

The quality of a finished piece of software is often defined by its ability to meet the requirements laid out by developers at the outset of production Traditional QA models used to measure the Number of requirements committed vs. actual requirements delivered to Production as Requirements Coverage metric.

However, with focus on a Continuous Delivery model with minimal intervention after Code Build to Deployment in Production, the efficiency of QA teams to automate testing of the requirements ahead of Code Build becomes a success factor in increasing Frequency of Deployment, Change Volume & Lead Time

Requirement coverage through Automation Testing

= (No. of Requirements Tested through Automation/ Total Number of Requirements) *100

While a 100% testing through Automation is a Utopia, many organizations are still able to achieve more than 85% of testing through Automation. This comes from an effective test strategy where Testing teams are involved right from the beginning of Development and able to automate the test cases through API testing rather than GUI testing.

2. Automated Test Effectiveness

Most of the surveys find that organizations are struggling to match right level of quality with the demand for a faster testing. To address this, QA teams must take a business-centric approach to choosing right test focus areas and creating nimble processes, such as behavior-driven development (BDD) to define Acceptance criteria that can be automated.

While there is no simple formula to quantify the right amount of test coverage, increasing usage of test automation vs. Manual testing for Acceptance criteria with reducing test execution times is the simplest way of measuring Automated Test effectiveness.

3. Deployment Lead Time

Deployment Lead Time is a subset of the Lead Time measured as part of DevOps Metrics and measures the time taken from Creation of Code Build to Deployment of Code Build to Production. From a QA perspective, this is majorly affected by the amount of Manual, Automated API, GUI and Regression testing we do for each release cycle and proportionately impacts the time take for Production cycles.

Shift Left strategy where QA teams perform Smoke testing upon check-in, API/message layer testing after integration, end-to-end testing at the system level increases the Automated testing effectiveness at the same time reducing Deployment Lead time.

4. Performance Metrics

Defining Acceptance Criteria for Performance has become a norm in today Software Development models. However, with DevOps release cycles happening hourly or more frequently, having an automated performance monitor to measure Performance for each Release cycle as part of QA always helps moving right builds to Production and avoid System-level failures due to Peak Loads.

Working against the right and related metrics will help an organization provide better quality products, reduced go-to-market time and hence improve their ROI. Experts at Suneratech can help you understand what needs to be measured and tested to get the optimum outputs. Contact Suneratech for DevOps or Continuous Testing assistance.