Quality Metrics to Consider For Your SaaS Product Series — Part 1

Meghana Dwarakanath
Feb 2 · 4 min read
Image for post
Image for post

Back in the day, quality was synonymous with testing — how many tests were run, how many passed and how many failed. Quality of a feature being released was determined by these failures, and by the number of issues that the customer reported.

Cut short to the world of software today — we have moved to the world of services. Multitude of these services interact with other services in complex ecosystems, with third party integrations, accessing and manipulating terabytes of data at near real time speeds. Do the features work? — is no longer a sufficient metric to track the quality of a product. The questions about quality have gotten more diverse and complex, spanning multiple tools, teams and initiatives:

  • Does the feature work? How many test cases passed / failed?
  • How much time did the operations team spend in deploying / maintaining the new service?
  • How does the feature impact security posture for the product?
  • Do we have code hotspots that generate a lot of bugs?
  • Is the product maintaining its SLAs and SLOs?
  • How quickly can someone debug and fix a P0 issue for this feature?
  • What is my deployment cadence and how is it affecting incoming defect rates?

To sufficiently answer the quality question, today you need data from multiple engineering tools — Jira, SCM, PagerDuty, Security Tools, CICD tools to name a few. You need to be able to successfully co-relate metrics and draw insights on the success of your quality program.

In my previous roles, as I helped build and scale quality and performance teams for a rapidly growing SaaS product, along with regression results and test coverage, there were a few metrics that helped me track and answer quality questions around the product:

Change Failure Rate

In a SaaS world, deployments are frequent. Unfortunately, so are customer impacting outages. As we scaled teams, it became critical to find a balance between how much and how often we deploy vs how stable the product is. By tracking deployment frequency and deployment sizes against incoming outages and customer issues, we were able to make data driven decisions on how often we deploy. Transitioning from weekly to bi-weekly deployments immediately improved the stability, gave features more time to soak in staging and reduced outages

Image for post
Image for post
Image for post
Image for post

Deployments vs Incoming Bugs, Volume of Change being deployed

Code Hotspots

We all have parts of the code that are complex, critical or just simply magical. Any changes to these areas of code require rigorous and careful testing and even an altered deployment plan to cover for rollbacks. We had a 2 fold problem here — we did not fully know where these hotspots were and we did not know if any of those hotspots were touched in the upcoming deployment. By correlating incoming issues to the areas of code where the fixes went in, we were able to get a rudimentary map of hotspots, review the release change list against the map and determine the risk factor for a deployment.

Image for post
Image for post

Code Hotspot HeatMap

Release Security Metrics

It is a no-brainer that security is a critical aspect of every SaaS product. But with rapid development and deployment, we put security testing into the final steps of release planning. It was a hard lesson learnt when after a feature was developed and tested, we realized that we had overlooked some security aspects and had to go back to the drawing board. Incorporating security tools into the CICD pipeline and tracking vulnerabilities and best practice violations on the release development dashboard saved some unpleasant surprises downstream

How are quality metrics changing in your engineering environments? Let us know in comments below.

In the next part of the series, we will go over other metrics that impact your quality.

levelopsio

Bringing Analytics and Automation To Engineering Discipline

Meghana Dwarakanath

Written by

Director of Customer Success & Engineering @ Levelops

levelopsio

LevelOps is an industry first Engineering Success platform to drive velocity, quality and security outcomes by providing visibility and automation. LevelOps is used by Fortune 500 companies, multinational public companies and startups to manage their engineering teams effectively

Meghana Dwarakanath

Written by

Director of Customer Success & Engineering @ Levelops

levelopsio

LevelOps is an industry first Engineering Success platform to drive velocity, quality and security outcomes by providing visibility and automation. LevelOps is used by Fortune 500 companies, multinational public companies and startups to manage their engineering teams effectively

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store