Developing a Quality Strategy

Over the course of my career in testing and quality, I have seen many different quality strategies. All of them have been narrowly focused on testing, with perhaps a sprinkle of DevOps tossed in. I started to wonder why this is, and when I dove a bit deeper I realized that all of them were written by testers.

It’s a truism in the industry that you can’t test quality into a product. Why is it then that all of our quality strategies revolve around testing or development methodologies? I have a couple of theories around this:

1. Most companies don’t understand quality. Even though we as professional testers know that testing is only one part of quality, we still have to give in to the demands of the company we are working for. In order to move forward, we write a great testing strategy and slap a quality strategy label on to it to appease the powers that be.
2. The strategy itself is written by a tester, not a quality engineer.

If a quality strategy is not a testing strategy, what is it? A quality strategy is a holistic view of all the different thoughts, beliefs, and ideas that go into producing a quality product. If it sounds vague, that’s because it is.

To demonstrate how to put together a quality strategy and operationalize it, I’m going to draw a parallel between the test design pattern of Arrange-Act-Assert. You can use that pattern as a model when developing and implementing a quality strategy.

Arrange

When using the Arrange-Act-Assert pattern in testing, the Arrange steps set up your testing. Applying that to developing a quality strategy, the Arrange steps should arrange your thoughts around quality so that everyone has a clear understanding of and alignment around your quality strategy. You can organize and communicate your thoughts in a variety of ways; here, I’m going to focus on writing a Quality Strategy Document.

Components of a quality strategy document

There are nine elements to a Quality Strategy Document:

• Executive summary
• Purpose
• Principles
• Goals
• Strategy
• Drawbacks
• Anti-patterns
• Leading and trailing top indicators
• Open questions

Executive Summary

The Quality Strategy Document begins with an executive summary. Executives have a limited amount of time to read through an entire strategy document, but this is your opportunity to hook their attention. This synopsis should be well-written and succinct. I recommend saving this section for last, even though it’s at the top of the document.

Example:

The quality strategy is to embrace a culture of quality through a series of layers. In this approach, everyone involved with delivering software is responsible for its quality. This includes having processes in place to support effective testing that leads to confidence in the quality of our software. This is accomplished through a layered approach to quality where everyone has a clear understanding of the layers, as well as how each person has an impact on quality.

Purpose

The purpose of your test strategy should answer two very basic, but often difficult, questions:

1. What is this strategy intended to accomplish?
2. Why should readers care?

Write a short paragraph that summarizes the purpose of your strategy. The reader should be able to get an accurate, concrete understanding of your quality strategy and what they will gain from reading the document.

Example:

This document captures the quality strategy for delivering software products to our customers. It is intended to clearly describe our goals, principles, approaches, and tools. This includes concepts and ideas around quality that will be implemented throughout the organization.

Principles

Quality can mean different things to different people. In this section, clarify the organizational beliefs and principles that guide your quality strategy. I recommend talking to all of your stakeholders about their beliefs around quality and then summarizing that information into core guiding principles.

Example:

1. Quality is all about the customer.
2. Quality is owned by everyone and not a single team.
3. There is no silver bullet to quality.
4. Quality is more than testing.
5. Quality is not a religion.

Goals

What are you hoping to accomplish through the implementation of your quality strategy? Your goal section should list your long-term objectives. Much like needing to have a destination before getting driving directions, goals guide your entire quality strategy.

Example:

• Prevent defects from impairing the delight of customers
• Establish the standard by which all product development teams within the organization deliver products
• Establish a single strategy for the organization instead of a series of disjointed practices

Strategy

In this main section of your document, outline your model. Think of this document as your organization’s quality story with this section as the main character. Some questions that might assist you with determining your organization’s quality strategy are:

• Are you looking to move to a whole team model for quality?
• Will you have a quality center of excellence?
• Will there be one person embedded in each team that will be responsible for all of the tasks associated with quality?
• Are there different layers of quality that make up your quality strategy?

Whatever your strategy is, ensure that your strategy also helps people understand how they fit into it. Who is responsible for actually doing the work? Who is accountable for the work getting done correctly? Who do you need to consult in order to get buy-in or more information? Who should be informed about the strategy? It’s a good practice to include RACI (Responsible, Accountable, Consulted, and Informed) charts so that people understand what their responsibilities and roles are.

Example:

In this strategy, when a team pulls in work the entire team swarms to discover and document clarity and conformity around the acceptance criteria. From there, the team can determine which scenarios need to be covered for release and which tests can be automated. Work cannot be considered done until the team has coded, tested, and incorporated any automated tests into CI/CD. This includes and is not limited to hotfixes, patches, and any temporary solutions. Taking a layered approach to quality will enable the whole team to have a better understanding of the customer behavior driving what they are working on.

Drawbacks

As the saying goes, you can do something cheap, fast, or well. Are you sacrificing one or more of these with this strategy? Generally there is some sort of drawback and it’s important to call this out.

Example:

With this strategy it can initially take longer for new code to be released. This is due to quality being a part of the development process and happening within the same sprint as the development work. Teams will not only be responsible for developing code but also for understanding the behavior driving new features, performing testing, creating any new documentation, and ensuring measures are in place for monitoring and observability. Training will have to occur as well which will slow down development efforts as testing is a different craft and mindset from development.

Anti-patterns

In this section, you’ll outline other possible scenarios. It’s also important to note why you did not choose these options. This section demonstrates that you have thoroughly considered the alternatives.

Example:

Testing as the sole responsibility of one group outside of CI/CD: In this approach testing is considered the sole responsibility of one group, whether that be a specialized testing group, developers, product, or operations. With this approach:

• Silos can develop between different groups, causing bottlenecks in the software development life cycle
• There is a lack of fast feedback, causing developers to frequently context-switch between work items and bugs
• More testing tends to happen only in production, which is shown to be expensive
• There is a lack of accountability due to context being lost between groups
• This strategy is known industry-wide to lead to a higher count of bugs and for those bugs to be more expensive to fix

This is not an approach that we will accept or promote in this organization.

Leading and Trailing Top Indicators

In this section you’ll outline how you’ll measure your strategy’s success using leading and trailing indicators. Indicators are metrics that provide you with further insight into the quality at your organization. These metrics alone cannot determine if you are producing quality software. They indicate if your quality strategy is impactful to the overall quality at your organization. Leading indicators are the metrics you need to achieve success, and trailing indicators help you to determine whether or not the work you are doing is successful. If you meet your leading indicator metrics, your trailing indicator metrics should follow.

Looking at the example below, if you cover 90% of your services with one smoke test and 95% of the core functionality use case based tests are automated and running in CI/CD, then you would expect to achieve a defect rate of 4 per week maximum.

Example:

• [leading] 90% of services are covered by at least 1 smoke test in production
• [leading] 95% core functionality of software functionality use case based tests are running in CI/CD
• [leading] 95% of work is tested with test documentation in sprint
• [leading] Time to review quality standards decisions is less than 5 business days 90% of the time
• [trailing] Achieve defect escape rate of 4 per week max
• [trailing] 50%+ of “decision” reviews can be completed async (signals that early planning, knowledge sharing, and design quality are all high enough)

Open Questions

We can’t know everything about everything. In this section, list any outstanding questions about your strategy. What don’t you know that may lead to changes or have an impact on your strategy?

Example:

• Is the product team aligned with the expectations we have of them?
• Is engineering ready for a cultural change?
• Are there any other leading/trailing indicators that would help us measure success?

Act

The next step in the Arrange-Act-Assert pattern is to Act. Now that we have arranged our thoughts around quality into a quality strategy document, it’s time to Act on it. We do this by communicating our strategy to leadership to gain alignment around quality, and then socializing it out to the rest of the organization.

But gaining alignment and socializing your strategy is just the first step. In order to truly Act on your quality strategy, you may need to develop a quality program. Check back here for Parts 2, 3 and 4 of this blog. In Part 2, I’ll dig into Act by taking a deeper look into what a quality program is and how you can design a quality program within your organization. In Part 3, we’ll dig into Assert where we will explore some of the metrics that can be used to provide better insight on the impact of your quality strategy and program. Lastly, in Part 4, we will look at how we can also use the Arrange-Act-Assert pattern for your test strategy and how it ties in with your quality strategy and program.