How Paylocity Created a Culture of Quality, and You Can Too!

Introduction

In today’s Agile world, there is no time for extended manual testing sessions by QA engineers. Quality needs to be owned by the entire team, and software testers and developers alike need to participate in testing activities. In this post I will present the story of how we used a Quality Maturity Model at Paylocity to create a culture of quality.

The Problem

From 2016 to 2020, Paylocity grew from having ten development teams to over 40: a four-fold increase. This growth, combined with fast-paced development and a rapidly increasing customer base, resulted in concerns about releasing quality software quickly.

Every development team at Paylocity had between one and three QA engineers. In each team, testing was seen as an activity for the QAs rather than a whole-team activity. QA engineers were expected to do all the manual testing as well as write and maintain all of the test automation. As a result, QA often became a bottleneck that slowed down new software releases. Moreover, the lack of automation skill among some QAs meant that some teams had little or no test automation, further slowing down the release process as manual testing was greatly relied upon.

Because teams were so focused on getting features released on time, important qualities such as performance, reliability, and maintainability were added to existing tech debt, creating potential future problems for the next release and slowing the teams down even further.

The Strategy

In order to create a culture of quality at Paylocity, we decided to adopt a Quality Initiative. This initiative had three main steps: assessing our current QA engineers, asking each team to create a Quality Strategy, and creating a Quality Maturity Model.

We determined that we wanted each team to have at least one Software Test Engineer, who would act as a quality coach for the entire team. This Software Test Engineer would work with the software developers to create test automation, so that every developer and tester on the team was capable of contributing to and maintaining the automated tests.

We reassessed our QA engineers according to the new job descriptions we had created for the Software Test Engineer role. QA engineers with skill in writing test automation became Associate Software Test Engineers, Software Test Engineers, or Senior Software Test Engineers, depending on their skill level. QA engineers who either could not or did not want to write test automation became Product Analysts, who worked with Product Owners to determine the direction of new features.

Since the whole team was now responsible for quality, we wanted to make sure that each team had a conversation about what processes they would adopt to ensure quality. We asked each team to create their own Quality Strategy and provided a sample template for this purpose. Each team determined how stories would be created for development, what “done” would look like, what kinds of tests would be run in each environment and who was responsible for creating and running the tests, how bugs would be dealt with, and how features would be released. We created a wiki page that linked to each team’s Quality Strategy, which allowed teams to learn from each other’s examples.

In order to help teams develop behaviors that would ensure that they were producing good quality software, we decided to create a Quality Maturity Model that would outline those behaviors. First, we needed to determine what our definition of “quality” was. We decided that a quality application was one that was valuable, functional, reliable, secure, performant, usable, and maintainable. This definition was pressure-tested with senior managers to make sure that everyone agreed this was what quality looked like.

The Attributes of Quality at Paylocity

A quality application is one that is…

• Valuable- it meets the customer’s needs
• Functional- it does what we say it does, and we can measure those interactions
• Reliable- it is available when it is needed
• Secure- it protects customer and company information
• Performant- it responds within an acceptable time
• Usable- it is easy and intuitive to use
• Maintainable- it is easy to test, deploy, automate, monitor, update, and scale

The next step was to come up with a matrix of behaviors for each of these quality attributes. Each behavior had Minimum, Standard, and Excellent examples. Teams could assess their quality maturity by doing a self-evaluation with this matrix.

The Challenges

Creating our Quality Initiative and our Quality Maturity Model were challenging activities, but implementing them was even more challenging. Among the obstacles faced were resistance from developers and testers, and maintaining accountability and momentum over time.

Some developers were initially resistant to the idea of the whole team owning quality, because they felt that it was not their job to test. We assured the developers by reminding them that they were already contributing by adding unit tests. We also showed them that testers were helping out with the development process by participating in build and deployment activities. We adopted an automated test framework (Cypress) that the developers enjoyed using because it was easy to set up and was using a familiar language. (We’ll share more on why we chose Cypress in a future blog post.) Finally, we pointed out that when developers help with test automation and exploratory testing, they learn the product better and can therefore release new features more quickly.

Some software testers, especially those that were on a team with two or more testers, simply went about their business as usual by assuming all responsibility for both manual and automated testing. We showed these testers how other teams were releasing software more quickly due to developers’ participation in test automation.

Initially I met with each team once a month to check in on how well they were adopting the Quality Maturity Model behaviors. With dozens of teams, however, this became too onerous for one person. The solution was to create a cadre of volunteers called the Category Quality Leads. Each Category Quality Lead adopted between three and six teams in their product area and met with their teams each month. They made sure that teams were creating goals related to the Quality Maturity Model each quarter and were actively working on those goals. The Category Quality Leads additionally met once a month to share challenges, ideas, and successes with each other.

After the initial excitement of the new initiative had worn off, it was necessary to find ways to keep the momentum going so teams wouldn’t lose interest in adopting the Quality Maturity Model behaviors. During the first year, we did this by sharing team success stories, such as the story of the team that went from regular deployments once a month to regular deployments every two weeks, and the story of the team that lost both their testers for a sprint and were able to accomplish all their testing tasks anyway. During the second year, we kept momentum going through monthly videos and dashboards showing our progress, and publicly praising the most improved teams in company meetings.

The Results

To measure the effectiveness of the program, we first measured how well teams were making progress with achieving the Quality Maturity Model behaviors. We also chose some key indicators to assess our progress.

At the end of two years, 100% of teams were making progress with their Quality Maturity Model adoption, and the average increase in the number of quality behaviors adopted was 50%.

As a result of adopting these new quality behaviors, we were able to drop the average monthly number of release-related incidents to below 1% of deployments. Since security issues were found and fixed earlier in the development process, the average monthly number of issues found during penetration tests decreased by 18%. Furthermore, the average monthly number of critical customer complaints was cut by 45%, into the single digits.

Conclusion

Inspired by our initial success, we continue to move forward with our Quality Initiative. With the guidance of our Category Quality Leads, teams continue to set new challenges for themselves to meet more Quality Maturity Model behaviors. Our future looks bright as we continue to improve our quality processes, delivering high value to our customers.

How You Can Create a Culture of Quality at Your Organization

• Define and document what quality means for your organization
• Determine what part each role in your organization will play in your organization’s quality initiative; consider creating a RACI (responsibility assignment matrix)
• Create a Quality Maturity Model (QMM) to use as an objective scoring framework for measuring quality
• Ask each team to assess themselves with the QMM
• Ask each team to create a Quality Strategy that outlines how the team will support quality in their software development process
• Ask each team to create quarterly goals to help them adopt the QMM strategies
• Assign a quality champion to each team to help promote the QMM process and support the team in their goals