How Shift Left Testing Improved Software Development Quality Control at Q4

Rapid growth inspired new processes leading to enhanced quality in software development

Our team doubled its size in just a year and a half. With this influx of new people, ideas and past experiences, it was a good time for us to review and lock down core processes to ensure quality control of software development.

Our goals were to:

• Maintain the engineering best practices already in place and look for opportunities to improve our approach for better results.
• Increase stability, predictability, visibility, and, most of all, consistency in how we build Q4s software products, even as we scaled.

Quality 2.0: from Quality Assurance to Quality Engineering

The pace of change in the software industry demands the quick implementation of new ideas and new approaches to propel growth and achieve results. Software development has long embraced the practice of continuous iteration using agile methodologies. The term “fail fast, fail often” was born here.

We realized that fundamentally, we needed to transform our culture from one of Quality Assurance to Quality Engineering, and so began a project we called Quality 2.0.

We set project goals as follows:

• Increase confidence in new feature development by aligning cross-functionally, with shared responsibility around quality
• Build automation that aligns with our test strategy

“Quality 2.0 was the edge that my team needed for consistently thorough and high-quality features without the need to sacrifice an exceptional output”
~ Vitali Zatroutine, Team Lead, Engineering, Q4

To maintain current engineering practices as we scaled, we decided to invest in three key areas of quality: user story requirements, testing strategy, and development.

User Story Requirements

At Q4, autonomy is a significant factor in our strength as a team. But it is also our biggest challenge when it comes to cross-functional communication. We identified requirement-related issues stemming from a need for improved communications within the team. This led to the adoption of a behaviour-driven development process.

The behavior-driven development (BDD) process aligned positively with our existing development rituals and allowed us to improve collaboration, communication, and understanding from the outset.

Within our old team structure, a Product Owner was responsible for creating the user story, a Developer was responsible for writing the code, and once everything was complete, Quality Assurance (QA) would be responsible for testing.

We removed these silos and implemented a Squad structure to enable a more agile approach and enable BDD.

Squads is a concept inspired by Spotify — wherein small, flexible, cross-functional teams are responsible for the end-to-end delivery of each product.

From a post by Clearbridge Mobile, each squad has no more than eight members, is cross-functional, plans together, and sits together. In practice, it offers many advantages. Squads work in a collaborative, transparent environment and use the strengths of each team member to get the highest quality product to market in the least amount of time.

We made sure the entire team understood the importance of quality and that quality is a shared responsibility held by every member of the Squad, not only QA. We also made sure all software engineering teams knew the importance of architecture and design and all other roles contributing to the product whole.

We then introduced a new format for User Stories to help build empathy with the customer. This involved communicating requirements using realistic scenarios, using Gherkin.

Gherkin is a Domain Specific Language for writing acceptance criteria that has five main statements:

Scenario — a label for the behavior you’re going to describe
Given — the beginning state of the scenario
When — a specific action that the user takes
Then — a testable outcome, usually caused by the action in When
And — this continues any of the other three operators

Together these statements describe all of the actions that a user must take to perform a task and the result of those actions. For example, a real-world Gherkin example might look like this:

Scenario: User clicks the link
Given I am on the homepage
When I click the provided link
Then I should see the link click confirmation

Grounding the acceptance criteria in real-life situations makes it easier for a Squad to understand what to build and why, and then deliver the best solution for our business and our customers.

Our initial results have been excellent; with the entire Squad having a clear understanding of how a new feature should operate.

Testing Strategy

We saw two main challenges within our testing approach that diminished efficiency.

1. Our work struggled under the burden of an excessive amount of manual testing performed sprint over sprint — a side effect of fragile automation tests.
2. Traditionally, the testing phase for a software product happens once development is complete — just prior to release. This created a bottleneck at a critical juncture and meant going back to the teams who were involved at the very beginning to solve issues.

Our team identified that a Shift Left testing model would allow us to scale without sacrificing quality. We believed it would also empower a quality-focused culture that would underpin improved confidence in what we built and released.

Shift Left testing occurs within the development cycle itself, so portions of the product are being tested and iterated on at a much earlier stage in the process. In this way, teams are correcting issues before the product moves any further toward release. It is a far more efficient process from both a time and budget standpoint.

With Shift Left, we began communicating with engineers early in the development process, and focusing on quality from the inception of the task, instead of waiting for errors to surface later on in the cycle. In doing so, we made significant strides in reducing the amount of time spent manually testing each feature, and opened up opportunities to automate a more robust configuration for regression.

“It was easy to adapt to and served as a great set of practices for growing the strength of our engineers.” ~ Najeeb Arshad, Engineer, Q4

Before we started Shift Left, our testing activities were not unified and we did not benefit from each other’s work:

• Engineering was focused on the code test coverage
• Quality Assurance was focused on testing case coverage
• Project Management was focused on feature completeness

This was very obvious by looking at the User Story where there was not much context on how we approached testing. Most of the context was around what we build and how we build, but not much was recorded on what and how we test as a team.

We depended on the manual testing of the quality engineer to build confidence in our product. This led us towards an “ice cream cone” testing model (anti-pattern):

Our new Shift Left testing strategy resulted in a more ideal Pyramid state.

• It allowed us to reuse tests and test automations from different functional areas without duplication of effort.
• Increased use of automations accelerated the whole testing activity with a lower cost

Now, whenever a new product or a major change is introduced, we develop the test architecture while developing the architecture and design. Instead of considering test automation as a bunch of scripts, we encourage engineers to think of test infrastructure to be as important as product infrastructure. The availability, reliability, and stability of test infrastructure has a direct impact on the quality of testing.

Development

Historically, sprint over the sprint, moving faster meant validation was not a continuous process. With developers looking to accelerate task delivery and minimize the time between releases, engineers were dependent on a 3rd party to validate if the build fit the acceptance criteria and worked as expected.

Quality 2.0 empowered engineers to assume responsibility over the quality of the build process, whether in quality of the code, unit testing, or integration tests. The new process encouraged collaboration and testing early and often, and spreading the responsibility for quality across all stakeholders at all critical stages of the development lifecycle.

Testers are now involved early in the development to understand the requirements, architecture, design and functionality; developers are involved to provide contextual information, and to offer creative solutions in how they approach quality, avoid complacency and mitigate defects even before development is completed.

Quality in the development process is an investment in putting the client in the center of the final product experience. Testing early means increased focus on meeting the initial requirements and not removing expected components due to last minute issues, or otherwise managing the impact and cost of escaped bugs.

Benefits

After applying the Shift Left methodology to the development team, we have identified that the quality of our final product is much higher than before, and we have also decreased the number of production escapes.

“As a result of Quality 2.0, we have seen Q4’s product become more reliable than ever, created by a Squad that feels energized and empowered to build the right solution for our customers while meeting tight delivery timelines.”
~ Ashley Wilden, Product Manager, Q4.

At the same time, our automation testing coverage increased significantly which has led to much more efficient testing activities. In other words, we can conduct more tests with fewer resources in a shorter timeframe than before.

With a more efficient process in place, our testing team can now focus on more interesting and fulfilling activities — such as helping to onboard the continuous stream of new hires to R&D at Q4.