Taking a phased approach to adopting Quality Assistance

Faced with a rapidly growing development team (we’re always hiring!), increasing product complexity, and more users signing up every day across three markets, ClearScore QA needed to change the way we thought about Quality Assurance and test ownership. In March of this year, we started on a journey towards full Quality Assistance and came up with a phased approach to adoption that would allow teams to make the change without slowing momentum or requiring a hard-reset on existing processes and relationships.

What is Quality Assistance?

Quality Assistance is a term first popularised by Atlassian to describe the eventual QA model that they settled on after iterating through various stages that attempted to reduce the friction and time it took to get things from development into release. It promotes the idea that in an ideal world, developers should write code and release it to production themselves without the need for any separate function (i.e. Quality Assurance) to be involved in quality checking the work going out.

In an ideal world, there would be no need for a separate function to verify work being released

Atlassian describes the organic evolution towards, and subsequent adoption of, their Quality Assistance model in this 20-minute video better than I ever could, so it’s well worth watching!

How is this relevant to ClearScore?

We’ve come a long way from our stated target of having 1 QA for every 2 Developers on a team (we’ll be 1 for every 5 by the end of 2019) and we’re no longer doing the 10 releases a month we were doing at the end of 2016 (it’s more than 10 times a day!). Add to this the fact that every release has the potential to impact 10-million users across 3 markets and it’s no wonder that our approach to QA that served us so well for the past 4 years is starting to show some cracks.

In March of this year, we decided it was time to revisit our existing approach to testing and quality assurance and put together a new QA Strategy that would allow ClearScore to continue to scale its development team and release ever more frequently without having to double the size of its existing QA team. We even came up with a stated goal:

We’re looking for a QA model that can help ClearScore continue to scale, release more features, more often, and to more people in more places. We don’t want to simply maintain quality levels but continually raise them.

The approach we settled on was Quality Assistance and borrows heavily from Atlassian’s model (if it ain’t broke…). Rather than go through the 6 or more iterations that Atlassian went through to reach their current model, we’ve come up with a simplified, phased approach to adoption that should allow teams to shift gradually towards what can be a significant shift in mindset, rather than expect people to immediately abandon current ways of working.

The phases

Below are detailed the identified phases through which all teams at ClearScore aim to progress in order to transition to a true Quality Assistance approach.

Phase 0

Phase 0 is the default approach to testing that most teams before Quality Assistance were unknowingly following. When moving to a Quality Assistance approach, Phase 0 is the starting point before Quality Assistance practices are applied.

Our approach to testing before Quality Assistance — not terrible, but it can be better!

Active ingredients

Features and bug-fixes move from Product → Developer → QA → Production, often going backwards for rework. In this phase, developers write the code for the feature and QA engineers test the feature. Although developers may also write tests, it’s clear in this phase that QA engineers are seen as responsible for testing and have little if any visibility of what developers have tested before the story is handed over.

Common symptoms

• Acute QA bottleneck — multiple tickets are often “waiting for QA”, usually in a Ready for Test or In Test column.
• Tickets sit “In Test” for a long time, relying on QA to manually check the requirements of the feature/bug-fix have been met and/or to write automated tests for the feature.
• Automated tests are usually written after the feature/bug-fix is released if at all, reducing automated test coverage and increasing technical debt with each release.
• QA engineers handle all releases to Production.
• QA engineers identify when tests break as a result of the code change.
• QA engineers regularly find issues and have to pass the ticket back to the developer to fix.
• QA engineers often find issues that need input from the Product Owner to define the expected behaviour.

Phase 1

Reaching Phase 1 is the first step on a team’s adoption of the Quality Assistance model. Many of the changes at this phase are tweaks to the agile delivery process and borrow from Behaviour-Driven Development (BDD) and Test-Driven Development (TDD) methodologies.

Active ingredients

Teams use the BDD process of Example Mapping to align on and capture requirements for all Stories that aren’t self-explanatory. Developers perform a Kick-off with a QA engineer when they pick up a new story and hold a Review when development work is complete. Developers take charge of some smaller releases but the majority of testing and releasing of complex stories during this phase is still owned by QA in the traditional sense.

In Phase 1, the majority of testing is still performed by QA

How to apply

• The team runs Example Mapping sessions with the Three Amigos for complex stories in order to align on the specific requirements and behaviours of the feature.
• Developers perform Kick-offs with a QA Engineer before any code is written to reach agreement on what is going to be tested to prove that the feature/bug-fix has been delivered.
• Developers hold Reviews with a QA engineer once they are confident that they have delivered the requirements of the feature/bug-fix.
• If the QA engineer is happy at this point that there are tests in place to cover any new/changed functionality, then the developer should carry out the release.
• If the QA engineer is happy with the demo provided by the developer at the review stage, but there are no automated tests to cover the new functionality/fix, then the developer can assign the In Progress story to QA to write tests and perform additional testing to build confidence before release.
• The team no longer uses an In Test column as testing is considered part of the development stage of the Story/Bug.

Provides effective relief from:

• Rework — key team members are aligned on what the requirements are before any code is written and review the requirements.
• Acute QA bottleneck — tickets have to go through the review process between QA and Development before they are either released or left with QA to complete testing and release.
• Tickets no longer go all the way back to the Product Owner for clarification after the Developer has had a first attempt at the feature/fix.

Mild symptoms that could persist:

• Mild QA bottleneck — larger stories and some bug-fixes may still require additional testing from QA to provide confidence or to write new automated tests.
• QA may still spot bugs at the review stage, but this is less common than in the equivalent stage in Phase 0, and fixes should be quicker to implement and therefore less costly.

Phase 2

Teams that have mastered and become comfortable with Phase 1 can start to take ideas and elements from the Phase 2 approach. In this phase, there is a more pronounced shift away from QA owning testing towards developers taking complete ownership of the release of new features and bug-fixes into production.

Quality Assistance Phase 2 affords QA more bandwidth to assist the team in delivering more

Active ingredients

At the Kick-off, developers and QA engineers discuss not only what is going to be tested but how it is going to be tested — agreeing on what automated tests are going to be written and where those tests should live. Developers take complete ownership of testing, including the writing of new tests and the maintenance of existing tests. Automated tests are demoed to QA as part of the Review after which the developer should have the confidence to release into production. Stories and bugs rarely, if ever, get assigned to QA in Phase 2 teams.

How to apply

• Building on Phase 1, the team continues to run Example Mapping sessions on all complex requirements (stories and bug-fixes) before they make it into the sprint.
• The team runs Kick-offs and Reviews between QA and Development for every story and bug-fix.
• During the Kick-off, QA engineers and developers agree on what is going to be tested as well as how it is going to be tested.
• Developers write new tests and update existing tests for all new features/fixes.
• Developers demo the passing tests during the Review stage and add new tests if the need is highlighted by QA.
• The release of any code is carried out by the developer immediately after the Review, at which point the Story or Bug is considered Done.
• In the case of native iOS or Android development, where an immediate release to Production isn’t possible, the developer will merge to master following a successful review — ensuring the feature/fix makes it into the next release.
• Stories and bugs are only assigned to QA in exceptional circumstances.
• QA engineers can act as a second pair of eyes to help with debugging during the development stage.

Provides effective relief from:

• QA bottleneck — because stories and bugs are not assigned to QA when they are “ready for test” or even “ready for release”, there should be no QA bottlenecks.
• Rework — any issues are discovered at the point the code is written, in some cases, issues may be found during the Review but should be able to be addressed quickly.
• Going back to Product — because the developer is writing the tests based on the agreed requirements, and demoing those tests at the Review stage, there should be very few instances of ambiguity that needs to be settled by the Product Owner.

Phase 3

Of all the phases, Phase 3 represents the biggest shift in terms of mindset and day-to-day work for QA Engineers. Working within a Phase 3 team, QA engineers are now free from looking for bugs in newly written code and can actively move towards improving the quality of existing ClearScore products through monitoring in Production. Ownership of test automation in Phase 3 teams sits firmly with developers. Assuring quality has now transitioned from a tactical and reactive task to the strategic and proactive pursuit of quality.

Phase 3 teams can operate with a much higher Developer to QA ratio

Active ingredients

QA makes it a habit to constantly monitor the quality of the ClearScore product in Production. This includes reviewing monitoring tools daily to identify potential issues with the product and any new releases that have gone out. Reviews and Kick-offs still happen within the team, but seasoned developers can kick-off and review with each other without the need for QA. QA engineers continue to take part in Example Mapping sessions. QA devote time to ensuring the release process is optimised for speed and quality and performing root-cause analysis when bugs make their way into production to reduce the risk of future releases.

How to apply

• The team continues to apply the aspects of Phase 1 and Phase 2, such as Example Mapping and Kick-offs and Reviews.
• Developers experienced with Phase 2 Quality Assistance start to assist other developers, kicking-off stories and reviewing before releasing.
• QA engineers review production monitoring tools (Sentry, SonarQube, SignalFX, Amplitude etc.) daily for quality issues.
• QA engineers identify areas of bad quality and technical debt and highlight them to the team to address in future sprints.
• QA engineers devote more time to improving the speed of releases and reducing risk.
• Developers take full ownership of tests and existing test frameworks.
• QA continuously spike new tools for monitoring and testing that could improve quality.
• QA engineers continue to assist Developers in understanding, reproducing, and debugging complex bugs.
• QA continue to contribute to test automation in a more strategic sense, refactoring for performance and filling in existing gaps in testing.

Provides effective relief from:

• Manual testing — automated test coverage improves with each release to a point where no manual testing is required for the majority of releases.
• Long-standing bugs in production — QA’s mission to continually raise quality levels is most visible in this phase where QA engineers are helping identify and prioritise user issues.
• Subjective quality measures — the more the team is measuring and monitoring, the better understanding we will have of objective quality.

Self-diagnosis

Because reading a blog post each time you want to work out which phase you’re in is not really practical, we allow and expect squads to self-diagnose which phase of Quality Assistance they are in and map their own journey to the next phase.

Which phase is ClearScore in now?

One of the major benefits of a phased approach to implementing any radical change is the ability for teams to adopt at their own pace. The strategy was intended to be an evolution of our existing delivery processes, not a complete revolution that would likely disrupt the business’ ability to continue to deliver at speed.

Having codified our new QA Strategy around the beginning of April 2019, we began promoting it internally, asking teams to adopt the process changes of Phase 1 by mid-July — something most teams achieved, with the rest at least partially adopting the processes for that phase. Since then, we’ve been pushing those teams that have mastered Phase 1 to move towards Phase 2, and now have several squads operating at Phase 3.

We’re learning as we go, and inspecting and adapting to find what works and what doesn’t. Like everything at ClearScore, nothing is set in stone — if we find something that doesn’t work or could work better, we will change it — Quality Assistance is no exception!