Modern/Agile Software Testing Framework
Introduction
Imagine a Software Testing Life Cycle in which the business stakeholders write a Test Plan that the QA team could use for their Test Preparation phase.
In addition, during the Test Execution phase, Automation QA would help run the classical functional Test Plan along with the Regression suite; while Manual QA could focus on the Exploratory strategy to find real time bugs.
Sounds interesting, right?
This is definitely the dream Testing Life Cycle for any company… but could it be a reality? Yes, it can be done!
What are the challenges with the classical deployment pipeline?
We should admit that the impetus behind the creation of this flow can be attributed to two main reasons:
1. Lack of Time for Test Preparation Phase
First of all, Test Preparation Phase plays a key role during the STLC as far as writing test cases, planning the coverage, and scheduling the releases.
- In an Agile environment, with CI/CD deployment pipeline and continuous deployments, QA colleagues’ have packed schedules on any given day. Devoting more time to the test preparation phase would be very challenging.
- This sometimes leads to a stage where the development is ready but the QA colleagues don’t have time to prepare the test plans, and even if they do prepare them in a rush, these haphazard plans would end up in compromising the release’s quality — which puts everything at risk!
2. Blockers / Red bugs being reported by Management
The second reason is that blockers/Red bugs reported by the management team are often overlooked by the QA Team during the Deployment pipeline — more so for bigger deployments.
- During the Root Cause Analysis, some cases are found to stem from spotty test coverage. These bugs could’ve been easily caught with Exploratory testing.
- So, the basic goal of any QA expert is to find the red bugs during the deployment phase- but these are sometimes overlooked since with a classic deployment pipeline, bigger releases will always loom.
- So we wanted to give more time to the QA team to work on their Exploratory testing strategy during the deployment pipeline by having the proper test coverage during test preparation and also during the test execution.
Key phases to improve in the classical process
We got some hints on the problems. Let’s dig more into the classical Test preparation phase and Test Execution Phases and understand how these bottlenecks are formed.
I. Classical Test Preparation phase
As depicted in the below flowcharts, Classical Test Preparation phase contains around 6 phases — starting from brainstorming the requirements all the way to moving the test case to the regression folder after the release.
- During this flow, the crucial phases are: writing the test cases from use cases, and getting them reviewed with Business as a way to get “four-eye” validations for the test coverage.
II. Classical Test Execution Phase
As shown in the below flow chart for the Test Execution, QA colleagues would perform the Test case execution on different environments. They would start from the manual execution of the classic test plan until the final production validations.
- During this flow, the key phases are executing the classical Test plan, and using the Automation regression pack to get the most out of it.
How we transformed the STLC
We tried to completely automate and sometimes even remove the key phases from Test Preparation and Test Execution phase during the Deployment pipeline.
I. Modern Agile Test Preparation Phase
We tried reverse engineering to streamline and solve Test Preparation Phases. On high level, QA colleagues would write a test case based on a requirement.
- So, theoretically QA colleagues and BA colleagues write requirements using different terminologies.
- Along with this, writing and maintaining a test case from management’s point of view would be a redundant activity since all the information is already available in the form of requirements. So the main question is: “Why can’t all the stakeholders talk in same language?”
- So, if the key challenge is “To use the same language” — which means the language Business is using — how can QA colleagues, Developers, POs and other stakeholders use the same language?
- Getting to the bottom of this issue also unifies the Testing process across all product teams, and thanks to the Engineering of modern frameworks like Gherkin, this flow could be implemented without much groundwork.
The below flowchart depicts the high level differences between the Traditional Flow VS New Modern Agile STLC flow.
The main difference between the two flows are the stakeholders’ involvement and the teams’ roles.
In the Traditional flow:
- QA colleagues are be the main drivers of the Test Preparation phase: starting from writing the test cases, to reviewing them with the business team, and finally moving them to the regression folder.
In the new Agile STLC flow:
- BA colleagues are responsible for the Acceptance Criteria’s/Requirements and these AC’s would be used as Test Cases by both Manual QA colleagues and Automation QA colleagues.
II. Modern Agile Test Execution Phase
On a daily basis, QA teams have to face the challenge of not having enough time for test execution. This is a key parameter for finding the most important bugs before releasing the version to Production!
- This includes running the complete test plan on the supported environments; as well as performing exploratory validations to find the uncovered scenarios in the test plan.
- As a QA Engineer, it would be ideal to have full confidence in every single release. However, if there is no Automation support or no validations done with Exploratory testing, reasonable doubts can arise while approving the build. Certainly, this would be the situation of any QA colleague working on a bigger release.
- So, we need to make sure we get the most out of our Automation. One of the frameworks which helps us achieve this is the ATDD framework, which focuses mainly on using Acceptance criteria scenarios to drive the development.
- So the goal is that Automation scripts should be created even before the development starts, so these could be used during the deployment pipeline to validate the new feature/regression cases. This would give manual QA colleagues more conviction in the thoroughness of their testing, and also more time to perform the Exploratory validations.
The below flowchart depicts the difference in the activities a Manual QA colleague would have to perform during the deployment pipeline.
Welcome to the Modern Agile flow
I. Implementation of Modern Agile Test Preparation Phase
In the above sections, we discussed theoretical implementation. Now, it’s time to discuss real-time implementation. In regards to the UI:
- We designed a basic UI in the Test Management tool / Project management tool.
- The basic idea is that the BA colleagues should be able to work with the libraries / Feature suites. They can create new library suites, add Acceptance criteria(s) or scenarios, modify the existing ones to match evolving business needs, and also remove the redundant scenarios.
Just to give an example:
- Library suite is nothing but the feature on a given product. For instance, Login / Signup is one library suite with all the Acceptance Criteria scenarios related to Login / Signup.
- If there is a new business requirement or if there’s an existing business requirement changes, BA colleague should be able to add / modify / remove the respective Acceptance criteria accordingly in the respective suite.
We designed a plugin interface in our project management tool which would give visibility to all the stakeholders, in particular BA colleagues, so they can work with the library suites.
On high level, these would be the steps:
Step 1: BA Colleagues could start working with the library suites at any level of the ticket like EPIC level / Story / Task / Investigation / Bug etc.
The global idea is whenever there is a new feature to be implemented on a product — a BA colleague would create a new suite from the project management interface and all the respective scenarios would be added to this, in Gherkin language syntaxes.
Step 2: Once the scenarios / Acceptance criteria’s are added to the suite, they would be saved to the branch of the ticket and these scenarios would be visible under the Plugin in the Project management tool.
Step 3: To brainstorm these requirements, there would be a Grooming meeting called “3 Amigo’s meeting” where the changes would be discussed with all the stakeholders of the project BA / QA / Automation QA / EM / LD.
Step 4: Once the Acceptance criteria scenarios are done, BA colleagues would link them to the respective tickets with the plugin in the project management tool.
These scenarios would be visible and transparent to all the stakeholders since they are visible in the project management tool.
Step 5: Scenarios could be directly used by Developers for Unit Testing, Manual QA colleagues as Test Plans for Functional Testing, Automation QA colleagues to write the step definitions.
Important note: These changes wouldn’t be saved to the library folder / Test Management tool unless the deployment reaches the “Tested on Production” status.
This would help us track all the changes of any evolving product. Once the branch is merged, these changes would be updated in the library suite, and so providing all stakeholders with a single “source of truth” about the product requirements.
Below flow chart depicts the high level flow chart of the Requirements / Test Preparations.
II. Implementation of Agile Test Execution Phase
The main objective of the new Agile Test execution workflow is to give the manual QA team the most time possible to perform Exploratory validations rather than sticking to the classic test plan execution..
This approach would give QA colleagues the chance to spend time on the application like an end user by relying on the automation scripts with functional / Regression coverage.
Tooling
We should thank the evolving engineering team which made this dream flow possible.
- As the Project management tool, we used JIRA and as a Test Management tool, we used Testomat.
- In the Project management tool, we developed a plugin which would give the feasibility to access the libraries/ feature files directly without opening any other Test management or CI/CD tools.
- Whenever there is an update, the changes would be pushed back to library after the merge.
- This would all stakeholders the single “source of truth.”
Any cons
Although, the flow looks very promising, Below are some of the bottlenecks in regards to the Test Execution phase:
- This flow could be implemented only with the teams where there is a strong mix of Automation and Manual QA Engineers. Otherwise, lack of Automation could cause some bottlenecks during the Test Execution phase.
- Projects which cannot be Automated — If the feature couldn’t be automated, then Test Execution flow should be done with 100% manual efforts.
Conclusion
In the fast paced deployment pipeline, this flow gives liberty to both Manual and Automation QA colleagues to express their creative skills. Plus it lets all the stakeholders of project team be confident about the release. This STLC including Test Preparation and Test Execution phases would be an ideal solution for those looking to release better products by finding the Red issues as early as possible during the development phase.
