Reducing time to release: minimizing the manual Test Cycle
Testing is a very important part of the Software Development Lifecycle. There are a lot of basic guidelines on how to organize the testing cycle, but it is usually up to the team to develop the most efficient testing strategy and deliver results in a timely manner.
Quality Assurance is composed of a complex set of steps, which could be divided into four main parts: Analyzing, Planning, Testing, and Reporting. I would like to highlight “Planning”, as this step paves the path for the whole testing process. In order to plan a test cycle, one of the most time-saving and guiding instruments is a test case. Well formulated test cases have a great impact on the testing phase and writing them is as important as testing itself. Test cases are usually based on PRDs, design specs, user experience, and flow charts. Each feature could be separated into positive and negative flows to cover as many edge cases as possible.
As the application grows and the list of product functionalities expands, the test case tree expands as well. Eventually, it becomes very complex. This is a milestone that should be achieved, but, at the same time, it comes with a pretty big downside: along with the test case tree, the release cycle expands and the time for regression testing significantly increases, which leads to a delay in getting testing feedback and shortens the time frame for potential issue resolution.
So how do you maintain a high quality of testing in constrained time, if automation is still under development? How do you minimize the release cycle and simultaneously not miss out on anything important?
- Reduce the number of test cases in each testing suite by omitting low-risk scenarios
It’s good practice to assign the “type” to each test case. The most commonly used types are Sanity, Smoke, Functional and Regression. This approach helps to group the test cases into test suites and execute them based on priority. The testing process that I find most efficient is executed in the following order:
- Smoke & Sanity test suite (includes the most high-level scenarios and high-risk tests)
- Functional & Regression test suite (includes the main functionality of the application and regressions)
- Sanity test suite (includes only high-level scenarios, as this is the final check before the release)
Test cycle time can be shortened by reducing the number of test cases in each of the testing suites. It can be done by adding one more type called “Monthly”, the main purpose of which is to bundle all the test cases that won’t block the release if their status is “Failed” and execute them once a month separately from the release Cycle. Having fewer test cases in each test suite should significantly speed up the testing process while ensuring quality.
The test case tree needs to be closely monitored for changes in the product requirements and their corresponding impact on the functionality of the application. Adjustments should be done regularly in order to keep the process up-to-date.
2. Determine the combination of test devices and environments based on priority
The important thing to keep in mind is the variety of devices and environments on the market. For example, on the web, customers access the product in any number of browsers and browser versions. Also, there is a large variety of mobile devices, platforms and operating system combinations to consider. Comprehensive coverage means that the release cycle would be endless and inefficient.
Tools like Mixpanel and Google Analytics are used by QA Engineers to analyze the summary of devices customers use the most and set the priorities on what to test first based on customers’ impact.
If testing with a lot of different combinations is critical and necessary, the other useful approach would be to complete the Smoke & Sanity test suite by testing on the wider variety of devices or browsers and then move to the Functional & Regression test suite by testing only on the primary cohort. It should catch the device-specific or environment-specific issues in the early stage of the release cycle.
3. Pre-release testing process: Minimize the QA on RC build
Pre-release cycle testing helps to discover issues early in the process. The goal is to test new features along with the high-risk areas and provide the detailed feedback accordingly. Also, it is important to keep the queue of tickets which are ready for QA verification clear prior to the release cycle.
This approach significantly reduces the chances of critical issues slipping into the release candidate build and leads to a clean and blocker-free release cycle.
4. Use resources efficiently and remove the duplication of efforts
This method is largely related to efficient teamwork. It is very important to divide the work for the release between the QA Engineers since the more eyes there will be on the product, the fewer defects could end up in the released build.
A simple trick to try and avoid duplication of efforts in the early stage of the testing process is executing a different sequence of test cases at a given time. As an example, one team member could start from the bottom of the test suite and another one from the top. This approach will ensure that a potentially critical issue will be found way faster and the results will be provided in a timely manner.
Conclusion
The key benefit of minimizing the time of release cycle for QA Engineers is the additional time gained that can be directed towards the completion of other testing activities, thereby increasing the overall quality of the product, improving test documentation and expanding test coverage. The ability to manage the release cycle and keep the process up-to-date is a valuable skill set for any QA Engineer.
These methods allow Motiv to deliver more value to customers faster as well as save a lot of time for the company. Additionally, it significantly improved the performance of the test team and boosted confidence in a successful product launch.
