Test Automation Journey
As agile scrum teams, we value people more than tools and we respect our working agreement. As Quality Engineers (QEs) in the team, our main responsibility is to drive towards a “whole team approach and encouraging automate first” mindset. But how do we exactly begin? How can test automation be easily adopted by anyone, promoting a culture of having continuous automated testing?
As a start, let’s look at this problem statement, goal, and outcome below:
Before we write any automated tests, we must always prepare and present the test strategy to the team. It is important that everyone in the team is able to understand and contribute to these tests since QEs shouldn’t be the only gatekeeper for product quality. The QEs must be willing to accept any (constructive) feedback or ideas from teammates. And of course, this test strategy must align back to the team’s working agreement — it is everyone’s responsibility to ensure that the automated tests pass at any point of time.
After we’re done with the test strategy planning and discussion, it is time to execute it. We should decide on a suitable test stack that everyone is comfortable to work with and here are some considerations before making that decision.
Tests should be easy to set up and use
The set-up should be fast and painless; it should be ready to use within 10 minutes. A simple test or acceptance criteria (AC) should be created successfully without needing anyone to have strong technical knowledge. For example, writing acceptance criteria (AC) using the Behaviour-Driven Development (BDD) format, using Gherkin-like syntax. The test stack should be well-integrated with the product tech stack so as to reduce the learning curve when writing any automated tests and it should also provide for a debug mode when troubleshooting test scripts. A simple README can also help anyone to set-up and run the tests easily in their local environment.
Tests should be easy to maintain
The maintenance of tests should also be painless; writing tests is not a one-off effort since tests may break after modifying some code or configurations. We should consider having design patterns such as Page Object Models (POM) to reduce code duplication in the test scripts and/or data-driven models to store multiple combinations of test data.
Multiple testing tools in one
The ability to wrap around multiple testing tools without the worry of vendor lock-in is also something to consider. A common use case we’ve encountered so far is cross-automating among different web browsers and mobile web and native apps. This way, you can optimise your testing workflows to get the fastest and most comprehensive results.
Tool should be extensible
Sometimes, you’ll need to enhance the built-in helpers/keywords in order to meet your own custom automated tests design (e.g. verifying any click events that can happen after the loading icon disappears). Instead of repeating the test cases whenever there’s a need to perform any click event, you can just reuse these custom helpers.
Great community support
The tool used should be hosted in any open source platform and has at least a group of active contributors. The release cycle of the tool should be no less than once a month.
Once the testing tool has been decided, it’s time to write the tests. However, there are many different types of tests, so which ones should we write first? Most times, QEs will start to automate end-to-end (e2e) a.k.a acceptance tests first considering it gives the most coverage benefits to the business. However, if we don’t strategise the testing approach properly, some blind spots in testing might lead to gaps or issues with the test setup later on in the project. The impact of which could be a failure to detect bugs/issues early on in the development. QEs should not also expect to have 100% e2e test coverage. Instead, QEs should probably consider writing API tests. You’d be impressed with API test execution — it is much faster as compared to e2e tests, more stable, and less flaky. API testing has become even more critical than before and we should leverage them as much and as early as possible. With that being said, we should always develop a reasonable coverage for unit tests, followed by integration tests, and then the e2e tests!
Early feedback loop is one of the important methods to measure our success. We should ensure that the total execution time for each of the test types should be kept within the acceptable waiting time.
What can I do if I really have to run the whole suite of tests for more than 30 minutes of test execution duration? Well, for this case, having parallel execution is the way to speed up the feedback loop. However, the infrastructure cost to run tests might increase. We need to find a balance between speed and cost. So all these requirements should be taken into consideration when deciding on the testing tool.
After test planning, preparation and execution, we’ve reached the most important part on this journey — Reporting. Is the tool’s built-in reporting feature sufficient and easy to interpret? Choosing the right test reporter is a crucial factor to determine how fast can the root cause of the test run outcome be identified. Besides, we also believe that these test results can be translated into living documentation easily.
Lastly, continuous integration (CI) enables stronger collaboration between software engineers. This means that any issues found can be fixed faster in the earlier stage of the software development life cycle. This also gives the opportunity for more frequent test runs without burdening the team with more manual work. Therefore, we need to set up automated tests into the CI pipeline and trigger these tests when anyone checks in their code changes. With this quality gate in place, continuously delivering small application updates will be undetectable to the end users, resulting in happier customers who can benefit from new features or fixes with the release!
Writing automated tests should primarily be driven by QEs, while the maintenance should be the whole team’s responsibility. Due to relatively few QE personnels in projects, we believe in reducing the bus factor by designing test scripts using the same tech-stack as the system under test, and using version control to track changes and ease the cross-functional barrier for team members.
It is important to get the team to start planning the most suitable test automation strategy together. Having automated tests via the CI pipeline is an important quality gate to ensure features release with peace in mind. Implement a trial without fear of failure and keep improving through iterations. Last but not least, do keep up the spirit to continue innovating and challenge yourself along the way.
- Merlin 💛
