Determining The Priority for Test Automation
This story is co-authored together with Sivamoorthy Bose
Determining The Priority for Test Automation
As a First Step of test automation, most of them determine the need for an automation tool but before that identification of priority and feasibility of test cases is very important. Every aspect should be considered while analyzing the feasibility. It is also essential to perform a feasibility analysis on the manual test case pack that allows automation engineers to design the test scripts.
In this particular stage, the following questions should always be asked.
- Which modules of the applications can be automated and which cannot?
- Which tests can be automated and how do you automate them?
- Factors like cost, team size, and expertise should also be considered.
What to automate? What to explore?
Many software systems are quite complex. Sometimes we don’t understand or don’t have data regarding how the feature is being used by the end-users like it is shown below:
The test automation uses the principle of checking. It addresses the question “Does the system do what it is supposed to do?” It can bring the biggest value when it covers known knowns. When you know the clear use cases of a feature and you have data proving it (from google analytics/ AB testing or other researchers) it is the best scope to be automated.
What about other risks and system vulnerabilities that we are not aware of? This can not be automated, it should be discovered first by human-centric testing.
What about unknown knowns or known unknowns? Those areas should be researched first by humans. If we would “delegate to robots” those areas the tests can be useless. No one can use this feature or we could have a wrong hypothesis regarding how it is being used.
Lisa Crispin and Janet Gregory also advise to leave risks and vulnerabilities on their “How to Implement Testing and Agile in a Company” webinar (watch recording). They also introduce the term “human-centric testing” instead of manual testing to value human effort of mindful product discovery. We also share this mindset.
However, should we really automate all known knowns? Of course no, let’s dive into this deeply.
What kind of tests should NOT be automated?
- Tests that need to be executed only once.
- Tests that need to be verified with human discretion.
- Tests that are very quick to execute.
- It’s not always suggestible to automate tests during usability tests.
- Test that takes more time in automating than in executing manually.
What Test Cases to Automate :
Business Critical Flows :
The test cases or functions that are mostly used by the end-users in PRODUCTION. We have to make sure these cases are automated and tested thoroughly in regression.
High Priority Progression test cases:
Automating Critical or high priority progression test cases helps to cover the majority of code changes.
Test Cases to test with Large Data :
Any test cases that need large data input has to be automated, which would save huge ROI when compared to Manual effort.
Smoke or Sanity test cases:
The major test cases that have to be tested for every build has to be automated and these tests should be executed daily.
Complex Cases :
Automating complex test cases is almost a rule. Again, you do want to be on the safe side and to save your time when it comes to complexity. If you have some routine and long to execute tests, where humans can make mistakes, automate it as well. Test automation is a tool that helps to speed-up the testing activities.
Test Cases to be executed on Cross browsers:
Automation can be done for those test cases that have to be tested on multi browsers, typically browser capability testing.
Automatable Cases:
Validate test cases that can be automatable, look for automation feasibility with respect to the cost of automation, complexity, and reusability.
Maintain Smaller Test Suite :
Maintaining a smaller automated test suite is always important, since regression testing often demands that the test suite is executed with every build and for every change made to the project. Splitting the test suite into smaller chunks is a viable option to reduce execution time. Creating Jenkins jobs for these smaller chunk automated suites helps to integrate into the CI/CD pipeline.
Automate tests with broader Test Coverage :
By using a minimization technique, identify tests that cover the broadest spectrum of faults, prioritize the test cases that can be considered for automation. This would help in faster execution with broader test coverage.
How to start automation? Visualize a Test Strategy as a Team!
We shared some principles of selecting the automation scope from our experience. However, your project is unique and your team knows best what to automate. Even if you already have some automated steps, it is always useful to take a step back and think about priorities and on what priorities to focus on next. Using visual models would help you to get the whole team involved.
Test Automation Pyramid:
One of the well-known tools is the test automation pyramid. Usually, it is divided into 3 sections of unit, API and integration as well as E2E tests. Read about it more on Siva’s blog. Discuss with your team what tests you can “push lower”.
Testing Quadrants:
One more thinking and conversational tool is testing quadrants. It can help to understand what types of testing are needed and in which order they should be performed.
The key idea is to define 4 quadrants depending if the testing type is business/technology facing or if it is guiding development/critique of the product. After filling collaboratively the quadrants the team can define the priority order depending on the project needs.
Read more about the tool on Lisa Cryspin’s blog (image is taken from there).
Test automation is not a goal, it is a tool to achieve the goals of your team. A smart definition of scope to be automated and test strategy can help you to become more efficient and manage better the balance between its value and maintenance cost. Visualizing your thoughts in a team is also extremely helpful. Try it out, you would get a lot of value!
Useful references:
