Which test scenarios do we automate?
Continuing with our blog series on automation testing, we first talked about how to make use of manual testing and automation testing together to make the best out of it. In this post, we’ll talk about something that relates specifically to automation testing.
Over 10 years of experience in quality control has taught me, that mobile and web applications test automation has numerous advantages compared to manual testing. For one thing, automation provides authentic, reliable, and speedy results but more importantly, it eliminates the need to execute test cases manually every time new changes are deployed to production.
That being said, one of the main problems in automating test cases is deciding which test cases to automate and which test cases to not automate? How do we even make those decisions?
We’re going to list down some major attributes of a test case that will help us make that decision.
In this article, I have tried to list down all the methods we at Yapsody have employed in identifying which test scenarios to automate and which ones do not.
Candidate scenarios for automation
Let’s start with the attributes which will help us decide if the test case is to be automated.
Test scenarios that are part of the core of the product
There are many E-commerce sites and apps in the market and the common trait shared among them is the payment system. The payment system is the most critical aspect of any E-commerce website. To make sure this part of the system is always working as per requirements, we need a reliable and fast way to execute these test cases and test cases about system core must be automated.
Data-intensive test cases
Ever executed those test cases manually which had too many pre-requisites or required too many user inputs before you could start executing them? Those test cases consume too much time and that too every time whenever regression testing is coming up pending a production deployment. It is always a good idea to automate these test cases. Take an example of a form that asks for a user’s postal address. There are going to be at least 8–10 input fields with maybe different validation rules for each input field. By writing automated test scripts for test cases about this form and feeding it pre-arranged test data as per test requirements, we’re looking at saving a lot of time, effort, and money.
Agreed, at first, when these test cases are in process of being automated, they’ll consume a lot of time but also consider this. How much a few hours of your time as an initial investment will save you from the trouble of executing these test cases over and over again in the future?
Repetitive test cases
Ask this question to any manual tester working in the field for more than 2 years. Are you not bored yet? The answer will most definitely be a yes and you won’t even need to know the reason why they are bored because you already know the answer to that. They’ve been executing those same test cases day in, day out. Automate these test cases. If a set of test cases are to be executed frequently and repeatedly they are first in line to be automated. A whole lot of positive ROI is achieved in automating these test cases. A simple graph below represents advantages in terms of cost savings in automating repetitive test cases.
Stress testing scenarios
You wouldn’t hire tens of thousands of people to performance test your application, would you? JMeter for instance is an industry-standard when it comes to stress testing. We, at Yapsody, have used JMeter extensively for our load testing requirements. Even though JMeter’s user interface might not be appealing for many, it compensates with being open source and a large user community online.
Browser dependent test scenarios
For a web application, there must exist a set of test cases that checks for compatibility of the application in different browsers. It is not feasible to test these test cases manually on different browsers. Imagine the number of hours and resources it would take to execute them. It is prudent to automate these test cases as automation tools provide options to run the same set of test cases parallel on multiple browsers.
Candidate scenarios against automation
Test scenarios of product that is constantly changing
When a product is under development, it is not a wise decision to perform automation testing on it. The best way to go forward is manual testing as it is like the software development life cycle that changes will come into the picture. As the product changes, the automation scripts written for it become obsolete, and thus the time and resources invested in writing those test scripts also go to waste. Make sure that only those test cases are automated which belong to the stable version of the product.
Low return on investment
The whole point of automation testing is to increase return on investment in terms of cost and time in the domain of testing. If automating a test case isn’t providing either of them then it is of no use to convert those test cases to automation test scripts. E.g. A form that asks for an email address from its users to subscribe to newsletters does not need automation as it is neither a core part of the product nor is it data-intensive and thus its presence in the test suite may not be justifiable.
Test case execution where domain expertise is required
Machine learning algorithms such as some of the leading Chess engines evolve over time as they play against each other or against human players. Testing such entities is no easy task as the testing doesn’t always return the same answer because results are more than a simple pass or fail and automation of test scenarios for such entities simply doesn’t cut it as automation tools don’t deal in fuzzy outcomes. What is needed here to test such entities is subject matter expertise which only a human can provide.
Summary
So as we see, it is really not about automating everything you have; it is about being thoughtful about your return on your investment. In the next blog in this series we will talk about the manual testing scenarios and why manual testing is still going to be there in our lives!
Thank you for taking the time to read till the end. If you happen to stumble on this post and like it, please like or subscribe to encourage content writer to publish more of such content. It helps more than you can imagine.
Cheers!!
