Automation Testing: Functions, Tools, Cost & Benefits

Published in

Relia Software

10 min readAug 14, 2019

1. What is automation testing?

Definition

Automation testing is an Automatic technique where the tester writes scripts by own and uses suitable software to test the software. It is basically an automation process of a manual process. Like regression testing, Automation testing also used to test the application from load, performance and stress point of view.

In other words, Automation testing uses automation tools to write and execute test cases, no manual involvement is required while executing an automated test suite.
The main goal of Automation testing is to increase test efficiency and develop software value.

What kinds of functions need automation testing?

There is a fact that we can’t automate everything, things are best suitable to be automated include: login forms, registration forms, and the place where numbers of users access the Software simultaneously can be automated. Moreover, all GUI items, connections with databases, field validations and many-more can be efficiently tested automatically rather than manually.

Benefits of automation testing.

Below is the list of advantages that automation testing offers to us if we can apply it effectively:

Increase productivity.
Saves money.
Increases software quality.
Reduces testing time.
Support various applications.
Increase testing coverage.
Reduction of repetitive work.
Greater consistency.

Different automation testing tools.

There are some automation testing tools/frameworks which will help you have a look and find out the suitable one:
Selenium.
LoadRunner.
HP Quick Test Professional.
Visual Studio Test Professional.
WATIR.
IBM Rational Functional Tester.
TestComplete.
Testing Anywhere.
WinRunner.
SilkTest.

2. Why can’t Record and Play is a game changer?

a. What is Record and Play?

Record and Replay is known as codeless automation, is a way to run tests without programming knowledge. This is done using a tool, like CrossBrowserTesting, that allows you to manually perform actions in the browser and save them as a test. You can then take this test and rerun it with a click, making regression and browser testing infinitely faster.

This means that manual testers or testers that are still new to Selenium can easily create automated tests. In addition, this allows other roles such as product managers, developers, designers, and even marketers participate in testing without having to learn an entirely new tool.

b. Advantages of Record and Play.

Record and Replay is good for:

Individuals with little or no programming knowledge.
Filling in the gaps of Selenium tests and transitioning from mostly manual testing.
Lightweight automation for smaller tests.
Non-technical roles doing one-off tests.
Teams where members outside of QA take part in some testing.

Record and Play is as a teaching tool:

Record and playback tools are useful in training because they provide automated test exposure to traditionally manual QC teams.

Helps to get coordinates/names of objects (low-level).

c. Problems with Record and Play.

It records too much:

When you try to record a sequence of actions that you perform on a website or a desktop application, most recorders will record way too many actions.

It records too little:

The reverse can also be true, depending on the tool or your configuration of the recorder. Maybe you have to hover a dynamic item in a table for something to change inside the application and its id/position/ text changes along with the time.
Maybe next time, this record won’t work anymore.

It records the wrong things:

All recorders need a way to figure out what you meant when you performed a certain action, but they get it wrong a surprisingly large amount of the time.
Let’s take something simple. Imagine recording when you click on a button to save changes in an application. A recording of that might look like this:

Click on button “btnSave” (based on id)
But what if the button didn’t have an id? Or even worse, if it had an auto-generated id like “ab29c5984df79123de”.

Maybe the button has the text “Save changes” inside. That could work. But what if the application was really clever and the button instead had the text “Save 4 changes” inside?

You don’t know how it works, so you can’t fix it:

Commonly, Record and Play tools gather information about the user interface element that you interacted with to build a single Xpath or CSS selector to find the object while running test.

And for above not-in-depth technical knowledge targeted users, it’s not easy for them to change/update the test scripts to change test cases. In situations of in-depth technical knowledge user, they will not be able to rely on Record and Play tool at all.

Reusability and Parameterization Nightmares:

Once you start recording testcases, you’ll quickly discover the need to reuse certain sequences of actions across multiple test cases. But extracting sequences into reusable modules can be a very technical and confusing process.

You might also face the problem of parameterization:
If you manage to extract a “login” process into a reusable module, the next logical step would be to drive that module using data from eg. an Excel sheet. In most cases, you’ll get a sinking feeling when you discover that the only way to do that is to dive into the scripts that were generated behind the scenes and implement the parameterization with code.

It’s hard or impossible to create logic rules:

While testing, you will face many situations where logic rules are needed to function the test:

You may want to get a value in a certain field, and branches of working flows need to create depending on that value.
Or maybe your application contains a dynamically changing number of rows in a table and you want to loop through them, performing certain sequences of actions for each one.
Maybe you even need recursive logic in your test cases, which typically requires both branching and looping to work.
All of those things are entirely beyond the reach of record and playback mechanisms and require you to dive into the code.

Churn Baby, Churn:

Someday, the application/product you are testing starts changing its design a little bit, elements are added, UI is changed, and menus are reorganized. This makes many of Record and Play test cases turn into red and not able to be executed.
You can’t release with those tests being red, right? So you end up being pragmatic, turning off the red tests and recording new ones. And you miss some minor — but important — actions or forget to fix those xpaths like you did the last time.
It’s an endless churn and a yuge waste of time and money.

So that:

For above problems, you might start using a record and playback tool and initially see good progress. But after a while, you are most likely going to hit a brick wall that is impossible to climb.
Record and Play can’t be a game changer.

3. Common misconceptions about automation testing.

Automation will provide you with more free time.

The misconception that automated testing will give you more free time is both true and false. In manual testing, most of the time is devoted to exploratory and functional testing where you would manually search for errors. Once that process is complete, the manual tester must repeatedly go through the same steps over again.

With automated testing, that time is cut drastically. The work for automated testers is instead spent coding the tests and making improvements to these tests repeatedly as adjustments are needed. Once the test is complete, however, automated testing allows for the recycled use of tests so that they do not have to go through this whole process again. In essence, the time spent on the mundane tasks and repetition a manual tester would go through is instead spent focusing on larger, more important issues involving the software you’re developing.

The Cost of Automated Testing is Too High.

At first, the investment in automated testing might seem cost prohibitive, especially if you’re a smaller company. But analysis has shown that, over time, automated testing pays for itself.

As mentioned previously, automated testing frees you up to focus on larger issues such as customer needs, functionality and improvements. Automated testing also reduces the cost and need for multiple code revisions, so over the course of time, the investment pays out.

Automated Testing is Better Than Manual Testing.

The reality is, there is no “better” or “worse” in the automated vs. manual debate, there’s just “different.” Each approach has its own advantages and disadvantages. Manual testing is performed by a human sitting in front of a computer carefully going through application via SQL and log analysis, trying various usage and input combinations, comparing the results to the expected behavior and recording the results. Automated testing is often used after the initial software has been developed. Lengthy tests that are often avoided during manual testing can be run unattended. They can even be run on multiple computers with different configurations.

In the end, both manual and automated testing have their roles, especially if the software you’re developing is too large and too complex to rely just on the manual approach.

Automated Testing Inhibits Human Interaction.

Another common misconception about automated testing is that it undermines human interaction. In all honesty, automated testing is more clear-cut and faster than what humans could do without suffering extensive human errors, so this misconception is understandable.

On the other hand, automated testing provides QC and development teams with more time to focus on big — new — complex features. This creates chances for them to have discussion/interaction while finding a way to implement and test those features.

4. Apply automation testing to Relia’s projects.

Automation testing is the best way to fulfill most of the testing goals with effective resources and time, but no any tools can fulfill 100% requirement. The QC team and automation tools should be backed by skilled staff, so that there will be a perfect combination between automation and manual testing.

Relia Software develops both mobile and web applications, we use the data-driven approach to create data-driven Selenium / Appium tests using Python and Java libraries, test scripts are written in Python and Java programming languages. By using the data-driven testing approach, we can use a single test to verify different sets of testcases or test data by driving the test with input and expected values from an external data source instead of using hardcoded values every time a test is run.

That means we separate the test data from the test logic by replacing the hard coded test data with variables using the data from external sources such as CSV or a spreadsheet file. This also helps in creating reusable tests that can run with different sets of data, which can be kept outside of the test.

Taking below login screen for example:

To test that screen, we prepared following test data in an Excel file:

The above image shows to us test data for 11 testcases to test the login screen of an application which was developed by Relia. To read this data and send it to testcase script, Relia team defines “get_data” function and uses the DDT library:

The “get_data” reads data from Excel file, to be used by ddt library for the “test_Login” function as mention in next steps.

To use DDT library, we need to import it:

To apply DDT to testscript, the class should be decorated by DDT class decorator:

Then, test data is passed to testscript through the DDT method decorators which is parameterized by “get_data” function:

Relia applies automation testing and DDT to regression test and performance/stress test. Below are some simple demonstrations that we have done.

a. Automate regression test.

Regression testing was automated to help us test any already tested application and find defects resulting from changes on both mobile application and web application.

Test script is written in separate classes/methods to simulate the exact steps from manual testing. After that, they are pushed into “Load Test” and “Test Runner” to be executed as below: