Data-Driven Testing using Python and the Parameterized Module

When we need to run the same tests on multiple data sets, creating an individual test for each data set is out of the question — unless we enjoy tedious, repetitive tasks. To make this process less monotonous and time-consuming, we use data-driven testing that allows us to execute tests for a set of given data with a single test script.

Working with a data set in product development often requires changes such as editing, moving, adding or deleting data. Because of this, running tests on hard-coded data each time can become challenging. Testers have to create multiple test scripts or modify existing tests multiple times and run them individually. Such tasks can easily become boring and cumbersome, also affecting the pace of development.

Luckily, we can use data-driven testing which lets us manage multiple data sets with a single test script.

What is data-driven testing and how do we handle the data?

Data-driven testing is a framework in which test data is stored in an external location and kept separate from functional tests.
Instead of using the same hard-coded values each time the test runs, in data-driven testing, test data and output values are read from files that can be stored in single or multiple data sources such as comma-separated values (CSV) files, Excel sheets, XML, and databases. This allows automation engineers to have a single test script that can run tests for all the test data.
Data-driven tests run the same test logic and assertions, but they fetch different input data each time which increases the speed and productivity.
They perform operations in the following sequence:

• Fetch the test data from a source
• Use inputs for automation, e.g fill a form
• Verify results
• Continue testing with the next set of input data

When should we use data-driven testing?

Imagine a scenario in which you want to test a login form with multiple input fields using hundreds of different data sets, containing various emails, usernames, passwords, dates of birth, etc.
To test this, you can create one script for each data set and run each test separately, which means creating thousands of tests which will all be almost identical. So, if you want to change anything, you’ll have to do that manually and update each individual test. Not to mention that you’d have to run the script thousands of times.
As you can see, this approach is ineffective and would probably be a nightmare to maintain.

Instead, you can use data-driven testing to import the data from an Excel sheet, for example, then fetch test data one row at a time, and finally, execute the test for all the test data in a single script.

Apart from organizing multiple data sets in a single file and separating test cases from test data, data-driven tests also help automation engineers to reuse the same scripts numerous times with different data sets each time. These tests also reduce manual effort and unnecessary duplicates of tests and improve test coverage, maintenance, and efficiency.

Of course, the quality of the data-driven test will depend on the automation skills of a tester and the whole team. Also, data-driven tests may still require a large amount of coding, as well as time required to create and maintain data files, but these are small drawbacks considering all the benefits of this type of testing.

Are parameterized tests and data-driven tests the same thing?

Both terms are often used interchangeably, but they’re not the same thing. Parameterization is a part of the data-driven testing concept. Simply fetching test data from the external source is not enough to run our data-driven tests. In order to pass multiple data to the application, we also need to parameterize our test scripts.

Typical tests are static, which means they usually contain hard-coded values that are difficult to modify. They also use the same data sets and always work in the same predefined manner. So, we cannot simply re-use these tests, which is a hindrance when working with large data sets. For example, look at the code below:

Here, we have two methods that send a text to the username and password fields using the hard-coded values. To test each combination, we need to change the parameters in .send_keys() each time we run the test, i.e. username2 instead of username1, password2 instead of password1, etc.

Alternatively, we can define only two variables named username and password and then change their values for each test run. Or instead of creating variables, we can add values directly into .send_keys(), i.e. .send_keys(‘tomsmith’).

This method might not seem to be a problem with three username and password combinations. But, what would happen if you need to test 158 different combinations? You get the picture.

To make tests more efficient and manageable we need to parameterize them, i.e. replace all hard-coded input values with parameters. This allows us to run the same test over and over again using multiple data sets and different values.

Test parameterization depends on the programming language. We’ll explain how to use parameterized, data-driven tests with Python and the unittest framework below.

Data-driven testing using unittest and Python

Python’s unittest framework doesn’t have a simple way to run parameterized test cases because it doesn’t allow us to easily pass arguments into a unittest.TestCase class from the outside.
What a great way to start this demonstration, eh?

However, we can use a third-party module called parameterized, which allows parameterized testing with any Python test framework, including unittest.

We don’t want to test anything complex here, like thousands of different data sets that we mentioned before. Instead, we’ll use three sets of credentials stored in a CSV file to test this login form. The goal is to read data from the CSV file and use it in a real test.
The three sets of credentials will include one true and two false login attempts. Since the login page that we want to test suggests to use ‘tomsmith’ for the username and ‘SuperSecretPassword!’ for the password, we’ll store that info with ‘true’ in our comma-separated value file that we’ll name ‘data.csv.’
We need two more credentials that will include the wrong username and password, as well as ‘false’ value for each, so we can put anything other than correct credentials in our CSV file.
Keep in mind that the CSV file format can be slightly different for each test depending on the test requirements. The file may or may not contain the header information, values may be placed under single or double quotes, so the actual data may require some additional operations.
So, we now have a page we want to test, and the three sets of credentials stored in a CSV file.

In the next step, we need to take care of a testing framework. To skip all the boring code setup part, we will use Ranorex Webtestit, the testing toolset that will generate the unittest environment and all required parameters in an instant.
All we need to do is to create a new project in Ranorex Webtestit and allow it to finish the setup for us.

Then, we’ll create the ‘data’ folder in the project tree and add our ‘data.csv’ to it.

To test the data from our CSV file we need to download the parameterized module.

For this guide, we’re using the latest version of the parameterized module at the time of writing, which is 0.7.0. So, we’ll add the lineparameterized==0.7.0 to the ‘requirements.txt’ document in the project tree as shown in the image below. After hitting CTRL+S the dependency will be automatically installed.

Next, we’ll create a Page Object file named login_page_object.py. Ranorex Webtestit will again setup the code, so all we need to do is to add the elements for the username, password, and login success/fail message in the Page Object file, using the Ranorex Selocity extension that makes finding selectors and adding elements a lot simpler and faster.

The added elements will have generic names that we’ll change to _username, _password, _login and _result_message.

Now, Ranorex Webtestit allows you to simply drag and drop an element into the Page Object file to create a method reflecting the action. Using this approach, we’ll drag the elements we added and create appropriate actions for each. For _username and _password, we’ll select Do > Type into element from a drop-down menu. For _login, we’ll choose Do > Click on element. Finally, we want to view a text for _result_message so we’ll select Get > Element’s text.

After we’ve created actions, our Page Object file will look like this:

In the get_login_message() method, we used .replace() and .strip() because of the presence of a special character ‘×’ in the message text, that we don’t want to deal with when writing our assertions later.

After we created a Page Object file and added all elements to it, we can proceed and create our data-driven test.
Right-click the ‘tests’ folder in the project tree and select ‘New’ > ‘Test file’. Ranorex Webtestit will generate a new test file with an empty stub with comments explaining how to use the AAA pattern to create your test.

After our test file has been created, we’ll import our LoginPageObject file, CSV and parameterized modules by adding the following lines of code at the beginning of our test. Here, we have used Python’s built-in open() function extracted the data from the CSV file and stored it into the TEST_DATA variable:

We’ll also add some code* to show the test case names in the report:

*Note: Check the project’s page for further info on how to use the parameterized module.

Finally, we’ll write our test that will open the login page, enter the username and password, click on the login button, capture the login message, indicate the wrong username/password combination, and assert the values of each login attempt. The test takes the username, password and should_pass extracted from the CSV file and uses them in the actual test run.
Below is the code of our test, with comments explaining each step:

In the ‘Assert’ step, we have created a custom assertion message using the iteration_indicator that will inform us about the details of the data set that failed/passed the test.
In the end, our test will look like this:

Before we can run our test, we need to create an endpoint to configure the test runtime environment, such as browsers, operating systems, and devices to use. In the ‘Execution’ tab, click ‘Add endpoint’, and Ranorex Webtestit will take care of the webdrivers for you.

Since our CSV file contains three sets of credentials, our test will execute three times; once with correct credentials expecting the ‘true’ value and then two times with incorrect ones, expecting the ‘false’ value. The test will evaluate the login form with all the credentials and assert that the proper message is shown for each login attempt.

Ranorex Webtestit will generate a report once the test execution is finished, showing that all three test runs were successful, as we got expected values for each set of credentials.

The test sample we just explained in this article can be download as a sample project directly from within the Ranorex Webtestit application. You’ll find it under the ‘Samples’ category of the Welcome screen.

This sample project is also available for download from our Git repository.

Conclusion

In this article, we described how easily you can create a data-driven automated test with the parameterized module by adding it to the unittest framework generated by Ranorex Webtestit. Here, we explored three test scenarios using only three data sets. However, the same approach can be used even if you want to test files and databases containing thousands of different inputs.

You can see how simple the whole process is if you download a free, fully-featured trial of Ranorex Webtestit.