Intro to test framework Pytest

In this article we will learn about pytest, a python based test framework.

Before proceeding forward it is recommended to read Part 1 of this series which explains the basics of automation-

Getting started with API Automation: Part 1- Overview

Step 1: Installation-

We need to install python, either version 2 or 3 as pytest supports both versions. With python installation comes pip- a package management system used to install and manage software packages written in Python.

To install pytest we will use pip command-

pip install pytest

That’s it, pytest is installed and we can move forward with writing tests using pytest.

Step 2: Pytest syntax for writing tests-

• File names should start or end with “test”, as in test_example.py or example_test.py.
• If tests are defined as methods on a class, the class name should start with “Test”, as inTestExample. The class should not have an __init__ method.
• Test method names or function names should start with “test_”, as in test_example. Methods with names that don’t match this pattern won’t be executed as tests.

Now, that we are clear with the naming conventions let’s create the project directory-

- demo_tests/
    - test_example.py

Step 3: Write basic tests in pytest

In file test_example.py, let’s write a very simple function sum which takes two arguments num1 and num2 and returns their sum.

def sum(num1, num2):
    """It returns sum of two numbers"""
    return num1 + num2

Now we will write tests to test our sum function-

import pytest
#make sure to start function name with test
def test_sum():
    assert sum(1, 2) == 3

To run this test, traverse to demo_tests directory and run command:

#It will find all tests inside file test_example.py and run them
pytest test_example.py

Test run result

Here each dot represents one test case. Since we have only 1 test case as of now, we can see 1 dot and it passes in 0.01 seconds.

To get more info about the test run, use the above command with -v (verbose) option.

pytest test_example.py -v

Test run result with verbose mode on

Add one more test in file test_example.py, which will test the type of output which function sum gives i.e integer.

def test_sum_output_type():
    assert type(sum(1, 2)) is int

Now, if we run the tests again, we will get output of two tests and test_sum and test_sum_output_type

Till now we have seen all passing tests, let’s change assertion of test_sumto make it fail-

def test_sum():
    assert sum(1, 2) == 4

This will give result along with failure reason that we expected sum(1, 2) which is 3 to equal 4.

This covers the basics of pytest. Now we will dive into some advanced concepts which makes test writing in pytest more powerful.

Step 4: Parametrizing test methods

If you look at our test_sum function, it is testing our sum function with only one set of inputs (1, 2) and the test is hard-coded with this value.

A better approach to cover more scenarios would be to pass test data as parameters to our test function and then assert the expected outcome.

Let’s make changes to our test_sum function to use parameters.

import pytest
@pytest.mark.parametrize('num1, num2, expected', [(3,5,8)])
def test_sum(num1, num2, expected):
        assert sum(num1, num2) == expected

The builtin pytest.mark.parametrize decorator enables parametrization of arguments for a test function. We have passed following parameters to it-

• argnames — a comma-separated string denoting one or more argument names, or a list/tuple of argument strings. Here, we have passed num1, num2 and expected as 1st input , 2nd input and expected sum respectively.
• argvalues — The list of argvalues determines how often a test is invoked with different argument values. If only one argname was specified argvalues is a list of values. If N argnames were specified, argvalues must be a list of N-tuples, where each tuple-element specifies a value for its respective argname. Here, we have passed a tuple of (3,5,8) inside a list where 3 is num1,5 isnum2 and 8is expected sum.

Here, the @parametrize decorator defines one (num1, num2, expected) tuple so that the test_sum function will run one time.

We can add several tuples of (num1, num2, expected) in the list passed as 2nd argument in the above example.

import pytest
@pytest.mark.parametrize('num1, num2, expected',[(3,5,8),              (-2,-2,-4), (-1,5,4), (3,-5,-2), (0,5,5)])
def test_sum(num1, num2, expected):
        assert sum(num1, num2) == expected

This test_sumtest will run 5 times for above parameters-

Notice the parameter values in test_sum[]. It changes each time for each value.

In above code, we have passed the values of 2nd argument(which are actual test data) directly there. We can also make a function call to get those values.

import pytest
def get_sum_test_data():
        return [(3,5,8), (-2,-2,-4), (-1,5,4), (3,-5,-2), (0,5,5)]
@pytest.mark.parametrize('num1, num2, expected',get_sum_test_data())
def test_sum(num1, num2, expected):
        assert sum(num1, num2) == expected

This will also give the same result as above.

Step 5: Use of Fixtures

As per official pytest documentation: The purpose of test fixtures is to provide a fixed baseline upon which tests can reliably and repeatedly execute.

Fixtures can be used to share test data between tests, execute setup and teardown methods before and after test executions respectively.

To understand fixtures, we will re-write the above test_sum function and make use of fixtures to get test data-

import pytest
@pytest.fixture
def get_sum_test_data():
        return [(3,5,8), (-2,-2,-4), (-1,5,4), (3,-5,-2), (0,5,5)]
def test_sum(get_sum_test_data):
        for data in get_sum_test_data:
                num1 = data[0]
                num2 = data[1]
                expected = data[2]
                assert sum(num1, num2) == expected

This will give output-

Notice that different test parameters are not displayed now.

If you look at the above test result, you might get a doubt whether the test ran for all the values because it is not clear from the test run. So, let’s change one of the test data to include errors and re-run the test.

We can see the tests ran for each data and failed at the last one.

Scope of fixture- Scope controls how often a fixture gets called. The default is function.
Here are the options for scope:

• function: Run once per test
• class: Run once per class of tests
• module: Run once per module
• session: Run once per session

Possible scopes, from lowest to highest area are:

function < class <module<session.

A fixture can be marked as autouse=True, which will make every test in your suite use it by default.

Now, we will make use of fixtures to write setup_and_teardown function. The setup function reads some data from the database before the test starts and the teardown function writes the test run data in database after the test ends. For simplicity, our setup_and_teardown functions will simply print a message.

Notice the yield in setup_and_teardown. Anything written after yieldis executed after the tests finish executing.

@pytest.fixture(scope='session')
def get_sum_test_data():
        return [(3,5,8), (-2,-2,-4), (-1,5,4), (3,-5,-2), (0,5,5)]
@pytest.fixture(autouse=True)
def setup_and_teardown():
        print '\nFetching data from db'
        yield
        print '\nSaving test run data in db'
def test_sum(get_sum_test_data):
        for data in get_sum_test_data:
                num1 = data[0]
                num2 = data[1]
                expected = data[2]
                assert sum(num1, num2) == expected

Also notice that we have not passed setup_and_teardown as parameter to test_sum function because autouse=True is already set in the fixture definition. So, it will automatically be called before and after each test run. We need to run the test using -s option now to print to stdout.

pytest test_example.py -v -s

Step 6: Test case Markings

Pytest allows to mark tests and then selectively run them.

By using the pytest.mark helper you can easily set metadata on your test functions. There are some builtin markers, for example:

• skip — always skip a test function
• skipif — skip a test function if a certain condition is met
• xfail — produce an “expected failure” outcome if a certain condition is met
• parametrize to perform multiple calls to the same test function. (We already discussed this above.)

In the above examples, We can mark the 1st test function as slow and only run that function.

@pytest.mark.slow
def test_sum():
    assert sum(1, 2) == 3
def test_sum_output_type():
    assert type(sum(1, 2)) is int

Now, to run only slow marked tests inside file test_example.py we will use-

pytest test_example.py -m slow

Step 7: Cheatsheet to run pytest with different options

# keyword expressions 
# Run all tests with some string ‘validate’ in the name
pytest -k “validate”
# Exclude tests with ‘db’ in name but include 'validate'
pytest -k “validate and not db” 
#Run all test files inside a folder demo_tests
pytest demo_tests/
# Run a single method test_method of a test class TestClassDemo 
pytest demo_tests/test_example.py::TestClassDemo::test_method
# Run a single test class named TestClassDemo 
pytest demo_tests/test_example.py::TestClassDemo
# Run a single test function named test_sum
pytest demo_tests/test_example.py::test_sum
# Run tests in verbose mode: 
pytest -v demo_tests/
# Run tests including print statements: 
pytest -s demo_tests/
# Only run tests that failed during the last run 
pytest — lf

This covers the basics of pytest and after reading this article you should be good to start writing tests in python using pytest.

You can find the entire code used here in my 👉 github repo.