pytest.fail in a fixture and in a test

A subtle difference between ERROR and FAIL

I found one amazing property of the pytest recently. If you call pytest.fail inside a test, it’s a FAIL status for that test (or XFAIL, if it has xfail mark).

But, if you call it from a fixture, the test, which depends on this fixture, is getting ERROR state. This is an absolutely marvelous property for use in testinfra, or, more generally, for any side-effect-heavy integration test.

Pure world of unit-tests

When you write a unit test, or some light-headed integration test, you generally has a good control over environment where this test is run. Well, you can’t control if you have pytest installed, or which version of python you are running, but inside of your code you are the boss.

If you said that ‘foo=2’, foo is 2. If you have some quirky class, and you wrote a pile of mocks to test it, your test become less robust, but you are (mock of) the boss. If you said that foo is mock.MagicMock, it is.

In this situation, any failure of your test is about the same as failure of your code. Or, to be precise, the procedure of fixing is the same: open text file, find the bug, replace it with not-a-bug version, save. Job done. If it’s a bug in a test, fix the test. If it’s a bug in a code, fix the code. That’s the code idea of the unit tests.

Dirty side-effected world of integration tests

Now, let’s look to the generic infra test.

You need to check version of the server via its cli.

Your test (with testinfra help) looks like this:

def test_server_version(host):
  with host.sudo():
    assert host.check_output('cli srv version') == '1.2.3'

Generally, you are OK if this server fails if version is ‘1.2.1’. That’s what your test is all about.

With a little doubt you are ok if you get trace and FAIL test if exit code for cli was not zero. That’s check_output is all about.

But… If you get error because sudo is asking for a password, does ‘server version’ test failed? We have no clue. What about connection error? Is it a ‘FAIL’ for the test?

Generally, if you have many tests of this kind, having many of them failing because of transient unrelated problems is troubling. You open your CI, see what tests were failed. You start to think ‘why can I have incorrect version of server? What’s going on?’, to realize later, that you’ve just forgot to update ssh key for CI, or to add it to sudo passwordless group (which is an issue, but it can be purely CI-related).

Basically, having FAIL here is misleading. It’s not a bug in the test, it’s not a fail in the code. Some stinky side cause is to blame.

ERROR to the rescue

Having a different type of ‘not success’ for a test is a bliss. You can reason, that test in ERROR state just wasn’t able to test things you’d like it to test.

Look at the updated version of the same test:

@pytest.fixture()
def server_version(host):
    with host.sudo():
        return host.check_output('cli srv version')

def test_server_version(server_version):
    assert server_version == '1.2.3'

There are two big changes. 1st: our test is as simple as it can be. If it’s in a FAIL state, server version is not 1.2.3. If if’s ok, server version is 1.2.3.

But if it’s in ERROR state, that means, we don’t know what version server has. There are tons of reasons, but none of them is FAIL for the test.

This subtle division between ‘types of failure’ improves clarity of tests, making communication more precise.

assert VS pytest.fail

The last topic is difference between assert and pytest.fail. Pytest does great job of wrapping assert output into fancy clothes. There is a diff, if assert was about ==, some additional clues on why assert happened.

But assert, or any other exception is not always the best.

Consider those two fixtures:

def fixture1():
  data = get_data()
  assert data
  return data

Or this:

def username():
  data = get_data()
  if not data:
    pytest.fail(f"There is no username in get_data (got: {data})")

The second is a pleasure to read:

ERROR at setup of test_bar
...
E       Failed: There is no username in get_data (got: [])

Conclusion

I really like to use pytest.fail in setup/fixtures to clarify why this test can’t pass. It’s neither PASS, nor FAIL, it’s a test which wasn’t able to run due to external circumstances.