Accessing remote host at test discovery stage in testinfra/pytest

Normally you can’t access fixtures at test discovery stage in pytest. That means, if you want to use the fixture host to parametrize tests, you can’t.

But this is python. You can’t access fixture in pytest_generate_tests, but can you construct it? Yep, you can. It was unexpectedly easy.

Example task

We have built-in tests for alerts in Prometheus. They looks like /etc/prometheus/alert.*.test, and I want to test them all in a idiomatic way: one test per file. This begs for parametrization, but files are on the remote server, so it’s impossible to use host fixture to generate parametrization list.

The test itself (without parametrization) looks like this:

def test_rule(rule, host):
  host.run_test(['/bin/promtool', 'test', 'rules', rule])

So, we need to create rule fixture. This problem is spit into two parts: extracting file list from remote host, and finding the most expressive and beautiful way to write it.

Creating fixture manually

(this is a secret sauce of this post)

I found a way to force testinfra to create host fixture without usual pytest machinery:

ANSIBLE_INVENTORY=foo.yaml python3
>>> import testinfra
>>> hosts = testinfra.host.get_hosts('ansible://all')
>>> for h in hosts:
...    h.run('hostname')
srv1
srv2
srv3

Yep, that easy. There is get_host (get one any matching host) and `get_hosts` to get a list of all matching hosts (even if there is one).

You can throw in ?force_ansible=True into hostspec if you need Ansible transport, which invaluable in case of complex inventories.

(that is an idea, the proper implementation with inventory file is below).

Minor issue

Whilst I wrote this chapter I realized I hadn’t though about the fact that each host may have a different list of files, and we need to cover this.

We’ll construct a simple fixture to cover both host and rule fixtures. Our updated test would look like this:

def test_rule(rule_at_host):
  rule_at_host.host.run_test(
    ['/bin/promtool', 'test', 'rules', rule_at_host.rule]
  )

(the key difference is that there is a single ‘thing’ containing both host and rule, creating not-Cartesian product of host and rule).

A fixutre or to generate?

I tried to invent fixture, but as usual, no, you can’t cross this boundary. Fixture is called after collection, so no parametrization is possible. The solution, as usual, is pytest_generate_tests.

def get_alerts(hosts):
  for host in hosts:
    for f in host.file("/etc/prometheus").listdir():
      if f.startswith("alert.") and f.endswith(".test"):
        class Result:
          pass
        Result.host = host
        Result.rule = f
        Result.id = f"{host.backend.get_hostname()}, {f}"
        yield Result

def pytest_generate_tests(metafunc):
  if "rule_at_host" in metafunc.fixturenames:
    # copypaste from pytest_generate_tests from testinfra plugin
    hosts = testinfra.get_hosts(
      testinfra_hosts,
      connection=metafunc.config.option.connection,
      ssh_config=metafunc.config.option.ssh_config,
      ssh_identity_file=metafunc.config.option.ssh_identity_file,
      sudo=metafunc.config.option.sudo,
      sudo_user=metafunc.config.option.sudo_user,
      ansible_inventory=metafunc.config.option.ansible_inventory,
      force_ansible=metafunc.config.option.force_ansible,
    )
    alerts = list(get_alerts(hosts))
    ids = map(lambda x: x.id, alerts)
    metafunc.parametrize("rule_at_host", alerts, ids=ids)

The test is simple:

def test_alerts(rule_at_host):
  assert rule_at_host.host.run_test(
    f"/bin/promtool test {rule_at_host.rule}"
  )

The monstrosity around hosts= is a clean copy-paste from testinfra itself, as it take in account all possible inventory nuances. It cleanly integrates both in manual pytest with ANSIBLE_INVENTORY variable and molecule run with --inventory-file parameter.

Outside of that huge blemish, the rest I consider a good code.