Testing salt formulas with Test Kitchen and Testinfra

We’re using Salt a lot to provision out infrastructure, so we need the way to test states. I like the Test Kitchen way for “unit testing” Chef recipes and fortunately, there is a SaltStack provisioner for test kitchen, so you can test your salt states with it.

The default way to run tests in Test Kitchen is to run Serverspec tests with Busser plugin.

There are some cons with this configuration that I wanted to avoid:

  1. Busser plugin is executing tests on the tested node itself, so Busser+testing framework should be installed on the node. In case of Busser+Serverspec it means installing chef-omnibus package and it can heavily increase testing time. There are also some test environment isolation issues, described in this post.
  2. I wanted tests to be executed by our CI system (TeamCity in our case) and a nice way to view test results. So test results should be in a machine-readable format (JUnit as a very common format for this). There is an issue of doing it with Busser plugin.
  3. Serverspec is using Ruby, so engineer writing tests should know Ruby more or less, and in our environment Python knowledge is more common (and required for dealing with SaltStack in depth). I wanted to avoid burden to know both Ruby and Python for dealing with formulas.

So I decided to use shell_verifier+testinfra. With shell verifier tests are executed from local machine, so you don’t need anything to be installed on tested node. And testinfra is just a Serverspec equivalent in Python.

I will show how to setup this kind of testing on mongodb-formula:

  • Add Gemfile with Test Kitchen gems
source "https://rubygems.org"

gem "test-kitchen"
gem "kitchen-vagrant"
gem "kitchen-salt"
  • Install bundler (and Ruby, if you need to) and run bundle install
  • Add .kitchen.yml with Test Kitchen config
---
driver:
name: vagrant

platforms:
- name: ubuntu-12.04
- name: ubuntu-14.04

provisioner:
name: salt_solo
formula: mongodb
pillars-from-files:
mongodb.sls: pillar.example
pillars:
top.sls:
base:
'*':
- mongodb

suites:
- name: server
provisioner:
state_top:
base:
'*':
- mongodb

verifier:
name: shell
command: testinfra -vvv --junit-xml junit-$KITCHEN_INSTANCE.xml test/integration/$KITCHEN_SUITE
  • Add requirements.txt for installing testinfra via pip
testinfra
pytest-logging
paramiko
  • Install pip (and Python, if you need to) and run pip install -r requirements.txt
  • Make directory test/integration/server/testinfra. You don’t need to follow Busser convention here (test/integration/<suite_name>/<test_framework>), but I prefer to stick to it for compatability and to avoid reinventing the wheel.
  • Write test config to connect shell verifier and testinfra
# test/conftest.py
import pytest
import testinfra
import os
import yaml
SSH_CONFIG = '.ssh-config'
SSH_CONFIG_MAP = {
'KITCHEN_HOSTNAME': 'Hostname',
'KITCHEN_USERNAME': 'User',
'KITCHEN_PORT': 'Port',
'KITCHEN_SSH_KEY': 'IdentityFile',
}
@pytest.fixture
def TestinfraBackend(request, tmpdir):
# Override the TestinfraBackend fixture,
# all testinfra fixtures (i.e. modules) depend on it.
ssh_config = ['Host {0}'.format(os.environ['KITCHEN_INSTANCE'])]
for key in SSH_CONFIG_MAP.keys():
if key in os.environ:
ssh_config.append('{0} {1}'.format(SSH_CONFIG_MAP[key], os.environ[key]))
ssh_config_file = tmpdir.join(SSH_CONFIG)
ssh_config_file.write('\n'.join(ssh_config))
# Return a dynamic created backend
return testinfra.backend.paramiko.ParamikoBackend(os.environ['KITCHEN_INSTANCE'], str(ssh_config_file), sudo=True)
  • Write some tests:
# test/integration/server/testinfra/test_mongodb_server.py
def test_port_27017_is_listening(Socket):
socket = Socket("tcp://0.0.0.0:27017")
assert socket.is_listening

def test_mongodb_server_is_running(Service):
service = Service("mongod")
assert service.is_running
assert service.is_enabled
  • Run tests with bundle exec kitchen test
  • You will get JUnit output as junit*.xml files

You can see full example of formula in this branch: https://github.com/syndicut/mongodb-formula/tree/testinfra-testing

To further speed up testing, you can build custom images with salt included and/or use cloud drivers for spinning up instances. I’ve also tried docker driver, but due to upstart issues, it’s impossible to test upstart services with it.

Other time consuming operation is chef install, so you can precreate /opt/chef dir in test image to avoid it (chef is only needed for busser+serverspec configuration). There is also this PR for kitchen-salt.

There is also an excellent post about using Testinfra without Test Kitchen (and with Docker):

UPDATE 26.08: Changed shell verifier glue code in test/conftest.py to support kitchen-openstack backend also and added cloud example to repo.