Infrastructure Testing for Beginners: Getting Started with Test Kitchen and Salt

There comes a time when spinning up environments in Vagrant isn’t enough. At Linux Academy, I have a number of courses to maintain, which means a number of environments, and often new ones to spin up and throw away to test things in student tickets and posts — especially after one of the platforms I teach pushes a change.

Now, it might seem like Vagrant is ideal for this — and it is ideal to start with — but I don’t just want to be able to provision my test environments. I want to be able to make sure my test environments work as expected. And this is where infrastructure testing comes in.

You’re already probably familiar with the need for test coverage for your applications: Testing is, after all, nothing new to the development cycle. Unfortunately, as infrastructure became code, that need for testing didn’t originally come with it. But that doesn’t mean solutions haven’t arisen since then — one of which is Test Kitchen.

Most closely associated with Chef, Test Kitchen can work with a variety of config management tools and drivers to spin up and test our infrastructure-as-code setups and make sure everything is the way it should be. Which is great when someone reports an issue with one of my courses and I want to quickly spin up and check the setup to see what the issue is. Of course, you can also imagine how useful this can be for any infrastructure written with a config management tool: You can test your formulas across multiple distros (or even operating systems, depending on what the configuration management platform supports) and make sure everything is working exactly as planned before pushing it out to your actual fleet.

Setup

I’ll be using my work Mac to run Test Kitchen, but you can just as easily set up with Linux or Windows. Just make sure you have Ruby installed!

Once you have Ruby situated on your workstation, you can install Test Kitchen with a simple:

gem install test-kitchen

And we can add the related providers we need with a gem install too. For this article, I’ll be using Vagrant (if you’re unfamiliar with Vagrant, check out this Vagrant Cheat Sheet) as the driver and Salt as the provisioner, so we’ll need to run:

gem install kitchen-vagrant
gem install kitchen-salt

Then, create a directory to store your work, and initialize it with a:

kitchen init

Three files have been created:

• kitchen.yml is our configuration file; this is where we’ll be doing all of our work.

• chefignore is for files you wish to ignore.

• test is a series of directories in which we can store our tests.

Let’s open up that kitchen.yml file now and check it out.

kitchen.yml Breakdown

The kitchen.yml file is written in basic YAML and contains information about what is deploying our infra, what distros (or OSs) we’re testing against, what formulas are being pushed to our platforms, and which tests we’re running. Essentially, this is a nice overview of our platform and everything we have to test.

Now, let’s break this down.

driver

---
driver:
  name: vagrant
  provider: virtualbox

The driver in Test Kitchen is what deploys our infrastructure out. This can be a hypervisor, cloud provider, or abstraction layer to the hypervisor, such as Vagrant, which is what I’m using in this demo. I’m also going to specify the virtualbox provider here, although it’s the one chosen by default. In the future, I know I’ll want to switch this to KVM, so I want to include the option now.

provisioner

provisioner:
  name: salt_solo

If the driver creates our instances, then the provisioner makes sure they are set up how you want them. Generally, this is a configuration management platform, such as Chef, on which Test Kitchen is based. That said, we’re going to use Salt instead.

While simply supplying the chef_solo parameter is good enough for Chef, we do need to add some additional configuration options for Salt Solo.

With added functionality, my provisioner section looks like the following:

provisioner:
  name: salt_solo
  is_file_root: true
  require_chef: false
  salt_install: bootstrap
  salt_copy_filter:
    - .kitchen
    - .git
    - .vagrant

Where is_file_root tells Salt to treat the same directory our kitchen.yml file is in as the traditional /srv/salt directory, require_chef disables Chef, salt_install ensures we’re using the bootstrap script to install, and salt_copy_filter prevents any unneeded files from being pushed to our test servers.

platforms

platforms:
  - name: bento/ubuntu-18.04
  - name: bento/centos-7

These are what we want to test on — so, our distros or operating systems. By default, CentOS 7 and Ubuntu 16.04 are populated, but I want to specifically test on the Ubuntu 18.04 and CentOS 7 bento boxes for Vagrant. To do this, we can just supply the box names like we would in Vagrant itself.

suites

suites:
  - name: default
  - name: web

When I initially started using test kitchen, the idea of “suites” is what I had the most trouble with, predominantly because every example I saw seemed to be doing something different. But what a suite is, essentially, is a combination of the platforms you provide and a software suite to test on. We’re provided with a started default suite, but we can add more as our needs change, whether this means adding a suite to test to different versions of Salt, testing multiple pillar options, or simply making sure a collected component — like a LAMP stack where each server serves a different functions — is only running the formulas it needs.

In this instance, I’m going to add two suites: a default one that will run everything I want on every server — such as my Vim configuration — and a second one that tests my “basebox” formulas and an Apache formula.

Adding Formulas

While we have the base of our Test Kitchen testing environment set up, we’re not quite done yet. As you may have seen, we have no formulas or pillar data provided. So, to populate our setup with some formulas, I’m going to create a simple formula that makes sure Vim is downloaded and uses the correct vimrc, as well as a second formula that installs Apache.

vim Formula

In the same directory where the kitchen.yml file is located, create a new directory, called vim.

We’re going to end up with three states here: install.sls, vimrc.sls, and init.sls, as well as the files directory in which we can store our vimrc file, and map.jinja file to keep our OS mappings.

install.sls

{% from “vim/map.jinja” import vim with context %}
 
vim_install:
  pkg.installed:
    - name: {{ vim.package }}

vimrc.sls

{% from “vim/map.jinja” import vim with context %}
 
vimrc:
  file.managed:
    - name: {{ vim.config_root}}/vimrc
    - source: salt://vim/files/vimrc

map.jinja

{% set vim = salt['grains.filter_by']({
'Debian': {
    'package': 'vim',
    'config_root': '/etc/vim',
  },
  'RedHat': {
    'package': 'vim-enhanced',
    'config_root': '/etc',
  },
})%}

init.sls

include:
  - vim.install
  - vim.vimrc

files/vimrc

Finally, create a files directory within the vim folder, and add your preferred vimrc file. This is mine:

set nocompatible              " be iMproved, required
filetype off                  " required
" Environment
set ruler
set ignorecase
set hlsearch
set number
set undofile
set termguicolors
" Tabbing
set softtabstop=2
set tabstop=2
set shiftwidth=2
set expandtab
retab
" No Sound Editing
set noerrorbells
set visualbell
set t_vb=
set tm=500
" Colors!
syntax enable
colorscheme delek
hi LineNr ctermfg=gray
" Goodbye Annoying
set nobackup
set spell spelllang=en_us

Apache Formula

For Apache, we’re simply going to add an installation state to an init.sls and use Pillar to determine if the service is running. Make sure to add this all to an apache folder added to the root environment directory.

init.sls

{% from "apache/map.jinja" import apache with context %}
apache_install:
  pkg.installed:
    - name: {{ apache.package }}
  service.running:
    - name: {{ apache.service }}
    - enable: {{ pillar['apache']['enable'] }}

map.jinja

{% set apache = salt['grains.filter_by']({
'Debian': {
    'package': 'apache2',
    'service': 'apache2',
  },
  'RedHat': {
    'package': 'httpd',
    'service': 'httpd',
  },
})%}

Adding Pillar Data

Finally, we need to the pillar data for this formula. Back in the main directory for our environment, create a pillar directory.

In that directory, create a file called base.sls, where we’re going to store just one simple bit of pillar data:

apache:
  enable: true

Update kitchen.yml

With our formulas defined, we can now update our kitchen.yml file to use them. And we choose two formulas for a very specific reason: Because there’s no limitation to where we define what formulas are run.

provisioner

We’re going to add our vim formula to the provisioner section, because we want it on all our servers:

provisioner:
  name: salt_solo
  is_file_root: true
  require_chef: false
  salt_install: bootstrap
  salt_copy_filter:
    - kitchen
    - .git
    - .vagrant
  state_top:
    base:
      "*":
        - vim

To do this, all we have to do is add the state_top parameter, then within it provide what is essentially a regular top.sls file.

suites: web

But for our Apache module, we’re going to work a little differently. We just want it on that second web suite, so we add a provisioner parameter and provide it our states and pillar data:

suites:
  - name: default
  - name: web
    provisioner:
      state_top:
        base:
          "*":
            - apache
      pillars-from-files:
        base.sls: pillar/base.sls
      pillars:
        top.sls:
          base:
            "*":
              - base

Notice how we first have to define where our pillar data is located, and then we can assign the data to our minions.

Kitchen Converge

Now that we have everything written out, we can see if our states are working across all our test suites by running a simple:

kitchen converge

And waiting for everything to provision! If there are any issues anywhere, it will let us know at the end that our converge failed.

Once done, we can go ahead and run a kitchen list to see all our instances! Notice how there’s an Ubuntu and CentOS version of every suite we provided.

So, now we have a way to test our formulas across a variety of instances in one go, giving us our testing basis. Next up, we’ll expand this by writing some unit tests to make sure everything is working just how we intended.

Interested in Test Kitchen but the prospect of YAML got you down? My course on YAML is free on Linux Academy!