Infrastructure Testing for Beginners: Adding Tests with InSpec

In our previous guide, we set up Test Kitchen to allow us to test our Salt formulas across two different distros and for two different setups. But while we can see if our Salt formulas are failing, we can’t verify the changes we made actually happened. This is where InSpec comes in: InSpec lets us test the actual end state of our servers.

Setup

We’ll be using the same kitchen.yml file as last time. We do, however, need to install the associated Test Kitchen packages:

gem install kitchen-InSpec

We also want to add the following to our kitchen.yml file:

verifier:
  name: InSpec

Since the verifier is run after we converge, we'll also want to start our kitchen converge now, if you no longer have your servers up from last time:

kitchen converge

We also want to make sure we have the test/integration/ directories in our working directory for our setup. These should have been created when we initially ran kitchen init, along with another, default directory under the integration folder: This is where our default tests go for our default suite. Eventually, we'll be adding another one for our web suite.

But let’s start by moving into default and writing our first test!

Writing an InSpec Test

We’ll write a basic test that confirms the appropriate Vim package is installed on all our servers. Move into test/integrations/default and create a file called vim_spec.rb.

Next, we can write the test. At its most basic, it would look like:

describe package('vim') do
  it { should be_installed }
end

With InSpec, at its most basic, each test will start with describe and then go on to "explain" the desired outcome of the state. In this case, we use the package resource type to check that our desired package (vim) is installed (it {should be_installed }. Parameters such as be_installed are dependent upon the resource type itself. A full list can be found on InSpec's website.

However, this is only set to work on Debian-based servers, since we know the name of the package for Vim is different on Red Hat-based distros. To accommodate this, we can add a case statement to our tests. So instead of a single stanza to check on the package installation, we would have:

case os[:family]
  when 'debian'
    describe package('vim') do
      it { should be_installed }
    end
  when 'redhat'
    describe package('vim-enhanced') do
      it { should be_installed }
    end
end

Now, let’s go ahead and test this specifically against our servers in our default suite. We can specify which servers we want to test by defining them after the verify command:

kitchen verify default-*

We can see each server returns information similar to:

System Package vim-enhanced
     ✔  should be installed

With the checkmark denoting the package was installed successfully.

Expanding Our Tests

Installing Vim is not the only thing we did in our Salt formulas in the last guide. We now need to add tests for our vimrc file, and our apache formula, the second of which is located under a second suite. We'll talk about adding tests across suites in a moment. Until then, let's finish up with Vim.

Testing the vimrc

In the vim_spec.rb file, we now want to make sure the actual content in the vimrc file we provided has been updated to reflect the file we supply in the Salt state. For this, we would use InSpec's file resource type. However, we don't just want to test that the file exists — it exists even before we push the version we want via Salt — we want to match the data in the file against the desired data. To do this, we would use the content property for the file resource. This particular example I added to the debian case stanza:

describe file('/etc/vim/vimrc') do
  its('content') { should match /REGEX/ }
end

Wherein describe file('/etc/vim/vimrc') defines that we are checking the /etc/vim/vimrc and its('content') { should match /REGEX/ } informs InSpec what the content should match via a regular expression that we have yet to supply.

Now, in an ideal world we would supply all the information for our desired configurations in repeated its('content') lines. Something like:

describe file('/etc/vim/vimrc') do
  its('content') { should match /set nobackup/ }
  its('content') { should match /set number/ }
end

Or with the plain text translated into a fuller regular expression:

describe file('/etc/vim/vimrc') do
  its('content') { should match /set\s.+/ }
end

But since this isn’t a guide to writing regular expressions (although I have a free course on that), we’re instead just going to make sure everything is working by looking for a specific line that my vimrc contains and most do not: The heading I used for turning off backups is Goodbye Annoying, so I'll be looking for that to determine if the right file has been added:

describe file('/etc/vim/vimrc') do
  its('content') { should match /Goodbye Annoying/ }
end

And, of course, we want to also write this for our redhat stanza, leaving us with the following full vim_spec.rb file:

case os[:family]
  when 'debian'
    describe package('vim') do
      it { should be_installed }
    end
    describe file('/etc/vim/vimrc') do
      its('content') { should match /Goodbye Annoying/ }
    end
  when 'redhat'
    describe package('vim-enhanced') do
      it { should be_installed }
    end
    describe file('/etc/vimrc') do
      its('content') { should match /Goodbye Annoying/ }
    end
end

Run another kitchen verify default-* to check that everything is working properly.

Adding Tests for Apache

Up until this point, we have been working within our default suite — we can tell because of the location where we've been storing our tests: The default in test/integration/default references the suite name the tests are being run against, and as long as we follow this convention, we do not need to add any extra configurations to our kitchen.yml config. This also means if we want to test our web suite, we need to add a new directory at test/integration/web.

Next, we want to create an apache_spec.rb file under that same directory, and since we already know there will be differences between our Ubuntu and CentOS servers, we can start this by writing the skeleton of our case statements:

case os[:family]
  when 'debian'
  when 'redhat'
end

Next, we want to make sure Apache is installed. If you remember from the last guide, that means we’ll be checking for httpd on CentOS 7 and apache2 on Ubuntu 18.04. We would write these stanzas the same as we did for the vim_spec:

case os[:family]
  when 'debian'
    describe package('apache2') do
      it { should be_installed }
    end
  when 'redhat'
    describe package('httpd') do
      it { should be_installed }
    end
end

Next, according to our simple apache state, we also want to make sure we have Apache enabled on these servers. For this, we would use the service InSpec resource type, along with the be_enabled parameter. It should look like this for Debian-family servers:

describe service('apache2') do
  it { should be_enabled }
end

And this for Red Hat:

describe service('httpd') do
  it { should be_enabled }
end

Giving us a full script that looks like:

case os[:family]
  when 'debian'
    describe package('apache2') do
      it { should be_installed }
    end
    describe service('apache2') do
      it { should be_enabled }
    end
  when 'redhat'
    describe package('httpd') do
      it { should be_installed }
    end
    describe service('httpd') do
      it { should be_enabled }
    end
end

Now, let’s see if our tests are working:

kitchen verify web*

Finally, let’s open up the apache_spec.rb file again and add a second property to the service resource: that Apache should be running. We do not need to create a new describe stanza for this; we only have to add it to our existing one:

case os[:family]
  when 'debian'
    describe package('apache2') do
      it { should be_installed }
    end
    describe service('apache2') do
      it { should be_running }
      it { should be_enabled }
    end
  when 'redhat'
    describe package('httpd') do
      it { should be_installed }
    end
    describe service('httpd') do
      it { should be_enabled }
      it { should be_running }
    end
end

Run kitchen verify web* again to see that all tests succeeded.

Going Further

While these are some of the more basic examples of tests we can run to make sure our infrastructure is really in our desired state, we can actually use the InSpec DSL to write far more advanced tests than we see here. View the documentation to get started.

Otherwise, meet me back here in two weeks, where we’ll learn troubleshooting tactics for when our Test Kitchen setup goes wrong.

What to check out how all this works with Chef? Check out Linux Academy’s Chef Local Cookbook Development course, which uses these same tools only with — you guessed it — Chef!