Testing your Chef Code: It’s All About Confidence
Before introducing any piece of code into a live environment, I’m sure we can all agree on the importance of testing it thoroughly to ensure it behaves as expected. By subjecting newly developed code to various types of testing, we can push it to any instance with a high degree of confidence. It seems simple — who wouldn’t want to be confident in their deployments?
Chef cookbooks are often treated differently when it comes to testing, but this doesn’t have to be the case. The recipes, libraries, attribute files and other configurations you create should be handled just like any other piece of application code. Chef offers a number of tools within its Development Kit (ChefDK) that make it possible to develop a thorough and consistent testing strategy for your cookbooks. Using a combination of these tools, there’s no reason why your Chef cookbooks can’t be subjected to the same level of rigorous testing that you apply to other parts of your application.
The goal of the following sections is to provide more of the “what” and less of the “how” with regard to implementing an effective Chef testing strategy. I’ll be introducing you to three key types of testing that can and should be included as part of your Chef testing strategy. As I do so, I’ll discuss their respective benefits along with a number of do’s, dont’s and pitfalls. Finally, I’ll discuss how automation can be used to increase adoption of these testing best practices within your organization.
Lint Testing
In a nutshell, lint testing is used to check your Chef cookbook for correct structure and syntax. Lint testing is an important first step in Chef development as it ensures you are writing clean, efficient code following both Ruby and Chef best practices.
Within the Chef community, the two most commonly employed tools for performing lint testing are Rubocop and Foodcritic. Based on the community driven Ruby Style Guide, Rubocop is a command-line tool that performs static code analysis of all files within your cookbook, including attribute files, recipes, library helpers and custom resources. In a similar vein, Foodcritic is a command-line tool that will identify Chef specific syntax/structural problems within your cookbook. Both Rubocop and Foodcritic are included as part of the Chef Development Kit (ChefDK) and are executed using simple commands, so getting started is easy.
Ensuring that your cookbook passes both Rubocop and Foodcritic testing is just the first step towards a robust testing strategy, but provides you with a consistent foundation upon which to test the functionality of your code.
Unit Testing
Unit testing is a well known concept within the software testing space, designed to quickly validate the behavior and performance of individual components (units) of your source code. As an example of how this can be applied to Chef, you might want to validate that the correct version of a specific package was installed for each operating system that your recipe supports.
ChefSpec is a framework provided as part of the ChefDK to help create and execute unit tests for your cookbooks. An extension of RSpec, a behavior driven development (BDD) framework for Ruby, ChefSpec allows you to quickly test resources and recipes as part of a simulated chef-client run.
There’s a common misconception about the purpose of unit testing with ChefSpec. Consider the following Chef recipe to install the apache web server package:
and a ChefSpec unit test for that recipe:
Examining the ChefSpec test above, this may seem like it’s designed to validate that the package was installed correctly. However, what it’s really testing is “Did I tell Chef to install the package?”, rather than “Did Chef install the package?”. The key point to realize here is that the simulated chef-client run executed by ChefSpec would have failed if Chef had not been able to successfully install the specified package. Like any piece of well developed software, Chef has been thoroughly tested to ensure that it functions exactly as we would expect.
So if the goal isn’t to test the functionality of the underlying Chef resources, you’re probably wondering why we should unit test at all. The answer: regressions.
To be precise, the true value in Chef unit testing is to protect against mistakes (think typo’s or accidental deletions) in your code that could be difficult to detect otherwise. Misspelling a file path, creating a user with an incorrect name or accidentally removing a line from your recipe are just a few realistic examples. All three of these scenarios might still result in a successful execution of chef-client, but will leave your instance in an undesired state. In such cases, a detailed and accurate unit test will help you to detect these hidden errors before they get anywhere near to production. To summarize, the main benefit of unit testing is the protection that it offers you from the most common source of errors, yourself.
Integration Testing
While unit testing is designed to test individual components, integration testing builds upon this to test these units collectively as a group. Within the Chef domain, integration testing allows you to run chef-client with a specific runlist of cookbooks and recipes on a target operating system version.
Integration testing in the Chef domain is performed using Test Kitchen, a tool also included as part of the ChefDK. To use Test Kitchen, you create a YAML configuration file which outlines properties such as operating systems to test against, runlists and cookbook attributes. When executed by Test Kitchen, this configuration file generates on-demand test instances running the specified OS versions. Once available for use, these instances are automatically converged with chef-client using the runlists and attributes specified within your YAML config file. Test Kitchen employs a driver plugin architecture, making it easy to test your configured code on instances created in a variety of popular cloud providers or virtualization technologies such as Amazon AWS, Microsoft Azure, Vagrant and Docker.
Even if you’ve put together a solid suite of lint and unit tests, I cannot overstate the importance of investing time to configure integration tests for your cookbooks. Being able to examine how your cookbooks work together to converge a real instance hosted by your chosen cloud or virtualization provider is extremely beneficial. As a developer, successful integration tests allow you to feel confident that your proposed changes will execute successfully when introduced in a real environment.
Consolidate and Automate
Having covered lint, unit and integration testing, some of you may be thinking that executing each of these tests one by one could become a burden. However, employing tools to help you automate will make things a whole lot easier.
Instead of running each test individually, utilizing tools such as Rake or Thor will simplify the execution of a testing suite. Generating a Rakefile or Thorfile respectively will allow you to run lint, unit and integration tests as a series of consecutive steps from a single command. The obvious benefit of simplified execution will in turn help to drive adoption among team members.
Configuring your testing suite for a particular cookbook to run as a job within your chosen build system software (Jenkins, TeamCity, TFS etc.) is also highly recommended. These tools work closely with your chosen source control system and can be used to help automate the execution of Chef tests. For example, you could set up a trigger to automatically execute a testing job whenever a pull request is opened for a specific repository in GitHub. To ensure that proposed changes have been tested thoroughly, you could also enforce a condition which does not allow a pull request to be merged until all tests within it’s associated job have passed successfully.
Putting It All Together
We’ve covered three distinct types of testing that can and should be applied when performing any kind of Chef development. The initial time investment required to write and configure tests for your cookbook is far outweighed by the long term benefits of consistent code structure and known functionality. As detailed in the previous section, using tools to help you consolidate and automate the execution of these tests will greatly reduce the effort required to execute them. In the end it’s all about confidence: implementing lint, unit and integration tests as part of a testing suite will propel you and your organization towards predictable and consistently successful deployments.
