Smarter, Relevant
Chef Tests
Save hours on every commit with chef-relevant-tests. This new gem will find only your most-likely-to-have-been-affected Chef tests.
Do you run kitchen test when you push a new commit to your Chef repo? Testing everything is a good way to be confident that a change will work, but it takes a long time (especially with large Chef repos with many test-kitchen suites).
Today, we are able to open-source our answer to this problem. A relatively basic but tremendously effective part of our Chef workflow, chef-relevant-tests is a Ruby gem that analyzes the differences between two commits in a multi-cookbook Chef repo and outputs only the test suites that depend on things that have changed.
At Brigade, our single Chef repository has (as of writing) 19 test-kitchen suites, but many of them are not relevant to the average commit. It takes over an hour and a half to run all our suites, even with maximum concurrency allowed by our integration servers. We want to encourage the addition of more integration test coverage by reducing the per-commit cost of adding a new suite, so, for every commit we want to filter out the unaffected test suites as much as possible.
This commonly brings the test suite length to under ten minutes.
Without further adieu:
How it Works
chef-relevant-tests works through the usage of explicit dependencies in Chef cookbooks. Whenever comparing multiple commits, the following algorithm is applied:
1. Find changed cookbook versions between those commits.
By analyzing the Berkshelf dependency graph before and after a commit, we can determine which cookbooks have changed version in that commit.
2. Expand each test-kitchen suite’s run list into a full list of cookbooks that will be tested in that suite.
It may not be a perfect reproduction of your final configuration (in the case of environment-pinned cookbooks, or any diff between your Chef repo and server), but we’ve found that most changes are caught this way.
3. Return any suites that contain changed cookbooks.
Since various commands accept formats in different formats (kitchen test [suites…]), there is an interface that allows customization for new libraries.
Compatibility
Right now, the gem is only compatible with our specific use-case: limiting test-kitchen suites based on changes in your Berksfile.lock. But if you don’t use these technologies, fear not, adding new support is as easy as:
- Write a ChangeDetector (e.g.) that compares the current state of your Chef repository to a prior ref and returns a list of cookbooks.
- Write an Expander (e.g.) that takes a list of cookbooks and returns a list of strings to output to the calling script.
…or just make an issue for it.
There is usage information in the README, and pull requests are welcome!
