Stabilizing Your Test Suite Settings
Improving reliability by managing global test suite settings
I head up the Test Automation team at Greenhouse. As a team, we maintain the test suite, build automation tools, and push for best practices in testing. We've gotten pretty good at fixing unreliable tests that pass 90% of the time but randomly fail without obvious cause. We’ve seen enough of them to identify common pitfalls like non-deterministic SQL ordering or poor use of Capybara wait time within a few minutes of looking at a test . But when our entire test suite suddenly became unreliable a few weeks ago, we had a hard time determining the cause. After digging through every code change we’d made over about a week-long period, we found the culprit: an AWS.stub! inside of a test that prevented every test that ran afterward from reaching AWS.
It wasn’t the first time we’d suffered as a result of someone changing a global test suite setting inside of a test. Pinning down this class of issue can be difficult and time-consuming. They depend on test ordering, and test failures don’t obviously point back to the root cause. To prevent it from happening again, we decided to create a small class to manage the settings in the test suite.
The basic idea was to create some kind of manager that would check the values of registered settings after each test to make sure they hadn’t changed from the defaults. We also leveraged RSpec metadata to make the settings easy to change by just adding a tag to a test. If the developer changes something and doesn’t flip it back, we throw an exception that tells him exactly how to fix it in the test output. For example, consider the following spec where a developer changes the AWS.stub! setting without changing it back:
After running the spec, the developer will get a helpful error message:
1) Flipping settings without tags permanently changes test suite settings
You have changed test suite settings:
Use the tag :stub_aws => true instead of manually modifying settings.
The spec can be easily fixed with the following change:
The settings manager code itself isn't too complicated. We created two classes, Setting and Manager:
To utilize the manager, we added an around hook in an RSpec.configure block:
We store all of the different Settings in a tags directory. Most of them look something like this:
In order to guarantee that the settings manager around hook is called at the appropriate time, we make sure to load the initializer before the rest of the support folder is loaded. Our rails_helper looks something like this:
The implementation of the settings manager you’re seeing now is actually our second iteration. The first just checked settings before and after a test and made sure they matched up. But after we were bitten by someone changing a setting outside of a test entirely, we added the concept of the default setting. Instead of checking before and after every test for matching settings, we just check for the default at the end. There’s a little more overhead with setup, but it actually ends up being a bit faster than the previous implementation.
Obviously this only works if you actually register every setting that can come back to bite you. We don’t have a perfect list yet, but so far we’ve added tags for:
I'm sure we're missing a few more settings, but we can always add more as they become problematic.