17 Guidelines for healthy specs

One of the guiding principles of the engineering team at MYOB is automated testing. We strive to be fully confident that every bit of code we write will work as expected, without undesired side effects and regressions. As I have worked in different products and codebases with different teams, I have noticed a few patterns that can make tests better or worse.

Here are a few guidelines for “spec” style unit tests that developers in our team are encouraged to follow. These will be familiar if you use RSpec (Ruby), Jest (JavaScript), ESpec (Elixir) and so on.

1. Write your describe, context and it blocks in plain English. Avoid tech jargon that only developers understand and focus on using terms that are part of your application’s domain.
2. Don’t use words that imply conditions (“if”, “when”, “with”, etc) in it statements. Instead, put the condition in a context block and describe the expected behaviour for that context in the it statements within.
3. When defining a context to represent conditional behaviour, always create another context for the opposite case (ie. what happens when conditions are not what the first context says). This will naturally push you towards completeness.
4. Avoid multiple expectations in a single test . Ideally, each it block should contain only one expectation.
5. Avoid before(:all) or after(:all) blocks to perform setup for your tests — doing so opens the possibility that a “rogue” test may change the setup and impact the conditions in which the following tests are executed. Use before(:each) and after(:each) instead.
6. Inside each context, create a before(:each) block that puts your application in the desired state specified by the context statement. In other words, context will be a plain English description of a state and the before(:each) block will show how to programmatically arrive at that state.
7. When testing asynchronous behaviour, don’t add your expectation in the async function callback. If the function fails, the expectation will never run and the test will appear to pass.
8. Avoid using mocks where possible — they expose implementation details of your application to the tests, which restricts your ability to refactor code with confidence. Instead, try to assert that the desired effects happened. If you have to mock half the world to test your code, it can probably be refactored to have fewer dependencies, or perhaps it should be tested with a more coarse test type.
9. In some cases, mocks are needed for practical reasons (ie. to avoid network calls, IO-heavy operations, etc). In those cases, try to pass the functions or objects that make those calls as parameters in the function you’re testing. That makes it explicit that function/object is necessary, mitigating the implementation leak problem. More information about this approach can be found on this excellent blog post.
10. If setting up your test is a painful process that involves lots of steps, you should consider refactoring your code so it has fewer dependencies. As a rule of thumb, code that is hard to test will be hard to maintain.
11. Tests are code too! Treat them with the same respect you treat your production code: have them checked by your linter, refactor to avoid repetition (shared behaviours are great for this), don’t use quick hacks/shortcuts, make them readable and so on.
12. Don’t write tests for the sake of getting code coverage stats up. That kind of test is not only useless, it may end up masking a real lack of coverage in some important part of your codebase.
13. When finding a bug, write a test to reproduce it — and then fix it. That helps make sure it won’t resurface in the future.
14. If you’re thinking about writing a comment in your code to explain why you’re doing something in a particular way, consider writing a test instead; chances are no one will read your comment, but everyone will notice when a test fails.
15. If your code accepts user input, challenge it with weird and absurd data a person might enter — this is known as fuzzing. Another way of automating this is by using property-based testing.
16. Configure your test suite to run tests in random order. This helps you pick up flaky and misbehaving tests and/or code that has unexpected state dependencies.
17. If you do find a flaky test, don’t despair! It happens to the best of us. Put on your thinking cap, grab a cup of coffee and figure out what the root cause of the flakiness is. Avoid shortcuts (ie. adding a “sleep” statement before asserting in the hope that the stars will align) — flaky tests usually expose problems in the code under test!

What do you think? Please feel free to comment below if you think I’ve missed something!