Fast and slow tests
People that write tests like to categorise them.
There are a bunch of names. The ones I generally use are:
- Unit tests
- Integration tests
- Functional tests
- Acceptance tests
- Smoke tests
The most common disagreement I have with people around testing is the difference between unit tests and functional or integration tests.
Unit tests, we argue. Should test just one thing. They shouldn’t do things like connect across a network or talk to the database. Maybe they shouldn’t exercise the framework.
Integration tests on the other hand can do whatever they like.
But the boundary between these two types of tests is conceptually tricky for most people.
I’ve found a much easier distinction to maintain is FAST and SLOW tests. A fast test finishes in a couple of milliseconds or less. You can execute 500 of them in a second. Meaning you can run all of them every time you modify the code you’re working on. We don’t have to understand exactly why they’re fast. What boundaries of your software they’re not touching. We just know they’re very fast.
Slow tests take longer than that. Maybe they read a big sample file off the disk. Or load a web browser, start a web server, and hit a page.
This becomes easy to manage if you have a test framework that lets you tag tests and only run a subset of them.
For example, using pytest:
Fast tests can be run whenever you make a code change locally, and all the tests can be run whenever you push code to your CI system.
Using a simple distinction like this is straightforward for large teams to understand. It’s also easy for one person in a team to “fix” a slow test by marking it as such.
The definition of a fast vs a slow test is uncontroversial. And a team can set whatever arbitrary limit they want. For a me a fast test might mean 2 milliseconds. But for a different tech stack maybe it means 1 second.
Either way, once you’ve decided on that number you can all enjoy the benefits of a fast test suite. And an accompanying slow test suite that catches any issues that the fast tests will not.
