A pragmatic solution for detecting and eliminating dependent tests

Recently we had a discussion about fail-fast with my colleagues. We cannot fix organisation problems, monitoring shortcomings or log spam with just throwing an exception to show an error message for the end customer. That is true. I still prefer fail-fast since

• it can help the developer to identify and fix the problems faster, which in the end leads to higher customer satisfaction
• often I cannot fix the things mentioned above

So I tend to think about fail-fast pragmatically.

Dependent Tests

Similar problem is when tests are depending subtly on each other. It can be really time consuming and annoying to find and fix such tests. Let me mention some examples:

• Spring caches the test contexts by default. If I happen to replace a spring bean dependency with a mock inside some test, but I forget to add the @DirtiesContext annotation to the test, we have immediately have stolen precious minutes from someone else’s life. The test will of course always pass in my IDE when running alone, but it will sometimes pass sometimes fail in the build server, which is extra tiresome.
• Hopefully it never happens in your project, but some static singleton could have references to Spring beans. In production we usually will have only one context in our application, but in tests we can have several. If this happens, we will have multiple instances of the beans, but the static singleton will have a reference to a bean of one specific context, which might not work with some tests. Again. Test runs alone, but not with the buddies together.
• Another case when we use eh-cache with Spring. One context creates the cache, second context starts, first context stops because of a @DirtiesContext and BAMMMMM, the test using the second context will die as soon as it tries to reach the cache, which has been already stopped. We have to use unique cache manager names for each contexts to prevent this issue, or we have to set the cache manager to be not shared.
• Setting and not resetting system properties or other JVM settings runtime can also cause dependent tests.

Theoretical solution for dependent tests

It is a good idea to keep the tests independent from Java Virtual Machine (JVM) state. This could be achieved with extracting such state access into classes and use a mock of them in our tests. This way we can avoid both reading and altering the JVM state.

Especially when we work with legacy code, we still might need to read JVM state in our test code, or even we need to change it. If a test changes the JVM state, which can influence other tests, we should restore the original state after the test cases. Usually we would use methods marked with @Before and @After annotations to setup and tear down. The problem is that we might forget the @After method. A better solution is to use a JUnit Rule. Such a rule encapsulates the setup/tear down actions and takes care of executing them. If I can forget to restore something, then I will forget it. But not with a Rule.

@DirtiesContext annotation should be used with care, since it makes our tests a lot slower. A better solution would be to make a smaller test context using mocks or if possible we can inject the dependencies into the test and create our system under test class in the test itself.

A possible out of the box solution for tests using system properties and things like that is this great project called system-rules. Still keep in mind: it is better avoiding the need for it.

Pragmatic solution for dependent tests

It may happen, that the project we work on has several hundreds of test classes and some of them are flaky. We know how tests should theoretically do their setup and tear down phase. Still checking and fixing each tests can take a lot of time. Additionally the test which is failing is probably 100% correct. An other test running before the failing test can be responsible for the failure.

First we would like to know why a specific test failed. By default Maven runs the tests in the order determined by our file system. This is the default setting for runOrder property. This can be different on my Windows machine from the one on the Linux build system. Again frustration. A good aid could be if the build produces an easy to use output to tell, what was the real order of the tests. You may find this listener useful. It produces a suite file. Just drag and drop into my work-space, start, and it produces exactly the same error as the build server(of course if we have multi-threaded test execution, it does not help).

I mentioned the runOrder property of surefire/failsafe before. I think I would simply set the runOrder to random combined with the suite generating test listener mentioned above. If my tests use correct setup and tear down process, they should be able to pass ALWAYS independent from run order. Again: it might be frustrating first, but it could be a really good motivator to fix all the problematic tests.

What do you think?