JUnit auto verification

In this post I am not going to spend too much time on what JUnit is or how to use it. It is a unit test framework for Java and it is well used. I am going to focus on one thing that surprised me even it is described in the documentation.

Mockito 2.x introduced Strictness modes. The feature is there to write cleaner, better debuggable tests without duplicated code and so on. Strictness modes are three — LENIENT, WARN and STRICT_STUBS. The STRICT_STUBS is becoming the default Strictness mode. E.g. in JUnit 5 it is already default.

If you’re using the STRICT_STUBS mode, you’ve probably seen a test failing because there was an unnecessary stubbing made and so on. I am not going to describe this — there has already been written a lot about this. Instead I am going to focus on following example.

DummyExecutor calls add method on DummyAdder instance. Once in a time (after several executions) it calls reset().

In DummyExecutor implementation there is a bug on line 21 that results in method reset() being called every time.

Unit test has one test that verifies that reset method is not called during first 4 calls of execute method. Mockito method verifyNoMoreInteractions is used to verify that only the addOne method (that is verified before) was called.

But, when you run the test, it ends up with green check mark — passed. I accidentally noticed this behavior when I was trying to do something as TDD — write tests before the implementation, run them first to see that tests are failing and then write the implementation. And I noticed, that some of the tests I expected to fail just passed.

I started to investigate what the hell is going on and I figured, that in STRICT_STUBS mode, all the stubbed method calls are automatically marked as verified. So when you use the Mockito.verifyNoMoreInteractions(…) it actually verifies only calls, that were not stubbed (e.g. a completely different method on the mock).

If I want this test to fail, either I have to use the LENIENT mode (so no auto verification is done), or to really verify, that there were zero interactions on the reset() method.

The lenient mode stubbing can be done e.g.:

Mockito.lenient().doReturn(true).when(this.adder).reset();

The zero call verification can be done by:

Mockito.verify(this.adder, Mockito.never()).reset();

This behavior is documented so it is my fault not to read the documentation carefully. On the other hand, the test looks like it just should work. And in case I would have written the test after the implementation, I would not know that the test is wrong and the implementation does not work.

In case you did not know this either, next time be careful. Probably, it is a good idea to write tests first. Or at least sometimes when it does make sense…