Mockingbird — Part 3 — The Pitfalls
In the first part of this series, I defined what mocking is, and described some of the use cases where it is needed in order to make tests clearer and easier to write. In the second part I focused on some of the strategies used in the Server Guild at Wix, comparing some of the mocking libraries that we use. In this final part of the series, I will discuss some of the pitfalls related to using mocks.
In the previous parts of this series, we have seen how mocks can be used to help write clear concise tests. But of course using mocks is not the silver bullet to solve all of your testing difficulties. As we will see in this part, there are some pitfalls when using mocks, and these need to be taken into consideration when writing the tests.
Pitfalls of Mocking
Coupling to Implementation Details / Over specification
Mocks should be used as a tool to help test the external behaviour of a function. Often the test setup dictates which functions of the mocks must be called, and in which order, and which parameters must be passed. This ends up coupling the test very tightly to the internal implementation details of the function, where the test dictates how the function interacts with its collaborators, instead of testing what the function does and that it returns the expected result. This makes the function difficult to refactor, because although the functionality remains the same, changes in the order and/or number of times a mock is called, causes tests to fail, and forces us to make changes in the tests also. This is known as brittle or fragile tests.
Over-mocking
Sometimes mocks are used in places where it is better to use the real implementation of the class. This slows down the refactoring process because of unrelated tests that don’t compile or start failing. Another consequence of over-mocking is that it can lead to bugs that are introduced into the real class, but go unnoticed because the interactions with this class are tested via mocks. Also it can lead to tests which don’t execute any actual production code because all collaborators are mocked.
Exception Handling
Sometimes an interaction with a mock object can cause a runtime exception, for example when calling the mock with incorrect parameters, with JMock would throw an UnexpectedInvocationException or with Mockito throw a NullpointerException when trying to access the response. If this situation is not handled carefully, it can lead to tests passing when they should fail due to the mock being called incorrectly. The call to the mock throws an exception, and the test passes because it expects an exception to be thrown, but does not check the specific exception. In this case the test needs to be changed so that it is more specific in what exception is expected.
Another possibility is that the call to the mock is sometimes wrapped in a try … catch which means that when the unexpected call throws an exception it is caught and “swallowed” in the production code, which continues processing, and the test still passes. To avoid this situation, more care needs to be put into the tests, in order to ensure that when this happens, the test fails.
Test only one thing per test
It is a good practise / guideline that tests should only test one thing. However when using too many mocks, this becomes very difficult, because the tests require setting up multiple mock expectations, and without realising it, the tests are really checking many different things, specifically all the interactions with the mocks, making it difficult to identify exactly which use case is being tested.
Confusing / unclear terminology
Because there is no standard regarding the terminologies of stubs, spies, fakes etc, it is important that the language of the test communicates clearly what the intention of the mock is. Is it to supply state for the setup (given / arrange) or is to verify that it was called correctly (then / assert). Using descriptive variable and function names help in this regard, and teams should adopt some naming guidelines, in order to ensure that tests are clear and understandable. It is a good idea to wrap the mocking in a function called givenXYZ when it is part of the setup, and in a function called expectXYZ when it is part of the assertion. This makes the intention of the test more clear.
Missing Integration Tests
Integration tests are required in order to ensure that all the wiring and dependency injection is done correctly. It is necessary to have at least 1 integration test that tests the full flow, without any mocking of internal classes. This ensures that all integration points between internal collaborators are tested, and that these collaborators work correctly when connected to each other.
“Don’t mock what you don’t own”
This is a general guideline, and not a rule. For integration tests, it is usually necessary to mock external / third party classes because it is difficult to test the system without it. But for unit tests, it is a good practice to avoid mocking these classes. This leads to better design, where external dependencies are only accessed via wrappers or adapters, making the software more flexible, by not binding the business logic with specific implementation details and versions of external collaborators. It also stops the specific implementation details from leaking into the core business logic of the application, which simplifies the process of dealing with upgrades and API changes of the external dependency. Mocking only the interactions between internal classes, which can be changed as the design of the system evolves, leads to better separation of concerns, but the interactions with external classes already have a fixed behaviour which can’t be changed, which means the design cannot easily evolve.
Another problem with mocking external collaborators is that the tests do not necessarily prove that we are using the API correctly, they just confirm that we use it the way the test expects it to be used. We could be passing the incorrect value for a parameter, when the third party application really expects something else. For example a third party application expects a country code to be passed as 3 characters, and we call it with 2 characters, or it expects an amount in kilograms and we call it with the amount in grams. These kinds of integration issues are not solvable with mocking in tests. They need contract tests which make calls to the actual service, or “sandbox” version of the service, and sometimes even manual QA is required to ensure that the integration is 100% correct.
Conclusion
Mocks are a great tool to help write well structured tests that are easier to write, read and maintain and clearly communicate the intention of the test. Using mocks allows setting up the state of the environment, particularly in cases where the state is difficult or impossible to reproduce. We had a look at some examples of why mocks are needed, and we also compared some of the mocking tools and strategies in use in the Server Guild at Wix. Lastly we saw that there are a few pitfalls when using mocks, and so just like all tools, they should be used correctly and when appropriate. This series is based on my experiences from my journey with TDD over the last few years. I know that everyone has different experiences with this, and I would love to hear your feedback and opinions in the comments section.
