Mockingbird: The Why, When and How of Testing with Mocks — Part 2
In the previous part of this series, I defined what mocking is, and described some of the use cases where it is needed in order to make tests clearer and easier to write. In this part I take a look at some of the strategies used in the Server Guild at Wix, and compare some of the mocking libraries that we use.
Mocking in the Server Guild at Wix
At Wix, TDD is strongly encouraged, and the Framework and Infra teams have put in a lot of effort in order to make testing easier. Backend services at Wix are mostly written for the JVM and NodeJS platforms. I will discuss the JVM, specifically Scala, mocking strategies, and I will separate it into 2 sections, Unit Testing and Integration Testing.
Unit Testing
Unit testing, refers to the testing of single or small groups of related components. This is done by using mocking libraries, in order to mock the collaborators, and focus on testing only isolated units. At Wix, we use the Specs2 library to write our tests, and the 2 main libraries that are used for mocking are Mockito and JMock. Specs2 has built in integration with Mockito, but for JMock we have an adapter layer which provides integration with Specs2 and adds support for advanced Scala features such as default parameter values and passing parameters by name.
In the Guild there is an ongoing debate among the developers about which is the preferred library. This is related to the well known debate about how to write tests, “State Based” vs “Interaction Based” there is a lot of literature for example Mocks Aren’t Stubs about this debate, so I will not go into the details.
The “State Based” testers prefer Mockito, whereas JMock is more suited for “Interaction Based” testing. The difference being that Mockito is more “lenient” in the sense that it will not fail when called with unexpected parameters, whereas JMock will throw an exception if it is not called with the exact interactions that were setup in the test.
I will look at some common testing requirements, and discuss the main differences between performing these with Mockito and JMock.
Requirement: Setting up the return value of the mock, and verifying that the mock was called with the correct parameters.
Mockito: Requires 2 steps. One to specify the return value, and one to verify that the mock was called correctly.
JMock: Expectations are done in one step, which specifies what the return value will be, and also verifies that the mock was called correctly.
Requirement: Verifying that the mock was called correctly.
Mockito: Verifying that the mock was called correctly is done after the action, so fits well with the GWT pattern, verifying forms part of the “THEN” phase.
JMock: Defining the expected interactions needs to be before the action, which does not fit well with the GWT pattern because the expectation comes before the action.
Requirement: Checking that the mock was called with the correct parameters.
Mockito: Allows specifying either actual values, or matchers of the values. Does not allow mixing actual values and matchers in the same call. Also has the ability to “capture” the parameters, and then doing custom logic to verify that the correct parameters were passed to the function.
JMock: Allows specifying either actual values, or matchers of the values. Does not allow mixing actual values and matchers in the same call.
Requirement: Verifying that a mock was not called.
Mockito: Needs to be done explicitly.
JMock: Throws UnexpectedInvocationException if a mock is called without setting the expectation.
Requirement: Calling a mock that has not been set up.
Mockito: Returns “null” — so the production code becomes subject to unexpected NullPointerExceptions, sometimes making it difficult to find the problem.
JMock: Throws UnexpectedInvocationException, with a clear error message stating exactly which call and parameters did not match the expectation.
Requirement: Mocking Classes instead of Interfaces.
Mockito: Works.
JMock: Designed to mock interfaces only. In order to mock Classes requires using a class imposterizer.
Requirement: Mocking multiple instances of the same Class / Interface.
Mockito: Works.
JMock: Uses names for the mocked instances, and the default name is the interface name, so mocking multiple instances, throws an exception that the named mock already exists. To overcome this requires specifying a name for each instance for example orderDataStore1 = mock[OrderDataStore](name = "OrderDataStore1")
Requirement: Returning different results on consecutive invocations of the mock.
Mockito: Has the syntax returns(firstResponse).thenReturns(secondResponse), which is more readable. Also allows specifying multiple return values in one statement returns(firstResponse, secondResponse). Setting the return value multiple times, the last one “wins” and will be returned on all calls. This is more intuitive.
JMock: The correct and more readable syntax is to use onConsecutiveCalls, which allows for a list of return values. Setting the return value multiple times, has different behaviour depending whether the first time uses oneOf or allowing. When using oneOf to setup the first call, subsequent calls return the values as defined in subsequent setup. When using allowing to setup the first call, subsequent calls always return the first value. This is confusing, and not intuitive and can lead to unexpected results and hidden bugs.
Requirement: Resetting the mock.
Mockito: Has a reset function, to clear all the setup.
JMock: I have not found a way to reset mocks.
Integration Testing
Integration tests refer to a level of testing which should have very limited use of mocks, and use real implementations for all the internal classes. Usually these tests start up the application in an environment that is as close to production as possible, and tests the APIs as if being called by production clients. Integration tests ensure that the integration with external components is working correctly. This requires a running version of these resources. External components include databases, message queues, external third party APIs and other internal microservices.
The Wix integration testing infrastructure provides the ability to start up an embedded server with a running instance of the microservice under test, together with in-memory / embedded or dockerized versions of the databases and message queues within the test environment.
For handling HTTP requests to external third party APIs, there is an HTTP testkit which allows starting up an embedded server that responds to HTTP requests. The testkit allows developers to implement a “fake” version of the API, and control the response to specific requests.
So what remains is handling communications with other internal microservices. Getting a running version of other microservices is usually not feasible in the test environment, and so there is a need for a mock implementation that mimics the real microservice. There are 3 common strategies for handling this.
1 — A testkit developed and maintained by the microservice owners. This is usually an in-memory “light-weight” version of the actual service, and often includes some of the business rules and validations that exist in the real service, allowing callers to get build-time feedback that they are using the service correctly. Sometimes testkits also include other helper utilities such as builders for building the response objects of the service, making it easy to set up the responses to the API calls. When the APIs change, the owners need to make the relevant changes in the testkit also. One of the downsides is that there is additional development and maintenance overhead. Another disadvantage is that the testkit can only be used by clients from the same language platform, for example a testkit written in a JVM language cannot be used by Node JS clients.
2 — A testkit developed and maintained by the caller of the microservice. This is very similar to the first strategy, except that it will usually be missing the business rules and validations. But there are a few disadvantages. As in the first strategy there is an additional cost of development and maintenance of the testkit. Also when the API changes, and the testkit is in a different code repository, the service owner will not know to update the testkit, and results in a failing build for the client code repository. If the microservice is used by many clients, there will be multiple versions of testkits developed by each of the client developers.
3 — Using a mocking library to mock the interface of the API. This strategy has the benefit of not requiring any additional development and maintenance of testkits. It is a familiar process, because it is done in the unit tests too. Additionally it requires mocking only the API functions that are relevant for the test, instead of implementing all functions of the API, as is required for a testkit. The disadvantage is that each required function needs to be mocked, instead of using an in-memory datastore to keep the state. For example a product catalog API, could expose the functions listProducts, getById, getByName etc. Using an in-memory datastore will work for all the functions, whereas using a mock requires setting up the mock response for each function.
When choosing a mocking library for integration tests, Mockito is the preferred choice. JMock is not well suited for integration tests because it does not have the ability to reset the mock, and because of the unclear and inconsistent behaviour when specifying the same expectations multiple times. These problems cause tests to not be isolated, and the setup of the mock for one test leaves the mock in a state that can have an impact on other tests. When calling a function that has not been mocked as expected, JMock throws an UnexpectedInvocationException, but unlike in unit tests, the informative stacktrace gets lost, and it is difficult to determine which mock was not setup correctly. It also leaves the mock in an unstable state, and other non related tests also fail with UnexpectedInvocationExceptions. Also verification of expected calls does not work well in integration tests, when defining expectations that a certain function is called, the tests only fail if the wrong parameters are passed, if the function is not called at all, the tests will pass.
The second part of this series was focused on comparing Mockito and JMock, 2 of the JVM based mocking libraries that we use at Wix. We also discussed some of the other strategies for creating mocks for integration tests. Don’t miss the next part of the series where we will look at some of the pitfalls that come along with using mocks.
