Sep 7, 2016
Use Easy-Fix to Run Integration Tests Like Unit Tests
Software is complex, so testing it is difficult. To make tests comprehensible, engineers divide up responsibilities between different types of tests. Unit tests are comparatively simple — specific and granular, they narrowly target methods and functions in isolation. Integration tests focus on the interaction between software components. For web projects, this includes web browsers interacting with web services, though in this article we’ll simply focus on web services.
Compared with unit tests, integration tests are broader in scope and slower in execution. They may, for example, run “live” (in a test environment) where they interact with remote systems or mutate a database. Such complexities have a cost: the tests require more time and effort to set up, to run, and to understand when they fail. It seems tempting to avoid this complexity, by “mocking” the integration tests. Integration tests can be designed to run with curated “mock” data, loaded from local storage, which mimics the interactions with other components.
It is not immediately clear if testing with mock data is a good idea. Tests with mock data run quickly and consistently, like unit tests, which is appealing. However, mocked integration tests don’t actually interact with remote systems — the mock data limits the scope of the tests. If the remote system changes, the mock data can become stale, invalidating the test results and potentially providing false confidence in the components under test. Live integration testing takes time, but when it succeeds, it provides real confidence that the target software components actually work together.
Don’t compromise between mock and live integration tests
Both approaches to integration testing have merits, so it’s tempting to think of this as a trade-off. We suggest, however, that you don’t need to make any such compromises with your integration tests. Choosing between mock tests and live tests is a false alternative — it’s possible to write your integration tests such that they can run either way. Further, we’re happy to introduce a new nodejs module, “easy-fix” (https://github.com/walmartlabs/easy-fix) which encourages this uncompromising approach to integration tests. Easy-fix facilitates a single set of integration tests that can run live, capturing the data that passes between the target components, and then the same tests can be replayed with the captured (mock) data.
Easy-fix works like this:
- Write your integration tests such that they run live, allowing network requests, database mutations, etc;
- Wrap the side-affecting asynchronous tasks with a method from easy-fix.
- Run those tests, which have a new option:
- In ‘live’ mode, your tests run just as you wrote them, side effects allowed.
- In ‘capture’ mode, your tests run like ‘live’ mode, but the arguments and response for each wrapped task are serialized and saved.
- In ‘replay’ mode, the wrapped tasks are not called. Instead, they respond with the saved (mock) data.
In summary, easy-fix provides an easy way to capture test data, and feed this data back into the tests.
Usage Example
Here’s a code example using easy-fix. In the test setup method, we use easy-fix to wrap some asynchronous tasks. Of course, the test author decides what tasks should be wrapped. To test a walmart.com checkout, for example, we might wrap the internal tasks that authenticate the request, get fees & tax details, etc.
before( function () {
easyFix.wrapAsyncMethod(helpers, 'authenticate');
easyFix.wrapAsyncMethod(isd.IroRequest.prototype, 'getProductOffers');
easyFix.wrapAsyncMethod(isd.TaxRateRequest.prototype, 'getTaxRates');
easyFix.wrapAsyncMethod(restrictions, 'getRestrictonsForUpcs');
easyFix.wrapAsyncMethod(productFees, 'getFeesForUpcs');
});The actual test code doesn’t need to be modified for easy-fix. A test of a walmart.com checkout might look like this:
it('checkout one item', function (done) {
attemptCheckout(
testUser,
checkoutDetails,
function (err, resp, body) {
var bodyJson;
expect(err).to.be.null;
expect(resp.statusCode).to.equal(200);
bodyJson = typeof (body) === 'string' ? JSON.parse(body) : body;
expect(bodyJson.data).to.exist;
expect(bodyJson.error).to.not.exist;
done();
}
);
});Note that the test code doesn’t show any awareness of easy-fix. When we execute the test, though, the $TEST_MODE environment variable will determine if the tests interact with remote systems. In “live” and “capture” modes, the checkout service may make it’s related network requests and record the checkout in the database. In “replay” mode, the test does none of this, running only with fake mock data.
➜ export TEST_MODE=live
➜ mocha test/int/endpoints-with-auth.js -t 20000 -g 'checkout one item' ✓ checkout one item - UPCE (12704ms) 1 passing (16s)➜ export TEST_MODE=capture
➜ mocha test/int/endpoints-with-auth.js -t 20000 -g 'checkout one item' ✓ checkout one item - UPCE (14311ms) 1 passing (17s)➜ export TEST_MODE=replay
➜ mocha test/int/capture.js test/int/endpoints-with-auth.js -g 'checkout one item' ✓ checkout one item - UPCE (71ms) 1 passing (224ms)
Here at Walmart Labs, on the Store Services team, we started writing our integration tests to run live. An example test suite with a few dozen integration tests took around two minutes to run. That’s not the end of the world, but it’s slow enough that engineers would typically run them only when they considered committing new code. After integrating easy-fix, these tests run (in ‘replay’ mode) in 4 seconds. The tremendous speed-up allows the engineers to run the tests (in replay mode) much more frequently — such as every file save.
Fast but not magical
At first glance, this might seem like a fantastical improvement — execution time for our example test suite was cut from 120 seconds to 4 seconds. When you consider the difference between the test modes, though, the difference in speeds seems unsurprising. Tests running in the ‘live’ or ‘capture’ mode include the interactions between the target components. They run comparatively slowly, but provide the confidence that these components work together.
Tests in ‘replay’ mode, using mock data, typically test only one side of the target interaction. In practice, tests in replay mode still catch many (but not all) errors. The quick execution allows them to be run more frequently, finding many errors earlier in the programming process.
Conceptually, easy-fix is another record-playback system, which is nothing new. There are several examples of other projects that do this, such as
All these examples, though, focus exclusively on mocking HTTP calls. Strangely, we couldn’t find any existing record-playback systems that worked for other asynchronous tasks, which is why we created easy-fix.
To be clear, the crucial difference is that easy-fix will wrap any asynchronous nodejs method. Easy-fix can mock HTTP calls, database access, or anything else where you can serialize the requests and responses.
Software testing is still complex, but easy-fix provides one single, useful simplification. With easy-fix, you don’t need to compromise between mock or live integration tests.
The easy-fix source code is available at: https://github.com/walmartlabs/easy-fix
