Recording and playing back network traffic in Node.js
A handy way to bootstrap your integration tests
Integration tests play an important role in a service’s development life cycle, helping assert its correct behavior by simulating how it’ll be used in production. This means running the service with all its dependencies — which might be problematic if those dependencies are external network resources.
At AppsFlyer, we have hundreds of microservices that handle requests from the UI, external APIs and other internal services. To create effective integration tests that run smoothly in our CI environment, we need to make sure that a service’s external network connections are properly mocked.
You already know what this requires: recording and then playing back network traffic. Luckily, several Node.js tools provide this capability.
In this post, I’ll show you a quick and useful way to bootstrap your tests to use network recording and playback, which can be utilized as the basis for all your integration tests.
Just before we start — the ingredients
What you should already have in place
I assume you have a server implementation that exposes one or more HTTP endpoints. The implementation needs to be written in such a way that the server can be started and stopped programmatically.
Tools
In the example below, we will use Jest as the testing framework. Specifically, we’ll use Jest hooks to set up our network mock. If you’re using another testing framework, you should find the equivalent hooks your framework provides, and you’re good to go.
Additionally, we’ll use Nock to help us with our network activity. Nock is an npm library for overriding, recording and playing back network traffic. Nock takes control of the underlying native implementation of HTTP in Node.js, so no matter what HTTP client you use in your code, Nock will be able to override it.
Lastly, we’ll use Axios to act as the client that calls the tested endpoints.
What a test run looks like
We’ll create a test, and wrap two hooks around it — beforeAll() and afterAll(). Using these, we’ll set up the recording/playback of network traffic:
- When we run the test with a RECORD=true flag, the hooks will record live network traffic and save it to a file
- When we don’t add the RECORD flag (which is the usual way we’ll run it), the test will playback the saved network traffic
Are you ready? Then let’s start coding! We’ll do this in three coding iterations.
First iteration: write just the test itself
In this iteration, we write the test itself without mocking anything. We just run the server, call the endpoint under test, and assert that the response matches our expectation.
const axios = require('axios');
const myApp = require('...'); // point to your app code here
let baseUrl;
beforeAll(async () => {
const { port } = await myApp.startup();
baseUrl = `http://localhost:${port}`; // the base URL for your app
});
afterAll(async () => {
await myApp.shutdown();
});
describe('integration tests', () => {
test('get-data', async () => {
const headers = {
uuid: 'user@company.com',
accountId: 'acc-1234567'
};
const res = await axios({
method: 'get',
url: `${baseUrl}/my-app/get-data`,
headers
});
expect(res.data.user.accountId).toBe('acc-123456');
});
});As you can see, this is a fairly common test. When we run $ jest the following occurs:
- It starts up the server in the beforeAll() hook
- It runs the test by calling the /get-data endpoint using Axios and asserts that the response matches our expectations
- It shuts down the server in the afterAll() hook
Not much magic here — the server accesses live network resources as it responds to the request made by the test.
Second iteration: set up network recording
When we run the first iteration, the server is free to talk to any external resource, such as other servers or databases. However, in a CI environment, we may not have the luxury of having those network resources available, or the certainty that they’ll return the same expected responses every time. That is why we need to ensure that whatever passes through the network is recorded and then played back in subsequent tests.
To that end, we’ll add a recording mechanism. This needs to be used only once initially. After that, we use it only when we know that something has changed — either in the implementation of our own server, in the network traffic from external resources, or both.
We’ll add a flag to signal whether we’re in “record” mode, and when that flag is true, our code will write all network traffic to a specific mock file.
Add the following code to the beginning of the file:
const fs = require('fs');
const nock = require('nock');
const recordFilename = `${__dirname}/integration.get-data.recording.json`;Add this code to the beginning of the beforeAll() hook:
if (process.env.RECORD === 'true') {
// delete the recording file
fs.rmSync(recordFilename, { force: true });
// prepare nock to record
nock.recorder.rec({
output_objects: true,
dont_print: true,
enable_reqheaders_recording: true,
enable_respheaders_recording: true
});
} else {
// third iteration beforeAll code goes here
}Add this code to the end of the afterAll() hook:
if (process.env.RECORD === 'true') {
// get the playback from nock and write it to a file
const interactions = nock.recorder.play().filter(interaction => interaction.scope !== baseUrl);
fs.writeFileSync(recordFilename, JSON.stringify(interactions, null, 2));
} else {
// third iteration afterAll code goes here
}Note that we don’t want to record traffic from calls to baseUrl. This is our own server, and what the test actually calls.
Now, when running $ RECORD=true jest, our test code records all network traffic and saves it into the recordFilename, which will be used in the third iteration.
Third iteration: run the test using playback
In the third — and final — iteration, we’ll playback our recorded network traffic and block all the real network activity.
In the else section of the beforeAll() hook, add this code:
} else {
// only allow localhost connections
nock.disableNetConnect();
nock.enableNetConnect('localhost');
try {
nock.define(nock.loadDefs(recordFilename));
} catch (err) {
throw new Error('Cannot load the recordings file');
}
}And in the else section of the afterAll() hook, add this code:
} else {
// enable back all network traffic
nock.enableNetConnect();
}Now, when you run $ jest, the test will block all network traffic (excluding localhost — that’s our own server), run the test using the recorded traffic, and then unblock network traffic. We unblock the network traffic because other tests may still want to use it.
One last touch: update your npm scripts
You can add the following scripts to your package.json file:
"scripts": {
"test": "jest",
"test:record": "RECORD=TRUE jest"
}Use $ yarn test:record when you need to re-record your network traffic, for example, when other services have changed their behavior. Don’t forget to push the updated recording file to source control!
Use $ yarn test to run the tests using the recorded network traffic. They’ll be available to you and anyone else who cloned your repo — and, of course, to your build machine.
We’re done! But this is only the beginning
What you have right now is a useful way to set up recording and playback of network traffic for your integration tests.
Going forward, you can:
- Extract the beforeAll() and afterAll() hooks to a test utility file or library, and you’ve got a tool that can be used in any Node.js service in your company
- Use Sinon when network traffic isn’t reliable (for example, when sending requests to databases or queues)
- Pass a list of allowed network hosts (right now, we’re only allowing localhost) where traffic shouldn’t blocked — such as resources that are available to the CI build machines
- Add your integration tests to your test:watch script — now that you know that everything is local and no real network is involved, you can run them every time you save a file
Remember — good integration tests should be indistinguishable from unit tests!
Integration tests should not be considered heavy or cumbersome. In fact, they shouldn’t even take up too much time running. They should cover your whole flow, provide excellent coverage, and give you a quick and efficient way to know if you or someone else just broke something. And, with the setup you just learned, you have one less excuse not to use them every single day!
