Tips on unit testing AWS lambda’s SNS/SQS communication with Node

Marek Krzynówek
Dec 12, 2019 · 7 min read

In AWS, inter-lambda communication often is achieved by the SNS and SQS working together to provide a publish subscribe type messaging. Once your setup becomes larger, with more lambdas communicating to even more lambdas, you would like to quickly make sure correct communication is being sent and received within your code. I would like to share with you a few utility functions, and an approach to make testing this as easy as possible. We all know, tests that are hard to write don’t get written, so the easier it is, the better.

TL;DR

Below you can find all the utility functions in the test-utils.js, as well as the usage examples with hello.js and hello.test.js.

Test-utils.js -> https://gist.github.com/marekkrzynowek/72d4f7bf66f61ef278dc9383688516be

Hello.js -> https://gist.github.com/marekkrzynowek/23745c9777f2fd35ec049e941bd49949

Hello.test.js -> https://gist.github.com/marekkrzynowek/679e37c2fc3d44ec98f14429e53a9169

Let me walk you through the utility functions, with an example of a simple hello lamda. The lambda contains one function -> helloWorld(). This function is designed to be triggered by an SNS payload delivered via the SQS. The function extracts the payload from the SNS event, and repackages this payload into a new SNS message to be processed further. The return of the function should be HTTP code -> 200 with a summary of all messages delivered in this batch.

module.exports.helloWorld = async (event, context) => {
context.callbackWaitsForEmptyEventLoop = false;

try {
const msgs = [];
// SQS may deliver messages in batches of records.
for (const r of event.Records) {
const payload = _extractMessageFromPayload(r);
// For each input SNS publish an output SNS
await utils.publishToSNS(
new AWS.SNS({ apiVersion: '2010-03-31' }), {
Message: JSON.stringify(payload),
TopicArn: process.env.HELLO_TOPIC,
},
);
msgs.push(payload.message);
console.log('Payload', payload);
}
// Return all messages
return utils.buildResponse({
statusCode: 200,
body: {
message: msgs.join(',')
}
});
} catch(err) {
console.log('Error: ', err.message);
throw err;
}
};

Now we would like to test that method. We need to invoke the function with a simulated SNS message, ensure the return is correct, and make sure that the correct SNS is published. Normally this would involve a lot of painful glue code, parsing, and decorating, which can discourage one from testing this behaviour at all. However using few handy functions from the test-utils.js file can make the test easy to setup and focus on testing the business logic.

To start with, we need to prepare the SNS spy to be able to measure the SNS messages sent from our function under test. I like to do it before and after test hooks.

beforeEach( async function() {
// We need to setup sns spy
SNS_SPY = sinon.spy((params, callback) => {
callback(null, {});
});
AWS.mock('SNS', 'publish', SNS_SPY);
});

afterEach(async () => {
// We need to clear the SNS spy
AWS.restore('SNS', 'publish');
});

Now for the test itself:

it('Hello world', async () => {
const payloadFixture = { message: 'Hello World' };
// Wrap the payload into an SNS payload
const sqsPayload = utils.generateSNSFixture(payloadFixture);
// Wrap the payload into an event
const event = utils.createAwsEvent({
template: 'aws:SQS,
merge: sqsPayload
});
// Call the function passing the event
const resp = await hello.helloWorld(event, {});
// Expect return
expect(resp.statusCode).toBe(200);
expect(resp.body).toBe(JSON.stringify(payloadFixture));
// Expect an SNS to be fired with a payload
utils.expectSNSWithPayload(SNS_SPY, process.env.HELLO_TOPIC, payloadFixture);
});

A deeper dive.

Here I will go through all the utility functions one by one, and describe a potential use for each one.

generateSNSFixture(message)

This function will help with generating the fixtures for testing the lambdas that are invoked by the SNS/SQS messages. You can wrap any kind of payload to mimic an SNS message. For example this line

const sqsPayload = utils.generateSNSFixture(payloadFixture);

will assign wrapped payloadFixture to sqsPaylod that can be sent to the lambda as parameter.

Source code:

/**
* This function wraps the message object in SNS context so it simulates amazon SNS payload.
*
* @param {*} message: Payload to be included in th SNS
* @param {string} topicArn: Optional: topic the SNS was sent to
* @param {string} sourceArn: Optional: SNS source topic
* @returns SNS message
*/
fns.generateSNSFixture = function(message, topicArn = 'topic_Arn', sourceArn = 'source_arn') {
const body = {
Type: 'Notification',
MessageId: '694e2d00-afd6-5065-a525-7932afa3652e',
TopicArn: topicArn,
Message: JSON.stringify(message),
Timestamp: new Date().toISOString(),
SignatureVersion: '1',
Signature: 'K+szeNJUc/q8aQ4LcDzfwBJt1/Q==',
SigningCertURL: 'https://sns.us-east-1.amazonaws.com/url',
UnsubscribeURL: 'https://sns.us-east-1.amazonaws.com/?url'
};
const fixture = {
Records: [
{
messageId: 'eccfe821-0a8e-4ab8-aa8c-cb818c0dcc1a',
receiptHandle: 'N1254aOMj35YWpAVbUzzx25rvGL39CC3NbL9+ZDaSK43Rgafo3+LDJPjydxF0LkAQ1+e1giDiL0=',
body: JSON.stringify(body),
attributes: {},
messageAttributes: {},
md5OfBody: 'be61a9f98ddd5dd9ef53e5e0f62b67fe',
eventSource: 'aws:sqs',
eventSourceArn: sourceArn,
awsRegion: 'us-east-1'
}
]
};

return fixture;
};

createAwsEvent(config)

This method is a wrapper fixing the aws-mock implementation which has a bug that causes modifications to be remembered between calls.

Usage example:

const event = utils.createAwsEvent({
template: 'aws:SQS,
merge: sqsPayload
});

Source code:

/**
* @param {*} config event parameters
* @return {*} event mock
*/
fns.createAwsEvent = function(config) {
const event = createEvent({ template: config.template });
const eventClone = { ...event };
return _.has(config, 'merge') ? _.merge(eventClone, config.merge) : eventClone;
};

checkCorsHeadersPresent(headers)

Test for the existence of headers necessary for the endpoint to support CORS requests. This I found was often a test I would like to perform quickly on the API facing methods. It is easy to forget, and not very pleasant when the clients get errors trying to connect to your endpoints.

Usage example:

checkCorsHeadersPresent(resp.headers);

Source code:

/**
*
* @param {*} Response headers
*/
fns.checkCorsHeadersPresent = function(headers) {
expect(headers['Access-Control-Allow-Origin']).toBe('*');
expect(headers['Access-Control-Allow-Credentials']).toBe(true);
};

expectExactSNSMessages(spy, expectation)

This is the most complicated, but also the most powerful method. It will take in an array of predefined topic and payload pairs, and ensure that these, and only these messages were sent. It offers the most control over the function under test, but also requires the most setup.

Usage example:

const msgs = [
{ topic: process.env.SNS_UPDATED, payload: { model } },
{ topic: process.env.SNS_DONE, payload: { model } },
{ topic: process.env.SNS_CANCELLED, payload: { model } },
{ topic: process.env.SNS_UPDATED, payload: { model } }
];
utils.expectExactSNSMessages(SNS_SPY, msgs);

Source code:

/**
* Test the exact number of sns send to a spy with the correlating topics and optionally payloads.
* You should construct the 'expected' array to contain all Messages that the test method will fire
* Once you pass this array the tests will ensure that all of these messages were fired and only these messages.
* It does not test the message firing order.
*
* @param {*} spy: Sinon spy
* @param {*} expected: Array of Messages to test for:
* [
* {topic: 'SNS_USER_CREATED', payload: { user }},
* {topic: 'SNS_USER_UPDATED' }
* ]
*/
fns.expectExactSNSMessages = function(spy, expected) {
const snss = spy.getCalls().map(call => {
const params = call.args[0];
return { topic: params.TopicArn, payload: JSON.parse(params.Message) };
});
expect(expected.length).toBe(snss.length);
for (const msg of expected) {
const toRemove = _findSNSIndex(snss, msg);
if (toRemove !== undefined && toRemove >= 0) {
snss.splice(toRemove, 1);
}
}
expect(snss.length).toBe(0);
};

getSNSPayloads(spy, topic)

Sometimes you have various expectations about the payloads sent to the SNS. With this method, you can export all the SNS payloads, and make expectations without the noise of the SNS wrapper values.

Usage example:

const snsMemberships = utils.getSNSPayloads(SNS_SPY, process.env.SNS_MEMBERSHIP_ARCHIVED_TOPIC_ARN);

Source code:

/**
* Returns all the payloads for the topic
*
* @param {*} spy; Sinon spy
* @param {*} topic: SNS topic
*/
fns.getSNSPayloads = function(spy, topic) {
const payloads = [];
for (const spyCall of spy.getCalls()) {
const params = spyCall.args[0];
if (params.TopicArn === topic) {
payloads.push(JSON.parse(params.Message));
}
}
return payloads;
};

expectSNSTopic(spy, topic, count)

This function asserts that a message was passed to an SNS topic at some point during the test a `count` number of times.

Usage example:

utils.expectSNSTopic(publishToSNSSpy, process.env.SNS_DIMENSION_ARCHIVED_TOPIC_ARN, 0);

Source code:

/**
* Test that messages was send to the topic.
*
* @param {*} spy: Sinon spy
* @param {string} topic: SNS topic name
* @param {int} count: Expected number of messages sent to the topic defaults to 1
*/
fns.expectSNSTopic = function(spy, topic, count = 1) {
let found = 0;
for (const spyCall of spy.getCalls()) {
const params = spyCall.args[0];
if (params.TopicArn === topic) {
found += 1;
}
}
expect(found).toBe(count);
};

expectSNSWithPayload(spy, topic, payload)

This function will assert a defined SNS topic receives the message with the exact payload.

Usage example:

utils.expectSNSWithPayload(publishToSNSSpy, process.env.SNS_MEMBERSHIP_ACTIVATED,
{ user: userFixture,
membership: activeMembershipFixture
});

Source code:

/**
* Tests that the exact payload was sent to the topic.
* Prints a diff of payloads for all payloads sent to the topic for debugging.
*
* @param {*} spy: Sinon Spy.
* @param {string} topic: SNS topic name.
* @param {*} payload: Expected payload sent.
*/
fns.expectSNSWithPayload = function(spy, topic, payload) {
let found = 0;
for (const spyCall of spy.getCalls()) {
console.log('==');
const params = spyCall.args[0];
if (params.TopicArn === topic) {
const jsonSns = JSON.parse(params.Message);
console.log('Found a call for topic:', topic);
console.log('-');
console.log('Expected SNS Payload:', payload);
console.log('-');
console.log('Acutal Payload:', jsonSns);
console.log('-');
const differences = diff(jsonSns, JSON.parse(JSON.stringify(payload)));
if (!differences) {
console.log('Found a match');
found += 1;
} else {
console.log('Differences: ', differences);
}
console.log('==');
}
}
expect(found).toBe(1);
};

expectNoCallToSNSTopic(spy, topic)

It expects… well nothing really :) It will fail if the topic receives a message.

Usage example:

utils.expectNoCallToSNSTopic(publishToSNSSpy, process.env.SNS_MEMBERSHIP_ACTIVATED);

Source code:

/**
* Tests that the topic had no messages sent to it.
*
* @param {*} spy: Sinon spy.
* @param {string} topic: SNS topic.
*/
fns.expectNoCallToSNSTopic = function(spy, topic) {
for (const spyCall of spy.getCalls()) {
const params = spyCall.args[0];
expect(params.TopicArn).not.toBe(topic);
}
};

Summary

So this article is probably not the mount everest of software engineering, and makes no insightful investigation into the nature of computer science, but what it will do is make you test more. The methods I described here are simple parsing and decorating methods, but I found these extremely useful in the world of lambdas. At some point, our infrastructure was running over 100 lambdas, all communicating with each other with SNS/SQS mechanism, and making changes became very stressful. These few functions helped eliviating parts of this stress. I hope they will help you too.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade