How to send large SQS/SNS messages with Node.js

One of the common problems that people have with SQS and SNS is a message size limit. At the time of this writing, you can only send messages that are less than 256 KiB in size, which may be not enough in some scenarios and use cases. Since it’s a common problem there’s a common solution — use S3 to store the payload and only send a link to the S3 object via SQS or SNS. There’s even an extended Java client created by AWS Labs to do exactly that.

At Aspecto we use Node.js and TypeScript for our microservices. We also have a one-to-many scenario, where we need to send the same message to multiple queues, so we needed a solution that would cover the SNS-to-SQS case as well (which is not supported by Java library mentioned above). We couldn’t find a package that meets all our requirements on npm, so we decided to create one: https://www.npmjs.com/package/sns-sqs-big-payload. In this article, I’ll try to explain how it works and how to use it.

SQS producer to SQS consumer

SQS producer-consumer

Let’s start with SQS producer/consumer since it’s a little bit simpler. In order to send a big payload, we need to provide 2 options: largePayloadThroughS3 and s3Bucket. Then we can send messages with sendJSON method.

import { SqsProducer } from 'sns-sqs-big-payload';
const sqsProducer = SqsProducer.create({
    queueUrl: '...',
    region: 'us-east-1',
    largePayloadThoughS3: true,
    s3Bucket: 's3bucket-with-message-payloads',
});
await sqsProducer.sendJSON({
    // some very big json object > 256 KiB
});

largePayloadThoughS3 option works in the following way: if payload size is larger than SQS can handle (i.e. 256 KiB) then it’s gonna upload it to S3 first, and then it will send the metadata to SQS. In this case, the message body will look like this:

{
  "S3Payload": {
    "Id": "cda6498a-235d-4f7e-ae19-661d41bc154c",
    "Bucket": "s3bucket-with-message-payloads",
    "Key": "cda6498a-235d-4f7e-ae19-661d41bc154c.json",
    "Location": "https://s3bucket-with-message-payloads.s3-region.amazonaws.com/cda6498a-235d-4f7e-ae19-661d41bc154c.json"
  }
}

To receive and process this message you should use SqsConsumer. If you want the consumer to download the payload automatically for you — specify the getPayloadFromS3 option, like so:

import { SqsConsumer } from 'sns-sqs-big-payload';
const sqsConsumer = SqsConsumer.create({
    queueUrl: '...',
    region: 'us-east-1',
    // to enable loading payloads from S3 automatically
    getPayloadFromS3: true,
    parsePayload: (raw) => JSON.parse(raw),
    // message handler, payload already downloaded and parsed at this point
    handleMessage: async ({ payload }) => {
        // do whatever you want with the payload
    },
});
sqsConsumer.start();

There is no need to specify the s3 bucket location in SqsConsumer as it is already specified in the SQS message.

SNS producer to SQS consumer

SNS producer — multiple SNS consumers

Publishing message to SNS is almost exactly the same as for SQS:

import { SnsProducer } from 'sns-sqs-big-payload';
const snsProducer = SnsProducer.create({
    topicArn: '<topic name>',
    region: 'us-east-1',
    // to enable sending large payloads (>256KiB) though S3
    largePayloadThoughS3: true,
    s3Bucket: '...',
});
await snsProducer.sendJSON({
    // your big JSON payload
});

When SQS receives a message from SNS, it comes in an envelope which typically looks like this:

{
"Type" : "Notification",
  "MessageId" : "63a3f6b6-d533-4a47-aef9-fcf5cf758c76",
  "TopicArn" : "arn:aws:sns:us-west-2:123456789012:some-topic",
  "Subject" : "Testing publish to subscribed queues",
  "Message" : "<your message goes here>",
  "Timestamp" : "2012-03-29T05:12:16.901Z",
  "SignatureVersion" : "1",
  "Signature" : "EXAMPLEnTrFPa3...",
  "SigningCertURL" : "https://sns.us-west-2.amazonaws.com/SimpleNotificationService-f3ecfb7224c7233fe7bb5f59f96de52f.pem",
  "UnsubscribeURL" : "https://sns.us-west-2.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-west-2:123456789012:MyTopic:c7fe3a54-ab0e-4ec2-88e0-db410a0f2bee"
}

In order to get a payload you need to unwrap the envelope first, that’s where transformMessageBody callback comes in handy:

const sqsConsumer = SqsConsumer.create({
    queueUrl: '...',
    region: 'us-east-1',
    getPayloadFromS3: true,
    transformMessageBody: (body) => {
        const snsMessage = JSON.parse(body);
        return snsMessage.Message;
    },
    parsePayload: (raw) => JSON.parse(raw),
    handleMessage: async ({ payload }) => {
        // do whatever you want with the payload
    },
});

The rest of the options are exactly the same as in the previous case.

Deleting payloads from S3

When the payload is uploaded to S3 — it stays there forever, as the library never deletes it. Obviously it may become a problem in the long term if we don’t take care of it. There are two things we can do:

• delete the payload from S3 manually during processing (not recommended, because in case of one-to-many scenario, we don’t know if all consumers have processed it already)
• setup a lifecycle rule for the bucket to delete all objects that are older than 2 weeks (the easiest and the аmost reliable approach). You should also take into account, that the maximum message retention period for SQS is 2 weeks as well.

MessageRetentionPeriod – The length of time, in seconds, for which Amazon SQS retains a message. Valid values: An integer representing seconds, from 60 (1 minute) to 1,209,600 (14 days). Default: 345,600 (4 days).

A few other things to consider when using the library:

• Uploading and downloading the payload to S3 takes some time and usually is much slower than sending the message to SNS or SQS. Since we need to wait for the upload/download process to continue before sending/receiving the message — it may slow down a message processing rate. See if it’s ok for your use case.
• Currently, the library has a possibility to send a JSON payload because that’s our main and only use case. But it’s very easy to extend it to send any types of the payload, be it binary or textual. PR’s are welcome 🙂.

Conclusion

Sending larger messages over SNS or SQS is totally possible, it just requires a little bit more work. Moreover, S3 is not the only option, you may want to use Redis for example. We also thought about using DynamoDB for storing payloads, but it has a limitation of 400 KiB per item:

The maximum item size in DynamoDB is 400 KB, which includes both attribute name binary length (UTF-8 length) and attribute value lengths (again binary length). The attribute name counts towards the size limit.

So in my opinion, S3 is the most convenient since we’re working with AWS and it doesn’t require any additional setup.