Store your bounced email notifications from AWS Simple Email Service (SES) in an S3 bucket

Photo created by natanaelginting — www.freepik.com

Context

At Inato, we are using Auth0 to have a passwordless authentication to our app. When logging, our users receive an email with a code to use instead of keeping a password.

While Auth0 offers an email provider for testing purpose, they recommend you use your own email provider for production. We decided to use AWS SES as there’s an easy integration with Auth0 and we already had a company account for AWS.

Using this kind of authentication makes the emails critical. We need them to be delivered to our users and we need to know what happened when they are not, particularly on bounced emails. A email bounces, for example, when the email address does not exist or when a mail server rejects it for security reasons. In order to troubleshoot our user’s login/signup issues, we need a good observability on these bounces. Understanding the reason why they bounced can help us tell our users, for example, that they mistyped their emails or that they need to update the security rules of their email server so our emails are not rejected anymore.

You usually receive information about your bounced emails as a reply of the emails. Sadly Auth0 does not allow to set the Return-Path of the emails sent so we were not receiving these replies. The email address used to send our emails is also managed by our support team and we did not want our engineers to have access to it, anyway.

While SES provides some statistics about bounced emails, you cannot see any information about them directly in the SES console. It however provides notifications with some details on the bounced email (bounce reasons, email headers… see an example here). With a few AWS services, you are able to store the notifications in an S3 bucket that you can browse when you need.

I’ve used this very useful post from Trend Micro to build the solution. Thanks to them!

Solution

AWS provides an easy way to redirect the bounced emails notifications to an SNS (AWS message queue service) topic:

We’ll then add a lambda subscribed to the SNS topic that will get the notifications and store them as JSON in an S3 bucket.

Let’s do this!

We’ll describe here how to set up this solution using the AWS management console. You’ll need:

• An admin access to an AWS account (or write access to at least SES, SNS, IAM, lambda and S3)
• A configured SES with a verified email address (see this AWS tutorial if you need to set this up)

1. Create an SNS topic

• Go to SNS in the console and click on Topics in the left navigation panel
• Click on Create topic
• Enter a name (ses-bounced-emails for example)
• Click on Create topic

See this AWS tutorial for more information about topic creation if you need.

2. Create an S3 bucket

• Go to S3 in the console
• Click on Create bucket
• Enter a name (which needs to be unique among all the buckets that exist in all the AWS S3 in the world. I chose bounced-emails-{random number} but you’re free to chose whatever unique name that you want)
• Chose a region and click on next
• Click on next (you can set the options you want but none are necessary for this)
• Make sure Block all public access is ticked (you don’t want the whole world to be able to download your bounced emails) and click on next
• Click on Create bucket

See this AWS tutorial for more information about bucket creation

In the buckets list, click on the checkbox of your bucket to open a panel. Click on Copy Bucket ARN at the top of the panel and keep it somewhere as we’ll use it in the next step (ARN are like resource ids in AWS)

3. Create a role that will allow our lambda function to write documents in the S3 bucket

• Go to IAM in the console and click on Policies in the left navigation panel
• Click on Create policy
• Click on the JSON tab and paste (this is the time to use your bucket’s ARN):

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:PutObjectAcl"
            ],
            "Resource": [
                "{put your bucket ARN here}",
                "{put your bucket ARN here}/*"
            ]
        }
    ]
}

• Click on Review policy (don’t worry, you’ll enter the name of the policy just after that)
• Add a name (like S3BucketBouncedEmailsCreateDocumentsPolicy) and a description if you want
• Click on CreatePolicy
• Click on Roles in the left navigation panel and click on Create role
• Keep the AWS Service type, click on Lambda then Next: Permissions
• Search for your created policy and select it then Next: Tags
• Add some tags if you want (not required for this) and then Next: Review
• Add a name (like S3BucketBouncedEmailsCreateRole) and a description if you want
• Click on CreateRole

4. Create the lambda function

• Go to Lambda in the console and click on functions in the left navigation panel
• Click on Create function
• Click on Use a blueprint, search for sns-message in the list of blueprints and select it
• Give your function a name (like ForwardBouncedEmailsFromSNSToBucket)
• Select use an existing role, search for the role you created and select it
• In SNS trigger, select the SNS topic that you created and tick enable the trigger (no messages are published on the topic so it’s not an issue to enable it right now)
• Check that the runtime is Node.js 18.x or a higher version of node
• Click on Create function

In the page that appears after creation, find the code source part and update the code in the index.mjs file to this (replace S3-BUCKET-NAME with your bucket’s name, not the ARN):

import * as AWS from "@aws-sdk/client-s3";
const s3 = new AWS.S3({apiVersion: '2006-03-01'});
const fileExtension = '.json';
const bucketName = 'S3-BUCKET-NAME' // set your bucket name here
exports.handler = (sns, context) => {
  //retrieve the events from the sns json
  const message = sns.Records[0].Sns.Message;
  //extract the date to use in the file names
  const timestamp = sns.Records[0].Sns.Timestamp;
  // default file name in case we cannot find any email in the message
  let filename = timestamp + '-unknow-email' + fileExtension;
  // let' parse the message to find the destination email and put it in the filename
  const parsedMessage = JSON.parse(message);
  if (parsedMessage.mail && parsedMessage.mail.destination && parsedMessage.mail.destination[0]) {
    // 2020-08-07T07:58:08.070Z-destination@email.com.json
    filename = timestamp + '-' + parsedMessage.mail.destination[0] + fileExtension;
  }
  const params = {
    Bucket: bucketName,
    Key: filename,
    Body: message,
  };
  // create document in the S3
  s3.putObject(params, function(err) {
    if (err) console.log(err, err.stack); // an error occurred
  });
};

This will create a new document in your bucket every time there’s a bounced email notification published in the SNS topic.

The name of the file will be something like 2020–08–07T07:58:08.070Z-destination@email.com.json . If you want to organize the bucket with folders, just change the name of the files so they have the folder in them (2020–08–07/2020–08–07T07:58:08.070Z-destination@email.com.json for example, if you want a new folder for each day) The S3 sdk will automatically create the necessary folders.

5. Publish bounced emails in the SNS topic

• Go to SES in the console and click on Email addresses in the left navigation panel
• Click on the email address for which you want to transfer the bounced emails
• In the Notifications part, click on Edit configuration
• In SNS Topic Configuration, select the topic you created in the Bounces select (you can also tick the include original headers checkbox to have more information in your json document)
• Click on Save config and Voilà!

You can now test all this by sending an email to an unexisting email address (There’s a Send a Test Email button in SES -> Email addresses) You’ll have a new item in your Bucket 🎉

Moving forward

Use the lambda function to do anything

We have created a very simple handling of our bounced email notifications but the lambda code is javascript so you can do what you want with them, for example:

• send a slack notification for each bounced email instead of storing the notifications in a bucket (get inspiration from this article)
• handle bounced emails notifications differently based on the bounce type (see here for examples of the message content)

Do the same for complaints and deliveries notifications

AWS provides notifications for complaints and deliveries too. You can have the same flow if you want to be able to check these.

Conclusion

We have seen here how to:

• Get notifications about bounced emails sent to SNS
• Have a lambda function subscribed to the SNS topic
• Make this lambda store the notifications in an S3 bucket

With only three AWS services (not counting IAM here) we were able to increase the observability of the Email Service concerning bounced emails. Apart from the access rights that can be a bit tough, everything can be configured with a few clicks in the management console.

At Inato, we are using terraform to manage our infrastructure resources so the next step was to import all theses resources there. If you want to do the same, see this documentation about the terraform aws provider and you might also want to have a look at this great article to understand how to provide some code to a terraform resource.