Naukri Engineering
Published in

Naukri Engineering

Use Amazon SES to receive emails in S3 and Forward Incoming email to any external email id

Amazon Simple Email Service (SES) is a public cloud-based email service that allows sending emails from AWS SES Service to anywhere external emails id’s. It’s designed to support several email use cases, including transactional, marketing, or mass email communications, etc.

Amazon SES also has support for incoming emails too but one of the major challenges with incoming SES emails is we need AWS SES to forward email into an external email address.

Region supported for incoming emails are -

https://docs.aws.amazon.com/ses/latest/DeveloperGuide/regions.html#region-endpoints

Email receiving Endpoints

With the exception of Amazon S3 buckets, all of the AWS resources that you use for receiving an email with Amazon SES have to be in the same AWS Region as the Amazon SES endpoint. For example, if you use Amazon SES in the US West (Oregon) Region, then any Amazon SNS topics, AWS KMS keys, and Lambda function that you use also have to be in the US West (Oregon) Region. Similarly, to receive an email with Amazon SES within a Region, you have to create an active receipt rule set in that Region.

The following table lists the email receiving endpoints for all of the AWS Regions where

Amazon SES supports email receiving:

Note

Amazon SES doesn’t support email receiving in the following Regions: US East (Ohio), US West (N. California), Asia Pacific (Mumbai), Asia Pacific (Seoul), Asia Pacific (Singapore), Asia Pacific (Sydney), Asia Pacific (Tokyo), Canada (Central), Europe (Frankfurt), Europe (London), Europe (Paris), Europe (Stockholm), Middle East (Bahrain), South America (São Paulo), and AWS GovCloud (US).

We need to implement this because AWS SES doesn’t support IMAP/POP3 protocols and IMAP/POP3 are the services supported by most of the email clients.

My goal was — AWS SES service to received email from an email id & store it in S3 and further it forwards email into an external email address which supports IMAP/POP3 protocols.

We had used Amazon cloud solutions like SES, Amazon S3, SNS and AWS Lambda to create this email forwarding solution and it's heavily inspired from below 4 blog posts.

https://aws.amazon.com/blogs/messaging-and-targeting/forward-incoming-email-to-an-external-destination/

https://github.com/arithmetric/aws-lambda-ses-forwarder

https://supporthost.in/amazon-ses-forward-incoming-email/

https://www.qloudx.com/automate-incoming-email-processing-with-amazon-ses-aws-lambda/

Below are the functional Architecture diagram that we are following –

Below are the Mail flow –

The following are the steps or so-called actions -

  1. A new email is sent from an external sender to your domain like support@abc.com
  2. Amazon SES handles the incoming email for your domain and we create required DNS entries in Route 53 or any other DNS
  3. An Amazon SES receipt rule saves the incoming message in an S3 bucket
  4. An Amazon SES receipt rule triggers the execution of a Lambda function
  5. The Lambda function retrieves the message content from S3, and then creates a new message and sends it to Amazon SES
  6. Amazon SES sends the message to the destination server

Prerequisites

In order to complete this procedure, we need to have a domain that receives an incoming email. If you don’t already have a domain, you can purchase one through Amazon Route 53. You can also purchase a domain through one of several third-party vendors.

Procedures

Set up Your Domain

We need a verified domain in AWS SES with MX record points to SES. Also if the Amazon SES is in sandbox mode, submit a support request to have it removed, follow –

1. Open Amazon Route 53 to do this, access to AWS console and go to Route 53 >> Hosted zones >> Create Hosted Zone. You can harness this step to add some DNS entries if you need them, such as A, MX, records that point to your website or DKIM, SPF record as a security measure to prevent the email addresses within your domain from being spoofed or your messages from being considered as spam.

2. In Amazon SES, verify the domain that you want to use to receive incoming email

3. Add the following MX record to the DNS configuration for your domain:

10 inbound-smtp.<regionInboundUrl>.amazonaws.com

Replace <regionInboundUrl> with the URL of the email receiving endpoint for the AWS Region that you use in Amazon SES

Note : Route 53 DNS zones cost $0.50/month per domain

4. If your account is still in the Amazon SES sandbox, submit a request to have it removed

Create S3 Bucket and S3 Bucket Policy

1. Create an s3 bucket from AWS S3 bucket console. For more information, see Create a Bucket in the Amazon S3 Getting Started Guide.

Make sure you create an S3 bucket in the same region as your Amazon SES exists; as such not mandatory

2. Once S3 Bucket get created click on S3 bucket name >> Click Permissions tab >> Click Bucket policy. Add below policy to the bucket by edit -

{
"Version": "2012–10–17",
"Statement": [{
"Sid": "AllowSESPutObject",
"Effect": "Allow",
"Principal": {
"Service": "ses.amazonaws.com"
},
"Action": "s3:PutObject",
"Resource": "arn:aws:s3:::<awss3bucketname>/*",
"Condition": {
"StringEquals": {
"aws:Referer": "<awsaccountid>"
}
}
}]
}

In the policy, make the following changes –

Replace <awss3bucketname> with the name of your S3 bucket
Replace <awsaccountid> with your AWS Account ID

Create an IAM Policy and Role

We are creating IAM policy and associating it to an IAM role. We assign this newly created role to the Lambda Function we create later.

Go to the IAM console in AWS console >> Click on Policies available in left side >> Click Create Policy >> Switch to JSON tab. Copy paste below content and review and create the policy by giving a name you can identify.

{
"Version": "2012–10–17",
"Statement": [{
"Sid": "LogCreatePut",
"Effect": "Allow",
"Action": [
"logs:CreateLogStream",
"logs:CreateLogGroup",
"logs:PutLogEvents"
],
"Resource": "*"
},
{
"Sid": "S3GetPutSend",
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:PutObject",
"ses:SendRawEmail"
],
"Resource": [
"arn:aws:s3:::<awss3bucketname>/*",
"arn:aws:ses:<awsregion>:<awsccountid>:identity/*"
]
}
]
}

In the preceding policy, make the following changes:

Replace <awss3bucketname> with the name of the S3 bucket that you created earlier
Replace <awsregion> with the name of the AWS Region that you created the bucket e.g., us-east-1
Replace <awsaccountid> with your AWS account ID

Next, we’ll create a new IAM role.

Click Roles from IAM console >> click Create Role >> Select type of trusted entity as AWS Service >> Choose Lambda from it >> Click next Permissions >> select the policy that we just created to the new role >> Click review and create by giving a Name for the Role. Make sure the role name you gave is the identifiable amount other existing roles

Create the Lambda Function

Open AWS Lambda console, create a new Node.js 12.x function.

Click Create Function >> Choose runtime as Node.js 12.x >> Ensure Handler is set to index.handler >> In Choose or create an execution role, select use an existing role >> Select the role that we created for Lambda from drop-down menu >> Gave a name and click Create

In the Function code section, copy/paste below Node.js code.

"use strict";
var AWS = require('aws-sdk');
console.log("AWS Lambda SES Forwarder // @arithmetric // Version 5.0.0");
// Configure the S3 bucket and key prefix for stored raw emails, and the
// mapping of email addresses to forward from and to.
//
// Expected keys/values:
//
// - fromEmail: Forwarded emails will come from this verified address
//
// - subjectPrefix: Forwarded emails subject will contain this prefix
//
// - emailBucket: S3 bucket name where SES stores emails.
//
// - emailKeyPrefix: S3 key name prefix where SES stores email. Include the
// trailing slash.
//
// - allowPlusSign: Enables support for plus sign suffixes on email addresses.
// If set to `true`, the username/mailbox part of an email address is parsed
// to remove anything after a plus sign. For example, an email sent to
// `example+test@example.com` would be treated as if it was sent to
// `example@example.com`.
//
// - forwardMapping: Object where the key is the lowercase email address from
// which to forward and the value is an array of email addresses to which to
// send the message.
//
// ensure you verify both emails from email id & to email id in SES Verify Email // options
//
var defaultConfig = {
fromEmail: "noreply@abc.com",
subjectPrefix: "",
emailBucket: "s3-bucket-name",
emailKeyPrefix: "",
allowPlusSign: true,
forwardMapping: {
"support@abc.com": [
"yourforwarding@mailaddress.com"
]
"@abc.com": [
"yourforwarding@mailaddress.com"
],
}
};
/**
* Parses the SES event record provided for the `mail` and `recipients` data.
*
* @param {object} data - Data bundle with context, email, etc.
*
* @return {object} - Promise resolved with data.
*/
exports.parseEvent = function(data) {
// Validate characteristics of a SES event record.
if (!data.event ||
!data.event.hasOwnProperty('Records') ||
data.event.Records.length !== 1 ||
!data.event.Records[0].hasOwnProperty('eventSource') ||
data.event.Records[0].eventSource !== 'aws:ses' ||
data.event.Records[0].eventVersion !== '1.0') {
data.log({
message: "parseEvent() received invalid SES message:",
level: "error", event: JSON.stringify(data.event)
});
return Promise.reject(new Error('Error: Received invalid SES message.'));
}
data.email = data.event.Records[0].ses.mail;
data.recipients = data.event.Records[0].ses.receipt.recipients;
return Promise.resolve(data);
};
/**
* Transforms the original recipients to the desired forwarded destinations.
*
* @param {object} data - Data bundle with context, email, etc.
*
* @return {object} - Promise resolved with data.
*/
exports.transformRecipients = function(data) {
var newRecipients = [];
data.originalRecipients = data.recipients;
data.recipients.forEach(function(origEmail) {
var origEmailKey = origEmail.toLowerCase();
if (data.config.allowPlusSign) {
origEmailKey = origEmailKey.replace(/\+.*?@/, '@');
}
if (data.config.forwardMapping.hasOwnProperty(origEmailKey)) {
newRecipients = newRecipients.concat(
data.config.forwardMapping[origEmailKey]);
data.originalRecipient = origEmail;
} else {
var origEmailDomain;
var origEmailUser;
var pos = origEmailKey.lastIndexOf("@");
if (pos === -1) {
origEmailUser = origEmailKey;
} else {
origEmailDomain = origEmailKey.slice(pos);
origEmailUser = origEmailKey.slice(0, pos);
}
if (origEmailDomain &&
data.config.forwardMapping.hasOwnProperty(origEmailDomain)) {
newRecipients = newRecipients.concat(
data.config.forwardMapping[origEmailDomain]);
data.originalRecipient = origEmail;
} else if (origEmailUser &&
data.config.forwardMapping.hasOwnProperty(origEmailUser)) {
newRecipients = newRecipients.concat(
data.config.forwardMapping[origEmailUser]);
data.originalRecipient = origEmail;
} else if (data.config.forwardMapping.hasOwnProperty("@")) {
newRecipients = newRecipients.concat(
data.config.forwardMapping["@"]);
data.originalRecipient = origEmail;
}
}
});
if (!newRecipients.length) {
data.log({
message: "Finishing process. No new recipients found for " +
"original destinations: " + data.originalRecipients.join(", "),
level: "info"
});
return data.callback();
}
data.recipients = newRecipients;
return Promise.resolve(data);
};
/**
* Fetches the message data from S3.
*
* @param {object} data - Data bundle with context, email, etc.
*
* @return {object} - Promise resolved with data.
*/
exports.fetchMessage = function(data) {
// Copying email object to ensure read permission
data.log({
level: "info",
message: "Fetching email at s3://" + data.config.emailBucket + '/' +
data.config.emailKeyPrefix + data.email.messageId
});
return new Promise(function(resolve, reject) {
data.s3.copyObject({
Bucket: data.config.emailBucket,
CopySource: data.config.emailBucket + '/' + data.config.emailKeyPrefix +
data.email.messageId,
Key: data.config.emailKeyPrefix + data.email.messageId,
ACL: 'private',
ContentType: 'text/plain',
StorageClass: 'STANDARD'
}, function(err) {
if (err) {
data.log({
level: "error",
message: "copyObject() returned error:",
error: err,
stack: err.stack
});
return reject(
new Error("Error: Could not make readable copy of email."));
}
// Load the raw email from S3
data.s3.getObject({
Bucket: data.config.emailBucket,
Key: data.config.emailKeyPrefix + data.email.messageId
}, function(err, result) {
if (err) {
data.log({
level: "error",
message: "getObject() returned error:",
error: err,
stack: err.stack
});
return reject(
new Error("Error: Failed to load message body from S3."));
}
data.emailData = result.Body.toString();
return resolve(data);
});
});
});
};
/**
* Processes the message data, making updates to recipients and other headers
* before forwarding message.
*
* @param {object} data - Data bundle with context, email, etc.
*
* @return {object} - Promise resolved with data.
*/
exports.processMessage = function(data) {
var match = data.emailData.match(/^((?:.+\r?\n)*)(\r?\n(?:.*\s+)*)/m);
var header = match && match[1] ? match[1] : data.emailData;
var body = match && match[2] ? match[2] : '';
// Add "Reply-To:" with the "From" address if it doesn't already exists
if (!/^reply-to:[\t ]?/mi.test(header)) {
match = header.match(/^from:[\t ]?(.*(?:\r?\n\s+.*)*\r?\n)/mi);
var from = match && match[1] ? match[1] : '';
if (from) {
header = header + 'Reply-To: ' + from;
data.log({
level: "info",
message: "Added Reply-To address of: " + from
});
} else {
data.log({
level: "info",
message: "Reply-To address not added because From address was not " +
"properly extracted."
});
}
}
// SES does not allow sending messages from an unverified address,
// so replace the message's "From:" header with the original
// recipient (which is a verified domain)
header = header.replace(
/^from:[\t ]?(.*(?:\r?\n\s+.*)*)/mgi,
function(match, from) {
var fromText;
if (data.config.fromEmail) {
fromText = 'From: ' + from.replace(/<(.*)>/, '').trim() +
' <' + data.config.fromEmail + '>';
} else {
fromText = 'From: ' + from.replace('<', 'at ').replace('>', '') +
' <' + data.originalRecipient + '>';
}
return fromText;
});
// Add a prefix to the Subject
if (data.config.subjectPrefix) {
header = header.replace(
/^subject:[\t ]?(.*)/mgi,
function(match, subject) {
return 'Subject: ' + data.config.subjectPrefix + subject;
});
}
// Replace original 'To' header with a manually defined one
if (data.config.toEmail) {
header = header.replace(/^to:[\t ]?(.*)/mgi, () => 'To: ' + data.config.toEmail);
}
// Remove the Return-Path header.
header = header.replace(/^return-path:[\t ]?(.*)\r?\n/mgi, '');
// Remove Sender header.
header = header.replace(/^sender:[\t ]?(.*)\r?\n/mgi, '');
// Remove Message-ID header.
header = header.replace(/^message-id:[\t ]?(.*)\r?\n/mgi, '');
// Remove all DKIM-Signature headers to prevent triggering an
// "InvalidParameterValue: Duplicate header 'DKIM-Signature'" error.
// These signatures will likely be invalid anyways, since the From
// header was modified.
header = header.replace(/^dkim-signature:[\t ]?.*\r?\n(\s+.*\r?\n)*/mgi, '');
data.emailData = header + body;
return Promise.resolve(data);
};
/**
* Send email using the SES sendRawEmail command.
*
* @param {object} data - Data bundle with context, email, etc.
*
* @return {object} - Promise resolved with data.
*/
exports.sendMessage = function(data) {
var params = {
Destinations: data.recipients,
Source: data.originalRecipient,
RawMessage: {
Data: data.emailData
}
};
data.log({
level: "info",
message: "sendMessage: Sending email via SES. Original recipients: " +
data.originalRecipients.join(", ") + ". Transformed recipients: " +
data.recipients.join(", ") + "."
});
return new Promise(function(resolve, reject) {
data.ses.sendRawEmail(params, function(err, result) {
if (err) {
data.log({
level: "error",
message: "sendRawEmail() returned error.",
error: err,
stack: err.stack
});
return reject(new Error('Error: Email sending failed.'));
}
data.log({
level: "info",
message: "sendRawEmail() successful.",
result: result
});
resolve(data);
});
});
};
/**
* Handler function to be invoked by AWS Lambda with an inbound SES email as
* the event.
*
* @param {object} event - Lambda event from inbound email received by AWS SES.
* @param {object} context - Lambda context object.
* @param {object} callback - Lambda callback object.
* @param {object} overrides - Overrides for the default data, including the
* configuration, SES object, and S3 object.
*/
exports.handler = function(event, context, callback, overrides) {
var steps = overrides && overrides.steps ? overrides.steps :
[
exports.parseEvent,
exports.transformRecipients,
exports.fetchMessage,
exports.processMessage,
exports.sendMessage
];
var data = {
event: event,
callback: callback,
context: context,
config: overrides && overrides.config ? overrides.config : defaultConfig,
log: overrides && overrides.log ? overrides.log : console.log,
ses: overrides && overrides.ses ? overrides.ses : new AWS.SES(),
s3: overrides && overrides.s3 ?
overrides.s3 : new AWS.S3({signatureVersion: 'v4'})
};
Promise.series(steps, data)
.then(function(data) {
data.log({
level: "info",
message: "Process finished successfully."
});
return data.callback();
})
.catch(function(err) {
data.log({
level: "error",
message: "Step returned error: " + err.message,
error: err,
stack: err.stack
});
return data.callback(new Error("Error: Step returned error."));
});
};
Promise.series = function(promises, initValue) {
return promises.reduce(function(chain, promise) {
if (typeof promise !== 'function') {
return Promise.reject(new Error("Error: Invalid promise item: " +
promise));
}
return chain.then(promise);
}, Promise.resolve(initValue));
};

Also from AWS Lambda Basic settings section, increase the code execution time out from 3 Sec to 15 Sec and the memory as per need depends on large emails. The basic timeout and memory limit is enough for normal emails

Replace the Environment variables in AWS Lambda console -

emailBucket : S3 bucket name where email stores

fromEmail : Set an SES email address like noreply@yourdomain.com. This is the from an email address which will display in the destination forwarded email box

fromEmail : details are already mentioned above

emailBucket : details are already mentioned above

emailKeyPrefix : S3 key name prefix where SES stores email. Include the trailing slash. If we don’t have any such subfolder which stored emails in S3 bucket and we are directly keeping the emails in a single folder. Just leave it as blank and it code line will look like below

subjectPrefix : “”,

forwardMapping : Here the first email address is our SES forwarding verified email address and the second one will be our external destination verified email address

Create an Amazon SNS Topic

You can optionally create an Amazon SNS topic. This step is helpful for troubleshooting purposes, or if you just want to receive additional notifications when you receive a message.

Create a Receipt Rule Set

Go the AWS SES console (“Email Receiving” will be greyed out if the region doesn’t support AWS SES Email Receiving follow https://docs.aws.amazon.com/ses/latest/DeveloperGuide/regions.html#region-endpoints)

Click on Rule Sets >> Click Create a New Rule Set >> Give A name for the rule >> On the Recipients configuration page, add our SES verified from email addresses from which you want to forward an email

On the Actions configuration page >> add an S3 action and then >> a Lambda action

For the S3 action >> choose the existing S3 bucket we created above

Leave Encrypt Message unchecked and SNS Topic set to [none] or create an SNS Topic

For the Lambda action: Choose the Lambda function we created. Leave Invocation Type set to Event and SNS Topic set to [none]

If you are asked by SES to add permissions to access lambda:InvokeFunction, agree to it

Once created, it will list under the ruleset section. Select it and make it as Active one by clicking ”Set as Active rule set ” button

This concludes the setting up the email forwarding solution via AWS SES.

Now let’s test, send an email to SES from verified email address.

Check the AWS S3 bucket, email must have arrived. It hardly takes any time; in a few seconds, the email will forward to remote forwarded email address. Ensure you must have verified the forwarding email address too.

You will see how the message is stored as a new object in the S3 bucket

This Node.js function will forward the email to the forwarding email like the regular forwarded email we usually see

Now go to the SES Rule Sets we created earlier and edit it and change the lambda function into the new one we created. Test the working again.

Basic Troubleshooting –

The first thing we need do is, Open AWS CloudWatch console -

Click CloudWatch Logs >> Log groups >> Click on our AWS Lambda function name log >> Click on the latest log if there will be multiples of logs

Now try to test by sending email to support@abc.com and it will be forwarded to the forwarding email like the regular forwarded email we usually see

If you send a test message, but it is not forwarded to your destination email address, do the following:

  • Make sure that the Amazon SES Receipt Rule is active
  • Make sure that the email address that you specified in the MailRecipient variable of the Lambda function is correct
  • Subscribe an email address or phone number to the SNS topic. Send another test email to your domain. Make sure that SNS sends a Received notification to your subscribed email address or phone number
  • Check the CloudWatch Log for your Lambda function to see if any errors occurred

If you send a test email to your receiving domain, but you receive a bounce notification, do the following:

  • Make sure that the verification process for your domain completed successfully
  • Make sure that the MX record for your domain specifies the correct Amazon SES receiving endpoint
  • Make sure that you’re sending to an address that is handled by the receipt rule

Cost of using this AWS SES, S3, SNS, Route 53 Solution

The cost of implementing this solution is minimal. If you receive 10,000 emails per month, and each email is 1–2 KB in size, you pay $1.00 for your use of Amazon SES

We also pay a small charge to store incoming emails in Amazon S3, & SNS for Subscription of Notification, Cloudwatch Logs, etc. along with Route 53 charges for DNS Records and resolution. The charge for storing 1,000 emails that are each 2KB in size is less than one cent.

Finally, you pay for your use of AWS Lambda. With Lambda, you pay for the number of requests you make, for the amount of computing time that you use, and for the amount of memory that you use. If you use Lambda to forward 1,000 emails that are each 2KB in size, you pay no more than a few cents.

Note: These cost estimates don’t include the costs associated with purchasing a domain since many users already have their own domains. The cost of obtaining a domain is the most expensive part of implementing this solution.

Conclusion

This solution makes it possible to forward incoming email from one of your Amazon SES verified domains to an email address that isn’t necessarily verified. It’s also useful if you have multiple AWS accounts, and you want incoming messages to be sent from each of those accounts to a single destination and many more use cases. We hope you’ve found this Solution to be helpful !!!

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store