An Efficient way of Bulk Email Processing leveraging AWS SES, Lambda, and S3.
Overcome the mundane email processing with Automation and extract useful data and MetaData from them.
More often than not clients send us documents such as Excel, CSV, and other formats to our email. Initially, we would analyze these documents in the email Manually and store them according to the Document Category in our S3 Bucket under the proper prefix for that particular document(invoice, purchase orders, RFQ, etc).
As we scaled our clients and thereby our users, the sheer volume of such mails increased to more than 1000 emails/day which could in no way be handled by our team manually. Gomigo follows a multi-tenant architecture and hence handles a complete tenant-level separation of all data.
For example; when a user sends an email to tenant@yourcompany.com, we will fetch the email data and process it, and store the information extracted from it into a Database. For these kinds of processes three Amazon services are required; Simple Email Service (SES), Lambda, and S3. The solution we provide is compatible with a multi-tenant platform.
Firstly, I want to explain these services for newbies on AWS or these services.
Simple Email Service enables you to send and receive email using a reliable and scalable email platform.
Lambda is a compute service that lets you run code without provisioning or managing servers. AWS Lambda executes your code only when needed and scales automatically, from a few requests per day to thousands per second.
S3 is a storage for the Internet. It is designed to make web-scale computing easier for developers. Amazon S3 has a simple web services interface that you can use to store and retrieve any amount of data, at any time, from anywhere on the web
Step 1: Verify a New Domain
In the AWS SES console, click the Domain menu and verify your Domain. This allows us to create a unique email account for each tenant.
Step 2: Set up rule sets with AWS SES
After verifying your Domain Name, click Rule Sets in the AWS SES menu.
Rule-Set Contains a List of Rules to be applied on the mail that is received by that email domain or address mentioned in the ruleset.
Step 3: Create a new Rule
Once the ruleset is created click on it and then click on the Create Rule button.
Step 4: Add Recipients for the Rule
Configure a list of recipients for whom you want to receive emails. A recipient may be an email address or a domain. For our multi-tenant use case, we would use something like tenant@domain.com
Step 5: Setting an Action
After that, we have to set actions for email. First of all, we have to add the S3 action. It means when someone sends an email to your address. The email data is saved in S3. To create and set the S3 bucket as you see below. The prefix is defined as /tenant name/Mailbox this way in the S3 bucket we could maintain a tenant level separation and have different IAM roles and access defined for separate tenants.
This Process completes our email receiving and storing that raw email according to its messageId as a name in our S3 bucket for the tenant in his prefix.
In the Next Step, we would like to process this email Information so we need to trigger a lambda whenever a new object is written to the prefix of this S3 Bucket and get this email Parsing it and then processing it further for metadata and data from it.
Step 5: Create a Lambda
We Create a new Lambda in this example I will be using Nodejs as my Runtime for the lambda.
And in the Lambda we need to attach a trigger to it for the S3 bucket.
voila, now you have a multi-tenant mechanism to process emails from tenants now based on your business logic you can process the emails for my case it was processing to do with the attachments.
Note: The Utils.tenantcredentials is an internal function we have written to abstract and separate the tenant level operations. The function gets the appropriate tenant credential provided by its tenant name in the parameter.
Conclusion
This is a simple but effective architecture to process a large no of emails each day and use.
In conclusion with using this type of services, you will take advantage of these:
- You can categorize emails according to content data.
- You can process attachments and classify them according to content.
- You can track your support agents’ actions.
Delete your AWS resources if you are just practicing. That’s it for today, thanks for reading. Please give feedback about the article in the comments below. Feel free to ask any questions or give your suggestions.
