AWS S3 Cross Account Event Notification

Objective

To receive Amazon S3 bucket events notifications which is present in AWS Account A in an Amazon SNS/SQS/Lambda which is present in AWS Account B.

Business Standpoint

Whenever a client perform any of AWS S3 Bucket events like upload a file for example in AWS S3 Bucket in his/her AWS Account say AWS Account A, System should be able to receive notification for the same in the System AWS Account say AWS Account B to get the trigger to perform subsequent actions.

Let’s get hold of this!

Brief

The Amazon S3 notification feature enables us to receive notifications when certain events happen in the bucket. To enable notifications:

1. Add a notification configuration that identifies the events(see supported events below) we want Amazon S3 to publish
2. The destinations(see supported destinations below) where you want Amazon S3 to send the notifications.

You store this configuration in the notification sub-resource that is associated with a bucket. We will programatically configure this for the S3 bucket in Account A from User/Role/Group in Account B using the setup described below.

Supported Events

Currently, Amazon S3 can publish notifications for the following events:

• New object created events
• Object removal events
• Restore object events
• Reduced Redundancy Storage (RRS) object lost events
• Replication events

For details refer Supported event types.

Supported Destinations

Amazon S3 supports the following destinations where it can publish events:

• Amazon Simple Notification Service (Amazon SNS) topic
• Amazon Simple Queue Service (Amazon SQS) queue
• AWS Lambda

Walkthrough

Step 1: Requirements in Account A S3 Bucket Policy

Grant access to Account B Root/User/Role/Group to configure bucket event notifications.

Policy Elements:

Principal:
Used to specify the principal that is allowed or denied access to a resource. Here Principal can be Account B Root/User/Role/Group.

Action:
For Peak to programatically configure Client’s S3 bucket notification sub-resource configuration, action required is s3:PutBucketNotification

Effect:
Set the Effect element to Allow.

Step 2: Setup an Amazon SNS topic in Account B

1. Create an Amazon SNS topic say using AWS Console.
2. Subscribe to the topic. Say, use email as the communications protocol. An email requesting subscription confirmation to the topic will be received. Confirm the subscription.
3. Update the SNS Access Policy to allow Client’s S3 Bucket to Publish on it.
   Example:

4. Note the SNS topic ARN.

Step 3: Account B Root/User/Role/Group programatically stores notification sub-resource associated on the Account A AWS S3 bucket

Delegate S3 PutBucketNotification Action Permission from Principal in S3 Clients policy to User, Role or Group which will programatically stores notification sub-resource associated on the Client’s bucket.

Code to set notification sub-resource on Client’s Amazon S3 bucket to publish events to Peak Amazon SNS Topic.
Internally, both AWS console and the SDKs call the Amazon S3 REST API to manage notification sub-resources associated with the bucket.
Follow the link to see all configurable event properties using AWS SDK for JavaScript putBucketNotificationConfiguratin

All done. Subscribers of Account B Amazon SNS created in step 1 should receive events notifications on file uploads in Amazon S3 bucket in Account A.

Similarly, we can configure Amazon Simple Queue Service (Amazon SQS) and AWS Lambda and allow Amazon S3 to publish event messages to AWS Lambda or Amazon SQS.

Preferred architecture: Fan out events using Amazon SNS

Limitations

• Policy quotas of Amazon SNS
  Amazon SNS will have to include S3 Bucket Arn in its Access Policy, allowing them to publish. The following are the maximum quotas specific to AWS SNS policy statement.
  Bytes: 30 kb and Statements: 100 Source: SNS Policy Limits
• Amazon SQS as destination of Amazon S3 event notification
  Currently, Standard SQS queue is only allowed as an Amazon S3 event notification destination, whereas FIFO SQS queue is not allowed.
• Amazon S3 event notification
  If two writes are made to a single non-versioned object at the same time, it is possible that only a single event notification will be sent. If we want to ensure that an event notification is sent for every successful write, we can request client to enable versioning on the bucket.
  With versioning, every successful write will create a new version of the object and will also send an event notification.

Latency

Amazon S3 event notifications are designed to be delivered at least once.
Typically, event notifications are delivered in seconds but can sometimes take a minute or longer.

Pricing

A. Account A Side

There is no charge for Event Notifications for Amazon S3.
Standard S3 storage and data retrieval charges applicable which are subject to various storage classes and encryption used

B. Account B Side

Standard messaging and execution charges for SQS, SNS, and Lambda.

Conclusion

An AWS S3 bucket in Account A can send Bucket Event Notifications to AWS SNS, AWS SQS and AWS Lambda in Account B.