This article was originally published here: https://blog.kloudless.com/2019/02/12/how-to-monitor-activity-in-s3-using-sns-and-sqs/
Amazon S3 is one of the most popular object storage services that apps use today. Monitoring S3 buckets for activity can be very beneficial, depending on the reason the bucket stores data.
For example, the Kloudless File Explorer provides an easy way for users to upload content to an app’s S3 bucket. Apps can monitor S3 for new files to process rather than write client-side logic to trigger data processing when a user completes an upload.
AWS docs describe monitoring activity in S3 using SNS. In this article, we will describe a generalized version that enables events to be consumed by multiple receivers via SQS instead. Kloudless adopts this approach to scale to multiple environments and downstream processors rather than restrict the notifications to a single receiver. This is critical since S3 buckets do not support multiple notification configurations.
There are three steps to receive notifications from buckets:
- Create an SNS topic to receive notifications sent from a bucket.
- Create an SQS queue for each receiver, and subscribe to the SNS topic.
- Configure S3 buckets to publish events to the SNS topic.
We’ll use two AWS Accounts to demonstrate the process:
074962000000represents the ID of the developer’s AWS account
085962000000represents the ID of the bucket owner’s AWS account
The developer’s app creates and configures the SNS topic and SQS queue in its own AWS account, and retrieves events from the queue. The app accesses buckets using the bucket owner’s account to configure the buckets to publish events to the SNS topic.
Create an SNS topic
The app first creates an SNS topic in its own AWS account with the name
s3-event-085962000000. This represents a topic that will receive notifications from the bucket owner’s buckets. The presence of the bucket owner’s ARN ensures uniqueness. Set a policy similar to the one below on the SNS topic:
Condition above specifies three S3 buckets that can publish to this SNS topic.
Create an SQS queue and subscribe to the SNS topic
Continuing to use the app’s own AWS account, create a queue with name
s3-eventq-receiver-1-074962000000 and set its policy similar to the one below:
Similar to the first policy, the
Condition specifies an SNS topic that can send messages to the queue. The queue name includes
receiver-1 to uniquely identify the recipient of messages in this queue. Create additional SQS queues for other recipients.
Then, set the queue to subscribe to the SNS topic. Here’s an example of how to do this using the popular Python library,
The developer app’s IAM account provides the key and secret for the API requests above.
Update the S3 buckets’ notification configurations
Using the bucket owner’s AWS IAM credentials, set the buckets to publish
ObjectRemoved events to the SNS topic. Here is an example using
The bucket owner’s key and secret are used in this case.
The developer’s app can now poll the queue for new events! Here is a diagram of the event flow:
The queue’s presence ensures that multiple apps can subscribe to bucket notifications. For example, the subscriber described above,
receiver-1, could be the developer’s production application, but still leave room for a development and staging application to receive the same events. As mentioned above, buckets do not natively support more than a single notification configuration, even though the API endpoint accepts a list.
Simplify S3 activity monitoring using Kloudless
The process described above is a simplified version. Real-world use involves complexities such as the ones below:
- Buckets in different regions
- Detecting and monitoring newly added buckets
- Cleaning up SNS topics intended for deleted buckets and/or accounts
- Conflicts with existing notification configurations on buckets
Authored by and with permission from Chao-Shih Chen