EventBridge Notifications for S3

Have a read about using InputTransformer to start an ECS Task using CloudTrail.

We found an interesting scenario when using “AWS API Call via CloudTrail” as the detail-type. The whole idea was to process a file in S3 bucket without a lambda, so we were interested in “PutObject” eventName.

With CloudTrail as the source we would receive all PUT events, that means if there was a 403 Forbidden response to the PUT S3 request, the event would still get triggered, which will start an ECS task and when the ECS tries to process the file, it would get 404 NoSuchKey error.

If you want to monitor such occurrence then we are good, but if you want to only process files which are saved to the bucket, then you would want to ignore such events, Send notification to Amazon EventBridge is what we need.

The whole idea of InputTransformer remains the same, but we will use a new event pattern.

Enable send notification to EventBridge

Let’s assume you already have an S3 bucket my-bucket and we now need to enable EventBridge notification.

aws s3api put-bucket-notification-configuration \
 --bucket my-bucket \
 --notification-configuration '{ "EventBridgeConfiguration": {} }'

Check this link to see how you could do via AWS console or REST API.

Create Cloudwatch rule

Let’s create the cloudwatch rule, but this time with EventBridge event pattern.

aws events put-rule --name "MyS3SourceRule" \
 --event-pattern "{\"source\": [\"aws.s3\"], \"detail-type\": [\"Object Created\"], \"detail\": {\"bucket\": {\"name\": [\"my-bucket\"]}}}" \
 --role-arn "arn:aws:iam::XXXX:role/custom-cloudwatch-events-role"

Let’s understand the event pattern, I’ll not go through keys which are already discussed in my previous blog:

{
  "source": ["aws.s3"],
  "detail-type": ["Object Created"],
  "detail": {
    "bucket": {
      "name": ["my-bucket"]
    }
  }
}

With event bridge you can control things via “detail-type”, in our case we want to only process the file when it’s created, so we are filtering for “Object Created”. Check this mapping to see how S3 events are mapped to the EventBridge detail-type The EventBridge event for a file dropped in the “my-bucket” would look like this:

{
  "version": "0",
  "id": "17793124-05d4-b198-2fde-7ededc63b103",
  "detail-type": "Object Created",
  "source": "aws.s3",
  "account": "111122223333",
  "time": "2022-11-12T00:00:00Z",
  "region": "ca-central-1",
  "resources": ["arn:aws:s3:::my-bucket"],
  "detail": {
    "version": "0",
    "bucket": {
      "name": "my-bucket"
    },
    "object": {
      "key": "file-123.csv",
      "size": 5,
      "etag": "b1946ac92492d2347c6235b4d2611184",
      "version-id": "IYV3p45BT0ac8hjHg1houSdS1a.Mro8e",
      "sequencer": "617f08299329d189"
    },
    "request-id": "N4N7GDK58NMKJ12R",
    "requester": "123456789012",
    "source-ip-address": "1.2.3.4",
    "reason": "PutObject"
  }
}

Next you need to add a target, use InputTransformer and pass the bucket and filename as environment variables, which remains same as explained in this blog. But make sure you use the correct path to pass the bucket and key.

.....
"InputTransformer": {
      "InputPathsMap": {
        "s3bucket": "$.detail.bucket.name",
        "s3key": "$.detail.object.key"
      },
.....

Conclusion

Enabling EventBridge notification pattern for an S3 bucket allows for seamless integration and granular control that could streamline event driven architecture.

Reference

1. AWS EventBridge — S3
2. AWS CLI — create rule
3. AWS CLI — add target

Information has been prepared for information purposes only and does not constitute advice.