Cloudwatch event rule to trigger ECS on S3 event

Published in

Engineering the Skies: Qantas Tech Blog

4 min readAug 31, 2023

Once you finish reading this blog and before jumping in to implementation please have a read on EventBridge event notification

For use case where you want to process a file in a bucket you could use a cloudwatch rule and lambda, but more memory intensive and longer jobs require ECS. You also do not want to scan for files in the bucket in you ECS job, you want to have a setup where the job is told to process this file in this bucket.

With target as ECS for a cloudwatch rule, it becomes a bit tricky on how to pass the file as an environment variable to the ECS task.

Cloudwatch rule setup

The blog focus on cloudwatch rule and assume that you are familiar with S3, ECS, task definition, IAM policy, IAM roles and how to set them up.

For this blog we assume there is a task definition file-processor and an S3 bucket my-bucket with CloudTrail configured for logging.

Create Cloudwatch rule

Let’s first create a cloudwatch rule, I will be using the AWS CLI, the idea remains same, there would be similar API when terraform, ansible or cloudformation is used.

aws events put-rule \
    --name "MyS3SourceRule" \
    --event-pattern "{\"source\":[\"aws.s3\"],\"detail-type\":[\"AWS API Call via CloudTrail\"],\"detail\":{\"eventSource\":[\"s3.amazonaws.com\"],\"eventName\":[\"PutObject\"],\"requestParameters\":{\"bucketName\":[\"my-bucket\"]}}}" \
    --role-arn "arn:aws:iam::XXXX:role/custom-cloudwatch-events-role"

Let’s understand the event pattern:

{
  "source": ["aws.s3"],
  "detail-type": ["AWS API Call via CloudTrail"],
  "detail": {
    "eventSource": ["s3.amazonaws.com"],
    "eventName": ["PutObject", "CompleteMultipartUpload"],
    "requestParameters": {
      "bucketName": ["my-bucket"],
      "key": [{ "prefix": "input/" }]
    }
  }
}

The detail-type is set to AWS API Call via CloudTrail so that we constrain the events that we are interested.
The eventSource is further restricted to s3.amazonaws.com, so that out of all the Cloudtrail API events, it only looks about S3 API calls.
The eventName is PutObject. These are S3 API operation names, in our case when an object is added. Configure your rule based on the use case, more events can be found here. I have added CompleteMultipartUpload as well because for larger files, files are split into multiple parts.
Finally, let’s restrict the bucketName to the name of the bucket we are interested.
To make things a bit more interesting, I have added a filtering using prefix keyword so that the event is triggered when the file is prefixed with input/ or rather my-bucket/input/, you might have a use case where your application might produce a hand back file into the same bucket. You want to make sure that these are filtered out and ECS doesn’t get triggered for your handback file. We are using AWS content based filtering concept here. The concept is documented for EventBridge, and EventBridge uses the Cloudwatch API. There are other ways to match as well, which are documented here.

So we now have a rule MyS3SourceRule which provides a source for uploads in our bucket. We need to hook a target to trigger actions.

Add target

If we look at the Cloudwatch event delivered by CloudTrail as below you can see the requestParameters, this is what we are going to use while creating our target.

{
  "version": "0",
  "id": "36eb8523-97d0-4518-b33d-ee3579ff19f0",
  "detail-type": "AWS API Call via CloudTrail",
  "source": "aws.s3",
  "account": "123456789012",
  "time": "2016-02-20T01:09:13Z",
  "region": "us-east-1",
  "resources": [],
  "detail": {
    "eventVersion": "1.03",
    "userIdentity": {
      "type": "Root",
      "principalId": "123456789012",
      "arn": "arn:aws:iam::123456789012:root",
      "accountId": "123456789012",
      "sessionContext": {
        "attributes": {
          "mfaAuthenticated": "false",
          "creationDate": "2016-02-20T01:05:59Z"
        }
      }
    },
    "eventTime": "2016-02-20T01:09:13Z",
    "eventSource": "s3.amazonaws.com",
    "eventName": "CreateBucket",
    "awsRegion": "us-east-1",
    "sourceIPAddress": "100.100.100.100",
    "userAgent": "[S3Console/0.4]",
    "requestParameters": {
      "bucketName": "my-bucket",
      "key": "file-123.csv"
    },
    "responseElements": null,
    "requestID": "9D767BCC3B4E7487",
    "eventID": "24ba271e-d595-4e66-a7fd-9c16cbf8abae",
    "eventType": "AwsApiCall"
  }
}

We will add the ECS target for this rule:

aws events put-targets \
    --rule MyS3SourceRule \
    --targets  file://target.json

The target.json is as below and that’s where the magic happens.

[
  {
    "Id": "1",
    "Arn": "arn:aws:ecs:us-east-1:XXXXX:cluster/fargateclusters",
    "RoleArn": "arn:aws:iam::XXXXXX:role/service-role/AWS_Events_Invoke_ECS_826859824",
    "InputTransformer": {
      "InputPathsMap": {
        "s3bucket": "$.detail.requestParameters.bucketName",
        "s3key": "$.detail.requestParameters.key"
      },
      "InputTemplate": "{\"containerOverrides\": [{ \"name\": \"file-processor-container\", \"environment\": [ {\"name\":\"S3_BUCKET\", \"value\":<s3bucket>} , {\"name\":\"S3_KEY\", \"value\":<s3key> }] }]  }"
    },
    "EcsParameters": {
      "TaskDefinitionArn": "arn:aws:ecs:us-east-1:XXXXX:task-definition/file-processor",
      "TaskCount": 1,
      "LaunchType": "FARGATE",
      "NetworkConfiguration": {
        "awsvpcConfiguration": {
          "Subnets": ["subnet-abc", "subnet-abca"],
          "SecurityGroups": ["sg-asasasa"],
          "AssignPublicIp": "ENABLED"
        }
      }
    }
  }
]

AWS provides a feature called InputTransformer, that allows to customise the text that is taken from a Cloudwatch event and map it to a template. You define multiple JSON paths from the event and then assign it to different variables for your choice, these variables goes as the input template with <variable-name>. The characters < and > cannot be escaped.

In the above example we map bucketName and key from the event to template variables s3bucket and s3key.

"InputPathsMap": {
        "s3bucket": "$.detail.requestParameters.bucketName",
        "s3key": "$.detail.requestParameters.key"
      }

And we use these variables in the InputTemplate as <s3bucket> and <s3key>. Now your ECS Task would get the value my-bucket in environment variable S3_BUCKET and file-123.csv in S3_KEY.