Subscribe to AWS ECS Event Stream Using Serverless Framework

Made with https://draw.io

Do you have scheduled or long-running task on AWS ECS cluster and want to get notified when it fails? You can subscribe to ECS event stream with AWS CloudWatch Event rules and use Amazon SNS to send notifications to your email when container state changes.

The following example uses Serverless Framework to set up a service that sends an email to you when the container stops with the non-zero exit status. You find the sources for this example from GitHub. It is the same service that we are going to install here with Serverless Framework.

If you have not already installed Serverless Framework, do it now by running npm install serverless -g on the command line. For more information about Serverless Framework, visit https://www.serverless.com.

First, run sls install -u https://github.com/maasglobal/ecs-event-notifications -n my-ecs-events on the command line to fetch the service example to your computer. “my-ecs-events” will be the name of your service, so that can be almost anything you like. Next, change directory to “my-ecs-events” and run npm install to install dependencies. If you like you can also run npm test to run unit tests to find out whether the installation was successful or not.

The serverless.yml contains a Lambda function handler, “state-change”, which is used to filter out the events that we don’t want to receive and then send the message to SNS topic.

functions:
  state-change:
    handler: state-change/index.handler
    environment:
      TOPIC_ARN:
        Ref: ECSStateChangeTopic
      MODE: errors
    events:
      - cloudwatchEvent:
          event:
            source:
              - aws.ecs
            detail-type:
              - ECS Task State Change
            detail:
              lastStatus:
                - STOPPED
              stoppedReason:
                - Essential container in task exited

This configuration will trigger the Lambda function when the last status of the task is “STOPPED”, and the reason for that is “Essential contain in task exited”. The environmental variable “MODE” is passed to Lambda function and “errors” is the mode that filters out the events in which all the containers in the task have exited with exit code 0. More info about the AWS ECS Events can be found from AWS documentation and Serverless Framework documentation.

To set up SNS subscription so that you’ll get the notifications on ECS events, you need to add SNS Subscription resource to serverless.yml. Here is the snippet that you also find from resources block in the serverless.yml. The endpoint should be changed to match your email address.

ECSStateChangeSubscription:
  Type: AWS::SNS::Subscription
  Properties:
    Endpoint: my.email@example.com
    Protocol: email
    TopicArn:
      Ref: ECSStateChangeTopic

Now you are ready to deploy the service to AWS! Run sls deploy and sit back, it takes couple minutes in the first round. Once the service is deployed, go to your email account and look for an email titled “AWS Notification - Subscription Confirmation”. Open the email and follow the instructions to allow AWS to send emails to your address.

You should be all set up and receive notifications when your tasks are failing… but as that will hardly ever happen, so you can test the setup by running a failing container in your cluster.

To do that, create a new task definition in AWS console, you can use Fargate or EC2, both of them should work fine. With Fargate, start by adding a name to the task definition. The “Task role” can be set to “None”, as well as the “Task execution role”. Then add some memory, 0.5GB should be enough and 0.25 vCPU. Click “Add container” and add container name, use “alpine:3.7” for the container image. Fill in the “Healthcheck” block with default values (Interval 30, Timeout 5, Start period 0 and Retries 3) and use, e.g. “echo” as health check command. To the “Command” field in the “Environment” block use sh,-c,exit 1, so that when the container starts, it immediately stops with exit status 1. Now click “Add”, “Create” and finally “View task definition”.

To run the task in you cluster press “Actions” button and select “Run Task”. Select launch type “FARGATE” and configure cluster VPC, subnets and security groups with the ones that you have. The container is not connecting to anywhere nor doing anything, so it really doesn’t matter what kind of network configuration you have for this task. Now click “Run Task”.

After a while, you should receive an email with pretty JSON formatter message. If you want to fine-tune the message, you can construct it in the “state-change” Lambda function.