ECS Task Runner with Lambda

Background

When you need to execute a long-running or periodic jobs in AWS, it’s been very conventional to write your scripts and configure them as cron job on EC2 instances. However, you risk the failure of EC2 instances and you may also have problem in its scalability.

Solution

An interesting approach is to wrap your business logic of periodic long-running jobs into Docker images and compile them into AWS ECS Task Definitions and use the ECS CLI or SDK to run-task with them. The ECS scheduler will try to allocate resources for the task on proper ECS instance and execute the Docker command you wrapped in.

ECS CLI

Let’s assume you already wrap your codes of business logic into the docker image and created your ECS task definition. It’s very easy to run the task for one-shot just like this:

aws --region ap-northeast-1 ecs run-task --task-definition my_task_definition

ECS, Cloudwatch Events and Lambda

Moreover, if you prefer the serverless fashion, you can configure Cloudwatch Events with scheduling rules, invoking Lambda to run the ECS task for you.

Considering the following example(ecs-task-runner.js):

In CloudWatch Events, configure the schedule expression like this

50 6 ? * SAT *

The above means execute the custom event Every Saturday 6:50AM UTC. Then configure the constant JSON input payload just like this

{
“ecs_task_def”: “my_task_def”,
“region”: “us-west-2”,
“cluster”: “default”,
“count”: 1
}

When invoking Lambda function, the input payload above will construct itself as the event object for Lambda, and Lambda will hence perform the one-shot task running on ECS as you expect.

Have Fun!

One clap, two clap, three clap, forty?

By clapping more or less, you can signal to us which stories really stand out.