Set up scheduled tasks with AWS Fargate using CloudFormation Templates

Out of the multiple options available in AWS to run containers, Fargate is the best option to deploy a scheduled task that only runs a couple of hours max per day. I’ll tell you why!

The most important thing to consider here is that in most cases a scheduled task doesn’t need to run the whole day, which means it doesn’t need resources allocated to it the whole day. So, serverless options like Lambda and Fargate make great sense here. But which one?

Use Lambda if your application is short-lived and doesn’t require much memory.
If your application is a long compute job and requires a lot of memory, use Fargate.

Let’s go with Fargate for the sake of this article.

Running ECS with Fargate eliminates the need to manually provision, scale, and manage compute instances. You need to create a cluster, add tasks to it and specify resource requirements (CPU and memory), and when ECS containers are deployed, Fargate will launch, run and manage pre-configured servers that meet container requirements.

Even with Fargate, you have two options to run tasks. By creating a service or a scheduled task. With scheduled tasks, you can schedule a time for the task to start up and make sure that there are no resources provisioned after the task is finished by making the task self-terminating.

Now let’s see how we can do this.

The first thing to do is to create an ECS cluster to deploy our task in.

MyECSCluster:
    Type: "AWS::ECS::Cluster"
    Properties:
      ClusterName: my-ecs-cluster

Next, we need a task definition to spin up a task within the created cluster.

MyEcsTaskDefinition:
    Type: AWS::ECS::TaskDefinition
    Properties:
      Cpu: 1024
      Memory: 2048
      ExecutionRoleArn: !GetAtt MyECSTaskExecutionRole.Arn
      Family: my-task-definition
      NetworkMode: awsvpc
      RequiresCompatibilities:
        - FARGATE
      TaskRoleArn: !GetAtt MyECSRole.Arn
      ContainerDefinitions:
        - Image: !Sub "${AWS::AccountId}.dkr.ecr.us-east-1.amazonaws.com/${MyECRRepo}:latest"
          Name: MyTaskContainer
          Memory: 2048
          Cpu: 1024
          PortMappings:
            - { ContainerPort: 80 }
    DependsOn:
      - MyECRRepo

When deploying an application in Fargate, in comparison to EC2 there are a lot fewer configurations. Here, we only have to provide two configurations namely CPU and Memory. You can find a set of accepted values for these two configurations here.

The next two important things are the ExecutionRoleArn and the TaskRoleArn. Why do we need two different roles for the same task? This is because AWS differentiates between a task execution role, which is a general role that grants permissions to start the containers defined in a task, and a task role that grants permissions to the actual application once the container is started.

For example, the task execution role might include permissions to pull the container image from ECR, manage the logs for the task, etc.

MyECSTaskExecutionRole:
    Type: "AWS::IAM::Role"
    Properties:
      RoleName: Task-ExecutionRole"
      Policies:
        - PolicyName: MyTaskExecutionPolicy
          PolicyDocument:
            Version: "2012-10-17"
            Statement:
              - Effect: Allow
                Action:
                  - ecr:GetAuthorizationToken
                  - ecr:BatchCheckLayerAvailability
                  - ecr:GetDownloadUrlForLayer
                  - ecr:BatchGetImage
                Resource: "*"
      Path: "/"

In contrast, the task role might include permissions like access to AWS S3 or other services(only if any service is needed within the application).

MyECSTaskRole:
    Type: "AWS::IAM::Role"
    Properties:
      RoleName: MyECSRole
      Policies:
        - PolicyName: MyECSPolicy
          PolicyDocument:
            Version: "2012-10-17"
            Statement:
              - Effect: Allow
                Action:
                  - ssm:DescribeParameters
                  - ssm:GetParametersByPath
                Resource: "*"
      Path: "/"

The next important configuration here is the image under ContainerDefinitions. This is the docker image that will be used to run the scheduled task. If you are planning to use a CI/CD pipeline to deploy this scheduled job into multiple environments, it’s highly recommended to use different images for each environment because each time the task restarts, it will use the image specified in this configuration. If you use the same image for every environment, you will have no control over the deployment.

Now, you need to create a rule which will run the scheduled task according to the rule you define.

MyServiceScheduleRule:
    Type: AWS::Events::Rule
    Properties:
      Description: "Schedule rule for My Service"
      Name: !Sub my-task-schedule-rule-${Stage}
      State: ENABLED
      ScheduleExpression: cron(0/45 06-18 ? * FRI *)
      Targets:
        - Arn: !GetAtt MyECSCluster.Arn
          Id: ScheduledTask
          Input: !Sub '{ "containerOverrides": [{"name": "MyTaskContainer", "environment": [{"name":"APPLICATION_ENV","value":"${Stage}"}]}'
          EcsParameters:
            TaskDefinitionArn: !Ref MyEcsTaskDefinition
            LaunchType: FARGATE
            NetworkConfiguration:
              AwsVpcConfiguration:
                AssignPublicIp: DISABLED
                SecurityGroups:
                  - !Ref MyServiceFargateSecurityGroup
                Subnets:
                  - !Ref SubnetId1
                  - !Ref SubnetId2
                  - !Ref SubnetId3

That covers the main components you have to create when deploying a scheduled task in Fargate.

You can find the complete CloudFormation Template here.

Thank you for reading!