Continuous deployment of one-time job to AWS ECS Fargate from CircleCI

Overview

Sometimes we need to execute a one-time job in AWS without additional configuration, operational support, and deployment of the new services. Using Kubernetes in this scope is overhead, so we should find a solution on how to do it!

AWS Fargate is a technology that you can use with Amazon ECS to run containers without having to manage servers or clusters of Amazon EC2 instances. With Fargate, you no longer have to provision, configure, or scale clusters of virtual machines to run containers. This removes the need to choose server types, decide when to scale your clusters, or optimize cluster packing.

Systems architecture diagram

The diagram shows the workflow and components which are used by the solution. The main idea is to execute database migrations using AWS ECS Fargate. This implementation helps to reduce the costs of infrastructure, and it doesn’t waste time on operational support of the AWS services.

1. Shell script

This script deploys ECS one task with a proper Docker container. It returns the CloudWatch logs URL of the TaskDefinition to see how the execution was.

#!/bin/bash

AWS_PROFILE=default 
aws ecs run-task --cli-input-json file://${PWD}/ecs-db-migration.json > ecs-log
cat ecs-log

task_id=$(cat ecs-log | jq -r ".tasks[].taskArn" | awk -F'/' '{print $(NF)}')
echo 'https://console.aws.amazon.com/cloudwatch/home?region=$region#logsV2:log-groups/log-group/$252Fecs$252Fmigrations/log-events/ecs$252Fmigration-latest$252F'${task_id}

Task Definition config

cat ecs-db-migration.json
{
  "cluster": "migrations",
  "count": 1,
  "enableECSManagedTags": true,
  "enableExecuteCommand": true,
  "launchType": "FARGATE",
  "networkConfiguration": {
    "awsvpcConfiguration": {
      "assignPublicIp": "DISABLED",
      "securityGroups": [ "sg-id" ],
      "subnets": [ "subnet-id" ]
    }
  },
  "platformVersion": "LATEST",
  "startedBy": "circle-ci-db-migration",
  "taskDefinition": "migrations"
}

2. CircleCI modules

A simple example of the CircleCI pipeline is provided here. It uses modules instead of scripts for building and deploying one-time ECS tasks.

Modules:

• aws-cli
• aws-ecr
• aws-ecs
• jq

Example of the pipeline:

# Note that the following stanza uses CircleCI 2.1 to make use of a Reusable Executor
# This allows defining a docker image to reuse across jobs.
# visit https://circleci.com/docs/2.0/reusing-config/#authoring-reusable-executors to learn more.

version: 2.1
orbs:
  aws-cli: circleci/aws-cli@3.1.1
  aws-ecr: circleci/aws-ecr@7.3.0
  aws-ecs: circleci/aws-ecs@2.2.1
  jq: circleci/jq@2.2.0
executors:
  node-executor:
    docker:
      - image: cimg/node:16.13.1

jobs:
  aws-setup:
    executor: aws-cli/default
    working_directory: ~/application
    steps:
      - aws-cli/setup:
          profile-name: default
      # Persist the specified paths (workspace/echo-output) into the workspace for use in downstream job.
      - persist_to_workspace:
          # Must be an absolute path, or relative path from working_directory. This is a directory on the container which is
          # taken to be the root directory of the workspace.
          root: ~/application
          # Must be relative path from root
          paths:
            - .
  build-migration-image:
    executor:
      name: aws-ecr/default
      use-docker-layer-caching: true
    working_directory: ~/application
    environment:
      ACCOUNT_URL: $account-id.dkr.ecr.$region.amazonaws.com
    steps:
      - attach_workspace:
          # Must be absolute path or relative path from working_directory
          at: .
      - aws-ecr/build-and-push-image:
          account-url: ACCOUNT_URL
          dockerfile: Dockerfile
          profile-name: default
          region: AWS_DEFAULT_REGION
          repo: $repo
          tag: 'migration-latest,migration-${CIRCLE_BUILD_NUM}'
  run-migrations:
    executor: aws-cli/default
    working_directory: ~/application
    steps:
      - aws-cli/setup
      - attach_workspace:
          # Must be absolute path or relative path from working_directory
          at: .
      - aws-cli/setup:
          profile-name: default
      - run:
          name: Executing migrations
          command: ./scripts/run-migrations.sh $env

workflows:
  version: 2
  build-deploy-master:
    jobs:
      - aws-setup:
          filters:
            branches:
              only:
                - master
          context: master
      - run-migrations:
          filters:
            branches:
              only:
                - master
          context: master
          requires:
            - aws-setup
            - build-migration-image
      - build-migration-image:
          filters:
            branches:
              only:
                - master
          context: master
          requires:
            - aws-setup

Dockerfile

Here is an example of the Dockerfile. This example deploys Hello World! container from https://github.com/docker-library/hello-world

FROM scratch
COPY hello /
CMD ["/hello"]

Summary

The provided solution is used for cost optimization and more efficient release management. All operational support activities are covered by AWS ECS Fargate, you just need to develop your own one-time job and run it in the existing ECS cluster. AWS services work together and don’t need complex configurations (ECS, ECR, CloudWatch, etc.).

CircleCI helps to automate all steps and speed up time to market. It uses modules for standardized deployment processes. It helps to avoid the development and support of custom scripts bash/shell/python etc.

URLs:

https://docs.aws.amazon.com/AmazonECS/latest/userguide/what-is-fargate.html
https://circleci.com/docs/2.0/reusing-config/#authoring-reusable-executors
https://hub.docker.com/_/hello-world
https://github.com/docker-library/hello-world