Overcoming Deployment Challenges: Deploying Docker Containers on AWS EC2 Instances

In my exploration of cloud infrastructure, I’ve found AWS EC2 instances to be the backbone for deploying a myriad of applications, providing the flexibility, scalability, and reliability that businesses crave. Yet, the process of deploying Docker containers on EC2 instances comes with its set of hurdles, especially when it comes to automation and updating container images. This blog post is my dive into these challenges, as I offer a structured approach to deploying Docker containers on EC2 instances using AWS CloudFormation. Throughout, I’ll also address the limitations that come with UserData initialization, paving the way for smoother deployments in the future.

Problem Statement

When deploying Docker containers on EC2 instances, relying solely on UserData for script execution presents several limitations. UserData scripts execute only during instance initialization, making subsequent updates or changes challenging. This limitation becomes particularly problematic when needing to update the Docker image running inside the EC2 instance. Even though UserData contains the necessary scripts for updating the Docker image, they won’t execute again after the initial instance launch, rendering the update process manual and error-prone.

Solution Overview

To tackle these issues, I use AWS CloudFormation, along with, cfn-init, and cfn-hup to automate deploying and updating Docker containers on EC2 instances. CloudFormation is great for managing AWS resources using code, while cfn-init and cfn-hup help automate and manage configurations on EC2 instances.

What is cfn-init and cfn-hup?

cfn-init: AWS CloudFormation Init (cfn-init) is a tool designed to help bootstrap and configure instances that are launched as part of an AWS CloudFormation stack. It allows you to define configuration sets, which contain instructions for installing packages, creating files, and executing scripts on EC2 instances. These instructions are specified in the CloudFormation template metadata and are executed during the instance initialization process.

cfn-hup: AWS CloudFormation Helper Scripts (cfn-hup) is another essential component that works in conjunction with cfn-init. Its primary purpose is to detect changes in the CloudFormation stack and trigger updates on EC2 instances accordingly. cfn-hup runs as a daemon on the EC2 instance, periodically checking for updates in the CloudFormation stack metadata. When changes are detected, cfn-hup can execute specified actions, such as running scripts or updating configurations, ensuring that the instance remains in sync with the desired state defined in the CloudFormation template.

How do cfn-init and cfn-hup work together?

During the initialization of an EC2 instance launched via CloudFormation, cfn-init reads the metadata specified in the CloudFormation template and executes the defined configuration sets. These sets encompass various instructions crucial for system configuration, such as installing dependencies, configuring services, and executing setup tasks required for the instance.

In our diagram, cfn-init is depicted as the component responsible for configuring the system based on the instructions provided in the CloudFormation template. This includes setting up system configurations, installing packages, configuring files, and other necessary tasks outlined in the CloudFormation template.

Once the instance is initialized, cfn-hup continues to monitor the CloudFormation stack metadata for any changes. If changes are detected, cfn-hup triggers the execution of specified actions, ensuring that the instance remains up-to-date with the latest configuration defined in the CloudFormation template.

By leveraging the capabilities of cfn-init and cfn-hup within AWS CloudFormation, we can automate the deployment and configuration of EC2 instances, including the provisioning and updating of Docker containers, thereby streamlining our deployment workflows and enhancing the reliability and scalability of our infrastructure.

Implementation Steps

Step 1: Defining the EC2 Instance and Metadata:

EC2Service:
  Type: AWS::EC2::Instance
  Metadata:
    AWS::CloudFormation::Init:
      configSets:
        default:
          - install_docker
      install_docker:
        files:
          "/tmp/install.sh":
            content: !Sub
              - |
                #!/bin/bash
                # Install Docker 
                sudo yum update -y
                sudo amazon-linux-extras install docker
                sudo service docker start
                sudo usermod -a -G docker ec2-user

                # Pulling and deploying new Docker image
                sudo docker pull ${ImageName}
                # Restarting Docker container with the new image
                sudo docker stop ${ContainerName} && sudo docker rm ${ContainerName}
                sudo docker run -d --name ${ContainerName} -p 8000:9000 ${ImageName}
            mode: "000755"
            owner: root
            group: root

Explanation:

• We define an EC2 instance named “EC2Service” using AWS::EC2::Instance.
• In the Metadata section, we utilize AWS::CloudFormation::Init to specify configuration sets.
• The “install_docker” configSet includes instructions for installing Docker and deploying a new Docker image.
• We define a script (install.sh) in the "files" section. This script installs Docker, pulls the specified Docker image (${ImageName}), stops and removes the existing Docker container (${ContainerName}), and runs a new container with the pulled image, mapping port 8000 on the host to port 9000 inside the container.

Note: ImageName and ContainerName are placeholders for the actual image name and container name. They should be defined in the parameters section of the CloudFormation template.

Step 2: Configuring cfn-hup:
In this step, we are configuring cfn-hup, which is a part of the AWS CloudFormation helper scripts designed to detect changes in the CloudFormation stack and trigger updates on EC2 instances accordingly. Let’s break down the configuration:

        /etc/cfn/cfn-hup.conf:
          content: !Join
            - ""
            - - |
                [main]
              - stack=
              - !Ref "AWS::StackId"
              - |+
              - region=
              - !Ref "AWS::Region"
              - |+
              - interval=
              - 3
              - |+
          mode: "000400"
          owner: root
          group: root
        /etc/cfn/hooks.d/cfn-auto-reloader.conf:
          content: !Join
            - ""
            - - |
                [cfn-auto-reloader-hook]
              - |
                triggers=post.update
              - >
                path=Resources.EC2Service.Metadata.AWS::CloudFormation::Init
              - "action=/opt/aws/bin/cfn-init -s "
              - !Ref "AWS::StackId"
              - " -r EC2Service "
              - " --region     "
              - !Ref "AWS::Region"
              - |+
              - |
                runas=root
          mode: "000644"
          owner: apache
          group: apache

cfn-hup Configuration (/etc/cfn/cfn-hup.conf):

• This file specifies the configuration settings for cfn-hup, which is responsible for detecting changes in the CloudFormation stack and triggering updates on the EC2 instance.
• The content attribute defines the content of the cfn-hup.conf file.
• Within the content, we have sections like [main] that define various parameters:
• stack=: Typically contains the name of the CloudFormation stack, but here it's left empty.
• region=: Normally contains the AWS region where the stack is deployed, also left empty here.
• interval=: Specifies the frequency (in minutes) at which cfn-hup checks for updates. In this case, it's set to 3 minutes.
• The mode, owner, and group attributes define the file permissions, owner, and group for the cfn-hup.conf file, respectively.

cfn-auto-reloader Configuration (/etc/cfn/hooks.d/cfn-auto-reloader.conf):

• This file configures a hook called cfn-auto-reloader, which triggers actions after CloudFormation stack updates.
• The content attribute defines the content of the cfn-auto-reloader.conf file.
• Within the content, we have sections like [cfn-auto-reloader-hook] that define the hook configuration:
• triggers=post.update: Specifies that this hook should be triggered after a CloudFormation stack update.
• path=Resources.EC2Service.Metadata.AWS::CloudFormation::Init: Specifies the path where CloudFormation init metadata is located.
• action=/opt/aws/bin/cfn-init -s <StackId> -r EC2Service --region <Region>: Defines the action to perform when the hook is triggered. It typically involves running cfn-init to apply new configurations. Here, <StackId> and <Region> are placeholders for the actual CloudFormation stack ID and AWS region.
• runas=root: Indicates that the action should be executed with root privileges.
• The mode, owner, and group attributes define the file permissions, owner, and group for the cfn-auto-reloader.conf file, respectively.

Step 3: Configuring UserData:

    UserData:
      "Fn::Base64": !Sub |
        #!/bin/bash
        yum install -y aws-cfn-bootstrap
        /opt/aws/bin/cfn-init -v --stack ${AWS::StackId} --resource EC2Service --region ${AWS::Region} --configsets default
        /opt/aws/bin/cfn-hup || error_exit 'Failed to start cfn-hup'
        /opt/aws/bin/cfn-signal -e $? --stack ${AWS::StackId} --region ${AWS::Region} --resource EC2Service

Explanation:

1. Installation of AWS CloudFormation Bootstrap Tools: The script begins by installing the AWS CloudFormation bootstrap tools (aws-cfn-bootstrap) on the EC2 instance. These tools are essential for executing further CloudFormation-initiated actions on the instance.
2. Initialization of CloudFormation Metadata: The script uses cfn-init to initialize CloudFormation metadata on the EC2 instance. This metadata includes configurations defined in the CloudFormation template, which are applied to the instance.
3. Starting CloudFormation Helper Script (cfn-hup): The script starts the CloudFormation helper script (cfn-hup). This script continuously checks for changes in the CloudFormation metadata and applies them to the instance, ensuring that its configuration stays up-to-date.
4. Signaling CloudFormation: Finally, the script sends a signal back to CloudFormation, indicating whether the UserData execution was successful or encountered errors. This signal allows CloudFormation to proceed with subsequent stack operations based on the outcome of the initialization process.

Conclusion

In conclusion, the UserData script within the AWS CloudFormation template executes only during the initial launch of an EC2 instance, setting up essential configurations such as installing necessary tools and initializing CloudFormation metadata. The CloudFormation helper daemon (cfn-hup) is started to continuously monitor changes in the CloudFormation metadata, facilitating automatic updates to the EC2 instance's configuration.

cfn-hup detects changes in the CloudFormation metadata, including updates to Docker image specifications. Therefore, modifications to the Docker image referenced in the CloudFormation template will trigger cfn-hup to update the EC2 instance, ensuring that the latest Docker image is pulled and utilized by the container running on the instance.

Overall, this orchestrated mechanism streamlines the management of EC2 instances, facilitating automatic updates and maintaining synchronization with the CloudFormation template, thereby enhancing operational efficiency and ensuring consistency in deployment configurations.

Here is the complete cloudformation yaml code:

EC2Service:
  Type: AWS::EC2::Instance
  Metadata:
    AWS::CloudFormation::Init:
      configSets:
        default:
          - install_docker
      install_docker:
        files:
          "/tmp/install.sh":
            content: !Sub
              - |
                #!/bin/bash
                # Install Docker 
                sudo yum update -y
                sudo amazon-linux-extras install docker
                sudo service docker start
                sudo usermod -a -G docker ec2-user

                # Pulling and deploying new Docker image
                sudo docker pull ${ImageName}
                # Restarting Docker container with the new image
                sudo docker stop ${ContainerName} && sudo docker rm ${ContainerName}
                sudo docker run -d --name ${ContainerName} -p 8000:9000 ${ImageName}
              mode: "000755"
              owner: root
              group: root

          "/etc/cfn/cfn-hup.conf":
            content: !Join
              - ""
              - - |
                  [main]
                - stack=
                - !Ref "AWS::StackId"
                - |+
                - region=
                - !Ref "AWS::Region"
                - |+
                - interval=
                - 3
                - |+
            mode: "000400"
            owner: root
            group: root

          "/etc/cfn/hooks.d/cfn-auto-reloader.conf":
            content: !Join
              - ""
              - - |
                  [cfn-auto-reloader-hook]
                - |
                  triggers=post.update
                - >
                  path=Resources.EC2Service.Metadata.AWS::CloudFormation::Init
                - "action=/opt/aws/bin/cfn-init -s "
                - !Ref "AWS::StackId"
                - " -r EC2Service "
                - " --region     "
                - !Ref "AWS::Region"
                - |+
                - |
                  runas=root
            mode: "000644"
            owner: apache
            group: apache

        services:
          sysvinit:
            cfn-hup:
              enabled: "true"
              ensureRunning: "true"
              files:
                - /etc/cfn/cfn-hup.conf
                - /etc/cfn/hooks.d/cfn-auto-reloader.conf

        commands:
          00_run_install:
            command: "bash /tmp/install.sh"

    Properties:
      InstanceType: !Ref InstanceType
      ImageId: !Ref ImageId
      KeyName: !Ref YourEC2KeyPair
      IamInstanceProfile: !Ref EC2InstanceProfile
      UserData:
        "Fn::Base64": !Sub |
          #!/bin/bash
          yum install -y aws-cfn-bootstrap
          /opt/aws/bin/cfn-init -v --stack ${AWS::StackId} --resource EC2Service --region ${AWS::Region} --configsets default
          /opt/aws/bin/cfn-hup || error_exit 'Failed to start cfn-hup'
          /opt/aws/bin/cfn-signal -e $? --stack ${AWS::StackId} --region ${AWS::Region} --resource EC2Service