Open in app

Sign in

Write

Sign in

Encrypt Internal Container Traffic in Amazon ECS with AWS Private CA

Securing Internal Container traffic in Amazon ECS with Internal Amazon ELB and ACM Private CA

Olawale Olaleye
CloudAdventure
9 min readApr 9, 2023

--

Before we take a plunge at this solution, It may be beneficial to walk you through what led to it. In my environment, I have several internal AWS load balancers that are the façade of my internal services which communicate over HTTPS as required by the IT Compliance team. We needed the containers in our services orchestrated with Amazon ECS to communicate with each other via private ALBs as well as other services deployed to EC2 instances.

AWS Elastic Load Balancer creates a default Domain Name System (DNS) name when you create a load balancer. When an internal load balancer is created, it receives a DNS name with the following form:
internal-name-123456789.region.elb.amazonaws.com

For this technical write-up, I have created an internal load balancer named internal in the eu-west-1 region, my load balancer receives a DNS name internal-internal-1090337730.eu-west-1.elb.amazonaws.com. For best practice, you can go a step further to create a custom DNS with the help of Amazon Route 53 to route traffic to your private ELB.

To access this internal load balancer over HTTPS, ACM Private Certificate Authority can be used to create the private certificate.

AWS Certificate Manager (ACM) Private Certificate Authority (CA) is a private CA service that extends ACM’s certificate management capabilities to both public and private certificates. With AWS Certificate Manager, you can quickly request a certificate, deploy it on ACM-integrated AWS resources, such as Elastic Load Balancers, Amazon CloudFront distributions, and APIs on API Gateway, and let AWS Certificate Manager handle certificate renewals. Private certificates are used for identifying and securing communication between connected resources on private networks, such as servers, mobile and IoT devices, and applications.

The Architecture:

Configuring Containers in ECS to Trust ACM Private CA with AWS ELB DNS

Assumptions

  • You have an existing Amazon ECS cluster and an existing container service running or refer to the steps to create an Amazon ECS service.
  • You have an existing Windows EC2 instance to test the default ELB URL over HTTPS. See the steps to launch the Windows instance.
  • You have an existing internal load balancer. See steps to create your own Application loadbalancer.

AWS Services involved in this solution:

  • ACM Private CA
  • Amazon ELB
  • Amazon ECS
  • Amazon CodeBuild
  • Amazon S3
  • Amazon ECR

Getting Started

Provisioning the ACM Private CA and Requesting SSL

Step 1: Create a Private CA

You can do this from AWS Console and it installs CA certificate as well or use AWS CLI as demonstrated below:
Create file: ca_config.json. This file is just enough for this article but you can define more parameters. See example

{
   "KeyAlgorithm":"RSA_2048",
   "SigningAlgorithm":"SHA256WITHRSA",
   "Subject":{
      "CommonName":"Whalecloud" # change this to your name
   }
}

Create file: revoke_config.json for OCSP

{
    "CrlConfiguration": {
        "Enabled": true,
        "ExpirationInDays": 7,
        "S3BucketName": "whalecloudrootca"  # <---- S3 bucket already created: you must have a secured Amazon S3 bucket in place before you issue the create-certificate-authority command
    },
   "OcspConfiguration":{
      "Enabled":false
   }
}

Run the AWS CLI Command

aws acm-pca create-certificate-authority \
     --certificate-authority-configuration file://ca_config.json \
     --revocation-configuration file://revoke_config.json \
     --certificate-authority-type "ROOT" \
     --idempotency-token 01234567 \
     --tags  Key=Name,Value=MyPCA

If you are using AWS CLI version 1.6.3 or later, use the prefix fileb:// when specifying the required input file. This ensures that ACM Private CA parses the Base64-encoded data correctly.

From AWS Console

Create Private CA from AWS Console

Note: A root CA does not have a certificate chain.

Step 2: Create a Subordinate Private CA

Follow the same steps above but using the command and ca_config.json below:

ca_config.json

{
   "KeyAlgorithm":"RSA_2048",
   "SigningAlgorithm":"SHA256WITHRSA",
   "Subject":{
      "CommonName":"Whalecloudsub"
   }
}

Run the AWS CLI Command

aws acm-pca create-certificate-authority \
     --certificate-authority-configuration file://ca_config.json \
     --revocation-configuration file://revoke_config.json \
     --certificate-authority-type "SUBORDINATE" \
     --idempotency-token 01234567 \
     --tags  Key=Name,Value=MyPCASUB

For the latest ACM Private CA pricing information, see the ACM Pricing page on the AWS website. You can also use the AWS pricing calculator to estimate costs.

Step 3: Install a subordinate CA certificate hosted by ACM Private CA

You can use the AWS Management Console to create and install a certificate for your ACM Private CA hosted subordinate CA. See the steps here or Run the following AWS CLI Commands

Generate a certificate signing request (CSR) of the subordinate CA.

aws acm-pca get-certificate-authority-csr \
     --certificate-authority-arn arn:aws:acm-pca:eu-west-1:<ACCOUNT_ID>:certificate-authority/<CA_ID> \
     --output text \
     --region eu-west-1 > sub/ca.csr

Using the CSR from the previous step as the argument for the — csr parameter, issue the subordinate a certificate.

aws acm-pca issue-certificate \
     --certificate-authority-arn arn:aws:acm-pca:eu-west-1:<ACCOUNT_ID>:certificate-authority/<CA_ID> \
     --csr fileb://sub/ca.csr \
     --signing-algorithm SHA256WITHRSA \
     --validity Value=365,Type=DAYS

Retrieve the subordinate certificate. The resulting file cert.pem, is a PEM file encoded in base64 format. This will be used for ECS later on.

aws acm-pca get-certificate \
    --certificate-authority-arn arn:aws:acm-pca:eu-west-1:<ACCOUNT_ID>:certificate-authority/<CA_ID> \
    --certificate-arn arn:aws:acm-pca:eu-west-1:<ACCOUNT_ID>:certificate-authority/<CA_ID>/certificate/certificate_ID \
    --output text > sub/cert.pem

Result of the 2 Private CAs we’ve created.

Step 4: On ACM, request a certificate from the subordinate ACM Private CA

You can do this from the AWS console or from AWS CLI. Get the certificate-authority-arn from AWS Certificate Manager Private Certificate Authority.
Run the AWS CLI Command

aws acm request-certificate \
    --domain-name "*.eu-west-1.elb.amazonaws.com" \
    --certificate-authority-arn arn:aws:acm-pca:eu-west-1:<ACCOUNT_ID>:certificate-authority/<CA_ID> \
    --validation-method DNS \
    --idempotency-token 91adc45q
    
{
    "CertificateArn": "arn:aws:acm:eu-west-1:<ACCOUNT_ID>:certificate/<certificate_id>"
}

The result:

Associate the ACM SSL certificate with the internal load balancer

Use the certificate above to create an HTTPS listener on the internal loadbalancer.

aws elbv2 create-listener \
    --load-balancer-arn arn:aws:elasticloadbalancing:eu-west-1:<ACCOUNT_ID>:loadbalancer/app/my-load-balancer/50dc6c495c0c9188 \
    --protocol HTTPS \
    --port 443 \
    --certificates CertificateArn=arn:aws:acm:eu-west-1:<ACCOUNT_ID>:certificate/<certificate_id> \
    --ssl-policy ELBSecurityPolicy-2016-08 \
    --default-actions Type=forward,TargetGroupArn=arn:aws:elasticloadbalancing:eu-west-1:<ACCOUNT_ID>:targetgroup/my-targets/73e2d6bc24d8a067

To add an HTTPS listener using the console, refer here.

Let’s test the default ELB URL from Windows EC2 instances.

The images showed that the Windows instance cannot trust the certificate on the URL.

To address the issue, complete the steps below and install the trusted root certificate cert.pem we generated earlier.

  • Launch MMC (mmc.exe).
  • Choose File > Add/Remove Snap-ins.
  • Choose Certificates, then choose Add.
  • Choose My user account.
  • Choose Add again and this time select Computer Account.
  • Move the new certificate from the Certificates-Current User > Trusted Root Certification Authorities into Certificates (Local Computer) > Trusted Root Certification Authorities.

Access the internal load balancer URL again.

Let’s test the default ELB URL from Containers in ECS.

The curl result below showed that the container cannot trust the certificate on the URL.

root@674e2d69e0b3:/# curl https://internal-internal-1090337730.eu-west-1.elb.amazonaws.com -v
*   Trying 192.168.97.231:443...
* Connected to internal-internal-1090337730.eu-west-1.elb.amazonaws.com (192.168.97.231) port 443 (#0)
* SSL certificate problem: self signed certificate in certificate chain
* Closing connection 0
curl: (60) SSL certificate problem: self signed certificate in certificate chain
More details here: https://curl.se/docs/sslcerts.html
curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it.

Building the ECS container image to trust the root certificate

Retrieve the root certificate for your private certificate authority (CA).

aws acm-pca get-certificate-authority-certificate \
    --certificate-authority-arn arn:aws:acm-pca:eu-west-1:<ACCOUNT_ID>:certificate-authority/<CA_ID> \
    --output text > root.crt

In the Dockerfile used to build your containers’ image, add the two lines below

ADD root.crt /usr/local/share/ca-certificates/root.crt
RUN chmod 644 /usr/local/share/ca-certificates/root.crt && update-ca-certificates

(Optional) If you are automating the Docker Image Build process with CodeBuild, you can store the CA cert in a secured S3 bucket and copy it from S3 during the build phase. See the sample Dockerfile Buildspec.yaml below:
Note: CodeBuild IAM role should be granted S3 permission to the bucket.

# create bucket
aws s3 mb s3://myprivatecabucket --region eu-west-1
make_bucket: myprivatecabucket

Sample Dockerfile

FROM nginx
RUN apt-get update
# openssl is the only required thing to install
RUN apt-get -y install openssl
ADD $CodeBuild_SRC_DIR/root.crt /usr/local/share/ca-certificates/root.crt
RUN chmod 644 /usr/local/share/ca-certificates/root.crt && update-ca-certificates

Sample buildspec.yml

version: 0.2
phases:
  pre_build:
    commands:
      - echo Logging in to Amazon ECR...
      - aws ecr get-login-password --region $AWS_DEFAULT_REGION | docker login --username AWS --password-stdin $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com
  build:
    commands:
      - aws s3 cp s3://myprivatecabucket/root.crt $CodeBuild_SRC_DIR/root.crt    
      - echo Build started on `date`
      - echo Building the Docker image...          
      - docker build -t $IMAGE_REPO_NAME:$IMAGE_TAG .
      - docker tag $IMAGE_REPO_NAME:$IMAGE_TAG $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$IMAGE_REPO_NAME:$IMAGE_TAG      
  post_build:
    commands:
      - echo Build completed on `date`
      - echo Pushing the Docker image...
      - docker push $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$IMAGE_REPO_NAME:$IMAGE_TAG

Create a working directory, zip the two files, and transfer them to S3 bucket

# create a directory to store the sample files above
mkdir CodeBuild
# directory structure should look like this
(CodeBuild)
├── buildspec.yml
└── Dockerfile
# compress the files
zip -r privateCAecsdemo.zip .
# upload to S3 for CodeBuild
aws s3 cp privateCAecsdemo.zip s3://myprivatecabucket/ --region eu-west-1
upload: ./privateCAecsdemo.zip to s3://myprivatecabucket/privateCAecsdemo.zip

Create a create-project.json file with the content below:

{
    "name": "demo-acm-pca-ecs-project",
    "source": {
        "type": "S3",
        "location": "myprivatecabucket/privateCAecsdemo.zip"
    },
    "artifacts": {
        "type": "NO_ARTIFACTS"
    },
    "environment": {
        "type": "LINUX_CONTAINER",
        "image": "aws/CodeBuild/amazonlinux2-x86_64-standard:3.0",
        "computeType": "BUILD_GENERAL1_SMALL",
        "environmentVariables": [
            {
                "name": "AWS_DEFAULT_REGION",
                "value": "eu-west-1",
                "type": "PLAINTEXT"
            },
            {
                "name": "AWS_ACCOUNT_ID",
                "value": "<YOUR_AWS_ACCOUNT_ID>",
                "type": "PLAINTEXT"
            },
            {
                "name": "IMAGE_REPO_NAME",
                "value": "<existing ecr repo name>",
                "type": "PLAINTEXT"
            },
            {
                "name": "IMAGE_TAG",
                "value": "acmca",
                "type": "PLAINTEXT"
            }
        ]        
   },
    "serviceRole": "arn:aws:iam::AWS_ACCOUNT_ID:role/service-role/<role_name>"
}

Create a CodeBuild project with the command below

aws CodeBuild create-project --cli-input-json file://create-project.json

Result:

From the AWS CodeBuild Console, start a build. The outcome of the successful CodeBuild project.

Update your existing service in ECS to use the newly build image.

Manually confirm that your Private CA is now on the list of trusted CAs. Exec into the container running on ECS and run the command below:

root@563e2d69e0bd:/# awk -v cmd='openssl x509 -noout -subject' '
    /BEGIN/{close(cmd)};{print | cmd}' < /etc/ssl/certs/ca-certificates.crt | grep whalecloudsub
subject=CN = whalecloudsub
root@563e2d69e0bd:/#

The container is able to communicate with the internal load balancer.

root@563e2d69e0bd:/# curl https://internal-internal-1090337730.eu-west-1.elb.amazonaws.com -v
*   Trying 192.168.97.231:443...
* Connected to internal-internal-1090337730.eu-west-1.elb.amazonaws.com (192.168.97.231) port 443 (#0)
* Server certificate:
*  subject: CN=*.eu-west-1.elb.amazonaws.com
*  start date: Jun  2 12:45:57 2022 GMT
*  expire date: Jul  2 13:45:57 2023 GMT
*  subjectAltName: host "internal-internal-1090337730.eu-west-1.elb.amazonaws.com" matched certs "*.eu-west-1.elb.amazonaws.com"
*  issuer: CN=whalecloudsub
*  SSL certificate verify ok.
...
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
</head>
<body>
<h1>Welcome to nginx!</h1>

Conclusion

This article covered how to configure containers running on Amazon ECS to trust the private CA certificate and to use the default DNS name generated for Amazon ELB. It is a recommended best practise to add an alternative DNS record when requesting for a private certificate in the certificate manager to trust custom DNS instead of using the default DNS name generated for Amazon ELB.

Containers
DevOps
AWS
Security
Architecture

--

--

Olawale Olaleye
CloudAdventure

Written by Olawale Olaleye

418 Followers

DevOps Pro | Cloud Solutions Architect | MultiCloud Specialist

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams