A Simple, Cost-effective Container Platform for AWS — Part 1

Allan Denot

Published in

DNX Labs

6 min readJun 26, 2019

Scalable, high-available, low cost and low maintenance. Is that possible?

Introduction

In this article, we are going to describe how to build a platform to run Docker images. The main technologies used are:

Docker
AWS
Terraform

Within the AWS ecosystem, we will leverage the following services:

EC2 with Spot instances and autoscaling for low-cost and scalable compute
ECS for scheduling containers
ALB for load balancing traffic between instances
CodeDeploy for blue-green deployments
EFS for those applications that need a persistent shared filesystem between all containers (such as Wordpress)
CloudFront for CDN
ACM, IAM, VPC, Security Groups as part of the stack

Architecture

Traffic flows to CloudFront that proxies to the ALB, which in turn distributes through the EC2 instances part of the autoscaling group.

EFS provides a mount target per Availability Zone, we are mounting the respective target to the EC2 depending which AZ it belongs.

For database, we will use RDS/Aurora MySQL. For non-production environments, Aurora Serverless can be used to reduce costs even more. It scales to zero when there are no connections active for a determined time. For production, we have the choice between RDS Multi-AZ and Aurora.

Terraform Setup

We are using 2 open source modules to make the deployment easier:

ECS Platform: https://github.com/DNXLabs/terraform-aws-ecs

ECS Service: https://github.com/DNXLabs/terraform-aws-ecs-app

Other services like the RDS DB and it’s security group will be created with Terraform but without using modules.

The full, working code for this article is at https://github.com/DNXLabs/terraform-aws-ecs/tree/master/example

Our files structure in terraform looks like this (files ordered as described in this article):

.
├── backend.tf
├── provider.tf
├── variables.tf
├── db.tf
├── ecs-cluster.tf
└── ecs-app.tf

Now let’s dig into those files.

backend.tf (for remote state storage) — optional

It’s a good practice to store Terraform state in a remote location. Since we are in the AWS ecosystem, we use the S3 backend type.

This is an optional step. Without this definition, terraform will save the state in a file in your local machine. Make sure to commit the file to GIT or store securely.

Check out article about Terraform Remote States at: https://medium.com/dnx-labs/terraform-remote-states-in-s3-d74edd24a2c4

Once you have the backend created, the backend configuration should look like this:

provider.tf

variables.tf and Terraform Workspaces

We recommend using workspaces so we can deploy multiple stacks using the same terraform code.

More details on how to use Workspaces and how we use it, at https://medium.com/dnx-labs/terraform-workspaces-101-f8e1a5054547

Let’s create and switch to a workspace with the commands:

# create a workspace called 'dev-apps':
terraform workspace new dev-apps# list:
terraform workspace list# switch to workspace:
terraform workspace select dev-apps

In our variables.tf file, we have parameters per workspace. This example shows values for workspaces dev-apps and prod-apps :

Those parameters will be available throughout the stack by using the syntax ${local.workspace["parameter_name"]} . Where parameter_name is the parameter and will resolve to different values per workspace.

Example: ${local.workspace["vpc_id"]} will return the value of vpc_id defined for the workspace.

Database

There are many flavours of managed DB that can be used. For simplicity we will use Aurora Serverless which also fits well on non-production environments as it scales down to zero and it’s costs are reasonable.

Under scaling_configuration you can se the min and max_capacity which will scale your DB accordingly. auto_pause when set to true scales the DB to zero when there’s no connections after 300 seconds defined.

The file also describes a security group attached to the DB instance so EC2 machines part of the ECS cluster can access it.

ACM Certificates

You will need at least 2 ACM certificates:

1 — Certificate for ALB (one per ECS cluster)

In the region you’re deploying the stack (in our example: ap-southeast-2)
For communication between CloudFront and ALB
Needs to sign domains used in the origin, we recommend a wildcard like: *.dev-apps.domain.com or *.prod-apps.domain.com.

2— Certificate for CloudFront (one per application)

In region us-east-1 (CloudFront can only use this region)
Certificate signs the public domain to access your application, example: myapp.domain.com or myapp-dev.domain.com.

This article will not cover the creation of the certificates using Terraform, create them manually using the AWS Console, verify and copy the ARN to use in variables.tf.

ECS Platform

As you notice, most parameters are coming from locals defined in the variables file, which change per workspace.

We are using Spot Fleet (with Launch Templates) to reduce costs, hence the 3 instance types passed.

AWS will use the instance types passed to avoid running out of capacity if a spike on the spot pricing occurs in one of the instance type.

on_demand_percentage defines how we want to distribute between on-demand and spot instances.

We recommend 0% on demand on non-production environments to reduce costs to the limit by using all spot.

In production, if your application responds well to failure, a value between 0 and 50% is safe to use.

If your application doesn’t handle failure well, it’s prudent to increase this value as spot instances can be unstable.

ECS Service (Application)

This module will create:

CloudFront distribution using the certificate ARN passed
Route53 Records in the hosted_zone passed
CodeDeploy application for deployments

The parameters hostname and hostname_blue point to the CloudFront distribution, while hostname_origin points to the ALB (everything using HTTPS).

[hostname]   [hostname_blue]
     |              |
    ┌────────────────┐
    |   CloudFront   |
    └────────────────┘
            |
    [hostname_origin]
            |
       ┌─────────┐          ┌─────┐┌─────┐
       |   ALB   |──────────| EC2 || EC2 |
       └─────────┘          └─────┘└─────┘

You might notice the image parameter is set to nginxdemos/hello:latest . We are using a placeholder container image for now as we just want the ECS service to come online and will deploy the right application later (in our case Wordpress).

Deploying the Infrastructure

Once all terraform files are setup, let’s deploy it:

terraform init                       # initialise terraform
terraform workspace new dev-apps     # create the new workspace
terraform workspace select dev-apps  # select it
terraform apply

All going well, once terraform finishes, you should be able to login to the AWS Console and see some of the resources created.

And if you browse to the hostname passed, you would see the NGINX Hello World below:

As we mention above, it’s just a placeholder for the application we are going to deploy in our next article.

Costs

EC2 Instances

At the time of this writing, a Spot t3.small (2GB RAM, 2 VCPU) costs $0.0079/hour or $5.70/month in Sydney.

Our cluster, running on 100% spot when setting on_demand_percentage = 0, will cost less than $12/month with 2x t3.small.

Managed Database

The cheapest RDS available by AWS is a db.t2.micro running MySQL. It has 1 vCPU and 1GB of RAM. In Sydney (ap-southeast-2) it costs $19/mo (plus storage costs).

Aurora Serverless (MySQL) charges per ACU (Aurora Capacity Unit), which is a 2GB instance. Each ACU costs $72/mo and the minimum is 2.

Running an Aurora Serverless DB 24/7 will cost $144/mo.

But if you run in a non-production environment only during business hours, as it scales to zero when not in use, it will cost $35/mo or less.

Conclusion:

Even though a db.t2.micro RDS is cheaper at $19/mo, you get much more capacity for $35/mo when using Aurora Serverless.

Source Code

Check our repo at https://github.com/DNXLabs/terraform-aws-ecs/tree/master/example for a working example of the stack shown in this tutorial.

Next Steps

In part 2 we will show how to we deploy an application to the ECS cluster using CodeDeploy and Gitlab CI.

Conclusion

Hopefully, this article was helpful in some way.

Please leave your feedback below as well as anything you would like to see added.

We work at DNX Solutions and help to bring a better cloud and application experience for digital native startups in Australia.

Our current focus areas are: AWS, Well-Architected Solutions, Containers, ECS, Kubernetes, Continuous Integration/Continuous Delivery and Service Mesh.

We are constantly hiring cloud engineers for our Sydney office, focusing on cloud-native concepts.

Check our open source projects at https://github.com/DNXLabs and follow us on our Twitter or Linkedin