A Simple, Cost-effective Container Platform for AWS — Part 1
Scalable, high-available, low cost and low maintenance. Is that possible?
Introduction
In this article, we are going to describe how to build a platform to run Docker images. The main technologies used are:
- Docker
- AWS
- Terraform
Within the AWS ecosystem, we will leverage the following services:
- EC2 with Spot instances and autoscaling for low-cost and scalable compute
- ECS for scheduling containers
- ALB for load balancing traffic between instances
- CodeDeploy for blue-green deployments
- EFS for those applications that need a persistent shared filesystem between all containers (such as Wordpress)
- CloudFront for CDN
- ACM, IAM, VPC, Security Groups as part of the stack
Architecture
Traffic flows to CloudFront that proxies to the ALB, which in turn distributes through the EC2 instances part of the autoscaling group.
EFS provides a mount target per Availability Zone, we are mounting the respective target to the EC2 depending which AZ it belongs.
For database, we will use RDS/Aurora MySQL. For non-production environments, Aurora Serverless can be used to reduce costs even more. It scales to zero when there are no connections active for a determined time. For production, we have the choice between RDS Multi-AZ and Aurora.
Terraform Setup
We are using 2 open source modules to make the deployment easier:
ECS Platform: https://github.com/DNXLabs/terraform-aws-ecs
ECS Service: https://github.com/DNXLabs/terraform-aws-ecs-app
Other services like the RDS DB and it’s security group will be created with Terraform but without using modules.
The full, working code for this article is at https://github.com/DNXLabs/terraform-aws-ecs/tree/master/example
Our files structure in terraform looks like this (files ordered as described in this article):
.
├── backend.tf
├── provider.tf
├── variables.tf
├── db.tf
├── ecs-cluster.tf
└── ecs-app.tf
Now let’s dig into those files.
backend.tf (for remote state storage) — optional
It’s a good practice to store Terraform state in a remote location. Since we are in the AWS ecosystem, we use the S3 backend type.
This is an optional step. Without this definition, terraform will save the state in a file in your local machine. Make sure to commit the file to GIT or store securely.
Check out article about Terraform Remote States at: https://medium.com/dnx-labs/terraform-remote-states-in-s3-d74edd24a2c4
Once you have the backend created, the backend configuration should look like this:
provider.tf
variables.tf and Terraform Workspaces
We recommend using workspaces so we can deploy multiple stacks using the same terraform code.
More details on how to use Workspaces and how we use it, at https://medium.com/dnx-labs/terraform-workspaces-101-f8e1a5054547
Let’s create and switch to a workspace with the commands:
# create a workspace called 'dev-apps':
terraform workspace new dev-apps# list:
terraform workspace list# switch to workspace:
terraform workspace select dev-apps
In our variables.tf file, we have parameters per workspace. This example shows values for workspaces dev-apps
and prod-apps
:
Those parameters will be available throughout the stack by using the syntax ${local.workspace["parameter_name"]}
. Where parameter_name
is the parameter and will resolve to different values per workspace.
Example: ${local.workspace["vpc_id"]}
will return the value of vpc_id
defined for the workspace.
Database
There are many flavours of managed DB that can be used. For simplicity we will use Aurora Serverless which also fits well on non-production environments as it scales down to zero and it’s costs are reasonable.
Under scaling_configuration
you can se the min
and max_capacity
which will scale your DB accordingly. auto_pause
when set to true
scales the DB to zero when there’s no connections after 300 seconds defined.
The file also describes a security group attached to the DB instance so EC2 machines part of the ECS cluster can access it.
ACM Certificates
You will need at least 2 ACM certificates:
1 — Certificate for ALB (one per ECS cluster)
- In the region you’re deploying the stack (in our example: ap-southeast-2)
- For communication between CloudFront and ALB
- Needs to sign domains used in the origin, we recommend a wildcard like:
*.dev-apps.domain.com
or*.prod-apps.domain.com
.
2— Certificate for CloudFront (one per application)
- In region us-east-1 (CloudFront can only use this region)
- Certificate signs the public domain to access your application, example:
myapp.domain.com
ormyapp-dev.domain.com
.
This article will not cover the creation of the certificates using Terraform, create them manually using the AWS Console, verify and copy the ARN to use in variables.tf
.
ECS Platform
As you notice, most parameters are coming from locals defined in the variables file, which change per workspace.
We are using Spot Fleet (with Launch Templates) to reduce costs, hence the 3 instance types passed.
AWS will use the instance types passed to avoid running out of capacity if a spike on the spot pricing occurs in one of the instance type.
on_demand_percentage
defines how we want to distribute between on-demand and spot instances.
We recommend 0% on demand on non-production environments to reduce costs to the limit by using all spot.
In production, if your application responds well to failure, a value between 0 and 50% is safe to use.
If your application doesn’t handle failure well, it’s prudent to increase this value as spot instances can be unstable.
ECS Service (Application)
This module will create:
- CloudFront distribution using the certificate ARN passed
- Route53 Records in the
hosted_zone
passed - CodeDeploy application for deployments
The parameters hostname
and hostname_blue
point to the CloudFront distribution, while hostname_origin
points to the ALB (everything using HTTPS).
[hostname] [hostname_blue]
| |
┌────────────────┐
| CloudFront |
└────────────────┘
|
[hostname_origin]
|
┌─────────┐ ┌─────┐┌─────┐
| ALB |──────────| EC2 || EC2 |
└─────────┘ └─────┘└─────┘
You might notice the image
parameter is set to nginxdemos/hello:latest
. We are using a placeholder container image for now as we just want the ECS service to come online and will deploy the right application later (in our case Wordpress).
Deploying the Infrastructure
Once all terraform files are setup, let’s deploy it:
terraform init # initialise terraform
terraform workspace new dev-apps # create the new workspace
terraform workspace select dev-apps # select it
terraform apply
All going well, once terraform finishes, you should be able to login to the AWS Console and see some of the resources created.
And if you browse to the hostname passed, you would see the NGINX Hello World below:
As we mention above, it’s just a placeholder for the application we are going to deploy in our next article.
Costs
EC2 Instances
At the time of this writing, a Spot t3.small (2GB RAM, 2 VCPU) costs $0.0079/hour or $5.70/month in Sydney.
Our cluster, running on 100% spot when setting on_demand_percentage = 0
, will cost less than $12/month with 2x t3.small.
Managed Database
The cheapest RDS available by AWS is a db.t2.micro running MySQL. It has 1 vCPU and 1GB of RAM. In Sydney (ap-southeast-2) it costs $19/mo (plus storage costs).
Aurora Serverless (MySQL) charges per ACU (Aurora Capacity Unit), which is a 2GB instance. Each ACU costs $72/mo and the minimum is 2.
Running an Aurora Serverless DB 24/7 will cost $144/mo.
But if you run in a non-production environment only during business hours, as it scales to zero when not in use, it will cost $35/mo or less.
Conclusion:
Even though a db.t2.micro RDS is cheaper at $19/mo, you get much more capacity for $35/mo when using Aurora Serverless.
Source Code
Check our repo at https://github.com/DNXLabs/terraform-aws-ecs/tree/master/example for a working example of the stack shown in this tutorial.
Next Steps
In part 2 we will show how to we deploy an application to the ECS cluster using CodeDeploy and Gitlab CI.
Conclusion
Hopefully, this article was helpful in some way.
Please leave your feedback below as well as anything you would like to see added.
We work at DNX Solutions and help to bring a better cloud and application experience for digital native startups in Australia.
Our current focus areas are: AWS, Well-Architected Solutions, Containers, ECS, Kubernetes, Continuous Integration/Continuous Delivery and Service Mesh.
We are constantly hiring cloud engineers for our Sydney office, focusing on cloud-native concepts.
Check our open source projects at https://github.com/DNXLabs and follow us on our Twitter or Linkedin