DevOps Automation with Terraform, Docker and AWS— Implementing a complete Terraform Workflow with GitHub Actions for Golang APIs.

Infrastructure Architectural Design

I would like to share my experience using Terraform, Docker and AWS. I have experience with various CD/CD platforms like Gitlab, Bitbucket, CircleCI, GitHub Actions etc. In this article, I would like to share my experience in implementing a complete infrastructure and deployment with Terraform, AWS and GitHub in a GitOps-style workflow. We will be implementing a production-grade AWS ECS environment and deploying a simple Go application on it.

Amazon ECS is a service provided by AWS that manages the orchestration and provisioning of the containers. You can read more about ECS here.

Amazon ECS comes with two deployment options, EC2 and Fargate.

With EC2 deployments, you need to manage the infrastructure i.e the number of EC2 instances that are required for your containers. Farget is a serverless compute engine provided by AWS.

Fargate is a serverless compute engine. This means your containers that are launched are managed by AWS. What you need to specify is the CPU and memory your containers will use. AWS will then provision an appropriate amount of the compute resources.

In this article we will focus on provisioning the infrastructure environment for AWS ECS Fargate with Terraform, and implement a CI/CD with GitOps workflow using GitHub Actions.

High-Level Architectural Design

HL Architecture

Prerequisites

You need to have some basic understanding of working with ECS and creating an ECS cluster using an AWS management console or CLI and basic knowledge of Terraform. You will need to have:

• A working AWS account. You can signup and use the free tier offered by AWS. Please note some of the resources created here may go beyond the free tier.
• AWS CLI and AWS Vault are installed and configured on your local machine. I will not cover how to do these here. If you have any issues installing and configuring, please let me know in the comment section. I will help you out.
• Terraform CLI is installed on your local machine. We use the v1.3.6 (latest version at the time of writing this article) in this article but feel free to use newer versions if you want. My recommendation is to use a docker image with a docker-compose file or tfenv then simplify the installation and usage of a specific version.
• Docker installed on your local machine.

Please note some of the resources created here may go beyond the free tier e.g DNS. So please be aware of this before applying the Terraform. You can as well make sure you immediately destroy the resources as soon as you are done with this tutorial.

Just to know the various components we will be creating:

I needed to demo how to deploy and run a 3-tier Golang application with an Nginx proxy. So I wrote a simple Golang service and Nginx Proxy for this demo. Please note this environment can be used to run any other containerized application. To build this environment on AWS, we will create the below services:

• VPC and Networking (Subnets, Internet Groups…)
• Elastic Container Service
• Application Load Balancer
• Cloud Watch
• Relational Database Service
• Route53
• A single NAT Gateway. This could create a single point of failure since the NAT Gateway is in one AZ. You can change to ensure Full Availability.
• A single Bastion EC2 Instance. This could create a single point of failure since Bastion Instance is in one AZ. You can change to ensure Full Availability.

Provider Configuration

The first thing we must create is provider configuration. Terraform relies on " providers " plugins to interact with cloud providers, SaaS providers, and other APIs. Terraform configurations must declare which providers they require so that Terraform can install and use them.

Provider configuration

✅ Recommendations:

• It is a good idea to declare the version of Terraform to be used to avoid any breaking changes that could affect our infrastructure if we use newer/older versions when running Terraform in the future.
• Resource providers can be handled automatically by Terraform with the init command. However, it’s a good idea to define them explicitly using version numbers the way we have done above to avoid Datasource/resource-breaking changes by future versions.
• It is advisable to lock the state of your backend to avoid others from acquiring the lock and potentially corrupting your state, especially when running this in a CI/CD pipeline. We are using Amazon DynamoDB for this.

⚠️Important: The S3 bucket and DynamoDB table need to exist before running terraform init command. Terraform will not create them if they do not exist in AWS. You can create a bucket manually or via a CI/CD tool running a command like this:

aws s3 mb s3://my-iac-bucket-name --region eu-west-1

Bucket names must be unique. Read more here.

For the DynamoDB table, the quickest way to do that is to create it manually via the console but you can as well create it via AWS CLI.

• Avoid defining AWS credentials in provider blocks. Instead, we could use environment variables for this purpose. Terraform will automatically use them to authenticate against AWS APIs.
  We will be using docker with this docker-compose file to run our Terraform commands.

docker-compose.yaml

We will create AWS IAM User with only needed permissions to run and create our infrastructure via Terraform.
Head to the AWS console and create an IAM Policy with the following content. Now you can create IAM User and attach the policy to that user then create credentials that you will use for your was vault setup. Please review the policy content and make sure the resources the policy is allowing for access e.g S3 Bucket, and DynamoDB names match what you have created.

As we mentioned above, we will be using aws-vault to securely store and access AWS credentials in our development environment.

⚠️Important: The version of the docker image used in the docker-compose file should be the same as the Terraform version we are using in the provider configuration.

We will be using Terraform Workspaces, for example, staging workspace for deploying to the staging environment. To initialize each workspace, we should run this command:

docker-compose -f docker-compose.yml run --rm terraform init

docker-compose -f docker-compose.yml run --rm terraform workspace new staging

Up to this point, we are ready now to start writing our infrastructure as code 😀. Take a break and grab some coffee ☕️.

Bastion EC2 Instance

If you refer to our architectural diagram above, you will see our ECS and DB Instances will be running in private subnets. For us to be able to connect to the database and do some tasks for example, running database scripts will need to establish a secure connection to it via a bastion instance.

We need to create SSH Key that we will use to connect to Bastion. You can follow these steps to import the Key Pair. Note the var.bastion_key_name and its default value golang-mux-api-devops-bastion in variables.tf file. Update that according to the name you will give your key pair

Here is the content for our bastion instance:

Bastion EC2 Instance

This configuration creates a bastion instance and attaches a security group that allows inbound and outbound access e.g able to SSH from our local machine and also connect to our RDS Postgres Instance.

We also create an instance profile policy assume role and attach it to the instance profile. We need docker installed on our instance, so we do that using the user data.

Instance Profile Assume Policy Role

Bastion Instance User Data

Network Configuration

We create a VPC and some Networking resources we gonna use in our set-up. For Networking, we need to create 2 Public and Private Subnets within VPC. We also need Internet Gateway and Route Tables. We need to Create a NAT Gateway in our Public Subnets to allow our ECS Cluster in Private Subnet to pull docker images for ECR. We create these with this configuration:

Network Configuration

Database Setup

Here we create a Postgres RDS without Multi AZ. You can enable Multi AZ if you need it. We are only creating a simple one for the demo purpose. We are allowing access for our Bastion Instance to this RDS. We create the database with this configuration:

RDS Configuration

Elastic Container Service (ECS) Setup

Now we’re going to create the ECS Cluster, Service and Task Definition.
A service is a configuration that enables us to run and maintain a number of tasks simultaneously in a cluster. The containers are defined by a Task Definition that is used to run tasks in a service.
We need to create IAM Policy that will enable the service to pull images from ECR and also create CloudWatch logs.

ECS Assume Role

Task Execution Role

We are running two containers so our Task Definition has two containers — api and proxy.

Task Definition Template

ECS Configuration

Application Load Balancer

We need to create a Load Balancer to handle HTTP requests to our services. In our ECS configuration, there is a reference to a load_balancer. This configuration creates an Application Load Balancer.

DNS Setup

The next step is to set up DNS. We use a new AWS-issued SSL Certificate to provide HTTPS in our ALB to be put in front of our ECS services.

DNS Setup

⚠️ Note: This configuration uses a data source to fetch a DNS Zone hosted in Route53 created outside of this Terraform configuration. If you don’t have one already, you can freely change this to create new DNS resources.

Variables

You noticed we have been referencing several variables. We need to define them in a file variables.tf with this content:

ECS Variables

Define Outputs

Let us define some outputs for our infrastructure.

Outputs

⚠️ Note: This configuration uses images from Elastic Container Registry (ECR), ecr_image_api for the main API and ecr_image_proxy for Nginx Proxy. Building and pushing these images to ECR is not being covered here. You can fork these repos go-mux-api and golang-api-proxy, build and push your own images and update the default values accordingly. Please give a star to those repos if you find them helpful.

⚠️ Note: db_username and db_password do not have default values. We need to supply them when running commands for plan and apply. You can also create terraform.tfvars file and add these values there. Please do not push that file to Github. Add it to .gitignore.

Validate and Formation our Terraform Code

Let us format and validate our code.

docker-compose -f docker-compose.yaml run --rm terraform fmt

docker-compose -f docker-compose.yaml run --rm terraform validate

These commands should run without error if you followed this article step by step.

CI/CD Design

Now that we are done writing our Terraform code, it’s time we build our CI/CD workflow.

This is how the workflow will be Terraform:

1. Develop code against a feature branch and locally run terraform fmt and terraform validate to format and validate your changes.
2. Develop code against a feature branch and locally run terraform plan as you make changes.
3. When You are happy with the plan description create a PR against the main branch in GitHub.
4. GitHub Actions will generate a Terraform Plan and put it in the PR comments for review.
5. Once the code and the plan output are reviewed and accepted it is merged into the main branch.
6. GitHub Actions will run terraform apply using the approved plan.

It’s good to run step 1 locally since it’s also integrated into the workflow and you don’t want the pipeline to fail with simple formatting issues 😂.

Example of Terraform Plan output in Github PR

There is also an email being sent for the same:

Example of Terraform Plan output sent via email notification

Example of Terraform Apply output sent via email notification

I will create a separate article explaining this GitOps-workflow in full detail

Testing our Infrastructure after successful Terraform Apply

Making a call to the API Health Check Endpoint. You can get the endpoint from the CI Terraform Apply Outputs part.

Curl Request to API API Health Check Endpoint

Using Bastion Instance to Connect to RDS

Grab the Bastion Host Endpoint and DB Host from the outputs.
Run the following command from your Linux/macOS machine to create a tunnel for connectivity from your machine

ssh  ec2-user@[BASTION_HOST] -L 5433:[DB_HOST]:5432

Substitute BASTION_HOST and DB_HOST with your values.

In my case:

Create a tunnel for connectivity

Now that SSH tunnelling is in place, you can connect to your DB instance from your local Linux/macOS machine. Then createproducts table.

DB Connection

Let us create a product and get products via Postman

Create a Product

Get Products

This is all for now. I hope you have learnt something and enjoyed reading the article.

Here is the repo for this article. Follow me on GitHub for more about DevOps and DevSecOps.

⚠️ Caution: Destroy the environment after you are done with this tutorial if you don't want to keep the environment running to avoid incurring AWS costs.

docker-compose -f docker-compose.yaml run --rm terraform destroy -auto-approve

Thanks for reading. Let’s connect on Twitter and LinkedIn 😁.