Amazon Virtual Private Clouds with Terraform: An Introduction & Tutorial

Run your AWS VPCs with Terraform. Live to tell about it.

Terraform is an open-source tool that allows you to build and maintain infrastructure in a declarative codebase. You design the infrastructure in a descriptive configuration language called HCL, and Terraform does all the grunt work: building, deploying, versioning, and managing state.

In Part 1 of this tutorial, we’ll tackle an issue every software team faces: maintaining two or more matching environments — one for production, and any number of others for things like staging, testing, or development. Using Terraform and AWS services, we’ll build and deploy two virtual private clouds (VPCs), each one corresponding to an environment and containing these standard network resources:

• Public subnet
• Private subnet
• Internet gateway (IG)
• Network address translation (NAT) gateway
• Network access control list (ACL)

If you want to jump ahead and see the final product from this tutorial, it’s available on Github. Part 2 of this tutorial (forthcoming) will address how to deploy OpenVPN access server into your VPC so you can connect to it securely.

Why Terraform?

At Stedi, we chose Terraform because we need several separate environments that mimic our production system. Often, a production environment suffers from configuration drift — a loathsome phenomenon in which undocumented or ad hoc changes lead, over time, to a unique configuration that’s tough to replicate. Terraform prevents configuration drift by forcing your team to define and provision your infrastructure in code, rather than clicking around in the AWS console. By codifying your infrastructure in this way, you’re able to track changes, document the current state of your environmental variables and configuration settings, and keep your environments in sync.

Prerequisites

Before jumping into our project, we need to install Terraform. We will also create an AWS user specifically for Terraform.

1. Install Terraform

Terraform has some pretty straightforward installation instructions, but here’s the TL;DR:

• Mac + Homebrew: Run brew install terraform.
• Mac/Linux: Download and unzip the correct package for your operating system and move the binary somewhere that’s on your PATH, e.g. /usr/local/bin.
• Windows: Download the correct package and set the binary on your PATH.

Confirm that the binary is accessible on the PATH by entering terraform -v at the terminal, which should output a version, e.g:

$ terraform -v
Terraform v0.11.1

2. Create an AWS user

Login to AWS, navigate to the ‘Users’ section of Identity Access and Management (IAM), and click the blue ‘Add user’ button. Enter terraform_user as the ‘User name’, select ‘Programmatic access’ as the ‘Access Type’, and click the blue ‘Next’ button. On the next screen, click ‘Create group’, select ‘AdministratorAccess’, and click the blue ‘Create group’ button. You will then be shown your new user’s ‘Access key ID’ and ‘Secret access key’. This is the only time you’ll be able to view your secret access key, so copy these two values temporarily, as we’ll need them in the next step.

Create or open ~/.aws/credentials (Linux/Mac) and add the following:

[terraform_user]
aws_access_key_id = XXXXXXXXXXXXXXXXXXXX
aws_secret_access_key = XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

Getting started

Let’s begin by launching a new AWS VPC. Create an empty directory that will contain your Terraform project, cd into it, and create the following file, named main.tf.

// main.tf
// Specify AWS as the provider and reference your credentials
provider "aws" {
  profile = "terraform_user"
  region = "us-east-1"
}
// Create a VPC
resource "aws_vpc" "vpc" {
  cidr_block = "10.0.0.0/16"
  enable_dns_support = true
  enable_dns_hostnames = true
  tags {
    Name = "my-first-vpc"
  }
}

At the terminal, navigate to the root of your project, and enter terraform init. Terraform will discover that we’re using AWS as our provider and download the AWS-specific plugin. Next, enter terraform plan, which will read the file we just created and then generate and display an execution plan. Note that Terraform wants to create one new resource: our VPC. To execute this plan, enter terraform apply and confirm yes when prompted. Once Terraform has completed, log into AWS, navigate to your VPC dashboard, and select ‘Your VPCs’ from the left rail menu. You should see your shiny new VPC listed under the tag ‘my-first-vpc’. Congratulations — you’ve just deployed your first resource with Terraform!

Reusing resources with modules

So far so good, but it’d be nice if we could use the same block of code to create our development and production VPCs, since only one line of code (the tag) changes between them. Fortunately, Terraform provides us with a way to do this: modules.

Let’s create a VPC module by describing an abstracted VPC resource that we can use to spin up individual VPCs for different environments. At the root of your Terraform project, create a new directory, modules, and within it, create another directory, vpc. Then create the following file, vpc.tf, inside modules/vpc:

//  modules/vpc/vpc.tf
variable "name" { }
variable "cidr" { default = "10.0.0.0/16" }
// Create a VPC with a configurable name (and cidr)
resource "aws_vpc" "vpc" {
  cidr_block = "${var.cidr}"
  enable_dns_support = true
  enable_dns_hostnames = true
  tags {
    Name = "${var.name}"
  }
}

The resource above accepts two parameters: name, which corresponds to an AWS tag, and cidr, which defaults to "10.0.0.1/16".

Update main.tf to use the VPC module and pass in a name argument:

// main.tf
// Specify AWS as the provider and reference credentials
provider "aws" {
  profile = "terraform_user"
  region = "us-east-1"
}
// Create a VPC tagged in AWS with "my-second-vpc"
module "vpc" {
  source = "./modules/vpc"
  name = "my-second-vpc"
}

From the root of your project, enter terraform get at the terminal to register the module we created, and then run terraform plan. Terraform should provide you with a plan that creates a VPC and destroys the one we previously created.

Now, we don’t want to just leave an unused VPC laying around, so let’s clean up. At the terminal, enter terraform destroy and confirm yes when prompted. Finally, delete main.tf before moving to the next section.

Creating the development VPC

We’re going to build our development environment by following the same module pattern we used above to create our first VPC. We’ll be adding modules for a public subnet, private subnet, and a NAT gateway. Here’s what our directory structure will look like by the end of this tutorial — go ahead and create these directories now:

- modules/                     // define shared components
-------- nat/                  
-------- private-subnet/
-------- public-subnet/
-------- vpc/
-------- environment/
- us-east-1-dev/               // our development environment              
- us-east-1-prod/              // our production environment

We are representing each environment as a directory, e.g. us-east-1-dev, and storing the instructions for creating resources in the modules directory, outside of any particular environment.

Public subnet
Let’s begin with the public subnet. Create a new file within us-east-1-dev called main.tf:

//  us-east-1-dev/main.tf
// Specify AWS as the provider and reference credentials
provider "aws" {
  region = "us-east-1"
  profile = "terraform_user"
}
// Create a VPC
module "vpc" {
  source = "../modules/vpc"
  name = "dev-vpc"
}
// Create a public subnet referencing the vpc
module "public_subnet" {
  source = "../modules/public-subnet"
  name = "public-subnet"
  vpc_id = "${module.vpc.vpc_id}"
  cidrs = "10.0.1.0/24,10.0.2.0/24"
  azs = "us-east-1a,us-east-1b"     // availability zones
}

With our public_subnet module, we’ve introduced two loose ends that we need to tie up. First, source = "../modules/public-subnet" references a file that doesn’t yet exist, so we’ll have to create that. Second, notice the line vpc_id = "${module.vpc.vpc_id}". We can’t create a subnet without giving it a VPC ID, but since we haven’t yet created a VPC , we’re dynamically referring to the ID of the VPC that we’ll create alongside the subnet at runtime. We need to add this output to the VPC resource.

Let’s tie up the first loose end by creating a public subnet module. Within modules/public-subnet, create a file named public-subnet.tf:

//   modules/public-subnet/public-subnet.tf
variable "name"   { default = "public" }
variable "vpc_id" { }
variable "cidrs"  { }
variable "azs"    { }
// Create the public subnet
resource "aws_subnet" "public" {
  vpc_id = "${var.vpc_id}"
  cidr_block = "${element(split(",", var.cidrs), count.index)}"
  availability_zone = "${element(split(",", var.azs), count.index)}"
  count = "${length(split(",", var.cidrs))}"
  tags {
    Name = "${var.name}.${element(split(",", var.azs),count.index)}"
  }
}
// Create an internet gateway
resource "aws_internet_gateway" "public" {
  vpc_id = "${var.vpc_id}"
  tags {
    Name = "${var.name}-subnet"
  }
}
// Create a public route table
// Allow outbound traffic to the internet gateway
resource "aws_route_table" "public" {
  vpc_id = "${var.vpc_id}"
  route {
      cidr_block = "0.0.0.0/0"
      gateway_id = "${aws_internet_gateway.public.id}"
  }
  tags {
    Name = "${var.name}.${element(split(",", var.azs),count.index)}"
  }
}
// Associate the subnet with the public route table
resource "aws_route_table_association" "public" {
  count          = "${length(split(",", var.cidrs))}"
  subnet_id      = "${element(aws_subnet.public.*.id, count.index)}"
  route_table_id = "${aws_route_table.public.id}"
}
// Export the subnet IDs so we can reference them from elsewhere
output "subnet_ids" {
  value = "${join(",", aws_subnet.public.*.id)}"
}

The public subnet above accepts four parameters: name (which defaults to ‘public’), vpc_id, cidrs, and azs (availability zones). The subnet is situated within our VPC and is public because it’s associated with a public route table. That route table contains a rule, i.e. route, allowing outbound traffic to the internet gateway.

Time to tie up the second loose end by adding a vpc_id output to the VPC resource. Append the following tovpc.tf:

...
output "vpc_id" {
  value = "${aws_vpc.vpc.id}"
}

Now, cd into us-east-1-dev and run terraform init to download the AWS provider plugin and initialize your modules. Then run terraform plan. You should see a plan to create 7 new resources.

Private Subnet and NAT Gateway
Conceptually, the private subnet sits within the VPC alongside the public subnet. It’s private because it isn’t associated with a public route table. Outgoing traffic, i.e. traffic bound for the internet or other AWS services, is pointed by a private route table to a NAT instance, which translates private IP addresses to public ones. The NAT instance must be located on the public subnet, and it needs an elastic IP address.

Create a private subnet and NAT gateway by adding the following to us-east-1-dev/main.tf:

...
module "private_subnet" {
  source = "../modules/private-subnet"
  name = "private-subnet"
  vpc_id = "${module.vpc.vpc_id}"
  cidrs = "10.0.11.0/24,10.0.12.0/24"
  azs = "us-east-1a,us-east-1b"
  nat_gateway_ids = "${module.nat.nat_gateway_ids}"
}
module "nat" {
  source = "../modules/nat"
  name = "nat"
  public_subnet_ids = "${module.public_subnet.subnet_ids}"
}

You might have noticed that both subnets reference the same availability zones. Let’s DRY this up. At the very top of the file, declare an availability zone variable and assign it a default string value:

// us-east-1-dev/main.tf
variable "azs" { default = “us-east-1a,us-east-1b” }
...

Now, in both subnets, pass the azs variable instead of a hard-coded string:

azs = "${var.azs}"

Much better! However, we’ve introduced two more loose ends by adding those two modules. First, we need to create a private subnet and a NAT resource. Second, we need to generate outputs from both of these resources so that we can reference nat_gateway_ids and public_subnet_ids from other modules in the same way we reference vpc_id.

To create the private subnet, add a file within modules/private-subnet called private-subnet.tf:

//   modules/private-subnet/private-subnet.tf
variable "name" { default = "private"}
variable "vpc_id" { }
variable "cidrs" { }
variable "azs" { }
variable "nat_gateway_ids" { }
resource "aws_subnet" "private" {
  vpc_id = "${var.vpc_id}"
  cidr_block = "${element(split(",", var.cidrs), count.index)}"
  availability_zone = "${element(split(",", var.azs), count.index)}"
  count = "${length(split(",", var.cidrs))}"
  lifecycle { create_before_destroy = true }
  tags {
    Name = "${var.name}.${element(split(",", var.azs),count.index)}"
  }
}
resource "aws_route_table" "private" {
  vpc_id = "${var.vpc_id}"
  count  = "${length(split(",", var.cidrs))}"
  lifecycle { create_before_destroy = true }
  route {
    cidr_block = "0.0.0.0/0"
    nat_gateway_id = "${element(split(",", var.nat_gateway_ids), count.index)}"
  }
  tags {
    Name = "${var.name}.${element(split(",", var.azs),count.index)}"
  }
}
resource "aws_route_table_association" "private" {
  count = "${length(split(",", var.cidrs))}"
  subnet_id = "${element(aws_subnet.private.*.id, count.index)}"
  lifecycle { create_before_destroy = true }
  route_table_id = "${element(aws_route_table.private.*.id, count.index)}"
}
output "subnet_ids" {
  value = "${join(",", aws_subnet.private.*.id)}"
}

Then, create the NAT by adding a file within modules/nat named nat.tf:

//   modules/nat/nat.tf
variable "name" { default = "nat" }
variable "public_subnet_ids" { }

resource "aws_eip" "nat" {
  vpc   = true
  lifecycle { create_before_destroy = true }
}

resource "aws_nat_gateway" "nat" {
  allocation_id = "${element(aws_eip.nat.*.id, count.index)}"
  subnet_id = "${element(split(",", var.public_subnet_ids), count.index)}"
  tags {
    Name = "${var.name}"
  }

  lifecycle { create_before_destroy = true }
}

output "nat_gateway_ids" {
  value = "${join(",", aws_nat_gateway.nat.*.id)}"
}

Finally, let’s create a network ACL. This is an added layer of security that we can use to regulate traffic flowing into and out of our subnets. We’re creating a basic ACL that will allow all inbound and outbound traffic, but in your application you might add additional ingress/egress rules to limit traffic to certain ports, protocols, etc. Add the following to the end of us-east-1-dev/main.tf:

...
resource "aws_network_acl" "nacl" {
  vpc_id = "${module.vpc.vpc_id}"
  subnet_ids = [
    "${split(",", module.private_subnet.subnet_ids)}",
    "${split(",", module.public_subnet.subnet_ids)}"
  ]
  egress {
    protocol = "-1"
    rule_no = 100
    action = "allow"
    cidr_block = "0.0.0.0/0"
    from_port = 0
    to_port = 0
  }
  ingress {
    protocol = "-1"
    rule_no = 100
    action = "allow"
    cidr_block = "0.0.0.0/0"
    from_port = 0
    to_port = 0
  }
  tags {
    Name = "network-acl"
  }
}

From us-east-1-dev, run terraform init, then terraform plan. Terraform will propose a plan to add 16 resources. There you have it: you’ve scaffolded a development environment! Your production environment is essentially going to look the same, but with a different VPC name. And that brings us to our last module: the environment module.

The Environment: A module of modules
From a structural perspective, the only real difference in our environments is the name. Let’s make it reusable by moving the infrastructure we described in us-east-1-dev into a module that accepts an environment_name parameter.

Move us-east-1-dev/main.tf to modules/environment/environment.tf. This ‘environment’ module is what we’ve been working up to — it’s essentially a module that aggregates our other modules. We only need to make a couple of changes. At the very top of the file, create a variable which will store the name we assign to an environment, and then pass that variable into the VPC module:

// Create a variable to store an environment's name
variable "environment_name" {}
...
module "vpc" {
  source = "../modules/vpc"
  name = "${var.environment_name}-vpc"
}

Next, replace every instance of source = "../modules/<...> with source = "../<..> since we’ve moved our file to within the modules directory. For example, the VPC module will now look like this:

module "vpc" {
  source = "../vpc"
  name = "${var.environment_name}-vpc"
}

Fantastic work. Now change back into us-east-1-dev and create the following file, main.tf:

// us-east-1-dev/main.tf
module "environment" {
 source = "../modules/environment"
 environment_name = "dev"
}

I’d call that concise! Now, still in this directory, run terraform init; terraform plan and marvel at the 16 resources you’ve conjured with your newfound Terrafoo. When you’re ready to spin up a configuration drift-resistant production environment, simply copy this file into us-east-1-prod and change the environment_name.

If you’ve deployed resources to AWS that you don’t plan to keep, remember to destroy them before reading on.

What’s Next?

At this point, you should be feeling pretty good about your devops chops — you now know how to spin up and tear down matching production and development environments from the command line. In Part 2 of this tutorial, we’ll learn how to access your VPC using OpenVPN Access Server.