Automating and Building your Infrastructure in the Cloud with Terraform

Introduction

In the last years, many concepts were introduced in IT area, and most of them will cover and build the path for the next big technologies of the century. For example, the cloud providers and the disruptive changes in migrating on premises and physical infrastructure to the huge cloud datacenters all around the world, bringing great advantages like:

• Reduced IT costs
• Scalability
• Business Continuity
• Collaboration Efficiency
• Data Security
• Disaster Recovery

Among many other advantages, one of them is the efficiency of provisioning infrastructure in a faster and automated way, which is the concept of infrastructure as code, that brings us to our main topic today, but before getting into practice, let’s delve into some conceptions.

The DevOps Wave

Infrastructure is just a branch of the notorious brand new trending topic, DevOps. Who went to IT environment to turn things effective, making developers focusing in their main activities, instead of losing time with another tasks.

Software and the Internet have transformed the world and its markets, ranging from commerce to entertainment to banking. Software no longer just supports a business activity, in fact it has become an integral component of every part of a business. Companies interact with their customers through software made available as services or applications online, and across all types of devices. They also use software to increase operational efficiency by transforming every part of the value chain, such as logistics, communication and operations. Similar to how these material goods companies transformed how they designed, created, and made available products using industrial automation during the 20th century, today’s companies must transform how they create and make software available.

Subjects like CI/CD, Infrastructure as Code, Configuration Management, Test Automation, Containerization and Orchestration, came to shorten the software development life cycle, improving the way the IT industry delivers solutions. Therefore, DevOps changed from a possibility, to a must-have to any company who wants to thrive in the market.

Infrastructure as Code

Our main topic today, which is one of the magic wands that came with the cloud providers, with the capability of generating instances in a question of seconds, and not only a single instance, but all the needed infrastructure like VPN, VPC, public and private subnets, internet gateways, route tables, NAT gateways, and many other components.

Thus, the cloud providers offers robust and consistent tools for provisioning our infrastructure, with all the capacity, scalability and security from those environments.

Summarizing, Infrastructure as code is like a bunch of files formatted in a general design, that describes how would be the characteristics of what you will provision in the cloud. After that, you just have to run some commands and have everything up and running in matter of seconds or minutes.

We have many tools available in the market today, using Amazon Web Services as our cloud provider, who has their own service called CloudFormation.

AWS CloudFormation - Infraestrutura como código e AWS Resource Provisioning

Cloudformation offers a lot of options for creating infrastructure in AWS, but it fails in the Configuration Management concept, although what the heck is Configuration Management?

Configuration Management

As a broader subject, configuration management (CM) refers to the process of systematically handling changes to a system in a way that it maintains integrity over time. Even though this process was not originated in the IT industry, the term is broadly used to refer to server configuration management.

By manipulating simple configuration files, a DevOps team can use application development best practices, such as version control, testing, small deployments, and design patterns. In short, this means code can be written to provision and manage an infrastructure as well as automate processes.

But what Configuration Management has to be with our infrastructure provisioning? Easily, everything, because we will not create configuration files to launch up our infrastructure in the cloud, without the intention of changing anything on that, or even versioning that.

Hence, there are tools that are more capable of handling Configuration Management, while others no. Today, the two best tools are Ansible and Terraform, and we will pick this last one, which is one of the wide used tool in the market today.

Terraform

Terraform is an open-source infrastructure as code software tool created by HashiCorp. It enables users to define and provision a datacenter infrastructure using a high-level configuration language known as Hashicorp Configuration Language, or optionally JSON.

Terraform by HashiCorp

Installing Terraform is very easy in Linux and Mac, although in Windows there are some issues, so we recommend using a unix system running in a Docker container or in Vagrant.

The installation tutorial can be found in the next provided link:

Installing Terraform

Infrastructure Description

As some considerations were given, and there is a lot of configurations to be done after launching our infrastructure, the first thing that has to be done, is analyzing how would be the infrastructure and network architecture, because there are some security constraints that need to be followed, and some parameters that have to be established after launching the EC2 instances.

As all the infrastructure is being provided by Amazon Web Services, which is not an on-premise infrastructure we don’t need to worry about scalability and availability, since that AWS can handle that for us, and we just need to alter some arguments to change sizes and other parameters.

Before using and explaining the details of Terraform, let’s delve into our infrastructure details and its components, because for launching an instance in the cloud we have to set up many components, let’s introduce some of them:

• VPC — Stands for Virtual Private Cloud, which will be your own private network inside the cloud, where you can define peering constraints and configurations to pair your network with another VPCs, VPNs or the public network.
• Subnets — When you create your VPC, you must specify a range of IPv4 addresses for the VPC in the form of a Classless Inter-Domain Routing (CIDR) block (example: 10.0.0.0/16). This is the primary CIDR block for your VPC. A subnet is a division of that VPC CIDR ranges. You can divide your VPC range of IPs allowed by the CIDR range in many as you can, as long that a subnet CIDR don’t pass out of the VPC range, and don’t go inside another subnet range.
• Subnets Routing — Each subnet must be associated with a Route table, which specifies that allowed routes for the outbound traffic leaving the subnet, and by default every subnet that you create is automatically associated with the VPC route table.
• Security Groups — Security groups are used to control inbound and outbound traffic for our instances, establishing some rules for those. But the security groups will operate at the instance level, while we have Network ACL (Access Control List) that operates at subnet level.

• Route tables — This one contains a set of rules, called routes, that are used to determine where network traffic is directed. A subnet can only be associated with one route table at a time, but you can associate multiple subnets with the same route table.
• Internet Gateway — This interface is responsible for allowing communication between instances in your VPC and the Internet, and provides a target in your VPC route tables for internet-routable traffic, and performs Network address translation for instances that have been assigned public IPv4 addresses.
• NAT Gateway — The NAT Gateway enable instances in a private subnet to connect to the internet or other AWS Services, but prevent the internet from initiating connections with the instances. The NAT Gateway will reside in a public subnet, and also an elastic IP should be associated.
• Elastic IP — An Elastic IP address is a static IPv4 address designed for dynamic cloud computing. An Elastic IP address is associated with your AWS account. With an Elastic IP address, you can mask the failure of an instance or software by rapidly remapping the address to another instance in your account.

So, as we have defined all those components for our infrastructure, summarizing everything together will be:

We can see in that tutorial, that we are creating and defining our network architecture, but we can use Terraform to launch up only a single instance, and AWS will take care of assign default VPCs, subnets, security groups to the created resources.

The Example

Then, now that we have already defined all our infrastructure components and architecture, we can start with our Terraform example. This sample will relay in the repository below:

felipemeriga/Terrraform-Example

Configuration Management with Terraform

For initiating, as we have talked about Configuration Management and the capability of handling in an effective way multiple changes. If we take a look in the repository root directory, we will see a file called variables.tf .This file is responsible for defining all the variables that will be used to build up the characteristics of our infrastructure. Therefore, instead of putting everything hardcoded inside the configuration files, we just have to change the variables of these files, and reference their values in the configuration files.

Let’s take a look in the file:

# Variables
	variable "access_key" {
	  default = ""
	}
	variable "secret_key" {
	  default = ""
	}
	variable "user_identification" {
	  default = "TerraformExampleUser"
	}
	variable "region" {
	  default = "us-east-1"
	}
	variable "cidr_vpc" {
	  description = "CIDR block for the VPC"
	  default = "10.0.0.0/16"
	}
	variable "cidr_subnet" {
	  description = "CIDR block for the subnet"
	  default = "10.0.0.0/24"
	}
	variable "cidr_subnet_private" {
	  description = "CIDR block for the private subnet"
	  default = "10.0.1.0/24"
	}
	variable "availability_zone" {
	  description = "availability zone to create subnet"
	  default = "us-east-1a"
	}
	variable "key_pair_name" {
	  description = "The name of the current key pair"
	  default = "mykey"
	}
	variable "public_key_path" {
	  description = "Public key path"
	  default = "mykey.pub"
	}
	variable "instance_ami" {
	  description = "AMI for aws EC2 instance"
	  default = "ami-6869aa05"
	}
	variable "instance_type" {
	  description = "type for aws EC2 instance"
	  default = "t2.micro"
	}
	variable "nat_instance_name" {
	  description = "The name of the Nat instance"
	  default = "NatInstance"
	}
	variable "private_instance" {
	  description = "The name of the private instance"
	  default = "Private Instance"
	}
	variable "environment_tag" {
	  description = "Environment tag"
	  default = "Development"
	}

If we take a look, we see that using variables in Infrastructure as Code tools is a very good practice, although as this is a sample example, we are defining simple variables, but Terraform let you create arrays, maps, conditionals and many other kind of variables.

Key Pairs

The key pair in AWS is the SSH keys for connecting to the EC2 instances, this enhances the system security, because even that if you have a security group with the SSH port 22 open for inbound connections, you will only connect the instance with the key.

There is a file called key_pair.tf, where the key name and the path of the public key is inserted. If we take a look in the file, we will see that it is using the values of two defined variables.

#resources
	resource "aws_key_pair" "ec2key" {
	  key_name = "${var.key_pair_name}"
	  public_key = "${file(var.public_key_path)}"
	}

For default, the variable will look for a key called mykey, and for sake of simplicity we will leave with that name, but if you want to change that, just remember to replace the variable name. Then, if you are on a unix shell system you can use the command below to generate the private and public key.

ssh-keygen -f mykey

If you are using Windows, you can use the git bash for creating the key, or you can even use the PuTTYgen software to generate them.

Finally, that we have already generated the key, let’s go to our base VPC.

VPC and Internet Gateway

The VPC is the base of our network architecture, let’s take a look in the vpc.tf file, where we configure the VPC and the Internet Gateway.

#resources
	resource "aws_vpc" "vpc" {
	  cidr_block = "${var.cidr_vpc}"
	  enable_dns_support = true
	  enable_dns_hostnames = true
	  tags {
	    Name = "VPC-${var.user_identification}"
	    Environment = "${var.environment_tag}"
	  }
	}
	
	resource "aws_internet_gateway" "igw" {
	  vpc_id = "${aws_vpc.vpc.id}"
	  tags {
	    Name = "Internet-Gateway-${var.user_identification}"
	    Environment = "${var.environment_tag}"
	  }
	
    }

For the VPC, we just have to pass the CIDR block for defining the allowed range of IPs that will have access to that resource, and enable the DNS support that will be used for the public instance.

After, we will create the Internet Gateway, pointing the VPC id that we just have created. Also, look that the two resources have tagging, which is a good way of identifying resources in AWS. Because, imagine in a collaborative environment, where you have a lot of developers. Tags for identifying the who created, and which kind of environment is this resource are a good practice to arrange, and don’t let everything into a total mess.

Public Network Resources

The public.tf file, will contain the public components of our infrastructure:

#resources
	resource "aws_subnet" "subnet_public" {
	  vpc_id = "${aws_vpc.vpc.id}"
	  cidr_block = "${var.cidr_subnet}"
	  map_public_ip_on_launch = "true"
	  availability_zone = "${var.availability_zone}"
	  tags {
	    Name = "Public-Subnet-${var.user_identification}"
	    Environment = "${var.environment_tag}"
	  }
	}
	
	resource "aws_route_table" "rtb_public" {
	  vpc_id = "${aws_vpc.vpc.id}"
	
	  route {
	    cidr_block = "0.0.0.0/0"
	    gateway_id = "${aws_internet_gateway.igw.id}"
	  }
	
	  tags {
	    Name = "Public-Route-Table-${var.user_identification}"
	    Environment = "${var.environment_tag}"
	  }
	}
	
	resource "aws_route_table_association" "rta_subnet_public" {
	  subnet_id = "${aws_subnet.subnet_public.id}"
	  route_table_id = "${aws_route_table.rtb_public.id}"
	}

In this section, we can see that we are creating a public subnet, passing the CIDR block, allowing it to have a public IP and telling an availability zone for this resource. Besides, a route table, and its association, for linking our public subnet with the Internet Gateway, thus the public network.

Private Network Resources

As we have created our public network architecture, let’s go to the private one:

#resources
	resource "aws_subnet" "private" {
	  vpc_id = "${aws_vpc.vpc.id}"
	
	  cidr_block = "${var.cidr_subnet_private}"
	
	  tags {
	    Name = "Private-Subnet-${var.user_identification}"
	    Environment = "${var.environment_tag}"
	  }
	}
	
	resource "aws_route_table" "private" {
	  vpc_id = "${aws_vpc.vpc.id}"
	
	  route {
	    cidr_block = "0.0.0.0/0"
	    nat_gateway_id = "${aws_nat_gateway.gw.id}"
	  }
	
	  tags {
	    Name = "Private-Route-Table-${var.user_identification}"
	    Environment = "${var.environment_tag}"
	  }
	}
	
	resource "aws_route_table_association" "private" {
	  subnet_id = "${aws_subnet.private.id}"
	  route_table_id = "${aws_route_table.private.id}" 
	}

Like the public one, we are specifying a CIDR block for this private subnet. Remember that the CIDR of each subnet can not overtake the CIDR of another one, while the VPC subnet has to cover all the range of IPs from the subnet’s CIDRs. For this example, we are using the overall CIDR 10.0.0.0/16 for the VPC. Wether you want to change that, you can use the link below, that helps you to divide your CIDR in multiple ones.

Visual Subnet Calculator

Moreover, there is also a route and an association for the private subnet, but this time we are routing from the private subnet to the NAT Gateway, that we will create in the next topic. Don’t forget, that the NAT Gateway is used for letting our instances in a private area to make requests to the public internet, but don’t let the public internet to make inbound connections with the private area.

NAT Gateway

Already introduced, the file nat.tf will contain the related configuration:

#resource
	resource "aws_nat_gateway" "gw" {
	  allocation_id = "${aws_eip.nat.id}"
	  subnet_id = "${aws_subnet.subnet_public.id}"
	  depends_on = [
	    "aws_internet_gateway.igw"]
	
	  tags {
	    Name = "NAT-Gateway-${var.user_identification}"
	    Environment = "${var.environment_tag}"
	  }
	}
	
	resource "aws_eip" "nat" {
	  vpc = true
	
	  tags {
	    Name = "Elastic-IP-${var.user_identification}"
	    Environment = "${var.environment_tag}"
	  }
	
    }

First of all we are defining an Elastic IP, which has a value associated with your AWS account, which will be attached to our NAT Gateway.

Pay attention that we are passing the public subnet id for the NAT Gateway, because is this entity that make the gateway between the public internet, and the private instances.

Security Groups

The security groups will be our guardians and fortress, defining the inbound and outbound rules, we can restrict access in many protocols, passing also CIDR IP blocks to narrow the range of IPs allowed to access the services. Let’s take a look on the security_groups.tf file:

#resources
	resource "aws_security_group" "db" {
	  name = "vpc_db"
	  description = "Allow incoming database connections."
	
	  ingress {
	    from_port = 22
	    to_port = 22
	    protocol = "tcp"
	    cidr_blocks = [
	      "${var.cidr_vpc}"]
	  }
	  ingress {
	    from_port = -1
	    to_port = -1
	    protocol = "icmp"
	    cidr_blocks = [
	      "${var.cidr_vpc}"]
	  }
	
	  egress {
	    from_port = 80
	    to_port = 80
	    protocol = "tcp"
	    cidr_blocks = [
	      "0.0.0.0/0"]
	  }
	  egress {
	    from_port = 443
	    to_port = 443
	    protocol = "tcp"
	    cidr_blocks = [
	      "0.0.0.0/0"]
	  }
	
	  vpc_id = "${aws_vpc.vpc.id}"
	
	  tags {
	    Name = "Private-SG-${var.user_identification}"
	  }
	}
	
	/*
	  NAT Instance Security Group
	*/
	resource "aws_security_group" "nat" {
	  name = "vpc_nat"
	  description = "Allow traffic to pass from the private subnet to the internet"
	
	  ingress {
	    from_port = 80
	    to_port = 80
	    protocol = "tcp"
	    cidr_blocks = [
	      "${var.cidr_subnet_private}"]
	  }
	  ingress {
	    from_port = 443
	    to_port = 443
	    protocol = "tcp"
	    cidr_blocks = [
	      "${var.cidr_subnet_private}"]
	  }
	  ingress {
	    from_port = 22
	    to_port = 22
	    protocol = "tcp"
	    cidr_blocks = [
	      "0.0.0.0/0"]
	  }
	  ingress {
	    from_port = -1
	    to_port = -1
	    protocol = "icmp"
	    cidr_blocks = [
	      "0.0.0.0/0"]
	  }
	
	  egress {
	    from_port = 80
	    to_port = 80
	    protocol = "tcp"
	    cidr_blocks = [
	      "0.0.0.0/0"]
	  }
	  egress {
	    from_port = 443
	    to_port = 443
	    protocol = "tcp"
	    cidr_blocks = [
	      "0.0.0.0/0"]
	  }
	  egress {
	    from_port = 22
	    to_port = 22
	    protocol = "tcp"
	    cidr_blocks = [
	      "${var.cidr_vpc}"]
	  }
	  egress {
	    from_port = -1
	    to_port = -1
	    protocol = "icmp"
	    cidr_blocks = [
	      "0.0.0.0/0"]
	  }
	
	  vpc_id = "${aws_vpc.vpc.id}"
	
	  tags {
	    Name = "NAT-SG-${var.user_identification}"
  
	}

We can see in this big configuration file, that we define ingress and egress rules. For SSH and ICMP in the private security group, note the usage of the VPC CIDR block. This means that only IPs from inside our VPC can ping or SSH our resources in the private area.

Instead, for the public security group we are putting the 0.0.0.0/0 CIDR block, which means that anyone is allowed to SSH and ping the instance from the public network.

However, if we want to allow only our computer to have access to the public security group? You just have to search for your public IP, and use the slash 32 for defining the CIDR block, which restricts for only one IP address. For example, if my public IP is 201.53.195.140, we will put 201.53.195.140/32 CIDR block in the security group.

Public Instance

We have finally defined all our network architecture, now it’s time to build up our instances. Firstly, let’s create an instance and put that on the public subnet. This EC2 instance will be like a gateway for our private network, in other words, for SSH into our private instance, we will have to SSH first in the public one.

#providers
	provider "aws" {
	  access_key = "${var.access_key}"
	  secret_key = "${var.secret_key}"
	  region = "${var.region}"
	}
	
	#resources
	resource "aws_instance" "nat" {
	  ami = "${var.instance_ami}"
	  # this is a special ami preconfigured to do NAT
	  instance_type = "m1.small"
	  key_name = "${aws_key_pair.ec2key.key_name}"
	  vpc_security_group_ids = [
	    "${aws_security_group.nat.id}"]
	  subnet_id = "${aws_subnet.subnet_public.id}"
	  associate_public_ip_address = true
	  source_dest_check = false
	  tags {
	    Name = "${var.nat_instance_name}"
	    Environment = "${var.environment_tag}"
	  }
	
    }

Check that we are passing an AMI (Amazon Machine Images), key name that we have created, public security group, public subnet and the associate_public_ip_address flag, to attach a public IP that will let this instance to be reachable from the public network.

Private Instance

Definitely, reaching our main point of our infrastructure, let’s create our private instance, where we can run a database or an internal application.

#resources
	resource "aws_instance" "testInstance" {
	  ami = "${var.instance_ami}"
	  instance_type = "${var.instance_type}"
	  subnet_id = "${aws_subnet.private.id}"
	  security_groups = [
	    "${aws_security_group.db.id}"]
	  source_dest_check = false
	  key_name = "${aws_key_pair.ec2key.key_name}"
	  tags {
	    Environment = "${var.environment_tag}"
	    Name = "${var.private_instance}"
	  }
	
    }

In the private_instance.tf file, simply all the configurations made in the public one, will be done again, but now we are referencing the private security group and the private subnet.

Take a look at the source_dest_check attribute that were used two times, which controls if traffic is routed to the instance when the destination address does not match the instance. Used for NAT or VPNs. Defaults true.

Running and Using Terraform

Lastly, it’s time for running our Infrastructure as Code using Terraform. First of all, Terraform uses plugins for building, so we have to run the following command that will download the plugins needed.

terraform init

If everything runs well, you will see the following screen:

Now you just have to put your AWS access key and private key in the variables.tf file. If you don’t know what is those keys, you can check the following tutorial that will help you in creating them.

Understanding and Getting Your Security Credentials

With the credentials inserted in to the variables file, you just have to run the next command, for building your infrastructure.

terraform init

This piece of command, will show you everything that will be created, and will ask you to confirm. If you want to override some variable value, you can run the command:

terraform apply -var="key_pair_name=awskeypair"  
-var="public_key_path=awskeypair.pub"

You can do this with many variables you want.

Testing

After everything is created, we can go to AWS EC2 console, and grab the public DNS from our public instance. First of all let’s SSH into that machine, with the following command:

ssh -i "mykey" ec2-user@<your-public-instance-dns>

After SSH into the public instance, you can use the scp command to copy the private key inside the public instance, to SSH into the private one. Then, exit the SSH session, and run on your machine the command:

scp -i "mykey" mykey ec2-user@<your-public-instance-dns>:/home/ec2-user

After copping the private key to the public EC2 instance, you may now go to the directory /home/ec2-user in your public instance, and SSH your private instance with the command:

ssh -i "mykey.pem" ec2-user@<private-instance-ip>

If everything goes well, it will be possible to SSH the private instance, and your infrastructure is ready to be used.

In the case you want to destroy everything, just run the command:

terraform destroy

Hope this article helped you in this topic, and if you have any consideration or things to add, just ping me!