Automating the Cloud: Terraform and Resource Provisioning for the Modern Developer

Quick guide using Terraform to deploy AWS EC2 instance

What is Terraform? Terraform is defined as an open-source infrastructure-as-code (IAC) tool developed my HashiCorp that allows users to provision infrastructure resources such as virtual machines, network interfaces, and storage across multiple cloud providers and on-premises data centers. One of the primary benefits is that it’s “cloud agnostic.” It utilizes a provider plugin model to interface with various cloud providers (AWS, Google, MS Azure, etc..), enabling users to create and manage infrastructure resources across multiple providers, consistently and repeatable.

Terraform is great for IAC, enabling users to define their infrastructure in the desired state, described in code that can be version-controlled, reviewed, and shared. It offers multi-cloud support and enables users to automate the creation and management of infrastructure resources, reducing risk of human error and ensuring consistency. Overall, Terraform is a tool that can simplify complex infrastructure deployments and enables users to make changes more efficiently.

Jenkins is an open-source automation server, used for building, testing, and deploying software projects. It is one of the most popular Continuous Integration (CI) and Continuous Deployment (CD) tools available today. We will be bootstrapping Jenkins on our EC2 Instance build.

Purpose:
This will be a quick guide using Terraform to deploy an AWS EC2 instance. As a business use case, your team would like to start using Jenkins as their CI/CD tool to create pipelines for DevOps projects. They need you to create the Jenkins server using Terraform so that it can be used in other environments and so that changes to the environment are better tracked. Prerequisites and Tasks below.

Prerequisites:

• AWS account
• AWS Cloud9 IDE
• Basic Terraform knowledge
• Basic Knowledge of Linux/CLI

Resources:

• AWS Cloud9 IDE Setup
• HashiCorp: Terraform Documentation
• Terraform Registry
• AWS IAM User Access Keys
• Jenkins on AWS

Tasks:

• Deploy 1 EC2 Instance in your default VPC
• Bootstrap the EC2 Instance with a script that will install and start Jenkins
• Create and assign a security group to the Jenkins EC2 that allows traffic on port 22 from your ip and allows traffic from port 8080.
• Create a S3 Bucket for your Jenkins Artifacts that is not open to the public
• Verify that you can reach your Jenkins Artifacts via port 8080 in your browser
• Push code to GitHub

So let’s get started!!!

AWS Cloud9 Integrated Development Environment (IDE)
First, we need to setup our AWS Cloud9 environment. Check out the resources section for documentation. Log into your AWS console, type “Cloud9” in the search menu. Note: also, I will be in the US-West-2 Region (Oregon). If your in the East Coast, I recommend using US-East-1 (N-Virginia) as it’s the most resilient region. You want to ensure you have enough availability zones for this project.

Once your in Cloud9 menu, hit the “Create environment” button.

Next, just fill in the details below. Create a name, description, select your environment and Instance Type. I will be using the t2.micro (Free Tier) on the Amazon Linux 2 platform. For network settings, use your default VPC. If you don’t have a VPC, see documentation here: Default VPC. Click “Create.”

You should have a Green success banner with newly created environment. I already had one setup, but just hit the “Open in Cloud” button to launch your IDE.

Could take few minutes!

You can verify IDE is running by clicking on the Instances tab.

Success!

Head back into your IDE! It will look like this below!

Deploy 1 EC2 Instance in default VPC:
Before we deploy our EC2, we need to create our main Terraform file or (main.tf) file. We are using a single configuration file and the standard name will be “main.tf.” You can name yours whatever you want, but the “.tf” is the required extension in Terraform. The beauty of AWS Cloud9, it comes preconfigured with Terraform. Type in following command to see what version.

terraform version

Now that our environment is setup, we can proceed with the tasks. First lets creating some working directories to keep our “.tf” files organized. Using the CLI, type in “mkdir Terraform.”

mkdir <terraform>

I already had the working directory created, but I’ll type in “ls” to show files in my current “environment” directory.

Within our environment, lets create a few working directories and create a main.tf file.

mkdir Terraform
mkdir luitw20 <create your own directory name>
touch main.tf

Now that our working directory is setup, we can use Terraform to deploy an EC2 instance. Make sure your using your default VPC. Refer to Terraform documentation in the resources section. HashiCorp has some really good examples of how to build configuration files. In the documentation, you an search for “AWS Provider.”

AWS Providers

First step in configuration build is selecting your provider. You can just copy and paste the code. AWS is my provider and I’m using the us-west-2 region (Oregon).

#Configure the AWS Provider
provider "aws" {
  version = "~> 4.0"
  region  = "us-west-2"
}

Next, we need to create our EC2 instance, but defining our resource values. These values consists of AWS Instance name, AMI, Instance type, and we will provide a Tag for naming. Very important for identifying once created. For AWS Instance name I have “jenkinsinstance1.” AMI is very important. Head over to EC2 in a new tab, and select Instances, then hit the “Launch EC2” button. You will see details of t2.micro. Copy and paste the AMI for t2.micro into your configuration build. For Tags, just give a name for identifying purposes.

t2.micro Amazon Linux 2 free tier

Below is a sample of the code.

#Create EC2 Instance
resource "aws_instance" "jenkinsinstance1" {
  ami           = "ami-0efa651876de2a5ce"
  instance_type = "t2.micro"

  tags = {
    Name = "Jenkins_instance"
  }

Task 1 is actually complete. We have enough information to deploy an EC2 instance using default VPC. It will look like this in Cloud9.

Before we deploy, we want to ensure we have all information from the Business Use Case. Task 2 is calling for us to bootstrap the EC2 instance with a shell script that will install and start Jenkins. We want update all packages, get latest system updates, install Java first, then install Jenkins. Finally, we will Enable and Start Jenkins. As a bonus, we will add “user_data_replace_on_change = true”, will serve as an action to apply if parameters have been changed, Terraform will terminate current instance, and apply updates to the new instances that gets launched. Below is the Script.

#Bootstrap Jenkins installation and start  
  user_data = <<-EOF

#!/bin/bash
#update all packages
sudo yum update -y

#Get latest updates
sudo wget -O /etc/yum.repos.d/jenkins.repo http://pkg.jenkins-ci.org/redhat-stable/jenkins.repo
sudo rpm --import https://pkg.jenkins.io/redhat-stable/jenkins.io.key

#Install Java and Jenkins
sudo amazon-linux-extras install java-openjdk11 -y && sudo yum install jenkins -y

#Enable and Start Jenkins
sudo systemctl enable jenkins
sudo systemctl start jenkins
EOF

 user_data_replace_on_change = true
}

Here is how the full configuration build looks in Cloud9.

Now lets move on to Task 3.

Create and Assign a Security Group to the Jenkins EC2 that Allows Traffic on Port 22 from your IP and Allows Traffic from Port 8080:
Configuring security groups is a way to control the traffic that is coming in and out of your environment through inbound and outbound rules. Essentially, security groups are virtual firewalls for instances within a specific network. These rules can be defined based on source IP, the destination IP, the protocol, and the port number. You can create and manage security groups using the Amazon Console, AWS CLI, or even AWS SDKs. Let’s go ahead and create ours. But instead of using the console, we are going to create and use a Terraform file that deploys to our infrastructure. See Terraform registry for security Groups.

#AWS Security Group Creation
resource "aws_security_group" "jenkins_securitygroup" {
  name        = "jenkins_securitygroup"
  description = "Allow SSH traffic and port 8080"
  vpc_id      = aws_vpc.main.id

  # ssh
  ingress {
    description = "ssh from IP"
    from_port   = 22
    to_port     = 22
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }
  # jenkins port 8080
  ingress {
    description = "jenkins default port"
    from_port   = 8080
    to_port     = 8080
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }
  # allow all outbound traffic
  egress {
    from_port        = 0
    to_port          = 0
    protocol         = "-1"
    cidr_blocks      = ["0.0.0.0/0"]
  }

  tags = {
    Name      = "jenkins_securitygroup"
  }
}

Here’s the cloud9 security group output.

Still on task 3. Lets assign our security group to the Jenkins EC2 that allows traffic from port 22, your IP, and port 8080. We will be working in our configuration file, by adding an expression, (vpc_security_group_id) to the (aws_instance resource), under AWS Security Group Creation section.

vpc_security_group_ids = [aws_security_group.jenkings_securitygroup]

Create a S3 Bucket for your Jenkins Artifacts that is not open to the public:
A Simple Storage Service (S3) is a cloud-based storage serviced provided by AWS, used to store objects, which can be files, folders, images, videos, or any other kind of data. It’s very popular storage solution for a wide range of applications, including backup and recovery. It also serves as a great tool to store artifacts generated by Jenkins.

Using S3 for Jenkins artifacts is a benefit, because it is reliable and provides durability. It also allows you to easily share artifacts across different environments and teams. To create, lets head back over to our Terraform S3 documentation. Just like in previous sections, we will find a template that meets our needs, and incorporate into the configuration file. Our task list requirement is for the Jenkins artifacts not be accessible to the public. Here’s a sample below. NOTE: “bucket” must be globally unique.

#Create Private S3 Bucket for Jenkins Artifacts 
resource "aws_s3_bucket" "jenkins_artifacts" {
  bucket = "my-terraform-cbeezy4-bucket" 

  tags = {
    Name        = "jenkins_artifacts"
    Environment = "Dev"
  }
}

resource "aws_s3_bucket_acl" "my-terraform-cbeezy4-bucket" {
  bucket = aws_s3_bucket.my-terraform-cbeezy4-bucket.id
  acl    = "private"
}

Now, let’s add to our main.tf configuration file.

Now, lets start deploying our Terraform configuration file we just built.

Deploy our resources using a few Terraform commands:

• Terraform Init — command initializes the Terraform environment and downloads any necessary plugins and modules

terraform init

• Terraform Validate — validates if the file syntax is valid.

terraform validate

• Terraform Plan — creates an execution plan that shows what Terraform will do when you apply the configuration.

terraform plan

• Terraform Apply — applies Terraform configuration and deploys the infrastructure.

terraform apply

Enter Yes

After running Terraform apply, you will get feedback of the resources created.

Now, lets verify that we successfully setup our infrastructure. See the next few screenshots.

Jenkins Security Group

Inbound Rules

“This Bucket isn’t Public”

Our security groups are set, with the correct inbound/outbound permissions. We have our S3 bucket that isn’t accessible to the public. Now, lets verify that we can reach the Jenkins server.

Verify Jenkins Server is Running:

http://your_ip_address:8080
http://54.70.205.27:8080

We’re In!

systemctl status jenkins

SUCCESS

It’s running! Now lets check through HTML and see if we get the Jenkins web landing page.

Lets Go!!! If you made it this far, congratulations, you did it! Before we close out, lets clean up our work.

• Terraform Destroy — tears down the infrastructure created by Terraform.

terraform destroy

Conclusion
Thank you for taking the time to read the article. This was a fun, but challenging project. A lot of attention and detail. But the troubleshooting required was so rewarding. I have much appreciation for Terraform and the power of CLI. All code pushed to my GitHub.