Automating Jenkins Deployment on AWS EC2 using a Terraform Monolith
Welcome everyone! Today we are creating the first project of our last technology covered in Level Up in Tech, and that is Terraform! Terraform is the most popular IaC tool that is currently used industry-wide.
What is IaC? Infrastructure as Code (IaC) refers to the practice of managing and provisioning infrastructure resources using code, replacing the need for repetitive manual builds.
https://www.redhat.com/en/topics/automation/what-is-infrastructure-as-code-iac
What is Terraform? HashiCorp Terraform is an IaC tool that lets you define both cloud and on-prem resources in human-readable configuration files that you can version, reuse, and share. You can then use a consistent workflow to provision and manage all your infrastructure throughout its lifecycle.
https://developer.hashicorp.com/terraform/intro
Today our managers provided the following task by email:
Your team would like to start using Jenkins as their CI/CD tool to create pipelines for DevOps projects. They need you to create the Jenkins server using Terraform so that it can be used in other environments and so that changes to the environment are better tracked. For the Foundational project, you are allowed to have all your code in a single main.tf file (known as a monolith) with hardcoded data. Push all your code to GitHub.
Our project:
- Deploy 1 EC2 Instance in your Default VPC.
- Bootstrap the EC2 instance with a script that will install and start Jenkins.
- Create and assign a Security Group to the Jenkins Security Group that allows traffic on port 22 from your IP and allows traffic from port 8080.
- Create an S3 bucket for your Jenkins Artifacts that is not open to the public.
- Verify that you can reach your Jenkins install via port 8080 in your browser. Be sure to include a screenshot of the Jenkins login screen in your documentation.
- Push your code to GitHub.
We will be using Cloud9 for this project as I have with my past projects. There are two benefits to this. First is that we do not need to worry about setting up programmatic access as we would for any other IDE, and second is that Terraform is already installed.
Let’s begin!
Just like any other Linux terminal we need to create our main.tf file.
A little bit about the main.tf file.
In Terraform, the main.tf file is a configuration file that plays a crucial role in defining and organizing your infrastructure resources. It is one of the main files used in a Terraform project. For this lab, we are only going to use a main.tf file to showcase its power.
The main.tf file contains the primary infrastructure code written in HashiCorp Configuration Language (HCL). It serves as the entry point for defining the desired state of your infrastructure. Within this file, you define the resources you want to create, configure, or manage using Terraform.
The above commands show that I created a new directory and a new main.tf file that I will be using to build out this project.
Now we can VIM into the editor and begin writing our code, but we’re not psychopaths so we will use the file above us.
I also wanted to reference one of the most useful sites that we are going to reference throughout this project.
The registry provides us with explanations and examples of the blocks of code we will need for whichever service we are building.
Let’s begin with the first block of our code.
Above, we are assigning AWS as the provider in my closest region, us-east-1.
Next is creating our security groups by ensuring that we are opening the ports we need to access. Ingress is for incoming traffic and egress is for outbound traffic.
As you can see, we have opened port 22 open for SSH, 8080 for Jenkins, and egress, as mentioned, is what allows outbound traffic. We have it set up to effectively allow unrestricted outbound communication to any IP address or destination.
Our next block is to create the EC2 instance that we need by providing our AMI, instance type, etc.
Followed by our Jenkins bootstrap.
In our next block, we have our S3 Bucket that will hold our Jenkins artifacts. These are files that are generated during the execution of a Jenkins pipeline.
That should do it for our main.tf file!
Now the moment of truth! Let’s see if we are able to build out what we created with our Terraform flow.
First, we start with the following command: terraform init
This is the first command we should always run after writing out our infrastructure. It ensures that all the required dependencies are available and sets up the necessary infrastructure for managing the Terraform state and executing subsequent commands in the working directory.
Next, we want to run: terraform fmt
This command helps us ensure that our file is written correctly.
Then we can run: terraform validate
Shoutout to my teammate for pointing out to use “vpc_security_group_ids” instead of “security_group_id”. I also love the error output Terraform creates because it makes it so easy to troubleshoot.
Since the other is a warning, I am going to choose to ignore it for now and move on to the next command.
terraform plan — this command creates an execution plan and shows us all the resources that will be created. This reminds me of the “What-If” command in PowerShell.
Now the real moment of truth: 🥁
terraform apply. This will create all the resources that were written out in the main.tf file.
Ok… not what we wanted to see…
I headed to the VPC console and chose a random subnet that was already created.
Then I realized that the information I need is in my VPC.
And added it in my EC2 instance block.
Then re-ran all previous Terraform commands… and… nope that wasn’t it.
I tried to add in just the subnet and continued to research the issue. No matter what I tried I could not get past this error.
Finally, after googling around I decided to add in a VPC and Subnet Block.
Once I added in the network portion, I re-ran the terraform flow commands, and finally!
That is what we want to see after applying Terraform apply. The registry was a big help as well.
hmmm…. 🤡 <- me
I noticed my instance wasn’t creating either, so I attempted to grab the public IP address however, it seems like I had the wrong script added in.
Now I realize this is what we want to see!
Why is Terraform playing with me? 😠
The CLI output shows created, however, the instance still was not showing up in my EC2 console until I realized that I was in us-east-2 instead of us-east-1. 🤦Rookie mistake. I created a new IAM user for this lab and completely forgot that was a thing.
I changed my region and finally got a sigh of relief. I did have to start all over creating a new cloud9 instance and creating the directories, files, etc. However, once all that was set up, I was not able to remote in and log into the Jenkins server.
I noticed I couldn’t SSH in either. After researching this issue (shoutout YouTube) I found that I had to adjust a setting within my VPC to enable DNS hostnames.
If you are running into the same issue, click into your VPC > Actions > Edit VPC settings.
On the next page, you will see an option to enable DNS hostnames. Make sure that is selected and click Save.
I was finally able to connect via the AWS console, but I was receiving an error starting Jenkins. I even attempted to manually run the script, one command at a time and that led me to understand that Java was not installing properly. I adjusted my script and changed my AMI back to the original (because I changed it about a half dozen times when I was troubleshooting). Reran all the terraform flow commands and it looks like I finally got it!
What a relief that was, I probably spent 4–5 hours troubleshooting. The Jenkins download page was also a big help in understanding what I could’ve been doing wrong.
Our last step is to go into our S3 bucket > Permissions. There, we can confirm that our Jenkins Artifacts are not open to the public.
Nothing like spending hours troubleshooting your environment just to run the next command. 😂
terraform destroy — This command ‘destroys’ all resources that were just created with the terraform apply command.
That does it for our terraform project! If you have your IDE connected to GitHub now would be a good time to run ‘git push’ and save your code. This project was specifically challenging for me and I am glad I was able to document my errors and how I overcame them.
If you are curious as to how the code came out after all my troubleshooting and adjusting, you can review it on my GitHub.
If you found this insightful, please give me a follow and connect with me on LinkedIn: https://www.linkedin.com/in/rgmartinez-cloud
