Setting up Terraform with AWS
Everything you need to setup, deploy and automate your workflow with AWS & Terraform
A world without IaC
Now you are probably asking what is IaC? Well, you came to the right place!
Infrastructure as Code (IaC) is a concept that originated back in 2005 and was a way for engineers to manage on-premise infrastructure in a declarative way (think of deploying VM’s etc in an automated way). With the release of AWS Elastic Compute (EC2) in 2006, its popularity skyrocketed. In an age where you had individuals manually managing the provisioning of hardware, there was a massive shift to Cloud services and with that came its own challenges. We encountered many issues with the rise of EC2 and quickly realized that this process can’t scale well in production systems. After all, who wants to set up hundreds of EC2 instances manually using an underwhelming user interface (UI) & SSH?
As a result, in a world moving more towards automation each day, there was a huge shift towards IaC and suddenly you can spin up an entire infrastructure stack in a matter of minutes – what a time saver ey!
What is Terraform
Terraform is a tool for building, changing and versioning infrastructure safely and efficiently – Terraform (www.terraform.io/intro)
It’s a tool that can seriously change the way you think about infrastructure – it sure has changed my life for the better! Long gone are the countless hours spent on clicking through Cloud providers UI and trying to set up your stack and new are the days where you do all that with the click of a button.
Here are some key features boasted by Terraform:
- Platform independent – this can run on your Windows, Linux, or Mac environments with a single executable
- Supports multiple providers – whether it's your on-prem infrastructure or a variety of Cloud providers, Terraform has you covered
- Maintains state – it can map your current infrastructure as a graph, understand when things change and how to safely/reliably change those components without human error
- Concurrent in nature – unless two components/resources are dependant on one another, Terraform can spin up infrastructure in parallel so you can get up and running as quickly as possible
- Execution plan – finally with terraform plan, you are able to get a detailed view of what changes are going to be made to your infrastructure, validate them and then decide whether or not to apply them
Still not a believer of IaC? Well follow this guide on setting up Terraform and by the end of it, you will never want to touch a UI again!
Downloading Terraform
Downloading Terraform is just as much a piece of cake as you thought… all you have to do is visit their website, download the executable and voila!
Step 1: Downloading the executable
Visit https://www.terraform.io/downloads.html and select your platform of choice (regardless of what you choose, you get a single executable file)
Once you have that, pop it into a directory on your machine that you are comfortable with.
Step 2: Creating a symlink (symbolic link) to the executable
What this will allow us to do is run terraform from any location on our machine, this way you won’t have to change into the directory containing the executable in order to run your code
Mac/Ubuntu users:
Paste the following line into the editor replacing the file path with the location to your Terraform executable
export PATH=<PATH_TO_TERRAFORM_EXECUTABLE:$PATH
Then hit save and run the following command in terminal to load your changes into your session
Step 3 (Optional alternative): Installing tfswitch with Homebrew
Thanks to Warrenbox, there is a neat tool that allows you to download and swap versions of terraform all from command line and without any additional setup. Follow the instructions over here:
Windows users:
You will have to set this up in your system path variables for which you can follow this guide
Setting up AWS Identity Access Management
At this point you will need to Sign up for an AWS account, don’t worry it won’t cost you a penny and most of the services are free for at least 12 months (you will need a debit card). To view a full list of services which are included in the free tier, visit this page
Now it’s never a good idea (nor good practice) to create users and use it directly in your Root account, more about that here. Ideally, you should create an IAM user for the project and then use that to perform all your infrastructure changes, so let’s do that first…
Once you have your account (make note of the Account ID), log in to the console and in the Services section type IAM and click the option in the drop down
Once you are in the IAM console, you will see an interface like the one below. What we want to do is head to the Users section so we can setup our first IAM user
Click on the Add user button, and fill out the form
Click Next: Permissions at the bottom right of the page and you should see the group we need in the table. Select the checkbox
Then follow through the menus (unless you want to add additional things such as Tags), you should hit the Review stage and then select Create user . Once that’s done your user will be created, log out of your account from the top right menu and head over to the Sign in page, this time selecting IAM User, enter the credentials we just created and Bob’s your uncle!
Setting up an IAM user for Terraform
Once you are logged into your brand new Administrator IAM account, head over to IAM again and similar to the above create a user, this time selecting Programmatic Access in the options. Move onto the Permissions section and this time select the Attach existing policies directly option
In the table below, search for AmazonS3FullAccess and select that checkbox (NB: Always restrict permissions to a more granular level in a Production system)
Follow the menu through once again and hit the Create user button. You will then be taken to a new screen where you will see some credentials that have been generated for you.
Make a note of these (if you lose them you will have to generate a new pair). Next, follow this tutorial on setting up your AWS CLI and you are finally done setting up IAM for Terraform, what a relief!
Terraform Resources
For those of you who are unfamiliar with Terraform and it’s workings, let’s cover a few concepts quickly…
- Resource describes one or more infrastructure objects, this could be anything from virtual networks, compute resources or DNS records
- Terraform state is used to track versioning, changes and keep track of meta data about your infrastructure
- Backend configuration is used to initialise Terraform and generates a state file for you
Great, now we have a basic understanding lets see what our configuration file looks like and break down what each section means
Provider: responsible for understanding API interactions and exposing resources (e.g. Amazon’s API will be different to Azure/Google etc.)
Region: the location of where we want our infrastructure to be deployed to (See this document for a list of available regions)
Backend: as described above, initialising our state file in an S3 bucket and adding encryption for security
Resource: defining the creation of a new S3 bucket with the name startup-chronicles-demo-account and assigning it default S3 encryption (always encrypt your data at REST using KMS)
Perfect… We have everything we need now to create an S3 bucket in our AWS account automagically! Let’s get that up and running with the following commands
Next, we need to run a plan so Terraform can figure out what needs to be done and show what will be added , changed & destroyed . You may notice the -out build.plan parameter at the end of the command which basically makes it easier for us to apply those changes without running a plan again
Finally, you should have a file generated by the output of the plan and it will show you exactly what is going to be added. Let’s go ahead and apply that to spin up our infrastucture
Head over to your AWS account and select the S3 service. You should see a newly created S3 bucket with whichever name you decided to give it…
Congratulations! Your first automated deployment of infrastructure!
Modularise everything
If you got this far, you are making fantastic progress! So now we have a basic idea of how Terraform works, how to setup resources and how to get it deployed to the cloud provider of our choice. It’s time to take it one step further and refactor what we’ve done so far to allow for various parameters to be passed in (rather than hard coding) and optimise our workflow. So let’s get started…
What is a module?
A module is a unit of clustering together parts of your infrastructure in a logical, lightweight and abstract way. Implementing modules will allow you to describe your infrastructure in terms of its architecture rather than the resources that it’s made up of.
What does it comprise of?
So a module has a few common logical concepts, namely:
- Input variables – define a set of parameters to be passed to the modules. This feature gives you the flexibility to pass dynamic parameters which only pertain to that logical part at runtime to spin up infrastructure
- Output variables – gives you the ability to output computed properties (i.e. subnet ids, database endpoints etc.) and share those outputs between modules/resources. For example, outputting Subnet IDs and passing them as an input parameter to a newly created RDS instance
How do you implement them?
Great, so now we know exactly what modules are, what they are made of and now it’s time to implement them. Let’s get started…
1. Create a new folder in your root and lets call it modules/s3(since we are dealing with S3 resources) and populate it with the following files
main.tf will be the entry point for our module, variables.tf will be all the inputs for our module and likewise outputs.tf will be all the accessible outputs so that we can share them between modules.
2. Let’s move the creation of our S3 resource from our <ROOT>/main.tf file and paste it into our <ROOT>/modules/s3/main.tf entry point
3. Next we want to change the input parameters so that we can dynamically populate the values for resource creation at runtime
4. Finally, we have to change our <ROOT>/main.tf file and import our newly created module like so
Fantastic, we have created a module that can now be populated with values through the command line at runtime. This will be useful for distinguishing between Development, Staging & Production environments which is common practice in the industry. We just have one more thing to do!
Setting up environment variables
- Create a new folder in the root level of your project and name it
env_vars
2. Next, create a profile for each of your environments (e.g. dev, staging, prod etc.). For this purpose I will show only development
3, Finally, define your variables here and change them accordingly per environment
Running the CLI with variables
- Initialise Terraform
2. Run the plan (you should see no visible infrastructure changes)
Voila! We are all done, whilst it seems like nothing has changed on the surface, we now have an abstract way of defining infrastructure and linking logical parts together.
Summary
I’m sure at this point you are trying to wrap your head around all the concepts we have learned, the setup we have implemented and how to apply infrastructure changes, so let’s wrap it up in TLDR fashion
- Resources: defines a new piece of infrastructure
- Modules: abstraction for a logical collection of resources (e.g. VPC module does everything VPC related)… use where possible
- Terraform State: keeps a record of infrastructure and tracks changes
- Terraform Plan: figures out what currently exists needs to be changed and shows you
- Terraform Apply: builds the infrastructure output from the plan
That’s all wrapped up in short, you are now well on your way to becoming a Terraform expert and there’s really not much left to learn. Remember, the more you use it — the better you will get so practice, practice & practice some more!
I hope you enjoyed this style of the walkthrough, join us in Episode 3 next Monday where we will go through setting up our CI server with Terraform so everything gets deployed automatically!
