AWS & Terraform
Most of us prefer that someone who is expert in installation and configuration of systems should install OS, software and can do required configuration in our laptops and desktops. Agreed? Through the evolution of IT infrastructure, we have depended on system administrators to configure the hardware, setting up and maintaining systems which require operational knowledge. The need for this knowledge is more in demand now as public cloud providers like AWS, given us access to large data centers and many other resources! we still needed someone to configure the hardware through the tools available in the cloud portals.
In AWS, the number of infrastructure resources to provision for a simple application goes easily above twenty five, not considering the need for different environments such as staging, development, and production!! This is simply impossible to maintain using the AWS web interface and it is where Infrastructure-as-a-Code (IaaC) software comes to the rescue.
With Infrastructure as Code and its tools like Terraform and AWS Cloudformation, this has changed. In this blog we will be discussing IaC, Terraform as one the best open source IaC tools and hands on using Terraform.
Let’s start with IaC and Terraform! As part of hands on, we will write Terraform scripts to integrate custom webhooks to send alarm notifications with AWS SNS and Lambda services.
In this blog, we will cover:
- What is Infrastructure as Code (IaC)?
- Why to use IaC
- Benefits of Infrastructure as Code
- Infrastructure as Code and DevOps
- Tools of Infrastructure-as-Code
- Terraform — An IaC tool
- Why to choose Terraform?
- Install Terraform
- Hands On
What is Infrastructure as Code (IaC)?
IaC is the process of managing and provisioning computer data centers through machine-readable definition files, rather than physical hardware configuration or interactive configuration tools. Wikipedia-based definition.
In simple words, IaC is the process of replacing manual effort required for IT resource management and provisioning by simple lines of code. A resource in this instance is any piece of infrastructure in a given environment, such as a virtual machine, security group, network interface, etc. IaC is a method to define, build and deploy vast environments within a few minutes. IaC files are easily readable, extremely portable and often serve as a documentation tool for IT administrators.
Why to use IaC?
When you define infrastructure as code, you can use a wide variety of software engineering practices that improve the software delivery process with following advantages:
- Speed: when automated, the entire deployment process happens way much faster that the traditional way
- Reuse: you can create reusable modules that can be used in every product, regardless of environment.
- Self-service: when infrastructure is defined in code, the entire deployment process can be automated. This means developers can kick off their own deployments whenever necessary;
- Safety: an automated process is by far more consistent, more repeatable, and not prone to manual error than a manual one.
- Easy access to infrastructure data: the state of the infrastructure is represented in source files that anyone can read.
- Version control: the IaC source files stored in the version control system make it easier to debug issues because the commit log history is available and you can easily switch to a previous version in case something goes wrong.
- Validation: every change on the IaC source files can be code-reviewed. You can run automated tests and pass the code through static analysis tools.
Benefits of Infrastructure as Code
Infrastructure as Code (IaC) and DevOps
DevOps is the combination of development and operations team aimed to reduce cycle time and ensure continuous delivery to the end-users. DevOps also offers more opportunities to improve the application and learn from the errors. Infrastructure as Code uses automation and code implementation to set up and tear down specific environments unlike traditional practice that involved humans managing VMs manually. This approach helps us in reducing development time and making it easier to deliver stable platforms to host applications.
Moreover, IaC stores all environment build-up commands in a repository allowing redeployment or faster roll back to the previous instance. This means that you no longer need to spend time fixing a server. All you need to do is either look into the version history to restore a previous deployment or redeploy it with a working copy.
Following are the major features of IaC which helps in DevOps:
- Manage infrastructure through source control to offer detailed audit tracks for modifications.
- Testing the infrastructure for any flaws or errors using unit testing, integration testing, and functional testing.
- Allow collaboration around infrastructure configuration and provisioning.
- Avoid written documentation as the code itself will document the state of the machine.
Tools of Infrastructure-as-Code
It is not possible to practice Infrastructure as a Code (IaC) in DevOps workflows without using tools. Here are a few common yet powerful IaC tools — AWS CloudFormation, Red Hat Ansible, Chef, Puppet, SaltStack and HashiCorp Terraform.
Some tools rely on a domain-specific language (DSL), while others use a standard template format, such as YAML and JSON.
Terraform — An IaC tool
Nowadays, the de-facto standard for IaC is Terraform which integrates with virtually all the available cloud providers and features a highly specialized domain-specific language called HCL. Terraform is a great tool with a very large community and a lot of open-source modules readily usable in the Terraform Registry.
It is developed by HashiCorp and written in GO language. It is one of the most popular Infrastructure-as-code (IaC) tools, used by DevOps teams to automate infrastructure tasks. It is used to automate the provisioning of your cloud resources.
Terraform allows you to safely provision and manage multi-cloud infrastructure at the scale. It is used to write, plan, and create Infrastructure as Code. Since this is codified it enables operators to automate infrastructure creation and changes. The configuration can be stored in version control systems and collaborated on by teams of operators. This provides more consistency, reduces errors, and decreases time.
Terraform allows us to declare configuration files that can be shared amongst team members, treated as code, edited, reviewed, and versioned. Using Infrastructure as Code, we can consistently provision your infrastructure. Terraform also enables users to easily provision AWS infrastructure on-demand with a library of approved infrastructure.
Why to choose Terraform?
As we discussed in the beginning of the blog, there are a lot of IaC tools available in the market like Terraform, Chef, Puppet, Ansible, SaltStack, and OpenStack Heat. As administrator, it is always a blog question in mind which tool to choose among all available tools. Here are the things you should consider before choosing which tool is the best:
- Mature versus cutting-edge
- Configuration management versus provisioning
- Mutable infrastructure versus immutable infrastructure
- Procedural language versus declarative language
- Master versus masterless
- Agent versus agentless
- Large community versus small community
The main reason to choose Terraform is, it is an open-source and cloud-agnostic provisioning tool that supported immutable infrastructure, a declarative language, a masterless and agentless architecture, and had a large community and a mature codebase.
Following table gives you a clear idea about different open source IaC tools.
Follow the instructions here to install Terraform.
When you’re done, you should be able to run the terraform command as shown in the image below:
In order for Terraform to be able to make changes in your AWS account, you will need to configure the AWS credentials for the user. There are several ways to do this, one of the easiest of which is to set the following environment variables.
Now let’s move on to the hands on part!
Let’s consider a scenario where one of your critical applications is running on AWS and you can’t afford downtime at all! So you want to set an alarm that will monitor the application’s health regularly. In case of any issues, you want to get a direct notification on your MS Teams channel so you can act upon it quickly. Let’s implement it using the Terraform template.
In this hands on, we will use Terraform to provision AWS Cloudwatch alarm, AWS SNS topic and subscription and AWS Lambda. You’ll need terraform to be installed on your system and an active AWS subscription. We’ll utilise AWS Lambda functionality to send custom notifications to MS teams channel.
Note: For the complete source code of this implementation, you can clone our following repository:
Let’s create a webhook in MS Teams to receive notifications from AWS Lambda.
Login to your MS teams account
Open MS Teams app and go to Apps section as shown below:
Now Search for incoming webhooks and click on it.
Select Add to a team button
In this screen you need to select your channel name and then click on setup a connector.
Provide the webhook name of your choice and click on create button as shown in the below screenshot
Copy the webhook endpoint populated and keep it safe in some editor because you have to use it in your Python code when you will setup your infrastructure using Terraform.
Lets create a directory for placing Terraform and Python code.
Create two files as below:
- Terraform file with name main.tf
- A python file with your preferred name. I have created as workfall.py
Place the below code in your terraform file (main.tf)
Note: If your python filename is different, you need to provide the same and edit line number 97,95,51,49,48.
Save the main.tf file.
Now let’s implements python code which will be used in Lambda service to send notification to MS Teams.
At line number 45 provide your webhook URL.
Our coding part is done, now let’s move toward creating Terraform infrastructure.
Let’s check terraform is installed or not by executing terraform version command as shown in the following screenshot
Now let’s initialize terraform using terraform init command as show in the below screenshot
The terraform init command is used to initialize a working directory containing Terraform configuration files.
As shown in the above screenshot, Terraform is initialized successfully. Now time to validate the Terraform template which we have created as the main.tf file. We will execute a terraform plan command for the same. The terraform plan command is used to create an execution plan. Terraform performs a refresh, unless explicitly disabled, and then determines what actions are necessary to achieve the desired state specified in the configuration files.
As shown in the above screenshot, our Terraform template is successfully validated and now we can proceed for actual implementation. For that we will execute terraform apply — auto-approve command.
The terraform apply command is used to apply the changes required to reach the desired state of the configuration, or the pre-determined set of actions generated by a terraform plan execution plan.
As shown in the above screenshot, necessary AWS resources have been created. Now let’s go to the ÁWS console to check if all resources are created.
As seen in the following screenshot, AWS Lambda got created.
As seen in the following screenshot, AWS SNS Topic got created.
As seen in the following screenshot, AWS SNS Topic subscription got created.
As seen in the following screenshot,.AWS Cloudwatch Alarm got created.
As seen in the following screenshot, IAM Role got created.
Here, we have seen that by executing Terraform templates, different services and IAM roles got created. WOW, so easy, really Terraform templates automate the different tasks which makes our development and integration processes really faster!!
Now, let’s check how this infrastructure works. Lets trigger this Alarm to test it and check if we receive the notification in MS Teams.
We’ll trigger this alarm through AWS CLI as shown in the following screenshot.
Now let’s open MS Teams to verify that we have received notification or not! As shown in the below screenshot, we have received a notification. WOW!!
So based on the notification, the support team can work and resolve the issue. After resolving the issue, Alarm State becomes OK. And we will receive a confirmation regarding the same in the MS Teams channel as shown in the below image.
Hope that you have noticed the difference between the above two images of MS Teams. Notice the red and green outline which helps for better representation and the Link to alarm button to directly navigate to the cloudwatch console.
This was the simplest implementation using the Terraform template. We can add more customization as needed using outlook connector cards and Microsoft graph API.
In this blog, we have understood that we can adopt infrastructure automation through Infrastructure as Code (IaC) tools like Terraform and decrease costs and manual labor, including human-prone errors and risks. We have discussed what is Terraform and how to use it to write templates to automate tasks. We have understood that using Terraform to provision infrastructure on AWS is very easy and fast. We have implemented how to send Cloudwatch events to a SNS service, how to trigger Lambda function with an SNS service and 4.Integrate AWS lambda with MS teams to send custom notifications on Alerts/alarms.
We will discuss more about Terraform and its other implementations, AWS Cloudformation and related hands on in our upcoming blogs. Stay tuned to keep getting all updates about our upcoming new blogs on AWS and relevant technologies.
For any further queries, feel free to post your comments, we are happy to help!
Keep Exploring -> Keep Learning -> Keep Mastering
This blog is part of our effort towards building a knowledgeable and kick-ass tech community. At Workfall, we strive to provide the best tech and pay opportunities to AWS-certified talents. If you’re looking to work with global clients, build kick-ass products while making big bucks doing so, give it a shot at workfall.com/partner today.