Automating Cloud Infrastructure with Terraform Cloud: A Deep Dive into its Capabilities

In this blog, we will discover how Terraform Cloud can help you streamline your cloud infrastructure management process and we will explore the powerful features that make it a must-have tool for cloud infrastructure management.

We will deliver the reasons why Terraform Cloud is essential, highlighting its capabilities and benefits, advanced workflows, and much more. We’ll take a closer look at the VCS-driven workflow, providing a detailed example to illustrate how it works in practice.

Plus, we’ll provide some practical recommendations for using Terraform Cloud in a production environment.

One of the main benefits of using Terraform to manage cloud infrastructure is that it allows for infrastructure automation. With Terraform, developers and operations teams can define and manage their infrastructure in code, rather than manually configure resources through a web console or command line interface. This makes it easier to manage large, complex environments and reduces the risk of human error.

Earlier, when using Terraform, the typical workflow involved developers defining their infrastructure as code using the Terraform configuration language(HCL), and then using the terraform plan and terraform apply commands to provision the infrastructure on their local machines.

The state of the infrastructure, which tracks the current state of the resources managed by Terraform, was often stored locally as well. However, as our cloud infrastructure grew, we started seeing inconsistencies and issues in the terraform state file as many team members were deploying resources at the same time.

That’s where Terraform cloud came in as a savior.

Terraform cloud is a managed solution offered by Hashicorp to run Terraform configurations.

Terraform Cloud is available as a hosted service at https://app.terraform.io.

It acts as a CI/CD tool for deploying Infrastructure as Code (IaC)

It offers a central UI for all your project and provides remote state management also it manages Terraform runs ( plan / apply ) in a consistent and reliable environment, access controls for approving changes to infrastructure, a private registry for sharing Terraform modules, Cost Estimation, Drift Detection, Sentinel for Governance (policy controls for governing the contents of Terraform configurations), and more.

You can use it to manage cloud infrastructure, including Amazon Web Services, Google Cloud Platform, and Microsoft Azure. It’s a recommended product if you needed to manage resources within an enterprise and you have a whole team contributing to it.

Why use Terraform Cloud?

1. Remote execution:

It allows you to run Terraform plan and apply operations on infrastructure hosted in the cloud (TFC) or on a central system that you can control (Terraform Cloud Enterprise)

2. Terraform cloud workspaces:

Workspaces are simply referring to environments. It works like separate working directories.

If we use multiple environments like development, QA, and production, we need to manage independent codes for each and every environment.

But with the help of Workspaces, we can use a single code for multiple environments and manage the State files of all the environments independently. It helps to create multiple State files with the same Terraform configurations.

Terraform cloud manages infrastructure collections with workspaces. It works like separate working directories. Cloud workspaces are not limited to state files, configurations, variables, state management & credentials & secrets.

In addition to the basic Terraform content, Terraform Cloud keeps some additional data for each workspace:

• State versions: Each workspace retains backups of its previous state files. Although only the current state is necessary for managing resources, the state history can be useful for tracking changes over time or recovering from problems.
• Run History: When Terraform Cloud manages a workspace’s Terraform runs, it retains a record of all run activity, including summaries, logs, a reference to the changes that caused the run, and user comments.

3. Version control integration and triggers:

As Terraform Cloud is an infrastructure-as-code service, It supports Version control system integration with some of VCS providers like GitHub, Gitlab, Azure DevOps, and Bitbucket. Terraform executions can be triggered upon new commits merged by pull requests or pushed in VCS. It can work as a CICD pipeline.

4. Private registry:

Reusable modules are very useful when needing to provision multiple environments from the same source configuration when the requirement is to store that module privately in a Terraform cloud. The private registry can be used in this scenario. A private registry allows us to store modules and use those modules in workspaces.

5. Access Control & Governance:

Large teams in organizations have many workspaces in Terraform cloud, And to manage access to those workspaces across team members is required most, Terraform cloud provides paid functionality to meet the control and governance needs of large organizations.

6. Sentinel policies & Cost estimation:

Terraform cloud provides sentinel, which is a policy-as-code framework. Which is used to enforce granular policies about how the infrastructure should get provisioned. Examples like limiting the size of virtual machines, confining major updates to defined maintenance windows, etc. These policies can act as firm requirements, advisory warnings, or soft requirements, which can be bypassed through explicit approval from the team.

7. Drift Detection:

It continuously checks against the infrastructure state to detect changes and provide alerts. This allows Terraform to provide a central pane of visibility to all your infrastructure and offers operators confidence that it matches the last known Terraform state.

8. Cost Optimization:

Cost Optimization is a unique feature of Terraform cloud. Before making changes to the provider, Terraform cloud can display an estimate of its total cost, as well as any change in cost caused by proposed updates.

Cost optimization can also be integrated with sentinel policies for warnings. It ensures that all your infrastructure is governed and has the proper security measures in place. It also reduces the potential for application downtime that could negatively impact the user experience and, eventually, revenue.

How does it work?

Terraform Cloud uses organizations and workspaces to organize your code and environments.

“Organizations are shared spaces for teams to collaborate on infrastructure”.

Each organization can have multiple workspaces that define groups of resources of your infrastructure.

Normally, a user will create one organization (common use case for small and medium companies) and have multiple workspaces describing your environments

Organizing Workspaces with Projects

A project is a container that holds multiple workspaces. Projects help to organize workspaces and one key benefit of using projects is that they allow for more granular control over permissions. Each project has separate permission set that you can use to grant teams access to all workspaces in the project.

Every workspace must belong to exactly one project. By default, all workspaces belong to an organization’s Default Project.

Enough of theory, Let’s start with the Practicals:

First, we will create the Organization itself:

Next, We will be Connecting to a version control provider

Connecting Terraform Cloud to your VCS involves four steps:

1. Go to your organization’s settings and then click Providers. The VCS Providers page appears.

2. Click Add VCS Provider. The VCS Providers page appears.

3. Select GitHub and then select GitHub.com from the menu.

This will take us to the page where have to configure OAuth authentication with your VCS provider.

Next thing on Github, we have to create a New OAuth Application This page is located at https://github.com/settings/applications/new.

You can also reach it through GitHub’s menus:

• Click your profile picture and choose “Settings.”
• Click “Developer settings,” then make sure you’re on the “OAuth Apps” page (not “GitHub Apps”).
• Click the “New OAuth App” button.

Fill it with the details you got above from the setup provider section and click on register application.

Next click on generate a new client secret.

Now go back to the Terraform Cloud, Enter the Client ID and Client Secret from the previous step, as well as an optional Name for this VCS connection. Click “Connect and continue.” This takes you to a page on GitHub.com, asking whether you want to authorize the app.

Click the green “Authorize <GITHUB USER>" button at the bottom of the authorization page. GitHub might request your password to confirm the operation.

Next, if the organization repositories include Git submodules that can only be accessed via SSH, an SSH key can be added along with the OAuth credentials. We will be skipping this step

And that’s it, we have created a new VCS provider to connect to GitHub

Terraform Cloud Workflows

Terraform Cloud has three workflows for managing Terraform runs.

• The UI/VCS-driven run workflow — here you are connecting your VCS to Terraform Cloud — easily integrate version control such as GitHub, GitLab, BitBucket or Azure DevOps and automatically initiate Terraform runs when changes are committed to the specified branch
• The API-driven run workflow — you can use your standard Terraform CLI to trigger remote runs.
• The CLI-driven run workflow — where you can manage and trigger runs through other tools by triggering calls to Terraform Cloud.

Here in this blog, we will be more focusing on the VCS-driven workflow.

VCS Driven Workflow

Some of the features:

• initiating speculative plans every time a PR is created against the default branch (this is set up by default so you don’t have to do anything)
• once PR is merged this will trigger the plan and apply, however by default applying will require manual approval.
• you have various triggers to choose from. I will discuss this more when coming to an example.

Sample repo containing some terraform code for testing can be found under: https://github.com/NITHIN-JOHN-GEORGE/terraform-cloud-demo

Creating a Workspace

Click Projects & workspaces → Click New, then select Workspace → Choose a workflow type: Version control → Choose a repository

Enter a Workspace Name → choose a Project to add the workspace to → Open Advanced options to configure Terraform Working Directory, Set up AutoRun Triggering, and setting up the VCS branch.

Note :

Terraform Cloud provides different types of triggers to start a run automatically based on specific events. These triggers can be customized to suit the needs of your workflow.

The first type of trigger is the path changes trigger, which is particularly useful for monorepos where multiple projects are stored in a single repository. With this trigger, you can configure Terraform Cloud to start a run whenever a change occurs in a specific directory or file path within the repository. This means you don’t have to run the entire codebase and can focus on the relevant changes.

The second and recommended type of trigger is a pattern-based trigger. This trigger uses glob patterns to select which changes should trigger a run and ignore others. For example, if you only require a run when there is a change in .tf files in the “tfe-test” directory, you can specify the pattern as “/tfe-test/*.tf”. This ensures that Terraform Cloud only runs when specific changes occur, reducing the risk of unintended consequences and speeding up the deployment process.

This is an important configuration to be noted, otherwise, the workspace will be triggered every time a code changes in any other sub-directory in the Github-Repo.

Next Click on Create Workspace

Here we can directly configure variables, or we can skip and do it afterward.

Next, Go to Workspace Overview → Click on Configure Variables

Here you can add new variables which are required !!

Here note we are adding a new variable called AWS_ACCESS_KEY. This is a variable that must be kept confidential so we have to click on Sensitive.

Here we can also directly declare variables as Environment variables .

Repeat the same for all variables you have.

As you can see the sensitive variables cannot be seen from the console, we also cannot see if we edit it, we can only update the value.

That’s it.

Now go to Actions → Start a new run

It will wait for our Confirmation for Applying. Click on Confirm & Apply

We can see the states got saved here.

And we can see a list of all Runs that happened:

Now, let's test the Auto triggering Functionality !!

Let's modify some parts of code and as soon as you push the code to GitHub, the Terraform runs will be triggered

And that's it, it got triggered !!

This confirms Everything Working Fine as Expected !!

There are other features of TF cloud like private module registry, configuring TF-agents, sentinel policies, drift detection,cost-estimation, etc .. we will be covering some of them in the upcoming blogs.

Configuring TF agents in our environment where can have complete control of the infrastructure, sentinel policies, drift detection, and cost estimation is paid feature and comes under the Business Plan of Terraform cloud

Recommendations while using TF cloud in production :

1. Add notifications for all events / certain events to a slack channel for workspaces with the prod prefix
2. Use self-hosted TF-agents
3. Use TF cloud for hosting our private modules
4. Alternatively, write a backup script that automatically sends the state to s3 and does cross-region replication for disaster-recovery purposes and create workspaces with the states from S3.
5. Provide granular access to terraform workspaces across team members with access controls.
6. Set the Terraform log level to debug
7. Use Features like Drift Detection ( which continuously checks against the infrastructure state to detect and notify when there are changes), Cost Estimation, and Sentinel Policies ( granular policies about how the infrastructure should get provisioned ).

Thanks for Reading !! Happy Learning