Part 6 — HumanGov Application — Terraform-4: Terraform Core Workflow, Subcommands and Terraform State

HumanGov is a software-as-a-service (SaaS) cloud company that will create a Human Resources Management SaaS application for the Department of Education across all 50 states in the US and host the application files and databases in the cloud.

In this following project series, we are going to transition the architecture from a traditional virtual machine architecture to a modern container-based architecture using Docker containers and Kubernetes running on AWS. In addition, we will also be responsible for automating the complete software delivery process using Pipelines CI/CD using AWS services such as AWS CodeCommit, AWS CodePipeline, AWS CodeBuild, and AWS CodeDeploy. Finally, we will learn how to monitor and observe the cloud environment in real-time using tools such as Prometheus, Grafana, and automate one-off cloud tasks using Python and the AWS SDK.

In this section, we are going to introduce working with Terraform State and practice Terraform with cloud provider AWS before using it in the implementation of the HumanGov application. Additionally, basically touch to Terraform Core Workflow and Subcommands and we will be using them as needed now and later sections. This is the 6th part of a project series. Check Part 1, Part 2, Part 3, Part 4 and Part 5 to follow up.

Terraform Core Workflow

1. Initialize the Terraform Configuration:terraform init initialize Terraform from the working directory, downloads the necessary provider plugins, checks configuration and modules, initializes the backend (if specified), and sets up the local environment.
2. Plan the Infrastructure Changes:terraform plan creates an execution plan analyzing the configuration and provides a review of resource creation, modification, or destruction.
3. Apply the Changes: terraform apply applies the changes proposed in the plan like provisioning, modifying, or destroying resources.

Terraform Subcommands

terraform destroy: Destroys all resources defined in the configuration.

terraform validate: Checks the syntax and configuration errors of Terraform files.

terraform fmt: Formats Terraform files according to the canonical format. You can see the changes below:

terraform providers: Displays information about the providers used in Terraform configuration with their plugins and versions.

TF_LOG=TRACE:TF_LOG is an environment variable. When it is set to TRACE, Terraform will provide detailed logging information during its execution. This can be helpful for troubleshooting. After we run Terraform commands it will generate verbose log output.

TF_LOG_PATH: It is an environment variable in order to specify a custom path that we can write Terraform logs in a specified file. As you can see below, we created a file named terraform.log and saved Terraform logs in it. We can share this file with someone else easily.

We can unset TF_LOG environment variable when we do not want to use it anymore.

terraform output: Displays the output of resources defined in your configuration.

terraform show: Displays current state from state file(terraform.tfstate).

terraform console: It’s a used tool for testing and debugging the Terraform code. The command opens a console where we can run prompts to test. You can work on Terraform code without executing it using this console.

Terraform State

The Terraform state is like a snapshot and a representation of the provisioned resources you’ve created, deleted, or updated in your infrastructure within a Terraform configuration. The existing stage of the Terraform is stored in terraform.tfstate file. This file will be updated whenever you make changes. Terraform first checks this file and if the intended changes are not already there, it will provision. This file is a read-only file managed by Terraform. You cannot make any manual changes to it.

Terraform state is also useful for dependency management. The state information is used to establish dependencies between resources. When resource dependencies are defined in configuration, Terraform uses the terraform.tfstate file to determine the order in which resources should be created or modified. Additionally, the Terraform state is used to prevent concurrent access and modification conflicts. Terraform locks the state file when operations are being executed to ensure that only one user or process can modify the state at a time.

Hands-on: Terraform State

Step 1: Open and Explore terraform.tfstate File

Enable Hidden Files on Cloud 9 (Settings > Show Hidden Files)

Now, you should be able to see terraform.tfstate file.

You cannot see any resources in it because we have not created any yet.

Lets run terraform init command.

As you can see it says “Initializing the backend…”. The backend is where the state file is located. This file is simply a text file and could be stored locally as you can see here or could be in a remote location like Amazon S3 bucket or Terraform cloud. When you run terraform init command Terraforms and it says “Initializing the backend…”, it starts looking state file either locally or remotely.

Step 2: Create random_password Resources

We are going to create a random password for a database.

Let's first go to Terraform Registry and then select Browse Providers.

Terraform Registry

Look for the Random provider and check its documentation.

Go to random_password resources. It generates random passwords or strings within the Terraform configuration. Then we can assign these passwords to the resources. One of the common use cases is once we need to create an Amazon Relational Database Service (Amazon RDS). We need to specify a password for the MySQL administrator. We can do all these inside the Terraform configuration. We can create random passwords and use these passwords while creating other AWS resources as you can see in the example in the documentation.

• We define a random_password resource named password with the length of 16 characters, indicating that we want to include special characters in the password using the special attribute, and override the default special characters with override_special attribute.
• When we run terraform apply, Terraform will generate a random password based on specifications and store it in the state file. The random_password resource generates random passwords each time you run terraform apply. The password should be captured in output or stored securely outside of Terraform to be persistent.

Create random_password resources in resources.tf file on AWS Cloud9, save and run the terraform init command to initialize the random provider.

Then run the terraform plan and terraform apply to create the random_password resources. Once you run the terraform plan, you can see that the password will not be exposed to the console because a password is a sensitive value.

if you run the terraform show command, no sensitive data will be displayed:

After we run the terraform apply a new file named terraform.tfstate.lock.info will be created before hitting ‘yes’. This file is really important and associated with the main Terraform state file (terraform.tfstate). Whenever we deploy a new configuration using the terraform apply or terraform destroy, it locks the operations state and employs lock files to ensure that only one Terraform operation can modify the state file at a time. When a Terraform operation is completed, the lock will be released allowing other operations to access the state file.

• ID: A unique identifier for the lock.
• Operation: The type of Terraform operation (e.g., "Apply" or "Destroy").
• Info: Additional information about the lock (usually empty).
• Who: The user or process that acquired the lock.
• Version: The Terraform version used.
• Created: The timestamp indicating when the lock was acquired.

In the shared file system, whenever you see this lock file in your file system that means someone else is running the Terraform in their system.

When we open the state file we can see created resources which is random_password in our case.

You can see the randomly generated password here:

"result": "dK$RJ(3:\u0026DCjS(\u003cR"

As a result, the state file is a really critical file because it includes sensitive information. Terraform actually recommends generating or storing passwords using external tools like AWS Secrets Manager or HashiCorp Consul. If you really need to do that in Terraform, store the state file encrypted remote location like an AWS S3 bucket. These are Terraform's best practices.

Step 3: Get the Password value as an Output

We can access this password elsewhere in our configuration using ${random_password.example_password.result}. We need to declare an output to retrieve the generated random password.

Add an output block inside the output.tf file. Make sure you set the sensitive attribute to true because the state file includes sensitive information like the password in this case and throws Error: Output refers to sensitive values error otherwise.

When you check the state file it now includes output.

Lastly, whenever we make a change to Terraform configuration, Terraform creates terraform.tfstate.backup file. Terraform backups the existing stage before making any changes. You can see the backup file is still in the previous step just before creating the output. So, it does not include the outputs.

If you need to roll back for any reason, you can use this backup file.

Step 4: Destroy Resources

terraform destroy

Make sure to destroy resources that you will not need and use, otherwise, you will be charged.

After destroying resources, the terraform.tfstate file will be empty but the terraform.tfstate.backup file will still have the information from the previous state just before destroying resources.

CONGRULATIOTIONS!!