Terraform Remote States in S3
A simple way to use S3 backend to store your states remotely
When a terraform stack is deployed, terraform creates a state file.
The state file keeps track of what resources have been deployed, all parameters, IDs, dependencies, failures and outputs defined in your stack.
In a not so distant past, this state file (JSON) would be committed to the repository containing your terraform stack code.
This created a few problems:
- Concurrency: If 2 or more developers are working in the stack, they won’t see the other state until it’s pushed to the repository
- Automated Deployment: A CI tool that deploys the stack automatically would need to commit the new state file to the repository
- Easily corruptible: In case of a merge conflict or human error, the state file can be corrupted or gone. If this happens, the stack becomes unmaintainable, resources need to be manually/individually imported or cleaned and then re-created with terraform
How remote state works
Instead of having a local JSON file holding the state, the state file is uploaded to an S3 bucket.
It’s all done automatically by the terraform S3 backend, example:
After deploying, terraform will create a state file called ecs-platform in the bucket dnx-terraform-backend.
If you use workspaces, it will prefix with /env:/<workspace_name>/.
You probably noticed the parameter dynamodb_table above. DynamoDB is used for locking and consistency checking, to prevent concurrent operations in a single workspace.
Creating the backend infrastructure (bucket, DynamoDB table and IAM roles)
We have a terraform module to facilitate the infrastructure creation:
Usage example:
The module creates the following resources:
- S3 Bucket named
<bucket_prefix>-terraform-backend - DynamoDB table named
terraform-lock - IAM Role:
terraform-backend
When deploying the module above, terraform will create a state file as it does for every stack. The state file for this stack has to be stored as a file in your repository as the bucket to store state files doesn’t exist yet. Classic “chicken and egg” problem 🐔🥚. All newer stacks can use the bucket created by this stack to store the state from now on.
Using multiple AWS accounts
If you deploy the S3 backend to a different AWS account from where your stacks are deployed, you can assume the terraform-backend role from the other account, as long as it’s allowed in the assume_policy parameter (it supports multiple roles separated by comma).
As an example, we have a network stack deployed to a different account than the S3 backend, the configuration on that stack would look like:
Note the new parameter role_arn and change <AWS_ACCOUNT_ID_OF_BACKEND> above to the AWS account ID where your S3 backend was deployed.
With the configuration above, when running terraform init/apply, it will:
- Assume the role defined in
role_arn - Reach the S3 bucket
dnx-terraform-backend - Create the file
networkif it doesn’t exist, or download it and use as state file - Run
apply: create/update the resources - Assume again the role in
role_arn - Upload the updated state file back to the S3 bucket
The module also automatically enables bucket versioning and disallows files being deleted using the role, making almost impossible to lose a state file.
Conclusion
Hopefully, this article was helpful in some way.
Please leave your feedback below as well as anything you would like to see added.
We work at DNX Solutions and help to bring a better cloud and application experience for digital native startups in Australia.
Our current focus areas are: AWS, Well-Architected Solutions, Containers, ECS, Kubernetes, Continuous Integration/Continuous Delivery and Service Mesh.
We are constantly hiring cloud engineers for our Sydney office, focusing on cloud-native concepts.
Check our open source projects at https://github.com/DNXLabs and follow us on our Twitter or Linkedin
