Terraform Modules 101: Create, version, and publish on GitHub
Tutorial on how to create and use a custom terraform module and publish it on GitHub
In this post, using terraform best practices I will demonstrate how to create a reusable terraform module from a sample configuration and publish a version on GitHub, after which we will create a root module to use it. Although a scenario will be used to present this practically the same principles can be applied in most scenarios.
What is a Terraform Module?
A Terraform module is a grouping of several resources that are used together. Terraform modules are similar to functions in programming, in that after it is created they can be reused anywhere in your code. A directory consisting of one or more terraform config files is a module.
Just like functions in programming, it accepts input parameters using input variables and can return values using output variables. The directory where you run terraform apply is also a module, called the root module. You can publish your modules to a registry such as the Terraform registry or a Version Control System. A list of supported sources can be found here.
Why Should I create a versioned module?
Versioned modules are especially useful when you use the same modules across various environments. For example, in your prod environment, you could use an old version of a module (e.g v1.0.0) and safely test a new version (e.g. v2.0.0) of the same module in your dev environment. Without versioning in place, it can be challenging to test new module releases.
Scenario
You’ve been given the terraform configuration below which was used to create an AWS S3 bucket to host a static website. The task is to make it a re-useable module and publish it to a new GitHub repository within your organization. You are also required to add the environment name as a suffix to any S3 bucket created.
Solution
To convert the configuration file above the following steps will be followed:
- Create a Standard Module Directory Structure
- Write terraform configuration i.e versions, input, output variables, etc
- Create a GitHub repository, push files and create a GitHub release
- Create a root module to use the GitHub module as a source
- Test and Apply terraform configuration and access the website
Requirements
- Basic knowledge of Terraform, AWS IAM, and S3.
- Terraform CLI (1.1+) installed.
- An AWS account and an IAM user with admin permissions.
- AWS CLI (2.0+) installed and configured with access keys.
- GitHub Account
You can find the completed demo module on my GitHub repository here.
Let's get started! 🍿
- Create a Standard Module Directory Structure
A terraform module to be distributed in separate repositories should use the standard module structure below. It is recommended for reusable modules to be placed at the root of their own repository. Hence, create the following files locally at the root of a directory.
$ tree module_directory/
.
├── README.md
├── main.tf
├── variables.tf
├── outputs.tf
├── LICENSE #optional
└── versions.tf The README.md file should describe what the module is used for, include the inputs or outputs of the module, and what resources it may create. The .tf file names above are recommended and widely used but not mandatory. If you are publishing a module, it is recommended you include a license, even if it’s an open-source license.
You can add an examples sub-directory to show examples of how to use the module. A complex directory module structure can also include a modules folder for nested sub-modules as shown below.
$ tree complete-module/
.
├── README.md
├── main.tf
├── variables.tf
├── outputs.tf
├── versions.tf
├── LICENSE.tf
├── ...
├── modules/
│ ├── nestedA/
│ │ ├── README.md
│ │ ├── variables.tf
│ │ ├── main.tf
│ │ ├── outputs.tf
│ ├── nestedB/
│ ├── .../
├── examples/
│ ├── exampleA/
│ │ ├── main.tf
│ ├── exampleB/
│ ├── .../2. Write the Terraform Configuration for the custom module
In the versions.tf file, remove any provider block as this should be configured by the user of the module. As a best practice, you should declare the minimum provider version your module is known to work with, using the >= version constraint syntax. The root module is where you should specify the maximum provider version using the ~> or = version constraints, to avoid accidental upgrades to incompatible new versions.
You add input variable(s) to make a module configurable and to be able to reference them as input arguments in the calling module which I will demonstrate later on. The variable named environment below was created to accept the environment name as an input to be used as a suffix for the bucket name. The bucket name is parameterized as bucket names must be globally unique.
My guiding rule when creating modules is that two resources of the same type created from the same child module should not result in conflict errors. This is why input variables can be critical to creating modules.
local variables were added to the main.tf file to create expressions to optionally create unique bucket names and add an environment name as a suffix. Use the locals block, if you don’t want the user of your custom module to pass input to a specific variable(s).
Resources created by a calling module are encapsulated and hence its attributes cannot be accessed directly from a root module without exporting the attribute(s) values using output variables. For example, without defining the bucket_id as an output variable below and the expression aws_s3_bucket.website.id the id of an S3 bucket created using the custom module cannot be accessed from the root module. Hence, create a output.tf file to export essential resource attribute values.
3. Push config files to a GitHub repository and create a release version
Create a GitHub repository (steps are available here if needed). Afterward, use the commands below to initialize the local directory you’ve created, commit the files to the local repo and push them to the new remote repo.
git init && git add -A
git commit -m "add all module files" && git branch -M main
git remote add origin https://github.com/<github_account_name>/<new_repo_name>.git
git push -u origin mainTo create a version of the new module, you can create a release on Github which uses tags or create tags locally and push them to the remote repo. Run the commands below to create v0.0.1 of our module.
git tag -a "v0.0.1" -m "First release of s3-website module"
git push --follow-tags4. Create a root module and use the GitHub module as a source
To call a module i.e to include the contents of another module in a module, you must create a module block and use the source meta-argument. The source value can be a local path to a module directory or a published module. Depending on the source, an optional version argument is used to specify the version of the module terraform should download when you run terraform init or terraform get.
Generally, for git repositories, the source argument value can be prefixed with git:: followed by the protocol (ssh or HTTPS). Unprefixed github.com URLs are also automatically recognized as git repositories. If a module is in a subdirectory of the repo, you will need to specify the path. Take note of the double slash // before the path.
source = "git::https://example.com/demorepo.git"
source = "git::ssh://username@example.com/demorepo.git"
#For modules in subdirectories
source = "github.com/example.com/demorepo//<PATH>?ref=v0.0.1"
#Example
source = "github.com/yemisprojects/s3_website_module_demo//submodule/service?ref=v0.0.1"To select a specific revision of a module from GitHub, the ref argument is used as shown in the main.tf file below. Without that argument, terraform will clone the default branch of the repository. The value of ref can be a tag name, branch or SHA-1 hash. v.0.0.1 below is the tag we created in the previous step.
The bucket_name, environment, tags arguments above are the exact same input variable names created previously in the custom module. To access a module’s output variable, use module.<MODULE_NAME>.<OUTPUT_NAME>.
For example, The S3 bucket id attribute is exposed in the child module using this expression aws_s3_bucket.website.id in the output variable named bucket_id. The same id attribute is accessed in the root module below using the module.my_static_website.bucket_id expression.
The versions.tf file includes a provider and a terraform block with updated version constraints.
Posting the variables.tf file for completeness 🙂.
5. Test and Apply terraform configuration and access the Website
You can clone my second repo which is a root module that uses the custom module we’ve created and apply the configuration. You can use the commands below to clone the repo and apply the configuration to verify our hard work has paid off.
git clone https://github.com/yemisprojects/use_s3_website_demo_module
terraform init && terraform plan
terraform apply --auto-approve After you run terraform init, the remote module will be cloned to your working directory to the .terraform folder as shown below.
$ tree .terraform
.terraform
├── modules
│ ├── modules.json
│ └── my_static_website
│ ├── LICENSE
│ ├── README.md
│ ├── main.tf
│ ├── outputs.tf
│ ├── variables.tf
│ └── versions.tf
└── providersAfter terraform apply you should see 5 resources added.
Upload the HTML files in the website-src folder of the same repo to the root of the S3 bucket.
Access the website URL in your browser as shown below.
Remember to destroy your resources when you are done with terraform destory --auto-approve.
Congratulations! 🙌 on creating your very own custom module and publishing a version on GitHub. We’ve also successfully tested the module and access the website deployed using the module. Great Job 🎉👏.
Till my next post 👋.
“The elevator to success is out of order. You’ll have to use the stairs, one step at a time.”- Joe Girard
