Part 01 - Publishing Terraform Modules to Gitlab Infra Registry

6 min readJan 23, 2023

Welcome to the first installment of our multi-part series on GitLab, Terraform, and everything in between. In this series, we will dive deep into the world of GitLab, a powerful platform for managing software development, and explore how it can be used in conjunction with Terraform, a tool for provisioning and managing infrastructure. Together, these two technologies provide a powerful combination for automating the entire software development lifecycle, from code management to infrastructure provisioning and deployment.

Whether you are a developer, DevOps engineer, or simply interested in learning more about these technologies, this series is for you. So join me as we explore the powerful world of GitLab and Terraform, and discover everything in between.

To kick off this series, we will be delving into the topic of GitLab’s Terraform Module Registry. Understanding the capabilities of this feature, it will serve as a solid foundation for further discussions in the upcoming articles of this series.

GitLab Terraform Module Registry

GitLab Terraform Module Registry is a Gitlab Feature for storing, versioning, and sharing Terraform modules. The terraform registry at registry.terraform.io is a public one, if your enterprise uses a self-hosted version of GitLab or you use the gitlab.com, you can use GitLab’s registry as a private one. The best part is that it is FREE!!

Initial Repository Set Up

Let’s assume you have a repository as shown below

where the src folder contains your super awesome terraform module. For the purpose of this demo, the contents of the “sc” folder can be disregarded. We will be discussing the organization and elements of the “src” module in a separate article.

The repository discussed in this article is available for public access and can be found on gitlab.com. Nevertheless, it’s worth mentioning that it can also be set as a private repository and hosted on a Gitlab instance that is managed in-house.

Publishing the module

To publish it to Gitlab’s terraform registry, you would simply run the package.sh script file

./package.sh

Let’s dissect the script file.

# Environment Variables 
TERRAFORM_MODULE_NAME="infra-registry" # The name of the module you would like to see
TERRAFORM_MODULE_VERSION="0.0.1" # The version of the module.  
TERRAFORM_TARGET_PLATFORM="aws" # What platform is this module targetting?
YOUR_TOKEN="<your-personal-access-token>" # Your personal access token
GITLAB_API_V4_URL="https://gitlab.com/api/v4" # if you are using GitLab.com else use your own GitLab instance URL https://<my-dedicated-gitlab-instance>/api/v4
PROJECT_ID="<project-id>" # Project ID  of your gitlab repository

GITLAB_URL="${GITLAB_API_V4_URL}/projects/${PROJECT_ID}/packages/terraform/modules/${TERRAFORM_MODULE_NAME}/${TERRAFORM_TARGET_PLATFORM}/${TERRAFORM_MODULE_VERSION}/file"
# ----------------------
echo "Preparing the package."
tar -vczf ${TERRAFORM_MODULE_NAME}-${TERRAFORM_TARGET_PLATFORM}-${TERRAFORM_MODULE_VERSION}.tgz -C ./src --exclude=./.git .

echo "Publishing the package to ${GITLAB_URL}"
curl --location --header "PRIVATE-TOKEN: ${YOUR_TOKEN}" \
    --upload-file ${TERRAFORM_MODULE_NAME}-${TERRAFORM_TARGET_PLATFORM}-${TERRAFORM_MODULE_VERSION}.tgz \
    ${GITLAB_URL}

echo "Terraform module ${TERRAFORM_MODULE_NAME} version ${TERRAFORM_MODULE_VERSION} has been published to GitLab Infrastructure Registry"

The script file is uncomplicated in its functionality, it performs the following tasks:

It creates a tar archive of the ‘src’ folder and compresses it using gzip.
It then publishes the resulting archive file to the Gitlab endpoint.

Upon successful execution of the script file, an HTTP status code of 201 will be returned, indicating that the file has been uploaded.

The published package can be located as illustrated below

Consuming the module

Consuming the module is straightforward. In your terraform code, you would consume the module as shown below. We will talk about finding the parameters for the module in the article that covers module documentation.

module "my_module_name" {
  source = "gitlab.com/transcendent-wisdom/infra-registry/aws" 
  version = "0.0.1" # The version you have published
  
  # Parameters to the module
  bucket_name = "my-bucket-name" # Ignore this for now
  context = var.context # Ignore this for now
  additional_tags = var.additional_tags # Ignore this for now
  force_destroy = true # Delete the bucket even if there are objects in it
}

If your repository is publicly accessible ( in our demo it is), the code provided above will work as is. However, if your repository is private or internal, an additional setting must be configured before running the code.

Private Modules

If you have a private repository and want to consume the module, there are 2 ways to do it.

Setting credentials in the ~/.terraformrc file

As per the documentation, set up your token as shown below in a ~/.terraformrc file

credentials "gitlab.com" {
  token = "<TOKEN>"
}

If you have a self-hosted GitLab instance, your credentials would be as shown

To authorize access to the Terraform registry:

credentials "<my-dedicated-gitlab-instance>" { # where <my-dedicated-gitlab-instance> can be anything like mycompany.com
  token = "<TOKEN>"
}

Note: It is important to keep in mind that the .terraformrc file must be placed in the home directory (~) of the currently logged-in user.

2. Using TF_TOKEN_<domain> environment variable

A better way of setting the credentials would be in your environment variable.

export TF_TOKEN_gitlab_com="<TOKEN>"

For your hosted instance, the environment name would be as shown below

export TF_TOKEN_mycompany_com="<TOKEN>"

where “<TOKEN>” can be any

A personal access token with at least read_api rights.
A CI/CD job token.

Note:

dots (“.”) in your domain would be replaced by underscores (“_”) in the token
hyphens (“-”) in your domain would be replaced with double underscores (“__”)

Considerations

Gitlab follows Terraform’s Module Registry Protocol. It means that the module address should be in the below format
hostname/namespace/name/system
eg: gitlab.com/transcendent-wisdom/infra-registry/aws
When multiple teams are creating packages in an enterprise setting, it is possible for package naming conflicts to arise due to shared hostnames and namespaces among teams. To avoid this, it is recommended that unique names be given to packages. This is important as, otherwise, another team member may publish a package with the same name, but with a different version, leading to confusion.

Conclusion:

Gitlab’s Terraform Package Infrastructure Registry is a powerful tool that can greatly improve the efficiency and automation of managing infrastructure. By using Terraform modules in conjunction with the Gitlab Infrastructure Registry and Gitlab CICD, teams can modularize infrastructure code, and store and manage modules in a centralized location. The use of Gitlab’s Terraform Package Infrastructure Registry can streamline the infrastructure management workflow. Overall, this is a highly recommended approach for managing infrastructure in an enterprise setting.

It is typical for the packages to be uploaded by a CI runner, rather than directly from a local machine. We will see how to do this while automating the package creation using Gitlab Pipelines in a future article.

In the next article, we see how to create documentation to support the module.