Deploy your Components with Terraform and Gitlab CI in Air-Gapped-Environments (Terraform CE)

Photo by JJ Ying on Unsplash

When using Terraform in Air-Gapped environments there are several challenges that you have to overcome. The following article describes how you setup and use terraform in an air-gapped-environment.

Why should I use Terraform in Air-Gapped Environments?

• Orchestrate your Applications on Kubernetes
• Build your Infrastructure as Code
• Avoid human errors by documentation of your infrastructure as code, have similar environments

Requirements for using terraform

For using terraform it is necessary to import several resources first.

• What Providers do I need for my Terraform Project?
• How do I want to execute Terraform?
• Import of Providers -> Providers available at https://releases.hashicorp.com/

Prepare your Terraform Workspace

Terraform is downloading its providers' files at runtime. In an air-gapped environment, it is necessary to build a “provider-mirror” and reference to the providers folder.

Terraform can be configured by using a Config-File (.terraformrc). (For more read https://www.terraform.io/cli/config/config-file). In the config file, you can define a filesystem mirror, where you can overwrite the path of the terraform registry. In my example, i will overwrite it to /usr/share/terraform/providers.

After defining the .terraformrc file, the File must be placed in the users-home-directory. In a Gitlab / Github-Pipeline you would simply move the file to the user's home directory.

# copy terraform rc file to users home directory (can also be automatically executed in the pipeline) 
cp ./.terraformrc ~/.terraformrc

Import Air-Gapped-Resources

For running terraform in a Gitlab-Runner it is necessary to import the following:

1. Provide Terraform-Container in an air-gapped docker-registry

You have to provide the latest terraform version in your docker-registry, which is accessible by the GitLab-Runner-Server.

Here is an example, how you pull your docker image from the docker-hub-registry and push it to your custom registry:

# Image available at https://hub.docker.com/r/hashicorp/terraform
docker pull hashicorp/terraform:latest
docker tag hashicorp/terraform:latest <yourregistryaddress>/hashicorp/terraform:latest
docker push <yourregistryaddress>/hashicorp/terraform:latest

2. Provide Terraform Providers in Git-Repository

After that, download the needed providers at https://releases.hashicorp.com/ and move it to the following structure:

./providers_source/registry.terraform.io/<provider>/<plugin-name>/<version>/<filename>/

For example for the kubernetes provider (available here: registry.terraform.io/providers/hashicorp/kubernetes/latest/doc) you need to execute:

# set plugin version
export PLUGIN_VERSION=2.11.0
export PLUGIN_FILE_NAME=terraform-provider-kubernetes_${PLUGIN_VERSION}_linux_amd64.zip
# download terraform kubernetes provider
wget https://releases.hashicorp.com/terraform-provider-kubernetes/${PLUGIN_VERSION}/${PLUGIN_FILE_NAME}
# move to directory
export FILEPATH=./providers_source/registry.terraform.io/hashicorp/kubernetes/${PLUGIN_VERSION}/
mkdir -p ./providers_source/registry.terraform.io/hashicorp/kubernetes/${PLUGIN_VERSION}/
# move plugin to target directory
cp ${PLUGIN_FILE_NAME} ./${FILEPATH}/${PLUGIN_FILE_NAME}

The provided script moves the terraform provider to the directory “providers_source”. If you do not have an additional file mirror in the network, you can provide the providers in the git-source repository (not recommended — but easy to use for the beginning!).

Building the Gitlab Pipeline

After the terraform-container is provided in the private registry and the needed providers are available in the air-gapped-environment you can start building the pipeline.

In this example I will describe a pipeline with 4 stages:

1. Terraform Project Initialization (terraform init)
2. Terraform Project Validation (terraform validate)
3. Terraform Change-Planning (terraform plan)
4. Terraform Deployment (terraform apply)

For the pipeline I first defined some basics that can be included in the following gitlab-ci-jobs:

# change to your custom private registry
image: hashicorp/terraform:latest
terraform_base:
  variables: 
    TERRAFORM_PROJECT_DIR: "./terraform"
  before_script:
    # copy terraform providers to providers path
    - mkdir -p ${TERRAFORM_PROVIDER_PATH}
    - cp -R providers_source/ ${TERRAFORM_PROVIDER_PATH}
    # add execution permissions
    - chmod -R +x ${TERRAFORM_PROVIDER_PATH}
    # apply terraform settings
    - cp .terraformrc ~/.terraformrc
    # move to terraform project dir
    - cd ${TERRAFORM_PROJECT_DIR}
# Cache modules in between jobs, if cache is not configured in gitlab runner, use artifacts!
  cache:
    key: $CI_COMMIT_REF_SLUG
    untracked: true
  paths:
    - ${TERRAFORM_PROJECT_DIR}/.terraform/

After that, the jobs for the multiple stages can be applied:

In the end, after adding my terraform project to /terrraform/, the directory structure of my terraform project looks like this:

.
├── Readme.md
├── download_providers.sh
├── providers_source
│   └── registry.terraform.io
│       └── hashicorp
│           └── kubernetes
│               └── 2.11.0
│                   ├── registry.terraform.io
│                   │   └── hashicorp
│                   │       └── kubernetes
│                   │           └── 2.11.0
│                   └── terraform-provider-kubernetes_2.11.0_linux_amd64.zip
└── terraform
    ├── main.tf
    └── provider.tf

By applying all these changes, our terraform-project with the kubernetes-provider (in /terraform folder) can now be applied to our air-gapped kubernetes environment!

The code can be reviewed at https://github.com/andrisro/terraform_airgapped_example