Executing Terraform Scripts on CI/CD Pipeline using Self-Hosted Runners
Authors: Niraj Satpute, Rajat Kulkarni.
In this tech blog, we will discuss how to execute Terraform Scripts on CI/CD Pipeline using Self-Hosted Runners. But, before getting into the actual implementation, let’s understand the basic concepts of GitLab and CI/CD.
What is GitLab?
GitLab is a repository hosting manager tool that is developed by Git Lab Inc. and used for the software development process.
It provides a variety of management by which you can streamline collaborative workflows for completing the software development life cycle. It also allows you to import the repository from Google Code, Bitbucket etc.
What is a CI/CD Pipeline?
A CI/CD pipeline automates the process of software delivery. It builds code, runs tests, and helps you to safely deploy a new version of the software. CI/CD pipeline reduces manual errors, provides feedback to developers, and allows fast product iterations.
CI/CD pipeline introduces automation and continuous monitoring throughout the lifecycle of a software product. It involves all stages from the integration and testing stage to delivery and deployment. These connected practices are referred to as CI/CD pipelines.
Continuous Integration, Continuous Delivery, and Continuous Deployment
- Continuous integration is a software development method where members of the team can integrate their work at least once a day. In this method, every integration is checked by an automated build to search for errors.
- Continuous delivery is a software engineering method in which a team develops software products in a short cycle. It ensures that the software can be easily released at any time.
- Continuous deployment is a software engineering process in which product functionalities are delivered using automatic deployment. It helps testers to validate whether the codebase changes are correct, and if it is stable or not.
What is GitLab Runners?
GitLab Runners is a build instance which is used to run jobs over multiple machines and send the results to GitLab, which can be placed on separate users, servers and local machines. You can register the runner as shared or specific after installing it.
In simple words, GitLab Runners helps you run the jobs that you define in .gitlab-ci.yml. A Runner can be a virtual machine, a VPS, a bare-metal machine, a Docker container or even a cluster of containers. GitLab and the Runners communicate through an API, so the only requirement is that the Runner’s machine has network access to the GitLab server.
A Runner can be specific to a certain project or serve multiple projects in GitLab. If it serves all projects, it is called a Shared Runner.
What are Self-Hosted Runners?
As the name suggests, self-hosted runners can be used to host your own runners– you can use your own machine with the Operating System of your choice, and execute git hub actions on those machines.
Why do we need Self-Hosted Runners?
With GitLab Runners, you don’t have full control over the Operating System or the tools. If you want to have more control over the hardware and the Operating System on the tools, you can make use of self-hosted runners.
It also brings flexibility to use operating systems which are not supported by or provided by GitLab Runners. Of course, all the major operating systems like Windows, Mac and Linux are available with GitLab Runners, but in case you are looking for a specific version of Linux which is not available, you could make use of self-hosted runners.
Difference between GitLab and Self hosted Runners
Advantages:
Self-hosted runners offer more control of hardware, operating system and software tools than GitLab-hosted runners provide.
With self-hosted runners, you can choose to create a custom hardware configuration with more processing power or memory to run larger jobs, install software available on your local network, and choose an operating system not offered by GitLab-hosted runners. Self-hosted runners can be physical, virtual, in a container, on-premises, or in a cloud.
You can add self-hosted runners at various levels in the management hierarchy:
- Repository-level runners are dedicated to a single repository.
- Organization-level runners can process jobs for multiple repositories in an organization.
- Enterprise-level runners can be assigned to multiple organizations in an enterprise account.
When not to use self-hosted runners:
Though self-hosted runners bring in a lot of flexibility, they also come with caveats– in the case of public repositories, it’s best not to use self-hosted runners as there’s a possibility that some part of the code may contain malicious code that may hinder the self-hosted runners.
It also enables access to the machine’s network environment because you are enabling the communication between GitLab where the code is hosted, and the machine where you are trying to execute the runners. If there is malicious code, it can access these machines, you may encounter some security risk and you might end up with unwanted data on your machine.
What is Terraform?
In simple words, Terraform helps us to automate the infrastructure and services that run on that platform. It is Open Source and uses declarative language, which means that you don’t have to define each and every step of the automation and how it is done.
For a more detailed understanding of Terraform, please visit our previous blog on the topic.
EXECUTION
1. Installation:
Follow the below commands to install GitLab Runner on Linux Machine:
curl -L “https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.rpm.sh “ | sudo bash
sudo yum install gitlab-runner
service gitlab-runner status
gitlab-runner register
(Enter the url of your GitLab account and select the executor as per your requirements)
(In the below screenshot, the GitLab url and token has been masked)
Installing Terraform
wget https://releases.hashicorp.com/terraform/0.11.14/terraform 0.11.14 linux amd64.zip
sudo yum install unzip
unzip terraform_0.12.2_linux_amd64.zip
sudo yum-config-manager — add-repo https://rpm.releases.hashicorp.com/RHEL/hashi
sudo yum install terraform
2. Build Configuration Files
In order to set up required infrastructure using Terraform on any cloud platform, it is required to build configuration files.
Here is an example of setting up the required infrastructure on a cloud platform.
Please visit this link for your reference for the same (the code provided in the link can be manipulated on any editor).
3. Detailed implementation of CI/CD Pipeline
I. Create a repository and add/upload all the code developed for setting up the infrastructure on the loud platform.
- On the top bar in GitLab, select Menu > Projects
- Select Create New Projects
- Enter the project details and create a README file so that the Git repository is initialized, has a default branch, and can be cloned. Now, select Initialize repository with a README.
II. Configure the security credentials as variables (Go to Side menu -> Settings -> CI/CD)
III. Create .gitlab-ci.yml file
A file called .gitlab-ci.yml in the root of your repository, which contains the CI/CD configuration.
In the file, you can define:
- The scripts you want to run.
- Other configuration files and templates you want to include.
- Dependencies and caches.
- The commands you want to run in sequence and those you want to run in parallel.
- The location to deploy your application to.
- Whether you want to run the scripts automatically, or trigger any of them manually.
Check this link to get the sample .gitlab-ci.yml file.
Parameters description for .gitlab-ci.yml file:
At first, the AWS credentials (Access Key and Secret key) are passed as variables so that connection with the AWS account can be achieved.
Before script is used to define the command that should be run before all jobs, including deploy jobs, but after the restoration of artifacts. This can be an array or a multi-line string.
A Cache is one or more files that a job downloads and saves. Subsequent jobs that use the same cache don’t have to download the files again, so they execute more quickly.
Stage can contain zero, one or more jobs to execute. All jobs in a single stage run in parallel. The next stage is executed only if all jobs from the previous stage complete successfully, or they are marked as allowed to fail.
In the above screenshot, you can see that only one stage (provision) is used where all the Terraform commands are specified, you can provide any name of your choice while defining stages and you can define any number of stages as per the requirements.
Tags are used to select a specific runner from the list of all runners that are available for the project.
In the above screenshot, you can see that “ rr ” is the tag used to trigger the GitLab runner.
Master branch is an initial branch. It’s the first branch to be created automatically when you create a new repository. By default, this initial branch is named Master.
IV. Execute the CI/CD pipeline:
When you are ready with required configuration files and .gitlab-ci.yml file, you can execute the pipeline by committing (at the bottom of each file you can find a button named “commit changes” for executing the pipeline). After committing the changes, runners will trigger the pipeline to get executed and you can observe the pipeline execution by going to the CI/CD pipeline section in GitLab.
When you click on the Pipeline id, you can see the execution in detail.
Once the pipeline gets executed correctly, you can see the above message and you can check the creation of an EC2 instance in your AWS console.
In the screenshot below, we can see that there is an existing ‘EC2’ instance and a new instance(-) is being initialized.
If you want to destroy the infrastructure that you had set up, you can use “terraform destroy -auto-approve“ command in the .gitlab-ci.yml file. The reason why “auto-approve” is used is because generally Terraform expects the user to give ‘Yes’ or ‘No’ as inputs, while executing “terraform apply” and “terraform destroy” commands.
Conclusion
In this tutorial, we set up automated testing with GitLab Self-Hosted Runner for a Terraform project to get started with continuous integration on Gitlab.com.
I hope this blog post helps you in configuring the runners in your infrastructure; let me know about your experiences and happy learning!
