A CI/CD solution in under 10 minutes, using Terraform & Ansible on GCP

As projects mature and complexity increases, integrating a CI/CD solution can make a lot of sense.

However, getting such a system up and running can be a surprisingly technical and time consuming process.

What follows is a hands-on guide for deploying Drone CI, a lightweight, container-centric CI/CD solution, in a manner of minutes as opposed to days.

To accomplish this, we will harness the power of Terraform and Ansible, a powerful combo for automating the creation and and provisioning of infrastructure resources.

We won’t dive too deep into any specific technologies or low level concepts, as there already exists a plethora of resources that address such topics.

Instead, we will take a more pragmatic approach and focus on a real-world scenario, designed to highlight the benefits of automation and provide an insightful glimpse into the world of “Infrastructure as code”.

If however, you favour a more direct approach, head over to Github and follow the accompanying instructions. The whole process shouldn’t take you more than 10 minutes.

In any case, be sure to give me a few claps, or a star on Github, should you find the content useful 🤩

Prereqs

We will be using Google Cloud Provider (GCP), however, the instructions that follow can easily be modified for other cloud providers. If you don’t yet have an account, GCP provides 300$ worth of credits for new users.

We also need Terraform and Ansible for our orchestration and provisioning needs, so be sure to have those installed before proceeding.

Lastly, I’ve opted to use Github, however Drone CI also supports other version control solutions including Bitbucket, GitLab, Gitea and Gogs. Check out the documentation for more information.

Gameplan

We are now ready to begin, here is our game plan:

1. Generate the required infrastructure with Terraform: a single VM, or compute instance in the vernacular of GCP.
2. Add a static IP to our machine and configure basic Firewall settings.
3. Setup an OAuth application on Github.
4. Install any necessary dependencies and run Docker using Ansible Playbooks.
5. Download and run the Drone agent and server images from Docker Hub.
6. Celebrate 🎉 (optional).

Authentication

In order for Terraform to securely interact with GCP, we first need to setup a service account.

Service account

The preferred method of provisioning resources with Terraform is to use a GCP service account, a “robot account” that can be granted a limited set of IAM permissions.

Head over to Google Cloud Console and create a new project or select an existing one.

Under the APIs & Services section in the main menu, click on Credentials. Select Service account key from the Create credentials dropdown menu and fill in the required details. Finally, click create to download the resulting JSON file.

For more information on setting up authentication on GCP, check out the link below.

Getting Started with Authentication | Authentication | Google Cloud

SSH

We also need to grant Ansible programmatic access to our VM by adding a public SSH key to the Terraform metadata.

We can easily generate a pair of keys by running the following command and entering a password when prompted.

ssh-keygen -t rsa -f ~/.ssh/[KEY_FILENAME] -C `whoami`

Let’s also restrict access to our private key to prevent others from modifying it.

chmod 400 ~/.ssh/[KEY_FILENAME]

For more details about creating SSH keys for resources on GCP, check out the following link.

Managing SSH keys in metadata | Compute Engine Documentation | Google Cloud

Terraform

Terraform uses machine-readable definition files to define, create, manage and update infrastructure resources such as physical hardware, VMs, containers and much more. This allows us to create reproducible environments, share configuration with other developers and subject our infrastructure to version control.

Infrastructure as Code

Let’s begin by creating a directory called terraform and adding the following files.

terraform/main.tf

Here, we simply declare a few input variables, and specify our chosen provider.

A provider is responsible for understanding API interactions and exposing resources. Providers generally are an IaaS (e.g. Alibaba Cloud, AWS, GCP, Microsoft Azure, OpenStack), PaaS (e.g. Heroku), or SaaS services (e.g. Terraform Cloud, DNSimple, CloudFlare).

Note that we are referencing the credentials file we generated earlier.

terraform/vm.tf

Now we can move on to configuring the main resources of our infrastructure. First, we specify a boot disk and machine type for our compute instance. Then, we set up some basic Firewall settings and expose the necessary ports. We also need to attach the public SSH key from earlier. Finally, we specify a static IP address and bind it to our machine.

terraform/terraform.tfvars

We also need to declare the actual values for the variables in main.tf. Go ahead and replace the placeholder text with your own values. Be sure to include the credentials file of your service account and public SSH key.

Note that the project name needs to match that of your selected project in GCP, otherwise our credentials file won’t have any effect.

For a complete list of regions and zones, check out https://cloud.google.com/compute/docs/regions-zones/

Orchestration

To initialize Terraform, hop into the terraform directory and run the following:

terraform init

To test out what we have so far, we can use the terraform plan command to get a preview of what resources Terraform will end up creating and/or modifying. If everything looks good, we can go ahead and apply the changes:

terraform apply

If the operation was a success, the resulting output should look similar to this:

Be sure to take note of the IP address of our new compute instance, as we will be using it shortly. For convenience, we can store it in a variable:

export VM_IP=$(terraform output ip)

OAuth

For Drone to eventually be able access our source code on Github, we need to make a quick detour and register an OAuth application. Head on over to Github’s settings page and click on Developer settings.

Select the OAuth Apps tab and click New OAuth App, then fill in the required details.

Note that the authorization callback URL must match the above format and path, and must use your exact server scheme and host.

If you don’t have a custom domain, go ahead and use the IP address of our compute instance.

echo $VM_IP

Ansible

With Ansible, we can easily automate a wide variety of tasks, ranging from application deployment to multi-node orchestration, all without installing any agents or additional software on remote systems.

To accomplish this, Ansible uses the existing SSH daemon in combination with Playbooks, instruction files that help us easily manage configuration and define our orchestration processes.

Configuration as code

We first need to add a reference to our VM in a so called inventory file. You can either edit the default inventory, which you can usually find at /etc/ansible/hosts, or create your own inventory file and reference it with the -i <path> flag.

Go ahead and add the following line.

droneci ansible_host=[IP]ansible_ssh_private_key_file=[SSH_KEY_PATH]

To test out the connection, we can use Ansible’s built in ping command.

ansible droneci -m ping

This should result in an output similar to this.

Environment variables

Before we start writing our Playbooks, we need to add a few environment variables in order to properly configure Drone CI.

Note that the DRONE_SERVER_HOST variable needs to match the domain we specified in our OAuth application on Github, though we do not need to specify the protocol.

Execute the bash script by running chmod +x env.sh followed by source env.sh.

For more information on the variables required to configure Drone, check out https://docs.drone.io/installation/reference/.

Playbooks

We’ll need a dedicated folder for our Playbooks. Let’s nest it within another directory called ansible, like so:

.
├── ansible
│   └── playbooks
├── env.sh
└── terraform
    ├── main.tf
    ├── terraform.tfvars
    └── vm.tf

At last we can start adding our Playbooks.

ansible/playbooks/install-docker.yaml

Our first Playbook will simply install Docker and Docker Compose. We also specify a Python interpreter and install the required system packages.

ansible/playbooks/start-drone.yaml

With everything installed, the only step left is to run the Drone images. To do this we can use Ansible’s built-in Docker Compose module. Notice how we are referencing the environment variables we added earlier.

ansible/playbooks/main.yaml

Finally, we simply add a reference to each file in main.yaml, which will serve as the main entrypoint for executing our Playbooks.

Provisioning

To begin provisioning, run the following command:

ansible-playbook playbooks/main.yaml

If you choose to use a custom inventory file, the same command might look like this:

ansible-playbook -i inventory.tpl playbooks/main.yaml -u jimericskogman — private-key ~/.ssh/drone-ci

This might take a couple of minutes. In the meantime, why not take a glance at your crypto portfolio and gasp in despair 😱.

To verify the success of our recent endeavours, fire up a browser and head over to the domain/ip that we specified earlier. You should be prompted with the following screen.

Once you’ve signed in to Github, you will be redirected to the main dashboard and asked to sync your repositories. Hoorah! 🎉

Conclusion

We’ve made it to the end of a hopefully informative and relatively painless journey. With our CI/CD all set up, the stage is set for us to start building, testing and deploying our code. A topic we will cover in future posts. Until then, I thank you for your time and bid thee farewell!

jadechip/infrastructure-as-code