How we built an auto scaling CICD solution for 57 cents/month

Published in

Helicap Engineering

10 min readMar 13, 2020

In this post, we will be covering how our company, Helicap, uses AWS and Gitlab to create a CICD solution to build and deploy our applications, for a grand total of 57 cents/month.

For those who do not know Helicap, we are a FinTech investment firm specialising in the alternative lending space in the Asia Pacific region. We help people improve their livelihoods by driving financial inclusion and enabling our partner platforms to transform the way consumers and SMEs access credit today.

What is CICD?

To better understand the importance of such a pipeline, we need to delve into the concept of what CICD is.

In software engineering, CICD generally refers to the combined practices of continuous integration and continuous deployment. Continuous integration is the practice of merging all developers’ working copies to a shared mainline several times a day.

What does that mean? You can imagine it as two people building a car. One will build the engine, another person takes the chassis. By being able to constantly merge, integrate, and fix issues that come up, the two people can ensure that the design of the engine and the chassis is able to synergise perfectly constantly. This is as compared to each of them building their own parts in a silo-ed environment, only to find out after a few months, that the engine and chassis built are incompatible with each other.

Continuous deployment is the practice of constantly releasing software. This is extremely important for demoing to stakeholders and business users. To reuse the car analogy, every so often, the engineering team will assemble the latest prototype of the car for the driver to drive. With a tighter feedback loop between the driver and engineers, the team will find it easier to build a better car.

By building a robust and scalable CICD workflow, the business and software teams at Helicap are able to synergise perfectly to deliver quality products for our clients.

Helicap’s CICD Architecture

For our architecture, we have decided to go with Amazon Web Services and Gitlab. This is what our CICD flow looks like:

Typically, when a developer needs to build and deploy their source code, they need build machines to do that. Depending on the source code that needs to be built and deployed, these machines may consume quite a fair bit of resource. However, there is a problem. These machines will not be building and/or deploying source code all the time. In fact, most of the time, these machines will remain idle, until the developers decide they want to integrate and deploy. Even so, we still have to pay for these machines even if they are idle.

And to make it worse, as the software engineering team scales up, we will have multiple developers wanting to build and deploy their source code at the same time. So the team either has to deal with a) new jobs being queued until previous build jobs are done, or b) having to pay for more machines to deal with any demand spikes.

The ideal solution is obvious at this point. Why don’t we purchase those machines only when we need them? We can purchase however many machines we need during demand spikes, and however large it needs to be to deal with the job.

So we reduce our need to a single auto-scaling runner. Our runner is a small 1 CPU core, 1GB RAM machine that listens out for any jobs that require CICD. Upon receiving such jobs, it is responsible for renting larger machines from AWS, that will be responsible for building and deploying the source code. Once the build jobs are done, the machines will be returned back to AWS.

By doing so, we reduce our overhead from multiple, large build machines, to a single small runner, yet still retain the functionality of being to scale to a) build large jobs, b) build multiple jobs at once.

Step-by-step Guide

To set this up, we will be primarily following this fantastic article written by the Gitlab team.

This is written for anyone who has no DevOps background, and would like to just follow a series of steps to set such a CICD pipeline up.

Prerequisites

Amazon Web Services account
Gitlab Account and Project

Setting up EC2 infrastructure

Log into your AWS account, go to EC2 and select ‘Launch Instance’
Select which operating system you are more comfortable with. For the purposes of this guide, we will be choosing an Ubuntu Server 18.04 LTS

3. Next, select the instance type you would want. As this runner scales automatically, you would not want a very large instance. For practical purposes, we will choose a t2.micro instance because it is free to use.

4. Next, follow through on all the steps to create an instance. When you come to this page, we will suggest creating a new key pair for your runner. In this case, we are using a new key pair called ‘gitlab’. Make sure that you store your .pem file in a safe and secure place.

And that’s it! Your machine is now ready to go.

Installing Gitlab Runner

Get the public IP of your EC2 instance, and SSH into your new EC2 instance. The command for our case is:

ssh -i "gitlab.pem" ubuntu@<ec2-public-ip>

2. The first thing we want to do is install the Gitlab runner. So we need to add the official repository:

curl -L https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh | sudo bash

3. Following which, we can install the runner with:

sudo apt-get install gitlab-runner

Installing Docker CE

1. Next, we need to install Docker CE into our machine. The way our runner automatically scales is via Docker Machine, where it will spawn new Docker machines on demand.

So first, we run a quick update.

sudo apt-get update

2. Then we need to allow apt to use a repository over HTTPS:

sudo apt-get install \
    apt-transport-https \
    ca-certificates \
    curl \
    gnupg-agent \
    software-properties-common

3. Then we add Docker’s GPG key. GPG is a tool that apt uses to sign files and check their signatures. In short, this allows you to know that the files you are going to download comes from a secure source.

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

4. Next, we will add Docker’s repository for us to retrieve the installation files. As our AWS instance isusing an x86 or amd64 architecture, we can use the following command:

sudo add-apt-repository \
   "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
   $(lsb_release -cs) \
   stable"

5. We run a quick update.

sudo apt-get update

6. And we install the latest version of Docker CE with the following command:

sudo apt-get install docker-ce docker-ce-cli containerd.io

Optionally, we can test that this is working with:

sudo docker run hello-world

This command downloads a test image and runs it in a container. When the container runs, it prints an informational message and exits.

Installing Docker Machine

We will install Docker Machine that allows our runner to scale up by spawning new machines on demand. The command is as follows:

base=https://github.com/docker/machine/releases/download/v0.16.0 && curl -L $base/docker-machine-$(uname -s)-$(uname -m) >/tmp/docker-machine && sudo mv /tmp/docker-machine /usr/local/bin/docker-machine && chmod +x /usr/local/bin/docker-machine

Getting Gitlab Authorisation Token

Before we can proceed further, we need to obtain the registration token for our Gitlab. This will then connect our Gitlab project to our machine.

Open your Gitlab project, navigate to Settings => CI/CD => Runners and expand the tab.

You should see something like this.

Setting up AWS IAM

We will now be setting up permissions for our runner to be able to programmatically access our AWS console to rent larger machines.

Go to “IAM”, then go to the “User” tab and select “Add a new User”
Give it the following permissions: AmazonEC2FullAccess, and AmazonS3FullAccess. The former allows the runner to rent new EC2 machines to build, and the latter allows the runner to use S3 as a distributed cache where selected directories and/or files are saved and shared between subsequent jobs.

3. Add the user. Take note of the Access Key ID and Secret Access Key!

Setting up distributed cache

Go to “S3”
Create a new bucket. Take note of the name of the bucket, as well as the region that this bucket is created under.

Setting up VPC

Go to “VPC”. There should already be a VPC created along with your EC2 machine. Go to “Subnet” and you should see this.

Take note of the following:

Subnet ID
VPC ID
Region (in this case it is found under the Availability Zone column, ap-southeast-1)
Zone (in this case, it is the alphabet behind, either a, b, or c)

Configuring the Auto-scaling Runner

Phew! We have finally reached the final step! We now need to take everything from we have assembled from before and configure our runner.

sudo gitlab-runner register

When asked for the token, use the registration token you got from Gitlab earlier. When asked for the executor type, enter docker+machine. For image you can just use alpine:latest

2. After all this is done. You can edit the configuration file with:

sudo vi /etc/gitlab-runner/config.toml

3. Edit the file as follows:

https://gist.github.com/yish91/f953bf110460d1dcfbc604ab53ab63e1

There are some points to note for this configuration file.

Under the [[runners]], we have to add in environment = ["DOCKER_TLS_CERTDIR="]. We also have to modify[runners.docker] to add tls_verify = false. This is due to some breaking changes in Docker 19.03.

[runners.cache] is where we configure our distributed cache where we will make use of the S3 that we configured earlier.

[runners.machine] and MachineOptions is where we configured the runner to rent machines. The runner will rent machines under the VPC we created earlier. For amazonec2-instance-type , we decided to go with t3a.small as that is the size that we are most comfortable with. Please upgrade or downgrade where you deem fit. amazonec2-spot-price is set to empty. In this case, the default recommended price by Amazon is used. This is set up in such a way so that we are able to rent our machines immediately, without having to wait for our bid to be matched.

4. Restart our runner with:

sudo service gitlab-runner restart

For you to debug and see service logs, you can use the following command:

journalctl -f

5. Navigate back to your Gitlab Runner’s page, and you should be able to see your runner with a green light next to it. You are now ready to use the runner!

Cost Benefit Analysis

For anyone who has been using Gitlab, you will notice that Gitlab does provide some default runners for users to use. However, as of the time of writing, you are only allowed 2000 free minutes. As a startup eventually starts to scale up, the CICD pipeline will become a critical bottleneck.

Hence, our engineering team here at Helicap has decided to experiment with implementing this auto-scaling runner sometime last year.

To our surprise, our final overall cost was extremely small. The image below is taken from our AWS billing from last month.

Auto-scaling Runner February 2020 Costs

Given that we do not have to pay for the t2.micro machine, our costs only comprises of the uptime of the t3a.small machines that we spin up for our builds.

As of the month of February 2020, we have approximately 4760 minutes of build time, and our final overall cost is USD 0.57.

To have an idea of how much cost savings you can get, we tried to compare it with the subscription plans of other CI solution providers, namely Gitlab, Travis CI and CircleCI. We selected the plans based on the number of concurrent jobs the plan allows, as well as the number of build minutes.

CI Solution Providers Benchmark as of March 2020

Please take note that the other CI solution providers also have a bunch of other features included in their subscription plans beyond what our autoscaling runner can provide, so this benchmarking is not an exact one-to-one value comparison. But if you are a startup simply looking for a CI solution that can offer you concurrent builds with a large number of build minutes, you will probably be looking at these plans as a baseline, even without all the bells and whistles that these plans will bring to the table.

Conclusion

Bootstrapping a startup is an extremely difficult task to accomplish. However, despite limited resources, there are a few things that are absolutely essential for a modern technology startup to harness and multiply the value of their team. A robust CICD pipeline is one such thing.

Thank you for reading this article, and we hope that this helps to provide some insights into implementing a scalable CICD solution at a minimal cost.