Building a continuous delivery pipeline with Gitlab CI, Cloudflare and AWS

6 min readMar 5, 2018

Building a continuous delivery pipeline can be a time consuming task requiring full stack development skills and above all patience to test the end -to-end workflow.

Setting up a CI / CD pipeline can be a lot of work up front however the long term benefits pay off in spades:

Reduces risk by using identical environments for test, review and production releases.
Faster development by automating test and review stages.
Infrastructure is managed as code and deployed atomically with the application.
Easily add additional jobs to automate testing other factors such as security or performance.

This guide will demonstrate a functional, proof of concept pipeline using Gitlab CI, Cloudflare and AWS Elastic Beanstalk to take automate the release of a PHP app (PHP is used, but any stack such as Ruby or Node.JS can be embedded into this process) from code into a live web application.

Getting started

To get started you will need the following:

A Gitlab installation with configured to run Pipelines with Docker runners. A hosted version of Gitlab can be use for free at https://about.gitlab.com/products
A Cloudflare account configured with a domain name. Register here.
An Amazon Web Services account. Register here.

Note: This implementation may consume more resources than is allowed within Amazon’s free tier and may add charges to your AWS account.

Why use Cloudflare?

Cloudflare offers many benefits which can improve the performance and security of the application.

Save costs on both review and production apps by caching static content at Cloudflare’s global content delivery network.
Secure review apps with Single Sign On with Cloudflare Access, or alternatively secure to authorised networks with Cloudflare Firewall.

About you:

This guide assumes you have existing knowledge and capability with the following tools:

Basic Git usage such as setting up a repository, pushing, pulling, branching, and merging.
Basic understanding of Gitlab CI. Learn more here.
Basic Amazon Web Services experience with Elastic Beanstalk and IAM Roles.
Knowledge of testing frameworks for your application. This guide does not dive into TDD / BDD frameworks.

Set up a Gitlab project

If you have an existing project you want to extend to support this pipeline, skip this step.

Create new Gitlab project by selecting “New project” on the Gitlab homepage.

Get your Cloudflare API Key

Cloudflare has a rich API which allows developers to automatically manage DNS records. This pipeline uses the API to automatically configure DNS for test and review applications.

To get your Cloudflare API Key login to your account at https://www.cloudflare.com

Drop down your account menu by selecting your email address at the top right of the page.
Select My Profile
Scroll to API Key and select View API Key for Global API Key
Confirm your password and copy your API key.

Set Cloudflare Crypto to Flexible

This guide requires that Cloudflare’s Crypto / SSL Mode is set to Flexible.

Flexible Crypto terminates HTTPS at Cloudflare’s edge, and connects via HTTP to the test and review app.

For simplicity, this guide does not implement HTTPS at on the elastic beanstalk / origin stack. This is not recommended for production environments which should be configured to support HTTPS. A free certificate can be obtained from Cloudflare’s Origin CA, Let’s Encrypt or Amazon Certificate Manager.

Configure CI / CD with credentials

In this step we will configure our Gitlab CI project with the basic settings required to allow Gitlab CI runners to communicate with Cloudflare and Amazon Web Services. This is done by adding Secret Variables to Gitlab which are exposed to the runner as Environment Variables.

Within your Gitlab project browse to Settings → CI / CD, in the CI /CD Settings page select “Expand” on “Secret Variables.”

Add the following variables:

AWS_ACCESS_KEY_ID = Your AWS Access Key ID.
AWS_SECRET_ACCESS_KEY = Your AWS Secret Access Key.
AWS_DEFAULT_REGION = e.g. “us-east-1”. AWS will create resources in the default region. You may specify any other AWS region here.
CF_API_EMAIL = Your Cloudflare account email address.
CF_API_KEY = Your Cloudflare API token.
CF_DOMAIN = Your domain name to use for newly created environments. e.g. “mydomain.net”.

Automated DNS configuration

The CI pipeline automates generation of DNS records using the following pattern:

Test apps use {project_name}-ci{id}.{CF_DOMAIN} e.g. myproject-ci-123.mydomain.com
Review apps use {project_name}-{branch_name}.{CF_DOMAIN} e.g. myproject-stage.mydomain.com

Enable CI / CD in your project using .gitlab-ci.yml

Gitlab CI functions by executing tasks defined in a .gitlab-ci.yml file in the root of your project directory.

An example .gitlab-ci.yml and source repository is available on Github.

This pipeline provides a few key template jobs which are used to build both test and branch apps.

.build-stack: Creates a Cloudformation + Elastic Beanstalk environment.
.deploy-stack: Deploys the application code to the Elastic Beanstalk environment.
.deploy-cloudflare-dns: Updates your Cloudflare account’s DNS to point to the deployed environment.
.destroy-cloudflare-dns: Removes the DNS record for an environment.
.destroy-stack: Destroys the Cloudformation + Elastic Beanstalk environment.

For the test stages the pipeline:

Creates a temporary CI environment.
Configures Cloudflare DNS to point to the temporary CI environment.
Tests the application.
Removes the DNS record and environment after testing is complete.

For simplicity, this guide does not dive into testing frameworks and uses a simple ‘curl + grep’ command to confirm the application returns “Hello world”.
Any testing browser or unit testing framework can be used in this place. Success / failure of the test only requires the selected tool to return an error / success code.

For the release stages the pipeline:

Creates / updates the branch environment.
Creates / updates the branch DNS.
Removes the DNS record and environment when the branch is removed from Gitlab.

Test and Branch apps

This pipeline deploys identical environments in AWS for both test and review apps. The only difference between the two types of applications is a test app is lives only during testing, while branch apps exist for as long as the branch exists in Gitlab.

Warning: Deleting a branch in Gitlab will permanently delete the environment.

Going to production

To make this a full production pipeline, I recommend firstly using the master branch production release and implement a pull request process to restrict who can merge code into the production branch.

Secondly, I recommend configuring .gitlab-ci.yml to exclude the dangerous destroy-branch-app task from the master branch and add a manual trigger for master deployments. See Gitlab CI docs for more information.

Thirdly, I recommend enabling Termination Protection to the master Cloudformation environment to prevent accidental deletion of the production stack.

This framework serves as a foundation to implement a end-to-end developer pipeline using Gitlab, Cloudflare and AWS.

Give it a go, and let me know your experiences.

A full example repository is available on Github.