Installing Open Source CI/CD tool Drone using Docker Compose

Running Drone with Docker

There are various open-source CI/CD tools like Travis CI, Circle CI, Jenkins, Gitlab, Github Actions, Drone, etc, but comparing all features and licensing options (open source applications also may require licenses at times) , we chose to go with self-managed Drone at Gradeup.

Why we preferred Drone over others

• It is open-source (although requires a license to run more than 5000 builds per month)
• It is written in Go which is trending for its performance and low resource consumption
• Ability to run on-demand workers in Kubernetes as Jobs (not officially supported by Drone for community users)
• Simple & Easy Configuration (we will come to it)
• An easy option to create custom Plugins (much needed)

How we use Drone

Initially, we deployed Drone in Kubernetes and set up auto-scaling of workers which was quite easy as per provided documentation. But months later, we started facing issues like some pipelines stuck in running state for days (even after the timeout). We suspected this might have been because of AWS Spot Instances being used. Later we decided to go with a single EC2 Instance running Drone server as well as Workers (limited no of workers). It works well on a single node (t2.large: 2 Core 8 GB Ram) and doesn’t add up much wait time in a developer’s day to day task; so we continued using EC2 without auto-scaling.

Setting up Drone using Docker-Compose

Prerequisites

• Install Docker and Docker Compose on Server
• Get Github Client ID and Secret: Login to Github and visit register a new OAuth application. Choose a name for your application (drone-ci). Provide Homepage URL as https://drone.example.com and authorization callback as https://drone.example.com/login. You can also use public IP for testing purposes.
• Point domain name to the Server IP (if using domain instead of IP)

And that's all for prerequisites! Let’s now jump to the main crux which is a mere 5 minute task!

Create a docker-compose.yaml file with the following snippet:

Some points to note in this config:

• To allow only members of a particular Github Organization to access the service, use DRONE_USER_FILTER. If you want to allow some other users as well, add a comma-separated list of usernames and organisation names
• Drone-server is the main backend which listens for Triggers and schedules workers; drone-agent are workers that do the work you requested for
• Never copy DRONE_SECRET and DRONE_RPC_SECRET from anywhere if copying the docker-compose or configs from any sites
• DRONE_SECRET and DRONE_RPC_SECRET should be the same in drone-server and drone-agent
• Setup DRONE_RUNNER_CAPACITY based on the number of CPU Cores available (although even 20–30 workers do work on a 4 core machine, we observed better performance limiting to 4 workers in parallel)
• Go for a License if your usage exceeds the Open Source limit
• Feel free to play around with the environment variables in local/staging env, but double-check config before going to production for Secrets and Public Access

Adding CI Pipelines in Git Repo

To use Drone for CI/CD, we first need to activate the repository from the Drone dashboard, and by default, Drone looks for a .drone.yml file in the repository. We can change the name of CI file as well as timeout for pipelines.

Repository settings for Drone CI

A CI configuration file can have multiple pipelines, and each pipeline can have one or multiple steps. Every step of a Pipeline runs based on certain conditions (if provided) and performs tasks it is configured for. Here is a sample config file. (Don't get scared away by the length of this file, I have tried to include all configuration options that you might need! :D )

The example contains a single pipeline with two steps: one to build and publish a docker image to ECR, another to update build status on Github using a plugin drone-github-comment (completely optional step)

Secrets can be managed from the Drone dashboard per repository with the option to make it accessible by pull requests or not. Instead of using environment for all variables and data sent to the plugins (as used in example), drone has currently shifted to using settings to pass data into plugins, and any variables starting with DRONE_ are no more required to pass into plugins- they are made available to the plugins automatically by drone. Plugins configurations have also been updated on Drone Plugin Index with settings in place of environments.

Some plugins that may be useful for your teams:

1. Slack Notification: This plugin can be used to send pipeline status updates to Slack channels
2. ECR: For teams using private repositories on AWS, docker builds can be pushed to ECR

There are plugins to help in CD processes like SSH, Kubernetes, Terraform, etc. The complete list of plugins can be found on the plugins index.

At times, developers have some requirements which are not available in any plugins already out on Github; with Drone we can easily work this out because of a much simpler process to build and use a custom plugin. Drone Github Comment is one such plugin I tried building and it was super easy (as well as fun :D ), and there are many more created by various open-source contributors, just needs a little digging up Google for that!