Golden Path Approach for CI/CD Using Drone.io
Introduction
In the rapidly evolving landscape of software development, the ability to continuously integrate and deploy applications is crucial for achieving shorter development cycles, faster time-to-market, and improved overall software quality. However, implementing effective and efficient continuous integration and deployment (CI/CD) processes can be a daunting challenge, often plagued by complex pipelines, manual configuration, and inconsistent practices across teams.
Recognizing the need for standardized CI/CD pipelines, the concept of “Golden Path,” also known as “Paved Road” from Netflix, has gained traction in the industry. This approach emphasizes the creation of predefined, reusable, and easily configurable pipelines that serve as the foundation for consistent CI/CD workflows.
The GumGum Verity™ engineering team has been an early adopter of Drone.io to handle CI/CD workflows and application deployments. While being happy with the flexibility that Drone provides through the use of containers to incorporate pre‑configured steps into a pipeline and the ability to write custom plugins, it was time to rethink certain aspects of the workflow to address some paint points the team was facing:
- How can we keep pipelines DRY and prevent copy pasting from projects to projects?
- How can we decouple CI / CD from application deployment using a promotion rollback mechanism?
- How can CI / CD deal with multi-arch support arm vs linux/amd64 or even the need of specialized hardware like a GPU while remaining cost efficient?
In this blog post, we will discuss the reconstruction of our Drone ecosystem, revolutionizing our CI/CD processes within the company. You will learn how we leveraged the versatility and flexibility of Drone and harnessed its potential to create an intuitive and standardized pipeline framework. By embracing the “Golden Path” approach, we envision a future where developers can effortlessly navigate the CI/CD landscape, focusing on delivering innovative features and enhancements rather than getting entangled in intricate pipeline configurations. Our goal is to provide a comprehensive solution that expedites software delivery, enhances collaboration, and elevates the overall quality of our applications.
📦 Pipelines and Building Blocks
As your team grows and the number of projects increases exponentially, it can become tedious to maintain consistent CI/CD pipelines across the board.
While Drone provides the ability to configure pipelines as code using a simple YAML file called .drone.yml that you commit to your git repository, it does not offer an easy way to reuse well-established building blocks to compose your pipeline.
As a result you may find yourself copy-pasting steps from other pipelines to craft your own, which is far from being ideal, especially if you make use of Drone environment variables to write generic steps that will be 100% identical.
Here is an example of a generic step to publish a Docker container to the AWS ECR registry that can be reused across many projects as-is. The registry name is derived from the Git repository name, and the image tag will be associated with a Git tag (if any) or fallback to a short Git SHA (8 digits).
### Generic step that can work for many projects with no edit required
- name: Publish Container to ECR
image: plugins/ecr
settings:
repo: "12345.ecr.amazonaws.com/namespace/${DRONE_REPO_NAME}"
tags:
- "${DRONE_TAG:-${DRONE_COMMIT_SHA:0:8}}"If we extrapolate this previous idea to other CI / CD building blocks, there is a good chance that the same team will reuse the same frameworks and the same build tools or dependency managers across projects, providing us with more opportunity to standardize workflows.
By mapping out your global CI/CD footprint you will be able to identify common patterns and building blocks that are necessary for creating your end to end pipelines, and ultimately your “Golden Paths”.
➰ Use Templates And Keep It DRY
Once you have mapped out the CI/CD building blocks, it’s time to think about templates. Drone offers an out-of-the-box solution to leverage templates which can be written in plain YAML, using Jsonnet or Starlark (a dialect of Python originally developed for the Bazel build tool).
We chose to retain Starlark as a template engine due to its close proximity with Python, which makes it really easy for newcomers to pick up. Now, it is time to stitch the pipeline pieces together.
In an ideal world, building a “Golden Path” for a Python project using templates should resemble something like this:
- CI Pipeline — Use the Python Building block to run CI steps
- Delivery Pipeline — Use the Docker building block to release the Python project as a container
- Deployment Pipeline — Use the Gitops Workflow building block to deploy the new version
According to the Drone templates documentation, templates can be used by setting the kind to template, the name of the base template to load, and a set of freeform template inputs in the drone.yml file:
We may be tempted to craft our “Golden Path” this way:
### - - - - - - - - - -###
### Python Golden Path ###
### - - - - - - - - - -###
### Continuous integration for Python projects using Tox
---
kind: template
load: python_ci.starlark
data:
image: python-tox:v3.10
### Continuous Delivery for Docker images
---
kind: template
load: container_cd.starlark
data:
registry: "12345.ecr.amazonaws.com/namespace/${DRONE_REPO_NAME}"
### Continuous Deployment to K8s using Gitops
---
kind: template
load: gitops.starlark
data:
artifact: "12345.ecr.amazonaws.com/namespace/${DRONE_REPO_NAME}"
application: python-demo-appUnfortunately this is where you may become greatly disappointed 😱 when you realize that Drone only considers the first template and ignores the rest. After a few minutes googling, you land on this Github proposal: #27 Allow using multiple templates per .drone.yml file. This renders the built-in templates provided by Drone unusable in our scenario.
🛟 Drone Extensions To The Rescue
Drone templates still remain the best option for implementing “Golden Paths”. Additionally, thanks to Drone extensibility, you can use a conversion extension to modify a .drone.yml configuration file before it gets parsed and processed by the system.
This is where you land on the Drone Convert Starlark conversion plugin and hope your dreams come true … 🙏
An interesting aspect of this project is to support the loading of other templates from the file system (to address the multi template import issue mentioned earlier). The only caveat here is to ensure that the templates are present on disk for the convert plugin to load.
If you treat your CI/CD templates as a separate project (which I believe you should, as it enables version control, deployment and rollback) it is relatively straightforward to package your custom templates in the Drone Convert Starlark plugin using the following Dockerfile:
FROM drone/drone-convert-starlark
### Make Starlark templates accessible using: load("@custom//<file.star>")
ENV DRONE_STARLARK_REPO_PATHS=custom:/var/lib/drone/starlark/templates
COPY templates /var/lib/drone/starlark/templatesThis component is relatively simple to deploy on a Kubernetes cluster as it is a standalone component with no additional dependencies. Here is an infrastructure diagram showing the integration of the conversion extension:
Let’s see what a “Golden Path” could look like with this new setup:
### Python Golden Path ###
load("@custom//integration:python_ci.star", "ci")
load("@custom//delivery:container_cd.star", "cd")
load("@custom//deployment:gitops.star", "deploy")
def main(ctx):
return (
ci(
image="python-tox:v3.10"
)
+ cd(
registry="12345.ecr.amazonaws.com/namespace/${DRONE_REPO_NAME}"
)
+ deploy(
artifact="12345.ecr.amazonaws.com/namespace/${DRONE_REPO_NAME}",
application=python-demo-app
)
)🌟 It’s Time To Build Golden Paths !
In the previous section, we showcased how to create a Drone pipeline using reusable building blocks and templates. Now, it’s time to organize the Drone template Git repository, which will serve as the hosting location for all the Starlark templates
Here is a suitable folder structure for building reusable blocks:
- Golden Path — Assemble different set of pipelines to run an end-to-end workflow from CI, delivery to deployment by stitching smaller building blocks together (eg. Python to K8s / Python to Lambda / Webapp to Cloudfront)
- Pipelines — Set of Drone pipelines associated with their respective triggers (execute on push / on tag / on pull-request / promotion or rollback) that executes one or more workflows (Python CI workflow / Docker container workflow + security scanning workflow)
- Workflows — A sequence of steps to execute to achieve a given goal (A CI workflow commonly runs linting / checkstyle / unit tests)
- Steps — A configurable Drone step that calls a certain Drone plugin or run simple shell commands inside a container (Publish container / Run Gradle version X / Execute AWS CLI commands / Cache artifact)
👉 Tips: The Drone Cli makes it easy to develop and test your golden paths! Simply start the Drone Convert Starlark container locally, mount your templates as a docker volume, and run the following command to validate your code:
# Exectute the drone-cli against a local server to ensure proper rendering
# of .drone.star -> .drone.yml
drone plugins convert \
--repository test \
--endpoint http://localhost:3000 \
<path>/<to>/.drone.star✂️ Decoupling CI / CD from Deployment
In our previous implementation of Drone, we tightly coupled the CI/CD steps of a project with its deployment based on a set of Git events:
- Code pushed to a dev branch runs CI steps
- Code changes merged to the main / master branch triggers CI + CD + Deployment to staging environment
- Applying a tag to a commit triggers CI + CD + Deployment to production environment
The downside of this approach is that it creates a tight coupling between CI / CD and deployment of your application. In the event of a deployment failure and the need for a rollback, the team is left with a single option: Restarting the build of the previous stable version to restore the service. This results in a full replay of the CI + CD + Deployment, while what is truly needed in this case is to rerun the only deployment part of the pipeline.
Drone provides and interesting feature called promotions/rollbacks that can be leveraged to take a previous build and deploy it in a given environment (called target)
It is now possible to update the deployment pipelines to use a promotion mechanism as part of its triggers:
- Only code from main / master branch or a tagged commit can be deployed to the staging environment
trigger:
event:
include: [ promote, rollback ]
target:
include: [ staging ]
ref: [ refs/heads/master, refs/heads/main, refs/tags/** ]- Only Code from a tagged commit can be deployed to the production environment
trigger:
event:
include: [ promote, rollback ]
target:
include: [ production ]
ref: [ refs/tags/** ]💰 Cost Effectiveness & Specialized Hardware
With the rise of AWS Graviton instances and ARM based systems, which provides better cost-to-performance ratio compared to their x86 counter parts, or even the potential need to run a pipeline on an expensive GPU instance, it is legitimate to ask ourselves if our CI / CD environment is ready to build on specialized hardware in a cost effective manner.
Drone Autoscaler is a standalone daemon that continuously polls your build queue and provisions or terminates server instances based on volume or pipeline needs.
When used in combination with pipeline routing, you can achieve just-in-time provisioning of runners to execute your pipeline on specific hardware and shutdown when they become idle.
This brings us to the final infrastructure diagram for this new Drone setup which includes all the components put together, including the persistent layer to run Drone in a HA fashion
👏 Wrapping up
The revamp of our build platform marks a significant milestone in our continuous integration and deployment journey. By embracing the concept of “Golden Paths” and standardized pipelines, we have paved the way for streamlined CI/CD practices within our team.
With the “Golden Path” platform we unlock new possibilities for scalability, maintainability, and reusability, ultimately leading to enhanced productivity and innovation.
Together, let us embrace the power of standardized pipelines, paving the way for a more efficient and seamless software delivery experience !
