Introduction to Drone.io CI/CD platform Part 3
Introduction
This is the third and last part of this “Introduction to Drone.io CI/CD” series. On this last article, we are going to discuss some last important topics to complete this Drone overview, like:
- The Drone command line interface
- Kubernetes’s secrets integration
- Build promotions and rollbacks
- How to do debugging
Like in the previous articles, we will demonstrate all these features with detailed instructions on how to set them up and a hands-on lab.
Prerequisites
Before continuing, remember to go through the previous articles (part one and two), to get the lab setup up and running to be able to perform the tasks from this article. As a quick recap, if you have been following these article’s series from your laptop and have already done all the needed setup and want to get it running for this article, you need to:
- Start ngrok and take note of the assigned hostname.
- Go to your GitHub OAuth application settings page and update the Homepage URL and Application callback URL with the new ngrok hostname.
- Go to the repository webhook settings and also update the hostname with the new ngrok hostname.
- Start minikube cluster (if you have done the previous labs, you should have it ready with Drone already installed and configured, and the example guestbook application deployed).
Let’s also review the commands needed to get the lab ready to continue with this article’s lab. First, run the following kubectl command to expose the Drone server using port forwarding:
$ sudo kubectl — namespace drone port-forward svc/drone 80:80
Access http://localhost, and you should have access to the Drone web interface. The .drone.yaml file is located at the root of the kubernetes-engine-samples repository. Inside this repository, we are using the guestbook application for this lab. The Kubernetes manifests are inside that folder, and the frontend code is inside the php-redis folder. To access the frontend application, you can use the minikube service command:
$ minikube service frontend -n guestbook-demo
Now we are ready to continue!
Drone Command Line Interface (CLI)
Drone provides a command line interface program to interact with a Drone server remotely. This CLI can be used to perform administrative tasks on the server, like viewing logs, managing users, secrets and pipeline templates. It can also be used to manage repositories, build jobs, cronjobs or execute local builds. In this section, we are going to review some of those tasks.
Installation and Configuration
The installation of the CLI program is fairly easy, basically, download your platform’s binary to a directory in your user’s path and give it execution permissions. For example, for Linux you can use these commands:
$ curl -L https://github.com/harness/drone-cli/releases/latest/download/drone_linux_amd64.tar.gz | tar zx$ sudo install -t /usr/local/bin drone
For other platforms, look at the official installation instructions.
Usage
After the binary is installed, it needs to be configured to connect to the Drone server using environment variables. First, export the Drone server address:
$ export DRONE_SERVER=http://drone.mycompany.com
For this lab, the server address would be the ngrok hostname. Then, export the Drone personal token:
$ export DRONE_TOKEN=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9…
Remember, the personal token is obtained from the user profile page in the Drone server web interface. After the previous variables have been exported, the CLI command can be executed. To test the CLI command, run this:
$ drone info
User: yourgithubusername
Email: juan@youremaildomain.com
Kubernetes Secrets Integration
Drone.io can be integrated with many external secret storage tools, and allow the secrets stored in those tools to be used in a pipeline. With this feature, Kubernetes’s secrets can be used instead of Drone secrets. For Drone to be able to access Kubernetes secrets, three things needs to be done:
- Install the Drone Kubernetes secrets extension
- Create the Kubernetes secrets
- Add the secrets resources to the Drone pipeline to access the Kubernetes secrets.
Extension Install
The extension can be installed too using a Helm chart. First, the extension needs to be configured with a Helm values.yaml file, like the following one:
Replace the value of SECRET_KEY with your own secret. You can create a random one with the openssl command:
$ openssl rand -hex 16
a6ca63b531427e3165224112e02b1697
Also, as you can see in the previous example, the extension will look for secrets in the default namespace, so if you want to use a different one, then the values of secretNamespace and KUBERNETES_NAMESPACE must be updated with the namespace to be used. Look at the documentation for other variables to set additional to the shared secret.
Save this file as drone-kubernetes-secrets.yaml, and then install the extension with the following Helm command:
$ helm install — namespace drone drone-kubernetes-secrets drone/drone-kubernetes-secrets -f drone-kubernetes-secrets-values.yamlNAME: drone-kubernetes-secrets
LAST DEPLOYED: Wed Jul 20 15:22:50 2022
NAMESPACE: drone
STATUS: deployed
REVISION: 1
TEST SUITE: None
To check the extension was successfully installed, check the pod logs:
$ kubectl — namespace drone logs \
-l ‘app.kubernetes.io/name=drone-kubernetes-secrets’ \
-l ‘app.kubernetes.io/component=drone-kubernetes-secrets’time=”2020–01–29T00:02:18Z” level=info msg=”server listening on address :3000"
If you see in the last line a message saying “server listening on address :3000”, the extension is working as expected. The next step is to update the runners and configure them to use the Kubernetes secrets extension. For this, the runner’s Helm values file needs to be updated, adding the following configuration variables in the env: section of the file:
DRONE_SECRET_PLUGIN_ENDPOINT: http://drone-kubernetes-secrets:3000
DRONE_SECRET_PLUGIN_TOKEN: your-shared-secret-value-here
Replace the value of DRONE_SECRET_PLUGIN_TOKEN variable with the secret value configured previously during the Kubernetes secrets installation. Leave the value of DRONE_SECRET_PLUGIN_ENDPOINT unmodified. Then, update the runner’s install using Helm:
$ helm upgrade — namespace drone drone-runner-kube drone/drone-runner-kube -f drone-runner-kube-values.yamlRelease “drone-runner-kube” has been upgraded. Happy Helming!
NAME: drone-runner-kube
LAST DEPLOYED: Wed Jul 20 15:26:32 2022
NAMESPACE: drone
STATUS: deployed
REVISION: 2
TEST SUITE: None
Kubernetes Secrets Creation
Now, create the Kubernetes secrets using kubectl. We are going to create all the secrets in use in the guestbook example application pipeline used in the previous article:
- Docker Hub credentials
- The minikube parameters (CA certificate, server address and token)
This manifest file can be used as a template to create the secrets:
All values need to be base 64 encoded. To encode simple values, the following command can be used:
$ echo -n value | base64
For example,
$ echo -n juanluisbaptiste | base64
anVhbmx1aXNiYXB0aXN0ZQ==
Use that command to encode the docker hub username and password, and the minikube server address. Note the use of the -n parameter to avoid adding an end of line, this is crucial.
For the minikube CA certificate we can refer to the previous article as we already encoded it before with this command:
$ cd ~/.minikube
$ cat ca.crt|base64 -w 0
Note the use of the -w 0 parameter, it is used to disable line wrapping, so we get the encoded value in a single line, needed because of the length of the certificate value.
To encode the minikube token, we can also use the command given in the previous article to obtain its value, but with one small modification. Before, the token value was already base 64 encoded, and was decoded when extracted, so the command will be modified to avoid the base64 decoding:
$ kubectl get secret `kubectl get secret -n kube-system|grep default|awk ‘{print $1}’|grep -v NAME` -o go-template=’{{ .data.token }}’ -n kube-system
S0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURCakNDQWU2Z0F3SUJBZ0lCQVRBTkJna3Foa2lHOXcwQkFR
[…]
After all values have been encoded, copy-paste them in the previous secrets manifests file and save it as kubernetes-secrets.yaml. Then, create the secrets with this command in the default namespace:
$ kubectl apply -f kubernetes-secrets.yaml -n default
secret/docker configured
secret/k8s configured
Add Secrets Resources to Pipeline
The last thing to do is to add the new secret resources to the Drone pipeline for the new Kubernetes secrets. Add the following secret definitions at the beginning of the .drone.yml file:
With those secrets resource definitions, the secret values used by the pipeline will be fetched from the Kubernetes secrets that were previously created.
Pipeline Promotions
Pipeline promotions is a very cool feature that can be used to move or “promote” a build output to a target environment. Promotions allow deploying code already built in a pipeline into different environments (dev, test, staging, prod, etc.), by running a new pipeline that only executes the deployment steps. These environments would be the infrastructure where applications should be deployed to be used, for example, a Kubernetes cluster. These image promotions have many advantages:
- Create repeatable deployments: be confident that every time a build is promoted to a different environment, it will always be the same code being deployed.
- Improve the release times of new versions: When using promotions, the image with the new application code is built once and then moved around between environments until it ends up in the desired environment with the least amount of friction between development and operation teams.
- Create an audit trail of versions being deployed: Know which image versions have been deployed on each environment at any time.
- Segregation of duties: Let the QA team promote a new version between environments after their quality tests have passed, instead of having the operations team do it when QA needs it. This way, development teams can make new application versions available in a versioned manner, and QA and release managers can test and release these new versions to production when they feel confident, at their own pace.
- Easily roll back a promoted image version to a previously one known to be working from a previous build.
How promotions work
When promoting a build, Drone launches a new build of the application’s pipeline. This new build has its own build number, and inherits all the build parameters from the build being promoted. The new build will have its event type set to promote, which can be used to target a deployment environment in your infrastructure by using it in a trigger condition in the pipeline.
The Drone pipeline should define the steps needed to deploy the application somewhere, and then, the deployment target can be selected using the promote event type in a trigger (at pipeline level) or a when condition (at step level), combined with any other desired conditions.
From the previous article, we already defined a pipeline that deploys to a local minikube cluster, so we will continue from there to demonstrate this feature.
Promoting a Pipeline
Continuing with the lab from the previous article, there are two tasks needed to be done to use the pipeline promote feature. First, we have to modify the Drone pipeline and split it in two different pipelines: one for building and tagging the image and another for deploying it to a specific environment. Second, the application’s frontend manifest file has to be modified to deploy the image tag that was built and tagged with a git tag in the build pipeline. This also will allow us to do version rollbacks to specific versions from previous deployment pipelines.
To split the pipeline, add a new pipeline definition, give it a name and move the deployment steps to it. This is an example of how the file should look, but you can see the finished file here:
[ … secrets definitions here … ]
---
kind: pipeline
type: kubernetes
name: buildsteps:
- name: Build docker image
image: plugins/docker
settings:
repo: juanluisbaptiste/guestbook-demo
dockerfile: guestbook/php-redis/Dockerfile
context: guestbook/php-redis
username:
from_secret: docker_username
password:
from_secret: docker_password
tags:
- latest
- ${DRONE_BUILD_NUMBER}
when:
event:
- push
- pull_request
- custom
- name: Tag docker image
image: plugins/docker
settings:
repo: juanluisbaptiste/guestbook-demo
dockerfile: guestbook/php-redis/Dockerfile
context: guestbook/php-redis
username:
from_secret: docker_username
password:
from_secret: docker_password
tags:
- latest
- ${DRONE_TAG}
when:
event:
- tag
---
kind: pipeline
type: kubernetes
name: deploysteps:
- name: Deploy frontend deployment
image: danielgormly/drone-plugin-kube:0.2.0
settings:
image_tag: ${DRONE_TAG}
template: guestbook/frontend-deployment.yaml
namespace: guestbook-demo
ca:
from_secret: k8s_crt
server:
from_secret: k8s_server
token:
from_secret: k8s_token
trigger:
event:
- promote
- rollback
target:
- production[ … rest of deployment steps … ]
I have highlighted in bold the new parts. Notice that we added another step to tag the image in the build pipeline. How it is defined is not the optimal way to do it as the image is being built twice (the image should be built in the build step only, but it is enough to demonstrate this feature). Also notice this new step has its condition defined to trigger with the tag event, and in the tags section, there’s a tag that uses the ${DRONE_TAG} variable. This variable contains the value of the git tag that triggered the build.
Now, onto the deployment pipeline. To keep things simple, we are going to have only one environment where to deploy the example application. Normally, a project would have at least a staging and production environments, but for this demonstration with only production will suffice.
In the deploy pipeline, notice that on the first of the deployment steps, the one that deploys the frontend application, we are defining a variable named image_tag that receives the ${DRONE_TAG} variable. This variable is passed to the frontend application manifest and used as a template variable. The trigger for this pipeline defines two events: promote and rollback, and adds another trigger called target. This target trigger is used to define the environment for which the pipeline (or step if defined at the step level) will be executed,. On his example, the target is the production environment.
Now, the frontend application manifest file has to be modified to use this variable:
The Kubernetes deployment plugin can parse the manifest file as a template and replace variables like image_tag with their values. This means that the frontend image to be deployed is the one that was tagged with the ${DRONE_TAG} variable value. After these modifications are done, we are ready to do a pipeline promotion!
Commit these changes to your repo first. Now, edit the guestbook/php-redis/index.html file and add some text somewhere to simulate a change that creates a new version of the frontend application. For example, add the text “v2.0.0” to the h2 header and commit and push the change:
[…]<h2>Guestbook v2.0.0</h2>[…]
The build image step from the build pipeline will trigger and build the image. Notice that now the deploy steps do not trigger because they are now in a separate pipeline:
Wait until the build finishes and create a git tag with the value “v2.0.0”:
$ git tag v.2.0.0
Push the tag, now the tag step from the build pipeline will trigger.
After it finishes, you can check in your docker hub repository that you have an image tagged with that tag. Now we are ready to promote the build. At the build triggered by the git tag, click on the three dots menu from the upper-right corner and select Promote:
On the target field write production (that is the name of the environment we defined in the target field of the trigger in the deployment pipeline), leave the rest of fields as it is and click ”Deploy”.
Now the Deploy pipeline will start and do the deployment of the frontend application image tagged with the git tag in the production target environment:
Go to the frontend application URL and the changes to the index.html file we just did should be visible:
The build was promoted and deployed as expected. You can check the pods, and see that they were re-created:
$ kubectl -n drone get pods
NAME READY STATUS RESTARTS AGE
frontend-56d74fbb4b-crj9t 1/1 Running 0 36s
frontend-56d74fbb4b-qcfz5 1/1 Running 0 48s
frontend-56d74fbb4b-zw9tt 1/1 Running 0 30s
redis-follower-594666cdcd-dbd4f 1/1 Running 6 2d
redis-follower-594666cdcd-l6s8c 1/1 Running 6 2d
redis-leader-fb76b4755-bhfzh 1/1 Running 6 2d
Also check the image tag that was used:
$ kubectl describe pod/frontend-56d74fbb4b-crj9t[...]
Containers:
php-redis:
Container ID: docker://426fb895586bd1415faaba8cb84b6a7edc475a5a7a82162132e2415f3270c86a
Image: juanluisbaptiste/guestbook-demo:v2.0.0
[...]
Now, let’s do another modification to the frontend application. Edit again the index.html file and change the version text to “v2.1.0”:
[…]<h2>Guestbook v2.1.0</h2>[…]
Commit and push the change, wait until the build finishes and create a git tag with the value “v2.1.0”:
$ git tag v.2.1.0
Push the tag to trigger the deployment, wait for it to finish and then promote again this new build (of v 2.1.0):
After the pipeline finishes, go back to the frontend application and reload the page, you should see the new changes:
As you can see, promoting a build is a straightforward and convenient way to move an application’s new version across environments. Promotions can also be executed using the CLI interface. This example could also have been done like this:
$ drone build promote juanluisbaptiste/kubernetes-engine-samples 68 productionNumber: 70
Status: pending
Event: promote
Commit: 1fbf81df2adf8753df698fc9ae1546127d54aa6e
Branch:
Ref: refs/tags/v2.1.0
Author: juanluisbaptiste <juan@xxxxxx.tech>
Message: Increase version text to v2.1.0
You could use the cli to programaticallt do the pipeline promotion after a successful set of tests finish. Now we can check how to roll back to a previous version.
Rolling back a Pipeline
A rollback is deploying a previous application version deployed in an environment, for example because after the promotion was done, some errors appeared that unfortunately were not caught during testing. Continuing with the example, we are going to roll back from v2.1.0 to v2.0.0.
Go to the promoted build of v2.0.0:
Click again on the three dots menu in the upper-right corner, but this time click on the rollback select box (I know, very weird to have the rollback option inside the promote window!), and again on target put production and click on ”Deploy”:
The Deploy pipeline will start. Click on the clone job and notice the git ref it checked out:
As you can see, is the v2.0.0 tag, the pipeline is using the correct version for the rollback deployment. After the pipeline finishes, go back to the frontend application and refresh, you should see again the old v2.0.0 version!
Debugging
If you are having issues with your Drone setup, be it the server setup or while executing a build pipeline, there are some tools that will help you debug the issue and find out what is going on. The first thing to do when debugging an issue is reviewing logs. Drone has two types of logs:
- Server and agent logs
- Pipeline build logs
To see the server or agent logs, use the kubectl command, for example:
$ kubectl -n drone get podsNAME READY STATUS RESTARTS AGE
drone-6c4c8dd65c-7rxdn 1/1 Running 2 4d
drone-runner-kube-7f79d87b75-fffn5 1/1 Running 2 4d$ kubectl -n drone logs drone-6c4c8dd65c-7rxdn — tail 10{“arch”:””,”kernel”:””,”kind”:”pipeline”,”level”:”debug”,”msg”:”manager: request queue item”,”os”:””,”time”:”2022–07-20T02:36:55Z”,”type”:”kubernetes”,”variant”:””}{“arch”:””,”kernel”:””,”kind”:”pipeline”,”level”:”debug”,”msg”:”manager: context canceled”,”os”:””,”time”:”2022–07–20T02:37:25Z”,”type”:”kubernetes”,”variant”:””}{“arch”:””,”kernel”:””,”kind”:”pipeline”,”level”:”debug”,”msg”:”manager: request queue item”,”os”:””,”time”:”2022–07-20T02:37:35Z”,”type”:”kubernetes”,”variant”:””}{“arch”:””,”kernel”:””,”kind”:”pipeline”,”level”:”debug”,”msg”:”manager: context canceled”,”os”:””,”time”:”2022–07–20T02:38:05Z”,”type”:”kubernetes”,”variant”:””}{“arch”:””,”kernel”:””,”kind”:”pipeline”,”level”:”debug”,”msg”:”manager: request queue item”,”os”:””,”time”:”2022–07-20T02:38:15Z”,”type”:”kubernetes”,”variant”:””}{“arch”:””,”kernel”:””,”kind”:”pipeline”,”level”:”debug”,”msg”:”manager: context canceled”,”os”:””,”time”:”2022–07–20T02:38:45Z”,”type”:”kubernetes”,”variant”:””}{“arch”:””,”kernel”:””,”kind”:”pipeline”,”level”:”debug”,”msg”:”manager: request queue item”,”os”:””,”time”:”2022–07-20T02:38:55Z”,”type”:”kubernetes”,”variant”:””}{“arch”:””,”kernel”:””,”kind”:”pipeline”,”level”:”debug”,”msg”:”manager: context canceled”,”os”:””,”time”:”2022–07–20T02:39:25Z”,”type”:”kubernetes”,”variant”:””}{“arch”:””,”kernel”:””,”kind”:”pipeline”,”level”:”debug”,”msg”:”manager: request queue item”,”os”:””,”time”:”2022–07-20T02:39:35Z”,”type”:”kubernetes”,”variant”:””}{“arch”:””,”kernel”:””,”kind”:”pipeline”,”level”:”debug”,”msg”:”manager: context canceled”,”os”:””,”time”:”2022–07–20T02:40:05Z”,”type”:”kubernetes”,”variant”:””}
To look at the agent logs, run the same command, but with the agent pod. If needed, logs verbosity can be increased with DRONE_DEBUG and DRONE_TRACE environment variables. To set those variables, add them to the server or agent Helm configuration files (drone-server-values.yaml and drone-runner-kube-values.yaml in this lab), and redeploy the server and agent. To view the logs of a Drone repository, use the following command:
drone log view <repo/name> <build> <stage> <step>
For example:
$ drone log view kubernetes-engine-samples 23 1 1Initialized empty Git repository in /drone/src/.git/
+ git fetch origin +refs/heads/main:
From https://github.com/juanluisbaptiste/kubernetes-engine-samples
* branch main -> FETCH_HEAD
* [new branch] main -> origin/main
+ git checkout dd284331e3d24b5c9ab1b29cf5a0c00c18afbb10 -b main
To tell the truth, it is a little weird to have to see the log of a build stage by stage, instead of being able to pull the whole log of a pipeline in one single call.
And of course, the CLI can also be used to trigger a rollback:
$ drone build rollback juanluisbaptiste/kubernetes-engine-samples 63 production
Conclusions
We have covered a lot of ground since we started this journey to explore Drone.io Continuous Integration and Delivery platform. We went from a simple docker deployment to test some of Drone’s core features, to deploy it in a local Kubernetes cluster using minikube, where we tested other more advanced features like, integration with Kubernetes secrets, pipeline promotions and rollbacks, and the usage of its versatile CLI tool. There are still many more features to explore, but what we have covered until now can give you a good grasp of some of its main features.
I have seen Drone.io improve by leaps and bounds since I started using it many years ago around 0.5 version. There were few plugins available, something that has drastically changed over the years, even thought that other more traditional CI/CD platforms like Jenkins have a wider support of integration with 3rd party tools, drone has come to have more than one hundred plugins available that cover the most popular tools. But in what Drone.io shines is in the simplicity of their pipelines, its small footprint and how simple is it to install it and get it going. Being a cloud native tool built with docker containers makes its installation trivial, even in Kubernetes, thanks to the Helm charts. Also, as plugins are built as a docker image, they can be written in any language, using environment variables to expose its configuration, making it fairly easy to extend its functionality as needed.
One feature that is still missing, thought, is giving some love to the Kubernetes deployment. Currently, the only way to deploy to a Kubernetes cluster is by using one of the Open Source plugins, that aren’t developed by Drone.io, but by third-party contributors. Unfortunately, I do not think this is going to improve as Harness, the parent company of Drone.io, offers its own Continuous Delivery solution to complement Drone.io as a Continuous integration tool. I understand this, but I hope that Drone.io could deliver a functioning Kubernetes deployment plugin for basic use cases.
I hope you enjoyed reading this series of articles about Drone.io, as much as I enjoyed writing them, finding them useful and insightful as well.