Going Great Guns with GoCD
Having a solid and extensible CI-CD pipeline is one of the core engineering strengths of any organization. We want developers to spend as much little time as possible on deployment procedures. These days we have plenty of ci-cd options available in the market, some available at a click of a button to complex ones that feel too intimidating to give it a try.
In this editorial, I’ll be discussing one such tool GoCD. I wouldn’t go into discussing or comparing features of GoCD to other tools. I reckon if you are reading this blog you have already narrowed down on it after deep considerations and analysis. So what's the agenda? Here is what we would be covering step by step
- Prerequisites on GoCD starting with installing GoCD on k8s. For this demo, I’ve used minikube to spin up a local k8s cluster. But steps remain the same whether it is a local k8s cluster or production k8s cluster.
- Setting up the Docker registry on AWS ECR where we can push the docker images of our awesome service.
- Configuring and setting up GoCD elastic agents to compile, package, and push the image to ECR. For the demo, I’ll be focussing on packaging a JAVA application but methodology remains the same irrespective of the languages and frameworks.
- Pipeline in itself, wherein this editorial we would cover stage-0 (detecting code changes) and stage-1 (compiling. packaging, creating docker image, and pushing it to AWS ECR repo) and stage-2/3 which is setting up the infra of your service as part of CI-CD. In the follow-up series, I would go about deploying the service on k8’s cluster.
Installing GoCD on k8s
Here we assume that you have a k8s cluster up and running upon which we will be installing the GoCD servers. For demo purposes, I’ve set up everything on a local k8s cluster. However, the methodology remains the same for a cluster hosted on the cloud cluster.
$ minikube start
$ kubectl create ns gocd
$ helm repo add stable https://kubernetes-charts.storage.googleapis.com
$ helm install gocd stable/gocd --namespace gocd
$ minikube ip
$ 192.168.64.2
Creating a docker registry on AWS
We would be using Amazon ECR as a docker image repository. This is where we will be pushing all our docker images after every successful build. To create a new repository login to your AWS account and go to AWS -> ECR -> Create repository. Choose a repository name (in our case myawesome-service-1) where we would store all the images of service-1.
You’ll see the newly created repositories in the list of repositories.
Setting up GoCD agents
Generally, as the first step of any CI pipeline, we need to take the source code and build an executable binary/package out of it. So we would need a host which has all the compile-time libraries and dependency to start with. This where GoCD agents come into the picture.
These GoCD agents act as workers which do the heavy lifting of actually running the tasks. These agents are highly elastic in nature, which means they would be brought up when needed and then terminated once the task is completed. But how would a GoCD agent get the compilation libraries and all the dependent environment? Well, that is where containers come to rescue. Each GoCD agent is a container in itself running inside a k8s pod. So all you need to do is specify the image of the container to be used for each step and we are done.
Since we plan to compile java application in this example we will create an image with Java 8 / maven installed. Depending upon the use cases, we can have multiple GoCD agents each for different source code. For our use case, we will start off from a standard go-cd base image with agent pre-installed and then install mvn/JDK 8/aws /terraform. So let’s create an image which will act as GoCD agent to compile JAVA applications
$ git clone https://github.com/praks-1529/myawesome-gocd.git$ cd myawesome-gocd/elastic_agents/base_images/java8$ vi aws.credentials // Replace your actual AWS key/secret$ docker build -t myawesome-gocd-agent-java-8:1.0 .$ aws ecr get-login-password \
--region ap-south-1 \
| docker login \
--username AWS \
--password-stdin 479580041174.dkr.ecr.ap-south-1.amazonaws.com/myawesome-java-8$ docker tag <image_id> 479580041174.dkr.ecr.ap-south-1.amazonaws.com/myawesome-java-8:1.0$ docker push 479580041174.dkr.ecr.ap-south-1.amazonaws.com/myawesome-java-8:1.0
Right, so now we have the base image of a GoCD agent which has java 8 compile and runtime environment.
We will use this image to configure GoCD agent which will act as a host on which the Java application would be built and packaged.
Configure GoCD agent
In this step, we just mention which image to be used when the GoCD agent container boots up. The Elastic profile name will be used to pick the agent.
Configure k8s POD and AWS ECR
To be able to pull the required images stored in the private docker image repository (AWS ECR) from k8s PODs we need to do additional setups. If we are using Dockerhub instead of AWS ECR then we can pass on the static user name and password as k8s secret configuration.
$ aws ecr get-login-password \
--region ap-south-1 \
| docker login \
--username AWS \
--password-stdin 479580041174.dkr.ecr.ap-south-1.amazonaws.com/myawesome-java-8$ ls ~/.docker/config.json$ kubectl create secret -n gocd generic regcred.qa \
--from-file=.dockerconfigjson=<path/to/.docker/config.json> \
--type=kubernetes.io/dockerconfigjson
Note*: Since AWS ECR passwords keep rotating every X hours we cannot use the same methodology here. We can set up a cron that keeps fetching the latest secrets. Also if you are on Mac please check this to get auth in your config.json instead of credStore
The Pipeline
We would be taking a real-world example where a spring boot application would be built and deployed on the k8s cluster, yes all automatically. For demo purposes, I’ll be using this repository which is a very simple CRUD service. Here is how the pipeline would look like
I’ll be explaining the above stages of pipeline one at a time and set up stuff as we proceed.
Stage-0: Detect code changes
GoCD provides an excellent way to set this up with the help of material. So all we need to do in this stage is to make material point to our demo repo. So this is how the pipeline file looks after this stage.
Quick explanation, we are constantly polling the develop branch and the repository is set to git key. Please note that the git repository is the SSH URL and not the HTTPS. Hence we may have to ensure that the GoCD agent has access to the GitHub. I’ll cover that in the later section
Well, we have a pipeline file, but how do we import this file in the GoCD server? GoCD picks these pipelines from a config repository that needs to be set. What it means is that all the pipelines are managed as code. This is extremely useful as the number of pipelines grows in the organization and managing them via UI becomes a pain. So you manage these pipelines just as a regular code with all the benefits of version control. For demo purposes, I’ll be using this config repository. To set a config repository we need to go to GoCD -> Config repositories -> Add
Also, configure rules on the config repository to allow it to access all the pipelines under group “TEAMNAME”. Once you “Save” the configuration, GoCD will try to fetch all the pipelines available in the repository and set them up. But wait we see an error¹ !! Good thing is that our config repository was set properly as we can see GoCD was able to fetch the latest commit, but whats that error? Hmm, GoCD expects every valid pipeline to have one or more stages and hence we enter stage-1 of our uber pipeline, the build stage.
Stage-1: Build and push the image
In this stage, we would compile/package our java application first, create a Docker image, and finally push the created image into AWS ECR. We will go into details of each sub-step one by one
Stage-1.a: Compile/Package an application
All we need to do is to update the pipeline file we had earlier and add a stage that would build the java application on the configured GoCD agent as shown below.
Note*: Here I’ve configured material to use HTTPS. In production since the repositories are private, you would use SSH instead. So you need to ensure that a SSH keys are part of the docker image of GoCD agent.
Now if you go to GoCD -> Admin -> Config repositories you must see the error¹ getting vanished now.
Also if we go to GoCD -> Dashboard, we can see our pipeline popping up there.
Stage 1.b: Build the docker image
In this step, we basically take the built jar in the previous step and create a docker image out of it. For this, we use the Dockerfile defined in the top-level directory. This will be the image of our service (myawesome_service_1) which will be deployed on to k8s cluster later. If you intend to run any other sidecar containers like log scrapper, metrics alongside your service you can update the Dockerfile accordingly.
A little explanation on the above tasks.
- First, we build a package using mvn clean package.
- Then we get the release version of the package defined by version inside pom.xml
- The image name and tag to be uploaded. The image consists of <repository_name>:v<tag>. For the tag, we can use the commit number. (git rev-list HEAD — count)
- Build the image
Stage 1.c: Push the image to AWS ECR
As a final sub-step, we need to push the built image on to AWS ECR.
All that is left now is to trigger the pipeline and wait for things to fall in place and things to turn GREEN.
So this brings an end to the first and most important stage of the pipeline which involves compiling source code, creating a docker image, and finally pushing to docker registry i.e AWS ECR. Finally, we can also verify if the image was pushed to AWS ECR. In the subsequent sessions, we would be using this same image to deploy on k8s.
You can also add a few more tasks like unit tests, running code coverage, lint checks, and then decide to push the image or reject it. All you need to do is to add these tasks prior to build and push steps.
Stage-2/3:
In this stage, we will update the infra. Often our services require various other components like DB, cache, load balancers, inbound/outbound rules on resources, and many more. It’s best if these resources are managed as code and what better tool than terraform. In this demo, we will go over creating a DB instance via terraform and how changes to this DB can be managed via code. Managing these resources via terraform allows us to have version control for infrastructure and can track how and when the infra changed Frequent examples of changes to infra include ELB inbound rules where we want to allow new services to access the service or when you want to scale up the DB/cache for an upcoming event. All that can be managed by terraform code.
Stage-2.a: Setting up terraform
First, let's create an S3 bucket on AWS where terraform would store the current state of the infra. Whenever we make any update to resources, terraform compares it with the current state files to find the diff. Hence it is extremely crucial to ensure that these state files are not messed up accidentally. I’ve witnessed instances where the state files were messed up by other pipelines and that lead to very very undesirable situations. Hence it is always recommended to enable versioning on that the bucket and make it private.
$ aws configure
$ aws s3 mb s3://tf-state-myawesome-sqewru13sStage-2.b: Terraform bootstrap code
Every service must have the Terraform module to manage the infrastructure of that code. All the terraform code is placed inside theterraform folder. Below is the short description of files
- tf_plan.sh: Shell script to do plan terraform. This script will be invoked from gocd stage
terraform-plan. The output of the terraform planning will be saved in the artifact tf_plan_output.txt which can be reviewed. - tf_apply.sh: Shell script to apply terraform. This script will actually execute the terraform and start adding/deleting/changing the infra.
Important to note that as the first step of both the above first we MUST select the terraform workspace which is represented by
${COMPONENT_NAME}-${PRODUCT_MARKET}-${ENV}-${REGION}Because of this templatization, we can set up any number of parallel infra (QA/PP/PROD) for different regions (IN/US/*) by just changing these variables in the gocd pipeline. Also, the same methodology can be used by a different service by just changing the component name.
- main.tf: This is where we configure the S3 bucket where the state files will be stored. This will be configured in
backend.bucket. Also here we specify the version ofawsandterraformthat would be expected. - tfvars/{PRODUCT_MARKET}-{ENVIRONMENT}-{AWS_REGION}.tfvars: This is where we override the terraform variables per infra.
Stage-2.c / Stage-3: Pipeline to plan and apply to terraform
Now that we have all the prerequisites set up to run terraform successfully we will add these stages in our gocd pipeline. The complete pipeline is available here. This mainly consists of below sections
- As with any pipeline we need to set the upstream. In this case, since we want to run this pipeline automatically the build is completed we will set the previous build pipeline as an upstream of this pipeline.
- terraform plan: In this stage, we would just output the changes for us to verify. This stage is crucial as we can review the infra changes before the changes are applied. The terraform plan outputs changes in terms of resources that will be added/deleted/changed in case applied.
- terraform apply: In this stage, the reviewed changes are applied. This will actually modify parts of your infra.
Now let's create a new resource i.e a MySQL database for our service using terraform and see how the DB is created automatically. I’ve created this RDS resource just for the demo. But you can easily create any other resource (cache/s3/elb) in a similar way.
Now in the next subsequent terraform run you would see that terraform would detect the changes and inform us that 2 new resources have to be added in the infra. (parameter group and RDS itself)
When you apply the terraform changes, it would actually lead to the creation of these resources.
Once terraform apply stage is complete you can verify if the resources were created on AWS console
This brings an end to the current part of the setup of our pipeline, which so far watches changes in code (branch=develop), builds a docker image, pushes the image on AWS ECR, and setup infra (dev infra). In the next part of this session, I would go over deploying the images we built in this part on a k8s cluster aka stage-4 of the pipeline.
Conclusion
To get your understanding concrete and realize the importance of concrete ci-cd setup, you must try setting up a new infra lets say QA or prod on top of this dev setup. A brand new infra can be brought up with less than 5 min of code. Compare this to the old way of creating pipelines manually via UI tools and creating infra using the AWS console. In a nutshell, below are the changes you would need for setting up a new pipeline (Check the number of lines added in each of below commit)
In the next series, I would go over deploying the images on k8s pods. Stay tuned and stay safe!