Kubernetes and Trunk-based development
Written by Marck Oemar - Aug 2022
(Example repository: https://gitlab.com/cohesion/public/tbd-k8s-example-pipeline/)
For many developers, CI/CD and Kubernetes are familiar topics. Once you start talking about Trunk-based development (in short; TBD), often the room will split in half, which is understandable.
At its core, Trunk-based development is a Shift-Left in our way of working, in which we want to communicate and cooperate more often and earlier in the development process. This can have quite an impact on a development team. (If you like to read up on all things TBD, https://trunkbaseddevelopment.com is a good resource)
Let’s look at a few ways to create a TBD pipeline for a typical containerized application deployed in Kubernetes.
As a subject, I’ve created a simple Flask application that provides the current temperature at a given location.
Guidelines to consider
Before we build our pipeline, let’s look at some TBD and DevOps guidelines that could help us:
- Developers should be able to work on the same component simultaneously
- Trunk is leading and should always be stable
- Every change is automatically unit-tested, built, tested in a staging environment, and deployed in the production environment
- The repository should be service centered, meaning, any code or configuration that relates to the service should exist in the repository
Given these guidelines, I’ve created three objectives:
- The pipeline should be able to run concurrently with isolated jobs and resources
- CI and CD should be implemented in a single seamless process
- The pipeline should be smart enough to execute certain jobs, based on what code is changed in a commit
Now, let’s look at each one and how to achieve them.
1 - Running concurrent pipelines
I’m using Gitlab CI to execute jobs in a containerized environment. This is great, because any test or build job that doesn’t require external resources, can be executed in isolation and concurrently.
Docker in Docker
As an example, if you want to build and test a container in the pipeline, you can implement Docker in Docker as a side-car service:
buildtest:
image: ekino/ci-dind:latest # lots of handy ci stuff in there
services:
— docker:19.03.12-dind
script:
— docker build -t ${CONTAINER_IMAGE}:latest .
— echo “starting container to be tested”
— docker run -d -p 8080:8080 ${CONTAINER_IMAGE}:latest # detach
— echo “execute integration tests”With the services directive, you can specify a side-car container that is exposed in the job container. In this case, we run a Docker Engine and use the Docker CLI to execute commands. We’re building the image and running the container so that we can run tests against it.
How cool is that! Because, once the job is completed, the containers are destroyed. Containers are by nature ephemeral environments, and you can use this ‘ephemeral’ approach throughout the pipeline.
Ephemeral deployments with namespaces and Kustomize
Another example of creating ephemeral environments is when we want to temporarily deploy our service in Kubernetes, so that we can perform an acceptance or end2end test. Remember, we want to be able to do concurrent deployments in isolation.
You could achieve this by creating a temporary namespace, and deploying Kubernetes resources in it.
kubectl create namespace “cicd-${CI_JOB_ID}”
kustomize edit set namespace “cicd-${CI_JOB_ID}”
kubectl apply -k .But how do we specify the namespace? The answer is Kustomize, which is a powerful configuration tool. You can use the ‘edit’ subcommand to reconfigure the namespace for all manifests that are in scope. Kustomize is built into the kubectl CLI, and with “kubectl apply -k” the manifests are customized and deployed.
To make sure things are always cleaned-up:
after_script:
— kubectl delete -k .
— sleep 15
— kubectl delete namespace “cicd-${CI_JOB_ID}”2 - CI and CD should be a single seamless process
The version of the container image is often pinned in a Kubernetes manifest or Helm chart, which causes a decoupling in the CI and CD process. If we want to couple CI and CD, we’ll need to identify the flows that could occur.
The first flow: A change in the source code should trigger the CI jobs *and* CD jobs. The pipeline needs to unit-test the source code, build and test the artifact and release the artifact. Very important: we must deploy the image artifact that was released inside the pipeline, regardless of what happens outside the pipeline.
The second flow: There is only a change in the deployment code (Kubernetes manifests). In this case, the pipeline needs to identify the latest image released.
imageTag and imageDigest with AWS ECR
In my pipeline example, I’m using AWS ECR as the container registry. To make sure the CD jobs use the same image that was built and released in the pipeline, we can use the attributes imageTag and imageDigest to refer to an image. The nice thing about imageDigest is that it’s immutable.
How do we glue CI and CD together?
During a release, the image will be tagged with the commit SHA (for tracing purposes) and ‘latest’. After pushing the image to ECR, we can look up the image digest from ECR.
release_digest:
script:
- echo "RELEASE_DIGEST=$(aws ecr list-images --region ${REGION}
--repository-name ${CONTAINER_IMAGE} --output json |
jq -r --arg RELEASE_COMMIT "${RELEASE_COMMIT}" '.imageIds[] |
select(.imageTag==$RELEASE_COMMIT) | .imageDigest')"
> release_digest.env artifacts:
reports:
dotenv: release_digest.env
By using Gitlab CI dotenv artifacts, we now have a reference to the image that needs to be deployed, available in the CD jobs.
When there is only a change in deployment code, we need to have a reference to the latest released image:
latest_digest:
script:
- echo "LATEST_DIGEST=$(aws ecr list-images --region ${REGION}
--repository-name ${CONTAINER_IMAGE} --output json |
jq -r '.imageIds[] | select(.imageTag=="latest") |
.imageDigest')" > latest_digest.env artifacts:
reports:
dotenv: latest_digest.env
Now it’s up to the CD jobs to decide if they want to use the released image or the latest image.
To specify the image that should be deployed, you can use Kustomize:
kustomize edit set image ${REGISTRY}/${CONTAINER_IMAGE}@${DIGEST}3 - The pipeline should be smart
That’s easy with Gitlab CI: You can specify the fileglob that will trigger a certain job, by using the ‘only’ directive in .gitlab-ci.yml:
only:
refs:
- main
changes:
- .gitlab-ci.yml
- src/**/*In our pipeline example, all CI jobs are triggered only by a change in the pipeline configuration, and the source code. All CD jobs are triggered by a change in the pipeline configuration, deployment code and source code.
Putting it all together
This is my example pipeline, with ephemeral environments, a seamless CI/CD process, and job conditions.
What I like about this implementation is that from a source code perspective, any change that is committed to Trunk and passes all the tests will be deployed into production.
In a sense, the artifact is abstracted: it’s all about code changes.
I’d like to hear your opinion on this approach: there is always room for improvement, so feel free to leave a comment!
Thank you for reading.
