Part III: Building Docker images in Kubernetes with Kaniko and Azure DevOps
In this article, we will continue from where we left off in the previous article.
We are not limited by the kind of CI pipelines we can run on our self-managed Kubernetes Agents, to most things anyway.
We can build Python, Go, Dotnet, or run automation tests using Robot Framework.. but what about our Docker images CI pipelines? We can’t just install a Docker on our Agent and perform a docker build because the Agent is running inside a container.
So the first option that came to my mind is implementing DinD(Docker in Docker), but security-wise it’s not recommended. We can discuss why not to, but let’s talk about one of the alternatives I chose to work with — Kaniko.
Kaniko: in a nutshell
Kaniko is an open-source tool for building container images inside a container without needing a Docker daemon or privileged access, making it ideal for use in Kubernetes or other cloud-native environments.
Kaniko vs. DinD
Kaniko and DinD are used to build Docker images within a Kubernetes cluster, but their approach differs. Kaniko is a containerized build tool within a pod, while DinD runs a Docker engine within a pod to build images. Kaniko is generally considered more secure as it does not require privileged access to the host machine, while DinD may have more flexibility and support for Docker-specific features.
For this article, I have prepared a simple dotnet application and a Dockerfile that builds it. We will build this application using Azure Pipelines, of course, and Kaniko. We will publish it to our ECR. Inside my GitHub repository, you will find all that is needed:
- Application code
- Dockerfile
- Azure Pipeline
clone or fork it and make it available on your Azure Git.
Before we start working with Kaniko in our pipelines, we must understand that there is no native way to run Kaniko for Azure DevOps. as opposed to other CI systems. Hence, using Kaniko requires an understanding of Azure DevOps and Kubernetes.
Without diving too much into the details, I’ll mention that when using Kaniko, we need to provide the location of the build context (code and Dockerfile). Currently, there’s support only for a local directory, local Tar Gz, StdIn, GCS bucket, Azure Blob, S3 Bucket, and Git repository (not Azure Git).
This means we need to pass the code somehow to Kaniko, and we will do this by running an init container before the Kaniko container starts that will clone our dotnet repository for Kaniko to use.
Let’s dive in.
In this pipeline, I’m going to implement the use of templates in Azure DevOps. One of the best practices is to use templates whenever possible to avoid code duplication. In the case of a common type of pipeline, such as one that builds Docker images, it’s advised to write a template. I suggest managing the templates on a separate repository. But this is usually very specific to the needs and use-case of the company.
You can take the following files or even use the repository as is:
and make sure you have these templates available on a separate repository because I’m referencing them from within the Kaniko pipeline. You can put them wherever you like as long as you update them accordingly in the pipeline resources section.
Here is a brief explanation of the files:
- build-docker-image-via-kaniko-job.yml: This is a main template, the only Job in our pipeline. So that you know, variables and logic are firmly based on the previous solution. If you only need the Kaniko pipeline, you may have to adjust to your specific needs.
- create-kaniko-pod-task.yml: This task will create a YAML called ‘deploy.yaml’ and create the Kaniko pod and ConfigMap needed for Kaniko. in the initContainer section, we will need to make sure the azure-devops-git-cloner image version is correct, I’ll elaborate about it up ahead.
- apply-and-monitor-kaniko-pod-steps.yml: This is the exciting part. In this template, we have several tasks: create the Kaniko pod, and monitor the progress of the Kaniko pod (both initContainer and main-container) while printing the output of the Docker build that is being done. Once finished, it deletes all build resources and will exit with an error in case it fails, with relevant logs printed.
Prerequisites:
- AWS Access Keys with permissions to read from and push to ECR
- A Kubernetes secret that contains the AWS keys and region (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_REGION) for allowing Kaniko to authenticate using ecr-login. Because I’m deploying my demo app on the same cluster that my agents are running on, I will update my already existing secret in AWS Secret Manager, and the External Secrets Operator will automatically update my Kubernetes secret that already exists, with the newly added keys. We need this as a Kubernetes secret because, in the Kaniko YAML definition file, we are mounting the access keys into environment variables.
- You can read some more information about pushing to ECR via Kaniko at the following URL:
- A custom tool called azure-devops-git-cloner Docker image needs to be available in your ECR to be pulled by Kaniko’s init-container to do the git clone from Azure Git.
azure-devops-git-cloner:
To use Kaniko with Azure DevOps Agent, we need to add an init-container to clone our Azure Git repository and pass the code context to Kaniko. Since there is no native way of doing so, the alternatives are either writing a custom bash script within the init-container definition or, as I preferred of doing — using prepared code as a Docker image that will clone the repository for us by using the pipeline’s running token as credentials to git clone the repository via HTTPS (thus, no need to save the PAT anywhere).
I have prepared a repository with all the needed files: the code itself, Dockerfile, and even an Azure DevOps YAML pipeline ready to be created.
The pipeline expects to get the AWS credentials keys from a library group, and we can use the same group as I did in the first article. If you don’t have then create one that contains the keys. Alternatively, you can modify the pipeline to use a service connection instead. This is a personal preference decision; feel free to use it as needed.
Before creating the pipeline, don’t forget to ensure you have a repository in ECR called azure-devops-git-cloner to push the Docker image.
Once the pipeline is created, run it and expect the following results:
(The version of the Docker will be updated in the VERSION file only if the pipeline will run as part of CI and not manually, just a note)
We are creating a Kaniko pod because running the binary itself is currently not supported, and the only supported method is running it as a container. Hence, the pipeline will create a pod that will run the Kaniko container for us, meaning doing a “Docker run” without actually using a Docker. But for this to happen, we must allow our Agent pod the proper permissions to create a Pod in the cluster and a configMap since it’s part of the requirements.
Luckily, if you have followed the previous steps, I’ve already prepared the configuration in our GitOps repository (azdo-agents-gitops). I’ll remind you that in the Argo branch (eks-azdo-agents), under the folder “agents/templates/configurations/cluster” we have two files:
These files contain the permissions level, which I found is needed for the agent to create the necessary resources in the cluster and bind the role to the relevant service accounts that will require the permissions to perform the actions.
Ensure proper permissions for the Builder user to commit the version change into the code. I write about it in chapter I, and these are the permissions required:
Let’s create the pipeline and run it! But before, ensure that the repository for the Docker image exists in ECR.
After the pipeline finishes running, the expected result would be something like this:
And now we have a Docker image in our ECR that was built with a containerized agent and published to ECR.
I hope this solution answers your needs or that you’ve learned something new :)
