Kaniko for build tasks — AKS as Azure DevOps agents

This is a continuation of the first part:
AKS as Azure DevOps agents- Intro | Medium

Using kaniko for build task was initially implemented as an experimental approach, However, upon evaluation, it has been determined that the ideal approach would be to utilize a dedicated build server using BuildKit. Check out the below article for details.

Buildkit — AKS as Azure DevOps agents | by Albin Sunney | Medium

Kaniko a powerful open-source tool for building container images from a Dockerfile without the need for the Docker daemon. This makes it particularly useful in environments where users don’t have root access, like in a Kubernetes cluster. Kaniko operates entirely in user-space within the executor image, unpacking the filesystem, executing commands, and snapshotting the filesystem. This is how it avoids requiring privileged access on your machine.

Two approaches are identified to use kaniko instead of docker in azure devops self hosted agents. Following are the same.

Approach 1: Running kaniko as a separate pod

This approach ensures that Kaniko runs as a separate pod whenever an image build/push is required. This involves the direct usage of the Kaniko image.

The disadvantage of this approach is that since it’s running as a separate pod, it requires multiple changes on the pipeline side as Kaniko can’t be run directly in a simple script task. It also requires an additional storage account to share the context between the Kaniko pod and the agent pod.

More details: Building Docker Images in Kubernetes Using Kaniko | by Kevin | KPMG UK Engineering | Medium

Disadvantages:

• Additional storage account to pass the build context
• Build task in pipeline gets complicated.

Approach 2: Extending Agent Image to Include Kaniko Binaries

This method proves to be more suitable as it incorporates Kaniko as an executable directly in the agent. This allows it to be invoked directly in an Azure DevOps script task, overcoming the requirement of additional storage account for passing build context.

Use the following dockerfile as agent image which has Kaniko binary files included:

FROM ubuntu:20.04
RUN DEBIAN_FRONTEND=noninteractive apt-get update
RUN DEBIAN_FRONTEND=noninteractive apt-get upgrade -y

RUN DEBIAN_FRONTEND=noninteractive apt-get install -y -qq --no-install-recommends \
    apt-transport-https \
    apt-utils \
    ca-certificates \
    curl \
    git \
    iputils-ping \
    jq \
    lsb-release \
    software-properties-common \
    libicu66 \
    wget
 
#Install powershell 
RUN wget -q "https://packages.microsoft.com/config/ubuntu/$(lsb_release -rs)/packages-microsoft-prod.deb"
RUN dpkg -i packages-microsoft-prod.deb
RUN DEBIAN_FRONTEND=noninteractive apt-get update
RUN DEBIAN_FRONTEND=noninteractive apt-get install -y -qq --no-install-recommends powershell

#Install az cli
RUN curl -sL https://aka.ms/InstallAzureCLIDeb | bash

# Can be 'linux-x64', 'linux-arm64', 'linux-arm', 'rhel.6-x64'.
ENV TARGETARCH=linux-x64

WORKDIR /azp
#COPY --from gcr.io/kaniko-project/executor /kaniko/executor /kaniko/executor
COPY ./start.sh .
RUN chmod +x start.sh


# Setting Enviroment Variables for Kaniko
ENV HOME /root
ENV USER root
ENV PATH="${PATH}:/kaniko"
ENV SSL_CERT_DIR=/kaniko/ssl/certs
ENV DOCKER_CONFIG /kaniko/.docker/
ENV DOCKER_CREDENTIAL_GCR_CONFIG /kaniko/.config/gcloud/docker_credential_gcr_config.json

# Copy Needed Files from Kaniko Image
COPY --from=gcr.io/kaniko-project/executor /kaniko/executor /kaniko/executor
COPY --from=gcr.io/kaniko-project/executor /kaniko/docker-credential-gcr /kaniko/docker-credential-gcr
COPY --from=gcr.io/kaniko-project/executor /kaniko/docker-credential-ecr-login /kaniko/docker-credential-ecr-login
COPY --from=gcr.io/kaniko-project/executor /kaniko/docker-credential-acr-env /kaniko/docker-credential-acr-env
COPY --from=gcr.io/kaniko-project/executor /kaniko/.docker /kaniko/.docker

# Generate latest ca-certificates
RUN apt-get update && \
    apt-get install -y ca-certificates && \
    mkdir -p /kaniko/ssl/certs/ && \
    cat /etc/ssl/certs/* > /kaniko/ssl/certs/ca-certificates.crt

#RUN  ln -s /kaniko/executor /usr/local/bin/kaniko 
ENTRYPOINT [ "./start.sh" ]

Reference: kaniko/deploy/Dockerfile at main GoogleContainerTools/kaniko (github.com)

The configuration of Kaniko to use Azure Container Registry (ACR) credentials for pushing the image is crucial. It is important to ensure that the workload identity has access to push the image into ACR, as these identity credentials will be used by Kaniko.

The config.json file contents are as follows:

{
  "credHelpers": {
    "<acrName>.azurecr.io": "acr-env"
  }
}

This can be applied as a configmap using the command:

kubectl create configmap docker-config --from-file="config.json"

Disadvantages:

• Kaniko doesn't recommend running kaniko as part of different image, hence lack of documentation on extending agent image with kaniko binaries.
• Automation to update the configmap ‘docker-config’ for new container registries.

Recommended solution

• Buildkit — AKS as Azure DevOps agents | by Albin Sunney | Medium