Buildkit — AKS as Azure DevOps agents

BuildKit is a toolkit for converting source code to build artifacts in an efficient, expressive and repeatable manner. BuildKit is composed of the buildkitd daemon and the buildctl client.

Note: BuildKit has been integrated to docker build since Docker 18.09.

This is a continuation of the first part:
AKS as Azure DevOps agents- Intro | Medium

In this article we will be focusing on configuring the buildkit for building dockerfile as an alternative for docker builds.

Configure buildkit daemon as a deployment

(Ref: https://github.com/moby/buildkit/blob/master/examples/kubernetes/deployment%2Bservice.rootless.yaml)

Use the above reference yaml for creating a rootless buildkitd deployment (build server) and its service. Also remove the certs mounts since we are not adding the same.

containers:
  - name: buildkitd
    image: moby/buildkit:master-rootless
    args:
      - --addr
      - unix:///run/user/1000/buildkit/buildkitd.sock
      - --addr
      - tcp://0.0.0.0:1234
      - --oci-worker-no-process-sandbox

Using — addr tcp://0.0.0.0:1234 , we are exposing the buildkit server as a TCP service. In the yaml file we are creating a Kubernetes service tat opens the Kubernetes service in same port 1234. Apply the same yaml file in your Kubernetes cluster.

kubectl apply -f "deployment service.rootless.yaml" -n <agentNamespace>

This will create our buildkit server running and ready for building images.

Configure azure devops agents with buildctl

buildctl ref:
buildkit/docs/reference/buildctl.md at master · moby/buildkit · GitHub

To access the buildkitd server, we have to make use of the buildctl, buildctl is the command-line interface to buildkitd.

The same build server can be accessed directly using docker buildx plugin as described in the illustration. Details are not discussed in this article, since its not usable for devops agent running in a pod.

buildctl connects to a running buildkitd instance. The connection can is in a URL format of <protocol>://<address> as shown below.

• tcp://<ipaddress>:<port>, e.g. tcp://10.0.0.1:2555
• unix://path/to/socket, e.g. unix:///run/buildkit/buildkitd.sock

To enable build from azure devops pipelines we will be installing the buildctl in agent images. Once installed we can call the same to do builds instead of builds using docker cli

For understanding the agent related configuration follow the below doc:
Run a self-hosted agent in Docker — Azure Pipelines | Microsoft Learn

Installing buildctl:

Add the following entry in azure devops agent image dockerfile to get the buildctl installed along with other required tools. Make sure its added above the start.sh run command as follows.

Agent dockerfile:

FROM ubuntu:20.04
#Install other required tools
<>

#Install buildkitcli
WORKDIR /buildkit
RUN wget https://github.com/moby/buildkit/releases/download/v0.12.1/buildkit-v0.12.1.linux-amd64.tar.gz
RUN tar xvf buildkit-v0.12.1.linux-amd64.tar.gz
RUN  ln -s /buildkit/bin/buildctl /usr/local/bin/buildctl

#Following are required for agent configuration
#Can be 'linux-x64', 'linux-arm64', 'linux-arm', 'rhel.6-x64'.
ENV TARGETARCH=linux-x64
WORKDIR /azp
COPY ./start.sh .
RUN chmod +x start.sh
#This should be the last command
ENTRYPOINT [ "./start.sh" ]

Building this docker file will ensure our agent has buildctl installed and accessible by running ‘buildctl’.

Apply the agent deployment yaml with the new image we build.

Refer below to get the initial deployment yaml for agent and other prerequisites related to agent
https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/docker?view=azure-devops#deploy-and-configure-azure-kubernetes-service

kubectl apply -f "agent Deployment.yaml" -n <agentNamespace>

By now both the azure devops agent pod and buildkitd pod should be running in our AKS cluster. Ensure the same before proceeding.

Azure devops build task with buildctl

Following can be used in azure tasks to use buildctl (for image buillding and pushing to registry). Inorder to push to the ACR or any other registry buildctl requires a .docker/config.json file containing the authentication requirements to a registry. Here we will have the config file created with acr access token obtianed from az acr commands.

  #Login to azure cli before running below , 
  #or use azure cli task in azure devops
  acr_name=<ACR name>
  acr_repo_name=<Repository name>
  image_tag=<Tag for image>
  context_loc=<Path to build context>
  dockerfile_loc=<Path to docker file>
  docker_config_file=~/.docker/config.json
  echo "Acr name: $acr_name"
  #Retrive token
  token_value=$(az acr login --name $acr_name --expose-token --output tsv --query accessToken)
  #Create a sample file not exists
  if [ ! -f $docker_config_file ]; then
      echo '{}' > $docker_config_file
  fi
  #Save the token in required docker config file format
  jq -n --arg token $token_value '{"auths":{"'$acr_name'.azurecr.io":{"identitytoken":$token}}}' > tmp && mv tmp $docker_config_file

Once the registry auth files are created buildctl can be triggered from the devops task using below commands:

#This section takes care connecting to build server 
#and building and pushing image
buildctl --addr tcp://buildkitd:1234 build --frontend=dockerfile.v0 \
--local context=$context_loc \
--local dockerfile=$dockerfile_loc \
--opt filename=Dockerfile --opt build-arg:<argName>=<argValue> \
--output type=image,name=$acr_name.azurecr.io/$acr_repo_name:$image_tag,push='true'

Scope of enhancement

• Enable the cert based auth for buildkitd server as mentioned in their github
• Enable autoscaling with keda with scaling event set to azure pipeline jobs
• Ensure registry-based caching to ensure the agent pods stay ephemeral and thus supporting shared caching for multiple agents.

Dig deeper…

• buildkit github: GitHub — moby/buildkit: concurrent, cache-efficient, and Dockerfile-agnostic builder toolkit
• buildctl: buildkit/docs/reference/buildctl.md at master · moby/buildkit · GitHub
• AKS as Azure DevOps agents- Intro | by aabeing | Aug, 2023 | Medium