Kaniko, How to Build Container Image with SSH

Photo by Abbie Parks on Unsplash

TL;DR how via the Kaniko context, we can git pull Private repositories and build Dockerfile with private git dependencies.
This would be valid for other dependencies like Python private repository dependency or other dependencies alike.

A bit on Kaniko, Kaniko is a tool to build container images from a Dockerfile, inside a container or Kubernetes cluster.
You can get the motivation for using Kaniko here.

Motivation
My motivation for this post came from the fact that there is a lot of documentation on Kaniko, but this use case(Private repository in the Dockerfile) had little to none documentation, and it would be great if we could create a discussion on this matter and also get the feedback.

Let’s start by defining the problem and eventually a possible Solution.

The Plot

(Sources found at https://github.com/yossicohn/kaniko-build-private-repo)

• You have a private repo named dummy-repo-kaniko-build.git, which you would like to build a container from, using Kaniko build.
  Now, the Dockerfile is dependent on other private repo other-private-repo. git.

Git Repositories

• In the first stage(pre solution) we will have the Kaniko Pod definition, defining the flow
  The Kaniko Pod includes:
  1. initContainer -to create a workspace folder for the Kaniko context, which would load the dummy-repo-kaniko-build.git with the Dockerfile to be built.
  The InitContainer would have all the needed credentials as secrets which are mounted
  2. Kaniko Container -which executes the actual Dockerfile build, and also is mounted with credentials
• We would like to use Kaniko to build our Container image and push it to the Docker Hub registry (which may be any other registry)

Note, the content of the application here is a Go app, but this is less important for the Problem/Solution and can be any other type.

Below is the scatch for the kaniko POD with volume and secrets

Kaniko Pod Structure

The problem

When using Kaniko, you can use the mounting of secrets for the SSH keys or basic auth to git pull or Docker registry.
The problem starts when you have a Dockerfile that is using another private dependency, e.g. private git repository.
In that case, it would not be sufficient and the build would fail.

The Kaniko build would fail in the Dockerfile build since the build process does not have any credentials(SSH keys) for the private git repository(although we mounted the SSH secrets to the Kaniko pod).
Meaning the 1st stage of the initContainer passes, and the private repository dummy-repo-kaniko-build.git is cloned, but the Kaniko build fails.

In this example plot, the Dockerfile would be as follows:
Dockerfile -no-ssh

https://github.com/yossicohn/kaniko-build-private-repo/blob/main/Dockerfile-no-ssh

Note, that it is cloning the private git repository: other-private-repo.git via:

RUN git clone git@github.com:<user>/other-private-repo.git — single-branch

Kaniko Pod

Note, the Kaniko pod definition contains 2 containers:
1. initContainer
2. Kaniko Container
The initContainer prepares the Kaniko workspace with the dummy-repo-kaniko-build.git

Secret Mounting
The Pod contains mounting for secrets, the git SSH keys, and the Docker registry credentials.
With these secrets, we can pull the Git private images and push them to the Docker Registry.

Create Docker Secret

kubectl create secret docker-registry regcred --docker-server=https://index.docker.io/v1/ --docker-username=<user> --docker-password=<password> --docker-email=<user@mail.com>

Create Git SSH Secret

1. We should create a git SSH key, to have the id_rsa and the id_rsa.pub

ssh-keygen -m PEM -t rsa -P ""

This will result in 2 files: id_rsa, d_rsa.pub

Now the public key id_rsa.pub should be set in the git server configuration

2. Create the known-hosts

ssh-keyscan -t rsa github.com >> ~/.ssh/known_hosts_github

3. Create the git config

Host github.com
  HostName github.com
  StrictHostKeyChecking no

4. Create the SSH keys secret from the files

kubectl create secret generic ssh-key-secret --from-file=ssh-privatekey=.ssh/id_rsa --from-file=ssh-publickey=.ssh/id_rsa.pub --from-file=known_hosts=known_hosts.github --from-file=config=~/.ssh/config

Now the Kubernetes cluster has the needed secrets for mounting in the Pod.

Pod Definition

file: kaniko-pod-no-ssh.yaml
Note, via the SSH keys (via the secret mounting) we can git pull the private repository dummy-repo-kaniko-build.git

Running the Pod would show failure:

kubectl create -f kaniko-pod-no-ssh.yaml

Explaining the Failure:
The initContainer would finish successfully as the SSH keys are available for the git clone of dummy-repo-kaniko-build.git.
But, unfortunately, the Kaniko container would fail.

The reason for the Kaniko build failure, is that the other-private-repo.git is private and the build of the dummy-repo-kaniko-build/Dockerfile would fail on the git clone of the private repo other-private-repo.git
This would occur since the SSH keys are not in the context of the Kaniko build.

And that brings us to the solution…just set the Dockerfile with the SSH keys.

The Solution

We should copy the SSH keys to the context (workspace folder)and use them in the Dockerfile while building the container.

1. In the Kaniko Pod definition, we will add in the InitContainer the copy of the SSH keys:

mkdir /workspace/.ssh
cp .ssh-src/* /workspace/.ssh/

2. Copy the SSH keys to the Dockerfile layer and running the ssh-agent to use the added SSH keys.

The following block shows how :
- we copy the ssh keys from the workspace (.ssh/* to /root/.ssh)
- create an ssh-agent and add the ssh keys and git clone the private repo

IMPORTANT !! Multi-stage build — (Security Concern)
Note, that the usage of the multi-stage build here is very important and crucial for image security.
Since we are using the last stage, and the SSH keys do not exist there we have secured the SSH key and it will not be exported along with the exported image.

file: kaniko-pod.yaml

file: Dockerfile

Note, the usage of the SSH key in the Dockerfile RUN command:
It is important to use them all in the same RUN Command since each RUN command creates a different Shell!

RUN eval $(ssh-agent -s) && ssh-add /root/.ssh/id_rsa && git clone git@github.com:<user>/other-private-repo.git --single-branch

Running this pod would succeed, and we will get the Container image pushed to the Docker registry.

kubectl create -f kaniko-pod.yaml

Summary

Via the SSH mount on the Kaniko workspace, we can populate the SSH key to the Dockerfile and use private repositories.
This is done by populating the SSH keys to Multi-stage Dockerfile, having the Private dependencies cloned/pulled in the first stage, having the last stage with the result only.