How Encrypted Images brings about compliance in Kubernetes (via CRI-O)

Compliance through securing workload compute with encryption, or 100% security by shorting the motherboard with an actual key

During the previous series of blogposts, we introduced encrypted container images, and showed how to encrypt and decrypt container images with both containerd and skopeo.

In this blogpost, we will talk about how image encryption, when used in a larger context of clusters and nodes, can be used to achieve compliance controls. We will also include a demo of doing this in Kubernetes with the current version of CRI-O, which is part of the current release candidate in version 1.17.

This blogpost is brought to you by Brandon Lum & Harshal Patil.

Encrypted Container Images in the context of a cluster

In the most common case, we talk about encrypting container images for the purposes of ensuring confidentiality of the container image code. The ability for individuals to protect the confidentiality of their data without relying on a trusted arbiter is the basis of encrypted images. However, the use of encrypted images can be extended much further when we bring in a larger context of systems outside individuals such as “Bob” and “Alice”.

In high assurance environments, encryption of images (like VM images) are used to provide guarantees of where an image will run. Instead of encrypting an image for “Alice”, I can perform encryption for “Cluster PROD”, or “PROD Clusters in EU datacenters”. This assists in providing the ability to assert that certain images are only run on particular locations. This can be used to enforce policy as well as data content protection and export control.

Key Models

However, encryption is only one side of the coin. In order to achieve this level of compliance controls, key management and distribution is required. This ensures that only the correctly identified compute nodes are able to decrypt the images. By providing access to a key to a restricted set of entities, we implicitly create cryptographically backed access control to the encrypted images. In the context of encrypted OCI images, we call this the “Key Model”.

One simple example of a key model would be node based key model. Ensuring that only a node has access to a key, we create an association of trust with the node. Doing this with every node in a cluster will then create a trust association with an entire cluster.

The mode in which the nodes are attested and the mode of decryption and secure key introduction is also an equally important question, but will be a question for a future blog post.

For the rest of this blog post we will assume the key model of “node”. Thus, the association of keys to nodes will be 1:1.

Process of Encryption & Decryption

Flow of Encryption/Decryption in a k8s cluster

Let us now look at a sample flow of a single node kubernetes cluster:

1. Before we start encrypting and running containers, we need to generate a public/private key pair for our node. We will make the public key accessible to the encryptor, and the private key should be accessible to the node running the encrypted container image.
2. The first step in the process is to encrypt the container image. This would be done by a developer or a process in a DevSecOps pipeline. The public key would be made available to this process to encrypt the image. The image is then uploaded to a registry.
3. The request of a pod with the encrypted image can then be created.
4. Kubernetes schedules the pod to a node, and the kubelet (located on the node) communicates with the container runtime via the Container Runtime Interface (CRI). Popular implementations of the runtime interface include CRI-O and containerd. The container runtime here then downloads and decrypts the image (with the access to the private key we generated earlier).

Try this out in Kubernetes!

We need to setup a single node kubernetes cluster which uses CRI-O as the runtime. Let’s start by setting up CRI-O.

Setting up CRI-O

Follow this link to build and setup CRI-O from source, https://github.com/cri-o/cri-o/blob/master/tutorials/setup.md

To ensure that we’ve setup CRI-O correctly, we will install crictl to verify our setup.

$ crictl -r unix:///run/crio/crio.sock info 
{
  "status": {
    "conditions": [
      {
        "type": "RuntimeReady",
        "status": true,
        "reason": "",
        "message": ""
      },
      {
        "type": "NetworkReady",
        "status": true,
        "reason": "",
        "message": ""
      }
    ]
  }
}

Setting up a single node k8s cluster

Once we have CRI-O up and running, we need to setup a single node kubernetes cluster to play with. While there are multiple ways to install single node kubernetes cluster, here we will quickly demonstrate how to do so by compiling it from the source.

1. Clone Kubernetes v1.17.1 or above
2. From the Kubernetes project directory, run (as root or with sudo):

# CONTAINER_RUNTIME=remote \
CONTAINER_RUNTIME_ENDPOINT='unix:///var/run/crio/crio.sock' \
./hack/local-up-cluster.sh

After that is done, we can verify our setup:

$ kubectl get nodes
NAME        STATUS   ROLES    AGE     VERSION
127.0.0.1   Ready    <none>   3d10h   v1.17.0-...

Alright, so now we have a single node kubernetes cluster that’s using CRI-O as the runtime.

Getting and running an encrypted image

For this demo, we have already hosted an encrypted image that can be used to quickly verify this feature. But if you would like to encrypt your own image please follow our earlier blogpost.

The sample encrypted image can be only be downloaded with read-only credentials. So please create a kubernetes ImagePullSecret with following command.

$ kubectl create secret docker-registry regcred \
--docker-server=us.icr.io \
--docker-username=iamapikey \ 
--docker-password=x_egxeGnaXi4GfKMbp0pYw0iErUAIjn5uYQIHTZ2RKof

And finally use this deployment yaml to submit the job. Save this file as enc-dply.yaml

$ cat <<EOF > enc-dply.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: enc-nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: us.icr.io/ecins/nginx:encrypted
        ports:
        - containerPort: 80
      imagePullSecrets:
        - name: regcred
EOF

Now submit the job using,

$ kubectl create -f enc-dply.yaml

Let’s see what happens:

$ kubectl get pods
NAMESPACE     NAME                                    READY   STATUS             RESTARTS   AGE
default       enc-nginx-deployment-759b78c658-55wl5   0/1     ImagePullBackOff   0          49s
default       enc-nginx-deployment-759b78c658-j8dj7   0/1     ImagePullBackOff   0          49s
default       enc-nginx-deployment-759b78c658-txrh9   0/1     ImagePullBackOff   0          49s

We notice that our pods failed to run due to failure in the image pull process. Let’s describe the pod to find out more.

$ kubectl describe pod enc-nginx-deployment-759b78c658-55wl5
<snip>
Events:
Type     Reason     Age                 From                Message
----     ------     ----                ----                -------
Normal   Scheduled  <unknown>           default-scheduler   Successfully assigned default/enc-nginx-deployment-759b78c658-55wl5 to 127.0.0.1
Normal   BackOff    26s (x2 over 70s)   kubelet, 127.0.0.1  Back-off pulling image "us.icr.io/ecins/nginx:encrypted"
Warning  Failed     26s (x2 over 70s)   kubelet, 127.0.0.1  Error: ImagePullBackOff
Normal   Pulling    12s (x3 over 102s)  kubelet, 127.0.0.1  Pulling image "us.icr.io/ecins/nginx:encrypted"
Warning  Failed     2s (x3 over 71s)    kubelet, 127.0.0.1  Failed to pull image "us.icr.io/ecins/nginx:encrypted": rpc error: code = Unknown desc = Error decrypting layer sha256:ecbef970c60906b9d4249b47273113ef008b91ce8046f6ae9d82761b9ffcc3c0: missing private key needed for decryption
Warning  Failed     2s (x3 over 71s)    kubelet, 127.0.0.1  Error: ErrImagePull

We see the error: “missing private key needed for decryption” which caused the failure! This is expected since we have not made the decryption key available yet!

Configuring decryption keys for CRI-O

In order to decrypt our sample image we need to save the following key where CRI-O can find it. By default CRI-O looks for keys in /etc/crio/keys folder. So go ahead and copy this file in that folder and save as myprivatekey.pem.

$ cat << EOF > /etc/crio/keys/myprivatekey.pem
-----BEGIN RSA PRIVATE KEY-----
MIICXAIBAAKBgQDoJBuK1hQ5aCbF93uE6jzRm8v5icUNFL5j+DO9hnM5j/8XFTzp
40N2M2/ObLf2qwmWSivwj5LJR/+5ceS8jqVBAcJpckwOXupu3A5o4KgJo15s6v57
4+0wfraNJ/OapqBc7lGFBsj+XwdmegwYYqy41DnYNSzYS4Mov+v7RI014wIDAQAB
AoGALCuiqfouAvZUWlrKv/Gp/OA+IY8bVW/bAj6Z6bgJeKxzhzrdSkuZ7IXBAnAh
WOgWfOhEEBPhhDcU635GXbJusuD/bLBJPOTxiwCFazffm8zVGSQCndfTVxgCM4hn
+5bH6o/cSGQ4E6SLJQeEr8y/J0bMlNMkOco9F1FL1ZgwXGECQQD9/mDwWLJjbdEa
jmGtoPspGz80XDb1jRI09jKDXB826/cBUD+X/P50aTkU+XSJXVfa5F6zhzf/O7C8
07bVnn2pAkEA6fmJ+Jx/Cupy7jRHzIdKAN/7T9QJBIXVDZLz5ulFWLjYkNotpkxk
f0ZSIOvlD7vv5lOifRFivd680XjxIATWqwJARJ35QFUl9DiRuhPnDYok8Cj9PT8A
VfwDhC1S3iv//s1mkINGeuANOhPHKQRvWEDQYEE72FJabWiJyamEhldn6QJAFjLw
3j+q5hQ8d1FKhqNHaDHYHEjX2jAAeNs6fOwhAjv3gDbTIfYZiuHXJPx8rTN9nXLN
9ePSZIVfkNhSuGD9JQJBAI+mobcxj7WkdLHuATdAso+N89Yt7xHoG49c8gz81ufP
vvLPtYytL4ftpiVO3fTfPP90ze8qYPiNaFqMHYDkQ+M=
-----END RSA PRIVATE KEY-----
EOF

Now let’s try to redeploy the job:

$ kubectl delete -f enc-dply.yml && kubectl create -f enc-dply.yaml

You should now be able to successfully see the encrypted images running on your kubernetes cluster!

$ kubectl get pods
NAMESPACE     NAME                                    READY   STATUS    RESTARTS   AGE
default       enc-nginx-deployment-759b78c658-5kwbq   1/1     Running   0          30s
default       enc-nginx-deployment-759b78c658-8mtgh   1/1     Running   0          30s
default       enc-nginx-deployment-759b78c658-d7zc6   1/1     Running   0          30s

Awesome — What’s next?

Congratulations on successfully running an encrypted image on kubernetes! This is just the tip of the iceberg of what can be done with encrypted images. As we’ve eluded to earlier, the mode in which the nodes are attested and the mode of decryption and secure key introduction are also an equally important part of how encrypted container images create compliance in the enterprise.