Validating Content Trust Metadata for Kubernetes Targeted Container Images at Deployment Time
An often ignored container image security best practice is to verify that the container images targeted for a Kubernetes cluster, have been created by a trusted publisher using Docker Content Trust.
The patterns on how the signed container images are created and published to a container registry like Azure Container Registry are very well documented (see links in the next section). On the image consumption side though, currently there is no native support/standard pattern in Kubernetes to validate this Content Trust information associated with a container image, each time it is being pulled on to a node.
In this post we take a look at some of the options available for us for validating the Content Trust Information associated with a container image, during different times, like when pulling the images to a node, or when pushing a new deployment to a K8s cluster. Then we go into details of one of the options where we validate the content trust information at deployment time using Azure Pipelines and Azure Container Registry
What is Content Trust
Before we get into the details of different options for pulling signed container images , and into details of pipelines for pushing and pulling signed container images, we need to have basic understanding of how content trust works, and type of keys associated with content trust (root keys, repository keys, delegation keys). The links below give details of these:
Kubernetes Native Support to Validate Content Trust Information for Container Images
It would be ideal if Kubernetes could be natively configured, such that signature of each container image is validated when being pulled to a Kubernetes Node / or when new resource (deployment, pod , etc) is created in the cluster, however such native support does not exist currently
There is a good discussion around this potential native support on this github issue. Once Notary v2 design/development is complete and registries like Azure Container Registry provide Notary v2 compliant APIs, then validation on a Kubernetes cluster should be possible as described in the Notary v2 signing scenarios. This native support is still a few months away in my opinion.
Let us look at some current options in absence of this native support.
Current options for Validating Image Content Trust Information for a container Image targeted for a Kubernetes Cluster
Let us consider a few of the options currently available to us:
Option 1: Connaisseur
- The Connaisseur project is an open source project
- At a high level the way it works is that it adds a mutating webhook to your Kubernetes cluster. This intercepts request for changes (such as creates /updates) to cluster resources like pods, deployments, daemonsets, etc, and validates that the container image associated with the Kubernetes resource has valid content trust data associated with it. If no valid content trust information is found then the mutating webhook signals a failure causing the resource modification to fail. If valid content trust information is found then the image digest associated with the image tag is fetched, and the image reference on the resource is updated to include the image digest. For instance if the we try to create a deployment with with “image: contrstacr.azurecr.io/myorg/signedrepo:133”, and from the content trust information we find that the image digest associated with this image is “b61a4a0a3a49ea8939a5…”, then the deployment would be patched to modify image to include the digest, like “image: contrstacr.azurecr.io/myorg/signedrepo:133@sha256:b61a4a0a3a49ea8939a5…”. This image detail (with the digest) for the resource is what gets persisted to etcd. The diagrams below illustrate how this works.
- To Summarize container image details are only persisted to etcd after validating the trust information, and this persisted information includes the image digest of the signed image tag
- For further details on how it works you can refer how-it-works
Option 2: Custom Admission Controller
- If using an open source project like Connaisseur is not an option for (for instance due to enterprise constraints)and you still want equivalent functionality, then one option would be to write your own Custom Admission webhook
- This may require substantial engineering effort, which needs to be considered
Option 3: Using Notary and Open Policy Agent
The post Ensure Content Trust on Kubernetes using Notary and Open Policy Agent describes how to enable content trust using Notary and Open Policy Agent . This post also goes through how Notary Server can be installed/configured in-cluster.
Option 4: Deployment time validation
Finally let us look at the deployment time validation option. If an open source project like Connaisseur is not an option, and neither is creating your own Admission Webhook, and neither is explicitly working with current version of Notary APIs, then this deployment time validation option may provide a good middle ground till the time Notary v2 is released.
- The validation involved in this option are similar to those performed by the mutating webhook
- The difference is that these checks are done at deployment time by a pipeline and not each time a relevant kubernetes resource (pod, deployment, etc) is modified.
- The Application Kubernetes deployment pipeline is triggered and the image tag to be deployed is passed on to the pipeline. The pipeline first checks the Content Trust Metadata for the given image tag. If no content trust data is found the pipeline fails giving the error that the Content Trust Metadata could not be found. If the content trust metadata is found then we modify the image tag to include the image digest, and then apply the changes to the Kubernetes Cluster.
- This approach is straight forward to implement, however we need to remember that if someone manually applies an un-trusted image tag by directly (say using kubectl) then unlike the admission webhook approach no content trust validation will take place. This approach only supports deployment time validation.
- Often this approach is combined with OPA Gatekeeper to validate that all images being pulled on the cluster are from the trusted container registry, and that the pulled images tags have embedded image digests.
In the next sections we will look at the Azure infrastructure setup and Azure pipelines with this approach in action.
Initial Setup for Pipelines
Before we get into details of pipelines for pushing and pulling signed container images, we need components like Azure Container Registry, Pipeline Service Connection, appropriate permissions etc in place. If you wish to follow along, these required components can be created by using the initialization script below.
Initialization Script
This initialization script configure the following:
At a high level our initialization script does the following:
- Creates a Resource Group where we will create the Azure Container Registry and Key Vault
- Creates an Azure Container Registry and enables content Trust for the registry
- Creates an Azure Key Vault, and then creates secrets which will be required by our push and pull pipelines like content trust delegation key / passpharase, and the content trust root and repository key pass phrases. Next these secrets are added to the key vault. Finally the generated delegation key is added as a trusted signer for our repository in Azure Container Registry
- Creates a service principal, and adds role assignments to enable access to read the key vault secrets created in the previous step. Additionally adds role assignments providing the service principal access to push, pull and sign images in the Azure Container Registry
- Creates a service connection in Azure DevOps associated with the service principal created in the above step
In order to use this script, the variable values at the top of this script need to be set to the relevant values for your Azure Subscription and Azure Pipelines Organization. The name prefix (NAME_PFX) variable value entered will determine the names of the resource group, Azure Container Registry, etc created
The full script is shown below.
The script
With this setup in place we can now look at our sample signed container image push pipeline.
Signed Container Image Push Pipeline
Here we look at a basic pipeline using a delegation key which has been granted AcrImageSigner permission on the Azure Container Registry repository.
- The first step of this pipeline makes the secret variables from Azure Key Vault available to the pipeline as variables. These include the delegation key used to sign the container image (acr-frompipeline-signing-key), the delegation key passphrase (deployment-key-passphrase), and the repository passphrase (repository-passphrase)
- The next steps builds a container image
- The final step (line 30 onwards) first prepares and loads the delegation key using the “docker trust load” command. After that the container image is signed using the delegation key and pushed to Azure Container Registry using the “docker trust sign command”.
The key portions of the pipeline logs of the final step are similar to the log abstract shown :
Login Succeeded
Loading key from "/home/vsts/.docker/trust/private/delegate.key"...Successfully imported key from /home/vsts/.docker/trust/private/delegate.keySigning and pushing trust data for local image contrstacr.azurecr.io/myorg/signedrepo:133...
.
.
digest: sha256:b61a4a0a3a49ea8939a5............... size: 1570Signing and pushing trust metadata
The output of the final line “docker trust inspect …” is similar to
SIGNED TAG DIGEST SIGNERS
133 b61a4a0a3a49ea8939a5....................... azdopipelineSIGNER KEYS
azdopipeline 0b00eb62b7ac
The output of the inspect command shows us the image digest (b61a….) associated with the signed tag 133, and also tells us the signing key (name: azdopipeline, key hash: 0b00……)which was used for signing this tag.
Deployment time Content Trust Information Validation Pipeline
As stated earlier it will be good idea to combine this approach with OPA Gatekeeper to validate that all images being pulled on the cluster are from the trusted container registry. The high level steps in this pipeline are as follows:
- Pull Content Trust information for image tag using “docker inspect command” (line 32 below)
- If no content trust information exists for the tag fail the pipeline (line 37 below)
- If content trust information exists and parse it to fetch the image digest (IMAGE_TAG_WITH_DIGEST), signing key (SIGNING_KEY), and then create the new image tag which includes the image digest (IMAGE_TAG_WITH_DIGEST). Please refer line 41–45 below for detail of this parsing.
- In the real world we would then modify or helm chart, deployment yaml, etc to use the image tag with the image digest (IMAGE_TAG_WITH_DIGEST). In our example in the last line (55) we do a docker pull of the image IMAGE_TAG_WITH_DIGEST.
Let us know look at the details of the script and output logs of the pipeline
Deployment Time validation Pull Pipeline Script
Sample log when no content trust information exists for tag (failure case)
For this scenario I uncommented line 12 (and commented line 9), since I had an unsigned container image unsignedrepo/unsignedapp:133 in my container registry
The tag contrstacr.azurecr.io/unsignedrepo/unsignedapp:133 is not signed##[error]Script failed with exit code: 1
Sample log when content trust information exists for tag (success)
Trust information for tag contrstacr.azurecr.io/myorg/signedrepo:133 found
.
IMAGE_DIGEST: b61a4a0a3a49ea8939a5......
SIGNED_TAG: 133
SIGNER_NAME: azdopipeline
SIGNING_KEY: 0b00eb62b7ac......
IMAGE_TAG_WITH_DIGEST: contrstacr.azurecr.io/myorg/signedrepo:133@sha256:b61a4a0a3a49ea8939a5......
.
.
contrstacr.azurecr.io/myorg/signedrepo@sha256:b61a4a0a3a49ea8939a5....: Pulling from myorg/signedrepo
.
.
Digest: sha256:b61a4a0a3a49ea8939a5...
Status: Downloaded newer image for contrstacr.azurecr.io/myorg/signedrepo@sha256:b61a4a0a3a49ea8939a5.......Thanks for reading this post. I hope you liked it. Please feel free to write your comments and views about the same over here or at @manisbindra
