Connaisseur meets Cosign

Verifying Container Image Signatures from an OCI Registry in Kubernetes

With Connaisseur adding experimental support for Cosign, you can now verify OCI container image signatures in Kubernetes. Here is how.

Software supply chain security is an important but often underrated topic that has recently attracted significant attention due to the Solarwinds hack. A critical part of supply chain security is integrity and provenance of software, i.e. that software deployed to production has not been tampered with and comes from a valid source. This is commonly achieved by signing software after building it and then verifying the resulting signatures before deployment. Though important and despite some discussions, there currently is no default solution to verify container images when deploying them to Kubernetes. Enter Connaisseur and Cosign!

Photo by Romain Dancre on Unsplash

In the following section, I will give a warp speed overview of the central concepts behind Connaisseur, Cosign and the underlying infrastructures, many of which are worth a separate read and I’ll try to point to additional resources for that. In case you are only interested in how to get started, feel free to directly jump to the next section.

Sign, Store, Verify

Connaisseur is an admission controller for Kubernetes with a core focus on two things: container image signature verification and trust pinning. It is described in great detail elsewhere. In short, resource creation or update requests sent to the Kubernetes cluster are intercepted by the admission controller that identifies all container images and verifies their signatures against pre-configured public keys. Based on the result, it either accepts or denies those requests. Thus, integrity and selected provenance of deployed container images are ensured. Besides this central function, Connaisseur also provides some convenience features for practical use like detection mode (warn but do not deny invalid requests) and alerting (issue validation-based alerts to webhooks). If you have ever worked in DevSecOps, you know these two in conjunction come in handy and might spare you some tough nights.

While developing Connaisseur, we took great care to ensure usability and compatibility with a wide variety of (managed) Kubernetes solutions and container registries. The supported image signatures, however, were so far solely based on Docker Content Trust (DCT) which, in turn, builds on Notary (v1), a server (somewhat¹) separate to the registry to store and manage the signature data. Together, these implement The Update Framework (TUF), a very solid design pattern to protect against various attacks on software distribution and update systems. You can find a very comprehensible description of TUF in a recent article and a talk explaining in detail how DCT, Notary and TUF all tie together.

Notary v1 has served as a solid basis for signing and verifying container images in the past few years. However, some limitations — mostly related to it being a (somewhat¹) separate server from the registry — have caused challenges to its usability and scalability in multi-registry scenarios which has hindered widespread adoption so far. As a result, the Open Container Initiative (OCI; responsible for designing open standards for containers, e.g. the runtime-spec and image-spec) adjusted the image specification in order to allow placing signatures alongside images in the same registry, and corresponding work on Notary v2 was started.

In parallel, the Linux Foundation Project Sigstore to improve usability and security of signing and verifying open source software was recently announced:

Just like how Let’s Encrypt provides free certificates and automation tooling for HTTPS, sigstore provides free certificates and tooling to automate and verify signatures of source code. Sigstore also has the added benefit of being backed by transparency logs, which means that all the certificates and attestations are globally visible, discoverable and auditable.

Among its tooling is Cosign for creating, storing and verifying container image signatures in such OCI registries. Within the registry, the signatures directly appear as tags of the image linked to the associated image via the digest:

Notice how the signature tag below corresponds to the sha256 digest of the image tag ‘latest’ above.

Furthermore, Cosign aims to integrate with the central Sigstore infrastructure such as Rekor, the signature transparency log to e.g. protect against key compromises, and Fulcio, the PKI for code signing certificates for OpenID connect-based signing. Besides its remarkably straight-forward usage, the commitment to compatibility and the general focus within Sigstore to deliver solid and user-friendly solutions to common supply chain security threats make Cosign a strong candidate for container image signatures.

Now with the latest release, Connaisseur adds support for Cosign-based signatures!

⚠️ As Cosign and Sigstore are currently still experimental and under heavy development and so is our integration, we also consider this an experimental feature that might be unstable over time ⚠️

How to verify container image signatures in Kubernetes

Signing images with cosign and verifying these with Connaisseur is only a matter of minutes:

Let’s go step by step! We will start with Cosign. You can download the latest binary from the release page. At the time of writing, we support version 0.2.0 and the signature specification is likely to change, so you may have to check for the currently supported version in our docs. We use the Cosign binary to create a private-public key pair:

cosign generate-key-pair

You will be prompted to set a password, after which a private (cosign.key) and public (cosign.pub) key file are created. These can now be used to sign and verify images. We will get to that later.

Next, we setup Connaisseur. I’ll assume you have a running Kubernetes cluster for testing, docker, git, helm, kubectl, make, openssl and yq(≥v4) are installed, you are logged into docker via docker login and your kubectl context is properly set. We start by cloning the repository:

git clone https://github.com/sse-secure-systems/connaisseur.git
cd connaisseur

Connaisseur is configured via the helm/values.yaml. Two changes are required here: Firstly, we need to set it to use the previously created public key for verification by copying the content of cosign.pub to the rootPubkey key:

# Replace the actual key part with your own key
  rootPubKey: |
    -----BEGIN PUBLIC KEY-----
    MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEvtc/qpHtx7iUUj+rRHR99a8mnGni
    qiGkmUb9YpWWTS4YwlvwdmMDiGzcsHiDOYz6f88u2hCRF5GUCvyiZAKrsA==
    -----END PUBLIC KEY-----

Secondly, we need to set the validation engine to use Cosign via the isCosign key:

  isCosign: true

Finally after installing Connaisseur, we are good to go:

make install

Once the installation has finished, we can check that Connaisseur is running via kubectl get all. To test our setup, we need to create and push a suitable image. You can use your own Dockerfile or use our test image:

# Here, $IMAGE is REPOSITORY/IMAGE_NAME:TAG
docker build -t $IMAGE -f setup/Dockerfile .
docker push $IMAGE

Since we have not signed this image, it will be denied when we try to run it:

kubectl run demo --image=$IMAGE
#RESULT> Error from server: admission webhook "connaisseur-svc.connaisseur.svc" denied the request: no trust data for image "$IMAGE".

So let’s use the private cosign.key we created earlier to sign the image:

cosign sign -key cosign.key $IMAGE

And try to run it again:

kubectl run demo --image=$IMAGE
#RESULT> pod/signed created

Voila! You made it and ran your first cosigned container image validated via Connaisseur in Kubernetes 🥳

What’s next?

We are excited to add initial support for Cosign to Connaisseur and aim to extend the compatibility. Specifically, we hope to tackle the following topics in the upcoming months:

• test and improve Cosign integration
• extend support to further Sigstore and Cosign features, such as Rekor support and keyless signatures
• provide multi-key, multi-validator support to facilitate usage in more complex settings, e.g. multiple registries, different signing solutions, or more structured development organization

In summary, it is all about usability and compatibility, and this is where Sigstore/Cosign and Connaisseur meet with the goal to “make signatures invisible infrastructure”². Security ultimately boils down to making it exorbitantly more expensive to attack your infrastructure than to defend it. This requires not only secure, but also highly user-friendly solutions.

Improving Connaisseur needs your support, so please share your feedback on the new feature with us and stay tuned 🚀

Further information, additional configuration and extensive guides can be found in the GitHub repositories of Connaisseur and Cosign.

¹Notary v1 is essentially a separate service with its own database and API. However, it is typically attached to a registry as a sidecar and shares its authentication to, for example, avoid unauthorized creation/modification of trust data or to not reveal information on private images. As a result, signatures cannot be simply moved between registries and increasingly common multi-registry scenarios also mean (somewhat) separate notaries with separate keys and separate authentication 😉

²Taken from Cosign Readme