Trust but verify

AKA Never trust your cloud native friends

This is a true story. Not in the style of the Cohen brothers. A story of ordinary security in the cloud native world.

You have your K8S cluster well managed both in terms of security and observability, but something unlikely will happen.

You are sure to have protected your supply chain factory:

• The source code is scanned to detect flaws.
• You have time to apply the rules of your organization with static analysis with https://www.conftest.dev/ for example
• You have installed operators in K8S to check at runtime that the rules of good practice are there: no container in root, no elevation of privileges allowed… all that is possible and recommended classically
• You even went to install your own registry and build your secure images and validate them during build process with Google Container Structure Tests
• Your image providers for your containers are identified and you can be sure that only your CD tools can deploy these images. Whether it’s push or pull GitOps, you’re confident.

Everything is going well but… you forgot Jean-Pierre (sorry if a Jean-Pierre reads this article) has just connected in BYOD mode in his favorite brewer. Like every day, he chats and works without worrying that he is potentially being spied on and will one day connect to the wrong wifi. A beautiful honeypot that was just waiting for him.

Jean-Pierre will have to patch a Docker image on a registry. For this nothing serious, he has automated everything and nothing bad should happen because his secrets are well protected.

What Jean-Pierre doesn’t know is that it is possible through” Man in the Middle“ attack to replace the image that he will push on the registry with another image.

This has unfortunately been known for a very long time, but Jean-Pierre never thought that this applied too to container images.

Docker historically dealt with this for a long time but the so-called historical solution “Notary” has never been a model of simplicity for its implementation and maintenance.

Once again the problem is not the tool but the thing between the keyboard and the screen.

Other actors have tackled this problem. Since Notary has seen a v2 and another Cosign project has since also appeared.

In my opinion, the projects of the Sigstore ecosystem are certainly the ones to follow in the months and years to come

• Cosign : For container signing, verification and storage in an Open Container Initiative (OCI) registry, making signatures invisible infrastructure.
• Rekor : A built in transparency and timestamping service, Rekor records signed metadata to a ledger that can be searched, but can’t be tampered with.
• Fulco : A free root certification authority, issuing temporary certificates to an authorized identity and publishing them in the Rekor transparency log.

Check out their entire ecosystem. The next article will be dedicated to Fulco and Rekor.

Let’s come back to our subject and especially to Jean-Pierre who will discover the wall of reality in his face.

Jean-Pierre thought to push an image of his workstation (whether it was built automatically or not does not change anything) and deployed it. On arrival the hacker replaced the OCI image with its own in the dev cluster.

The only thing that can save Jean-Pierre from then on would be K8S-specific mechanisms that will verify the artifact.

Fortunately for him, Emma, ​​CSO, had an operator installed in the cluster Kubernetes to guard against this kind of malicious action. As uncritical as it is, a cluster remains a cluster. At the security level, a dev cluster also deserves all the attention on the security side.

She asked the platform engineers teams to install an operator like Connaisseur.

Connaisseur ensures integrity and provenance of container images in a Kubernetes cluster. To do so, it intercepts resource creation or update requests sent to the Kubernetes cluster, identifies all container images and verifies their signatures against pre-configured public keys. Based on the result, it either accepts or denies those requests.

Connaisseur is developed under three core values: Security, Usability, Compatibility. It is built to be extendable and currently aims to support the following signing solutions:

• Notary (V1) / Docker Content Trust
• sigstore / Cosign
• Notary V2 (PLANNED)

Even if Emma has confidence in her colleagues, she still prefers to check on their actions.

To have the right to push directly into a cluster, you must prove your identity and for that have given your public key to the Platform Engineer teams who will have installed (automatically of course) the key.

• Generate a (private/public) key pair

cosign generate-key-pair
Enter password for private key:
Enter again:
Private key written to cosign.key
Public key written to cosign.pub

• Sign the container image with the private key & store the signature in the container registry

cosign sign — key cosign.key nmuller/congruence

• locate the signatures for container images and verify them using known public key

cosign triangulate nmuller/congruence
index.docker.io/nmuller/congruence:sha256-fae5b13fa6f516bddf840442d1fda5e7c48a5f4e5ce5da220d02d562aa72abad.sig

If we were to manually check the image, it would be done like this

cosign verify --key cosign.pub nmuller/congruence
Verification for index.docker.io/nmuller/congruence:latest --
The following checks were performed on each of these signatures:
  - The cosign claims were validated
  - The signatures were verified against the specified public key
[{"critical":{"identity":{"docker-reference":"index.docker.io/nmuller/congruence"},"image":{"docker-manifest-digest":"sha256:fae5b13fa6f516bddf840442d1fda5e7c48a5f4e5ce5da220d02d562aa72abad"},"type":"cosign container image signature"},"optional":null}]

In our case, we will add into Connaisseur configuration file the public key

### VALIDATORS ###
image policy
  - name: default
    type: cosign
    trust_roots:
    - name: default
      key: |
        -----BEGIN PUBLIC KEY-----
      MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEQwy0icMaSLoIk45w3VU8Buy/WPf7
        ZtQIsgd+dKLiMBrLTFkNza8SmMeEM5mh836nGcm+ICU9mDMXaDR7WfcSqg==
        -----END PUBLIC KEY-----
# in detection mode, deployment will not be denied, but only prompted
# and logged. this allows testing the functionality without
# interrupting operation.
detectionMode: false

So before to deploy the new image, the admission controller will verify the signature. In our case, Jean-Pierre is safe thanks to Connaisseur/Cosign and Emma. The image will be rejected because during MITN the performer cannot have access to the private key to calculate.

T

The couple Connaisseur/Cosign is a killer feature for your K8S cluster.

Never forget to put in a secure location your private key (with passphrase)

To go further, the core feature set of cosign is considered ready for production use. This core set includes:

Key Management

• fixed, text-based keys generated using cosign generate-key-pair
• cloud KMS-based keys generated using cosign generate-key-pair -kms
• keys generated on hardware tokens using the PIV interface using cosign piv-tool
• Kubernetes-secret based keys generated using cosign generate-key-pair k8s://namespace/secretName

Artifact Types

• OCI and Docker Images
• Other artifacts that can be stored in a container registry, including:
• Tekton Bundles
• Helm Charts
• WASM modules
• eBPF modules
• (probably anything else, feel free to add things to this list)
• Text files and other binary blobs, using cosign sign-blob

Enjoy trying Connaisseur and if you already use Kyverno it is now possible to do this too.

You have the choice between different operators coming from Gatekeeper, Sigstore, SSE Secure Systems… Kyverno. Let’s go !

Thanks to Sigstore Project and SSE Secure Systems.

CONNAISSEUR - Verify Container Image Signatures in Kubernetes

To finish about the image at the beginning of this article, feel free to read https://en.wikipedia.org/wiki/Trust,_but_verify

Nicolas