Open in app

Sign in

Write

Sign in

Container signing with Notary v2

Lachlan Evenson
9 min readNov 5, 2021

--

Photo by 戸山 神奈 on Unsplash

With the recent release of Notary v2 alpha 1 I wanted to dive in and share how the Notation CLI can be used to sign and verify container images (full disclosure — I am a collaborator on the Notary v2 project). You can find more details in the announcement blog. The blog also covers some interesting use cases including signing and attaching additional supply chain artifacts to a container image through the use of the ORAS artifacts spec. The attachment of these artifacts to a container image makes it easy to discover a graph of supply chain artifacts for example signatures, SBOMs, and image scan results using a tool to query the container registry.

As denoted via the alpha stability marker on the release it’s early days for Notary v2 so keep an eye on this space as things are moving fast. In this blog I’m going to cover the following:

  • Signing and verification workflows
  • Adding and verifying additional signatures
  • Signing and storage of additional supply chain artifacts to a container image

At the end of this blog you will be able to sign, store, and verify the full graph of supply chain artifacts associated with a container image using Notary v2. If the image below doesn’t already blow your mind then it soon will.

Full graph of signed supply chain artifacts associated with a container image

Let’s dive in…

Setup

First we’re going to download two different tools to run through this walkthrough. For the signing and verification part of the walkthrough we’re going to need the Notation CLI binary. Navigate to Notation releases page for details on how to download and install notation.

$ notation --versionnotation version 0.7.0-alpha.1

This next step is optional depending if you are interested in how to store a graph of supply chain artifacts in a registry and associate them with a specific container image.

(OPTIONAL) For the ORAS CLI — We’re going to build from source from the artifacts branch of the ORAS repo. I’m on a Mac so I used the make build-mac command. Feel free to take a look at the Makefile to build for your platform.

$ oras versionVersion:        0.11.1+unreleased
Go version:     go1.17.1
Git commit:     7e9d3a890899a190f96fac617ec20beb2926f276
Git tree state: clean

Finally, we can define some environment variables to help through the process. In this example I’m using a test Azure Container Registry endpoint which supports the ORAS artifacts spec. If you don’t have access to registry that support ORAS artifacts spec you can run a version of the CNCF Distribution registry that provides support for the ORAS Artifacts spec. Please refer to the Local Setup section below for more details.

export ACR_NAME=wabbitnetworks
export REGISTRY=$ACR_NAME.azurecr-test.io
export REPO=${REGISTRY}/lachie/net-monitor
export IMAGE=${REPO}:v1# AUTH TOKENS FOR REGISTRY ACCESS
export NOTATION_USERNAME='<TOKEN_USERNAME>'
export NOTATION_PASSWORD='<SECRET_TOKEN>'docker login $REGISTRY -u $NOTATION_USERNAME -p $NOTATION_PASSWORD

Local Setup

In the case that you don’t have access to a container registry that support ORAS artifacts spec you can run through the following demo using a local setup. You will need Docker setup on your machine for this work. Run a local container registry by running the following command:

docker run -d -p ${PORT}:5000 ghcr.io/oras-project/registry:v0.0.3-alpha

Now configure Notation to be able to use a registry without HTTPS (remember, this is only to test the demo). I’m using a Mac so the PATH to this file may be different, I’ve seen ~/.config/notation on Linux hosts.

echo '{"insecureRegistries": ["localhost:5000"]}' > /Users/<USER>/Library/Application Support/notation/config.json

Finally, set the following environment variables.

export PORT=5000
export REGISTRY=localhost:${PORT}
export REPO=${REGISTRY}/lachie/net-monitor
export IMAGE=${REPO}:v1

Signing your Container image

In this section we are going to use the Notation CLI to sign a container and push the signature to container registry with ORAS artifacts spec support. We will then verify the signature. Here is the container we are going to be working on throughout this blog (In order to change this to your own container registry simply update the REGISTRY environment variable to your own):

$ echo $IMAGEwabbitnetworks.azurecr-test.io/lachie/net-monitor:v1

Build and push the container image. (Again you can use a local registry to test this out — check out the setup section for details on how to run that locally).

$ docker build -t $IMAGE https://github.com/wabbit-networks/net-monitor.git#main$ docker push $IMAGE
...
The push refers to repository [wabbitnetworks.azurecr-test.io/lachie/net-monitor]
e2eb06d8af82: Mounted from net-monitor
v1: digest: sha256:a0fc570a245b09ed752c42d600ee3bb5b4f77bbd70d889... size: 527

Now that we have a container image in a container registry we can use the Notation CLI to sign that container image. First we need to generate a key and certificate to sign the image. You can do that using the following command:

$ notation cert generate-test --default "wabbit-networks.io"generating RSA Key with 2048 bits
generated certificates expiring on 2022-10-29T21:15:32Z
wrote key: /Users/<USER>/Library/Application Support/notation/key/wabbit-networks.io.key
wrote certificate: /Users/<USER>/Library/Application Support/notation/certificate/wabbit-networks.io.crt
wabbit-networks.io: added to the key list
wabbit-networks.io: marked as default

Now that we have a key and certificate we can sign the container image.

$ notation sign $IMAGEsha256:a0fc570a245b09ed752c42d600ee3bb5b4f77bbd70...

The previous step signed the container image and pushed that signature to the container registry we configured as part of the setup. Now we can list the signatures associated with the container registry using the following command:

$ notation list $IMAGEsha256:54553368a416f44a6bcb7aabbb14b5b6da9ceafbea02f0d34...

Verifying the signature

Now that we’ve signed the container image and pushed the signature to a container registry, let’s verify the signature using the Notation CLI.

$ notation verify $IMAGE2021/10/29 14:17:55 trust certificate not specified

The output of the command above indicates that we haven’t yet configured a certificate that we trust for verification. Let’s go ahead and add that then perform the verify again. We will use the notation cert add command to tell the Notation CLI that we want to trust the certificate we created earlier.

$ notation cert add --name "wabbit-networks.io" /Users/<USER>/Library/Application\ Support/notation/certificate/wabbit-networks.io.crt
wabbit-networks.io

If we run the verify command again we can now verify the signature on the container image.

$ notation verify $IMAGEsha256:a0fc570a245b09ed752c42d600ee3bb5b4f77bbd70d...

Let’s also take a look at the Notation CLI config file to see which keys we are using to sign and which ones we trust for verification (Remember I’m on a Mac so the PATH of this config file may change depending on your platform).

$ cat /Users/<USER>/Library/Application\ Support/notation/config.json{
    "verificationCerts": {
        "certs": [
            {
                "name": "wabbit-networks.io",
                "path": "/Users/<USER>/Library/Application Support/notation/certificate/wabbit-networks.io.crt"
            }
        ]
    },
    "signingKeys": {
        "default": "wabbit-networks.io",
        "keys": [
            {
                "name": "wabbit-networks.io",
                "keyPath": "/Users/<USER>/Library/Application Support/notation/key/wabbit-networks.io.key",
                "certPath": "/Users/<USER>/Library/Application Support/notation/certificate/wabbit-networks.io.crt"
            }
        ]
    },
    "insecureRegistries": []
}

Discovering supply chain artifacts associated with a container image

This part is really cool — using the ORAS cli we can query the container registry for supply chain artifacts associated with a specific container image. Remember, you need a registry that supports ORAS artifacts specification. There are details for how to run a local container registry with support in the setup section. In this example I’m using an Azure Container Registry endpoint that supports the ORAS artifacts spec.

$ oras discover -o tree -u $NOTATION_USERNAME -p $NOTATION_PASSWORD $IMAGEwabbitnetworks.azurecr-test.io/lachie/net-monitor:v1
└── application/vnd.cncf.notary.v2.signature
    └── sha256:db09eb93b235fd5135feff3cae49e48955e3d5...

The output above shows that there is a Notary v2 signature associated with the container image. We can also pull the manifests from the registry to take a look at how they are stored. In the output below we see the digest of the container image itself with the v1 tag. In addition, the signature digest without tags. You may have noticed that this digest is identical to the sha256 output from the command above.

$ az configure --default acr=wabbitnetworks$ az acr repository show-manifests -n wabbitnetworks  --repository lachie/net-monitor -o jsonc[
  {
    "digest": "sha256:a0fc570a245b09ed752c42d600ee3bb5b4f77bbd70d8898780b7ab43454530eb",
    "tags": [
      "v1"
    ],
    "timestamp": "2021-10-29T21:12:28.7396899Z"
  },
  {
    "digest": "sha256:db09eb93b235fd5135feff3cae49e48955e3d504fbac82683a853d5f1e32a0dc",
    "tags": [],
    "timestamp": "2021-10-29T21:16:16.6682125Z"
  }
]

Adding a Second Signature

In some cases you will want to use a signed image from an upstream source, store it in your container registry and sign it with your own signing key. Let’s walk through how that works using the Notation CLI. For the purposes of the demo I’m going to run the next commands on different dev machine without any prior Notation CLI configuration.

First lets generate a new signing key.

$ notation cert generate-test --default "acme-rockets.io-library"generating RSA Key with 2048 bits
generated certificates expiring on 2022-10-29T21:41:45Z
wrote key: /Users/<USER>/Library/Application Support/notation/key/acme-rockets.io-library.key
wrote certificate: /Users/<USER>/Library/Application Support/notation/certificate/acme-rockets.io-library.crt
acme-rockets.io-library: added to the key list
acme-rockets.io-library: marked as default

We can also add the signing key we just created for verification to Notation CLI.

$ notation cert add --name "acme-rockets.io-library" /Users/<USER>/Library/Application\ Support/notation/certificate/acme-rockets.io-library.crtacme-rockets.io-library

Let’s confirm our signing key and certificate in the Notation CLI configuration.

$ cat /Users/<USER>/Library/Application\ Support/notation/config.json{
    "verificationCerts": {
        "certs": [
            {
                "name": "acme-rockets.io-library",
                "path": "/Users/<USER>/Library/Application Support/notation/certificate/acme-rockets.io-library.crt"
            }
        ]
    },
    "signingKeys": {
        "default": "acme-rockets.io-library",
        "keys": [
            {
                "name": "acme-rockets.io-library",
                "keyPath": "/Users/<USER>/Library/Application Support/notation/key/acme-rockets.io-library.key",
                "certPath": "/Users/<USER>/Library/Application Support/notation/certificate/acme-rockets.io-library.crt"
            }
        ]
    },
    "insecureRegistries": []
}

Now we are ready to add an additional signature to the container image using the acme-rockets key that we just created.

$ notation sign $IMAGEsha256:a0fc570a245b09ed752c42d600ee3bb5b4f77bbd70d88...

Now let’s verify the signature.

$ notation verify $IMAGE sha256:a0fc570a245b09ed752c42d600ee3bb5b4f77bbd70d...

Finally we can confirm that the container image does indeed have two signatures associated with it.

$ oras discover -o tree \
  -u $NOTATION_USERNAME -p $NOTATION_PASSWORD $IMAGEwabbitnetworks.azurecr-test.io/lachie/net-monitor:v1
└── application/vnd.cncf.notary.v2.signature
    ├── sha256:db09eb93b235fd5135feff3cae49e48955e3d...
    └── sha256:d12da8f8c7c31177805018952cc99e9e0813f...

How cool is that! Adding multiple signatures enables you to verify those images using specific certificates.

Adding and signing additional supply artifacts

Another use case I would like to demonstrate is how to add and sign additional supply chain artifacts and associate them with a container image.

Keeping the importance of container supply chain security in mind we may want to add a signed SBOM (that we can verify) along with image scan results for example. Let’s walk through doing that.

We are going to generate an example SBOM and push that to the container registry.

echo '{"version": "0.0.0.0", "artifact": "'${IMAGE}'", "contents": "good"}' > sbom.jsonoras push $REPO \
  --artifact-type 'sbom/example' \
  --subject $IMAGE \
  -u $NOTATION_USERNAME -p $NOTATION_PASSWORD \
  ./sbom.json:application/jsonUploading 2a1ee8cf78a2 sbom.json
Pushed wabbitnetworks.azurecr-test.io/lachie/net-monitor
Digest: sha256:40eb5d4598166a735ca99f14a6f81456c284...

Now we can sign the SBOM which allows us to again verify that signature against a configured trusted source. To do that we need the digest of the SBOM (this could be made easier with an SBOM tool that discovers SBOMs associated with a container image) that we just pushed from the registry. Once we have that, we can sign it.

$ SBOM_DIGEST=$(oras discover -o json \
                --artifact-type sbom/example \
                -u $NOTATION_USERNAME -p $NOTATION_PASSWORD \
                $IMAGE | jq -r ".references[0].digest")$ notation sign $REPO@$SBOM_DIGESTsha256:40eb5d4598166a735ca99f14a6f81...

Now we can verify the graph of resources associated with the container image which should now include not only two signatures, but also an SBOM with an associated signature.

$ oras discover -o tree \
  -u $NOTATION_USERNAME -p $NOTATION_PASSWORD $IMAGEwabbitnetworks.azurecr-test.io/lachie/net-monitor:v1
├── application/vnd.cncf.notary.v2.signature
│   ├── sha256:db09eb93b235fd51...
│   └── sha256:d12da8f8c7c31177...
└── sbom/example
    └── sha256:40eb5d4598166a735...
        └── application/vnd.cncf.notary.v2.signature
            └── sha256:8c0192b56...

I cannot stress how important it is to be able to associate and discover the supply chain artifacts associated with a container image.

Now we are going to scan the container image using Snyk and again sign and store the scan results and signature in the container registry associated with that container image.

$ docker scan --json $IMAGE > scan-results.json$ cat scan-results.json | jq{
  "vulnerabilities": [],
  "ok": true,
  "dependencyCount": 14,
  "policy": "# Snyk (https://snyk.io) policy file, ...,
  "isPrivate": true,
  "licensesPolicy": null,
  "packageManager": "apk",
  "ignoreSettings": null,
  "docker": {
    "baseImage": "alpine:3.14.2",
    "baseImageRemediation": {
      "code": "NO_REMEDIATION_AVAILABLE",
      "advice": [
        {
          ...
        }
      ]
    }
  },
  "summary": "No known vulnerabilities",
  "filesystemPolicy": false,
  "uniqueCount": 0,
  "projectName": "docker-image|wabbitnetworks.azurecr-test.io/lachie/net-monitor",
  "platform": "linux/amd64",
  "path": "wabbitnetworks.azurecr-test.io/lachie/net-monitor:v1/lachie/net-monitor"
}

Now we can push the scan results to the container registry.

$ oras push $REPO \
  --artifact-type application/vnd.org.snyk.results.v0 \
  --subject $IMAGE \
  -u $NOTATION_USERNAME -p $NOTATION_PASSWORD \
  scan-results.json:application/jsonPushed wabbitnetworks.azurecr-test.io/lachie/net-monitor
Digest: sha256:84f0a6d3c75ea30b17cd8f269f1b153b...

We will sign the scan results and store the signature in the container registry associated with the container image.

$ SCAN_DIGEST=$(oras discover -o json \
               --artifact-type application/vnd.org.snyk.results.v0 \
               -u $NOTATION_USERNAME -p $NOTATION_PASSWORD \
               $IMAGE | jq -r ".references[0].digest")$ notation sign $REPO@$SCAN_DIGESTsha256:84f0a6d3c75ea30b17cd8f269f1b153b4a736d48df...

Finally, let’s confirm that we see the complete graph of supply chain artifacts associated with out container image.

$ oras discover -o tree -u $NOTATION_USERNAME -p $NOTATION_PASSWORD $IMAGEwabbitnetworks.azurecr-test.io/lachie/net-monitor:v1
├── application/vnd.cncf.notary.v2.signature
│   ├── sha256:db09eb93b235fd5135feff3cae49e489...
│   └── sha256:d12da8f8c7c31177805018952cc99e9e...
├── sbom/example
│   └── sha256:40eb5d4598166a735ca99f14a6f81456...
│       └── application/vnd.cncf.notary.v2.signature
│           └── sha256:8c0192b567004fa7461d33f7...
└── application/vnd.org.snyk.results.v0
    └── sha256:84f0a6d3c75ea30b17cd8f269f1b153b...
        └── application/vnd.cncf.notary.v2.signature
            └── sha256:53ee353f0be08e8edba731cc...

Summary

It’s still early days for the Notary v2 project and I’m really excited to see the progress. The additional work that’s happening in the ORAS artifacts spec is also really promising when used in context of container supply chain security.

If you’re interested in learning more of have any questions feel free to reach out to me on Twitter or in the Notary community. Also — Steve Lasker has a recorded version of the same demo if you would like to see it in action.

Kubernetes
Containers
Cloud Native
Cloud Native Security

--

--

Lachlan Evenson

Written by Lachlan Evenson

377 Followers

Husband | Father of three | Youtuber | Containers @Azure | 🇦🇺 | Time Traveller | CloudNative Ambassador + Mercenary | CKA | Opinions are my own.

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams