How to handle SBOMs at scale in K8S

Bad attempt #10 to prompt an image :)

Today there is a (good) hype about Secure Supply chain software in a cloud native environment.

Secure supply chain level security , are (mostly) about producing valuable security artefacts along your application, such as the signature, the Software Bill Of Material, attestation etc

However, when you are running multiple cloud native applications , you end up with hundreds of container running. And for each container you have a SBOM, signatures, attestation and so on.

In my previous article, I described how to build a secure supply chain software in a multi cluster environment. You can find in my Github an extended version of the secure supply chain to support Ratify and Dependency Track.

Ratify

Ratify a framework to integrate scenarios that require verification of reference artifacts and provides a set of interfaces that can be consumed by various systems that can participate in artifact ratification. Source : https://ratify.dev/

Ratify enrich OPA Gatekeeper with 3 kinds of Verifier

• Cosign used to verify signatures generated using cosign.
• SBOM used to verify SBOM (Software bill of material). In SPDX JSON format.
• Vulnerability Report used to verify vulnerability reports. In SARIF format (from Trivy or Grype ).

In the following deployment, we are going to use Cosign and SBOM .

Trusting flow (Design greatly inspired by dagger.io)

How to deploy it

1. Install Gatekeeper on your cluster

helm install gatekeeper/gatekeeper  \
    --name-template=gatekeeper \
    --namespace gatekeeper-system --create-namespace \
    --set enableExternalData=true \               
    --set validatingWebhookTimeoutSeconds=5 \
    --set mutatingWebhookTimeoutSeconds=2 \
    --set externaldataProviderResponseCacheTTL=10s   

2. Install Ratify

Create the following values file :

cosign:
  key: |
    -----BEGIN PUBLIC KEY-----
    <your Cosign Public Key>
    -----END PUBLIC KEY-----
featureFlags:
  RATIFY_CERT_ROTATION: true
sbom:
  enabled: true
  disallowedLicenses:
  - "MPL"
  disallowedPackages: 
  - name: "busybox"
    version: "1.36.1-r0"

Add your own Cosign public key into cosign.key

sbom block configure which licences are disallowed and which packages are disallowed.

You can set a version for the disallowed package or leave it empty and all packages with the specified name will be disallowed regardless of the version.

Run the following commands to deploy ratify

helm repo add ratify https://deislabs.github.io/ratify
helm install ratify -n gatekeeper-system -f <values.yaml> ratify/ratify

Note: An interesting upcoming feature will be the capability of validating a SBOM inside an attestation (https://github.com/deislabs/ratify/issues/777)

Note: If you want to check SBOM signature, only notation is supported at the moment.

ratify/charts/ratify/values.yaml at main · deislabs/ratify

Upload SBOM to container registry

As mentioned above, SBOM cannot be part of an attestation (yet).

You need to use oras to upload your SBOM and attach it to the main container.

oras attach \
    --artifact-type application/spdx+json \
    myregistry.io/sbom/alpine:3.18.2 \
    sbom.spdx.json

Deploy a constraint

Ratify uses and extend the concept of contraint from Gatekeeper.

The following constraint will apply Cosign verifier and SBOM verifier as defined in the values.yaml of the helm chart deployment.

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: RatifyVerification
metadata:
  name: ratify-constraint
spec:
  enforcementAction: deny
  match:
    kinds:
      - apiGroups: [""]
        kinds: ["Pod"]
    namespaces: ["app"]

The enforcementAction field defines the action for handling Constraint violations. By default, enforcementAction is set to deny as the default behavior is to deny admission requests with any violation. Other supported enforcementActions include dryrun and warn. Refer to Handling Constraint Violations for more details.

The match field defines which resources the constraint will be applied to. It supports the following types of matchers:

• kinds accepts a list of objects with apiGroups and kinds fields that list the groups/kinds of objects to which the constraint will apply. If multiple groups/kinds objects are specified, only one match is needed for the resource to be in scope.
• scope determines if cluster-scoped and/or namespaced-scoped resources are matched. Accepts *, Cluster, or Namespaced. (defaults to *)
• namespaces is a list of namespace names. If defined, a constraint only applies to resources in a listed namespace. Namespaces also supports a prefix-based glob. For example, namespaces: [kube-*] matches both kube-system and kube-public.
• excludedNamespaces is a list of namespace names. If defined, a constraint only applies to resources not in a listed namespace. ExcludedNamespaces also supports a prefix-based glob. For example, excludedNamespaces: [kube-*] matches both kube-system and kube-public.
• labelSelector is the combination of two optional fields: matchLabels and matchExpressions. These two fields provide different methods of selecting or excluding k8s objects based on the label keys and values included in object metadata. All selection expressions are ANDed to determine if an object meets the cumulative requirements of the selector.
• namespaceSelector is a label selector against an object's containing namespace or the object itself, if the object is a namespace.
• name is the name of a Kubernetes object. If defined, it matches against objects with the specified name. Name also supports a prefix-based glob. For example, name: pod-* matches both pod-a and pod-b.

More details on How to use Gatekeeper | Gatekeeper

Validate signature with Cosign

Since we provided a cosign.key in the value.yaml the chart already deployed the Cosign Verifier and the store.

apiVersion: config.ratify.deislabs.io/v1beta1
kind: Verifier
metadata:
  name: verifier-cosign
spec:
  name: cosign
  artifactTypes: application/vnd.dev.cosign.artifact.sig.v1+json
  parameters:
    key: /usr/local/ratify-certs/cosign/cosign.pub
---
apiVersion: config.ratify.deislabs.io/v1beta1
kind: Store
metadata:
  name: store-oras
spec:
  name: oras
  parameters:
    cacheEnabled: true
    cosignEnabled: true
    ttl: 10

If you use an authenticated container registry, you can follow this doc to configure it

Ratify in action

The following container image jpgouin/spring-project has it’s sbom attached and is signed with the following public key :

-----BEGIN PUBLIC KEY-----
MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEOYMk7g3WDKOOWsjyglpWGlb9qjGk
jGoRWx4efs3P0d0qk4uyDHybCXhqEIifiRFuZXCHSzdUi5egWF9PKfZbvQ==
-----END PUBLIC KEY-----

The container is correctly signed

cosign verify jpgouin/spring-project:latest --key k8s://argo/cosign

Verification for index.docker.io/jpgouin/spring-project:latest --
The following checks were performed on each of these signatures:
  - The cosign claims were validated
  - Existence of the claims in the transparency log was verified offline
  - The signatures were verified against the specified public key

The OCI tree of jpgouin/spring-project shows that the SBOM is attached to the container image

oras discover  index.docker.io/jpgouin/spring-project:latest -o tree
index.docker.io/jpgouin/spring-project@sha256:02af01e142a3f2c10ff6453669c6733affd45bad6cb53d1f258c53b1d1ca8f62
└── application/spdx+json
    └── sha256:3731776049a73c79af8e2ce62bb84c48cc9042ed6e930fc642f411988d543a70

Let’s deploy the jpgouin/spring-project in the app namespace

k run test-ok --image jpgouin/spring-project:latest -n app
pod/test-ok created

The pod is deployed successfully in the namespace, meaning it matches the cosign verifier and sbom verifier.

Here is the log of ratify

time=2024-04-04T13:50:05.311208642Z level=info msg=verify result for subject docker.io/jpgouin/spring-project@sha256:02af01e142a3f2c10ff6453669c6733affd45bad6cb53d1f258c53b1d1ca8f62: {
  "isSuccess": true,
  "verifierReports": [
    {
      "subject": "docker.io/jpgouin/spring-project@sha256:02af01e142a3f2c10ff6453669c6733affd45bad6cb53d1f258c53b1d1ca8f62",
      "isSuccess": true,
      "name": "verifier-cosign",
      "type": "cosign",
      "message": "cosign verification success. valid signatures found",
      "extensions": {
        "signatures": [
          {
            "bundleVerified": false,
            "isSuccess": true,
            "signatureDigest": "sha256:db4ade28b9ee6f896a452eb68bc6735eef40745c9fe43364bc11da0ac72c6c48"
          }
        ]
      },
      "artifactType": "application/vnd.dev.cosign.artifact.sig.v1+json"
    },
    {
      "subject": "docker.io/jpgouin/spring-project@sha256:02af01e142a3f2c10ff6453669c6733affd45bad6cb53d1f258c53b1d1ca8f62",
      "isSuccess": true,
      "name": "verifier-sbom",
      "type": "sbom",
      "message": "SBOM verification success. No license or package violation found.",
      "extensions": {
        "creationInfo": {
          "created": "2024-03-13T13:40:27Z",
          "creators": [
            "Organization: Anchore, Inc",
            "Tool: syft-1.0.1"
          ],
          "licenseListVersion": "3.23"
        }
      },
      "artifactType": "application/spdx+json"
    }
  ]
}

Now let’s deploy an invalid image

k run test-ko --image busybox -n app
Error from server (Forbidden): admission webhook "validation.gatekeeper.sh" denied the request: [ratify-constraint] Subject failed verification: docker.io/library/busybox@sha256:c3839dd800b9eb7603340509769c43e146a74c63dca3045a8e7dc8ee07e53966

The pod is blocked by gatekeeper using ratify verifier because the container do not have a valid signature

Here is the log of ratify :

time=2024-04-04T13:51:08.156301962Z level=info msg=verify result for subject docker.io/library/busybox@sha256:c3839dd800b9eb7603340509769c43e146a74c63dca3045a8e7dc8ee07e53966: {
  "verifierReports": [
    {
      "subject": "docker.io/library/busybox@sha256:c3839dd800b9eb7603340509769c43e146a74c63dca3045a8e7dc8ee07e53966",
      "isSuccess": false,
      "message": "verification failed: Error: referrers not found, Code: REFERRERS_NOT_FOUND, Component Type: executor"
    }
  ]
}

Now let’s update ratify ‘s values to disallowed GPL-2.0-only licence

...
sbom:
  enabled: true
  disallowedLicenses:
  - "MPL"
  - "GPL-2.0-only"

Run the pod again, and it’s blocked by gatekeeper because the container contains a package with a GPL-2.0 licence :

k run test-ok --image jpgouin/spring-project:latest -n app
Error from server (Forbidden): admission webhook "validation.gatekeeper.sh" denied the request: [ratify-constraint] Subject failed verification: docker.io/jpgouin/spring-project@sha256:02af01e142a3f2c10ff6453669c6733affd45bad6cb53d1f258c53b1d1ca8f62

time=2024-04-04T13:52:46.164907718Z level=info msg=verify result for subject docker.io/jpgouin/spring-petclinic@sha256:c0dd9561c5240018a1c7dad3afa2de6ca4dcacb28180e8a0e6c84f9328ce584b: {
  "verifierReports": [
    {
      "subject": "docker.io/jpgouin/spring-petclinic@sha256:c0dd9561c5240018a1c7dad3afa2de6ca4dcacb28180e8a0e6c84f9328ce584b",
      "isSuccess": false,
      "name": "verifier-sbom",
      "message": "SBOM validation failed. Please review extensions data for license and package violation found.",
      "extensions": {
        "creationInfo": {
          "created": "2024-03-19T13:15:43Z",
          "creators": [
            "Organization: Anchore, Inc",
            "Tool: syft-1.0.1"
          ],
          "licenseListVersion": "3.23"
        },
        "licenseViolations": [
          {
            "License": "BSD-3-Clause AND EPL-1.0 AND GPL-2.0-only AND LGPL-2.1-only",
            "Name": "mysql-connector-j",
            "Version": "8.1.0"
          }
        ]
      },
      "artifactType": "application/spdx+json"
    }
  ]
}

Let’s update ratify ‘s values to disallowed caffeine package

...
sbom:
  enabled: true
  disallowedLicenses:
  - "MPL"
  - "GPL-2.0-only"
  disallowedPackages: 
  - name: "caffeine"
    version: "3.1.6"

Run the pod again, and it’s blocked by gatekeeper because the container contains the caffeine package:

k run test-ok --image jpgouin/spring-project:latest -n app
Error from server (Forbidden): admission webhook "validation.gatekeeper.sh" denied the request: [ratify-constraint] Subject failed verification: docker.io/jpgouin/spring-project@sha256:02af01e142a3f2c10ff6453669c6733affd45bad6cb53d1f258c53b1d1ca8f62

time=2024-04-08T11:22:03.946061691Z level=info msg=verify result for subject docker.io/jpgouin/spring-project@sha256:02af01e142a3f2c10ff6453669c6733affd45bad6cb53d1f258c53b1d1ca8f62: {
  "verifierReports": [
    {
      "subject": "docker.io/jpgouin/spring-project@sha256:02af01e142a3f2c10ff6453669c6733affd45bad6cb53d1f258c53b1d1ca8f62",
      "isSuccess": false,
      "name": "verifier-sbom",
      "message": "SBOM validation failed. Please review extensions data for license and package violation found.",
      "extensions": {
        "creationInfo": {
          "created": "2024-03-13T13:40:27Z",
          "creators": [
            "Organization: Anchore, Inc",
            "Tool: syft-1.0.1"
          ],
          "licenseListVersion": "3.23"
        },
        "licenseViolations": [
          {
            "License": "BSD-3-Clause AND EPL-1.0 AND GPL-2.0-only AND LGPL-2.1-only",
            "Name": "mysql-connector-j",
            "Version": "8.0.33"
          }
        ],
        "packageViolations": [
          {
            "License": "NOASSERTION",
            "Name": "caffeine",
            "Version": "3.1.6"
          }
        ]
      },
      "artifactType": "application/spdx+json"
    },
    {
      "subject": "docker.io/jpgouin/spring-project@sha256:02af01e142a3f2c10ff6453669c6733affd45bad6cb53d1f258c53b1d1ca8f62",
      "isSuccess": true,
      "name": "verifier-cosign",
      "type": "cosign",
      "message": "cosign verification success. valid signatures found",
      "extensions": {
        "signatures": [
          {
            "bundleVerified": false,
            "isSuccess": true,
            "signatureDigest": "sha256:db4ade28b9ee6f896a452eb68bc6735eef40745c9fe43364bc11da0ac72c6c48"
          }
        ]
      },
      "artifactType": "application/vnd.dev.cosign.artifact.sig.v1+json"
    }
  ]
} component-type=server go.version=go1.20.12 trace-id=01f06857-c3e5-4aff-997e-25e3d2bcc4d7

Combining these three capabilities provides a holistic approach to security in production Kubernetes environments. It helps to establishe a secure software supply chain by ensuring the integrity and authenticity of container images, enforcing compliance with licensing requirements, and mitigating risks associated with vulnerable software components. Moreover, by integrating these capabilities into OPA Gatekeeper, you can automate and enforce security policies at scale, streamlining the management of SBOMs and ensuring consistent security across your Kubernetes deployments. This automation reduces manual effort, minimizes human errors, and enhances overall security posture in production environments.

Manage exceptions

While, running it at scale, you want to always enforce a high level of security and trust but you have to be pragmatic and balance security and operational needs in Kubernetes environments. Here’s how you could add flexibility in the enhanced security:

Have an “exception” namespace, so you establish a controlled environment where exceptions to security policies can be managed.

Deploy a constraint in the “exception” namespace with warn enforcementAction so deployments in this namespace are still subject to scrutiny and monitoring. As the constraints are set to warn mode, you can trigger notifications for non-compliance without blocking deployments outright.

Enforce fine grain RBAC, Network policies for the “exception” namespace.

Ratify has currently an open issue to support multi tenancy : https://github.com/deislabs/ratify/issues/743

Dependency Track

Dependency-Track is an intelligent Component Analysis platform that allows organizations to identify and reduce risk in the software supply chain. https://dependencytrack.org/

Dependency Track provide a comprehensive scoring solution based on SBOMs.

Because a SBOM can’t be read by a human and since your secure software supply chain will now generate a ton of SBOMs, Dependency Track can assist your team by establishing a scoring mechanism for each SBOM.

How to deploy it

You can deploy it in your Kubernetes cluster but it’s still WIP (still my favorite option)

GitHub - DependencyTrack/helm-charts: Helm Charts for Dependency-Track

You can also deploy it in a VM using docker using docker-compose .

Deploying Docker Container

Depencency Track in action

Login into Dependency Track , go to Administration and create a new team. Add BOM_UPLOAD permission to the team. And save the API key .

Create a new project

Get the project ID from Project Details

You can now stream SBOM to Dependency Track using the API.

curl -X "POST" "https://<apiendpoint.your-dependency.track>/api/v1/bom" \
     -H 'Content-Type: multipart/form-data' \
     -H 'X-API-Key: <api_key>' \
     -F "project=<project_id>" \
     -F 'bom=@payload.json'

payload.json should be the SBOM of your application.

You can easily track components, licences and risk score for each container and evaluate what needs to be blocked in production (e.g configure in ratify )

Conclusion

In conclusion combining Dependency-Track with Ratify provides a solid way to manage SBOMs at scale. Dependency-Track’s thorough examination of SBOM, coupled with its scoring system, facilitates efficient prioritisation of security measures. When you link it up with Ratify, you can leverage your secure software supply chain and set up strict security rules based on Dependency-Track’s scores, stopping riskiest deployments in your clusters.