How to Use Image Signing in IBM Cloud Private 3.2.1

The newly released IBM Cloud Private version 3.2.1 comes with an image signing technology preview feature that provides extra security. Image signing brings IBM Cloud Private security capabilities that include Docker notary service and image enforcement policy together with Docker content trust to provide a trust infrastructure for container images.

Learn how to set up the image signing feature and deploy an application image to the IBM Cloud Private cluster.

Prerequisites

Make sure that the following components are installed.

• Notary service is installed on your IBM Cloud Private cluster.
• Docker environment is set up on another system.

The following resources are used for the purposes of this blog.

• The system with Docker installed is referred to as the “build system”.
• The IBM Cloud Private cluster is referred to as the “target system”.
• The IP address for the IBM Cloud Private master node is 9.1.2.3.
• The notary server URL is https://9.1.2.3:8443/notary.
• The Docker registry is artifactory hyc-cloud-private-docker-local.artifactory.swg-devops.com. You can use docker.io or set up your own registry to try it out.
• The application image used is nginxdemos/hello from docker.io. You can also build your own sample image.
• The signer key and certificate are created using OpenSSL.

Sign the Docker image

First, complete the following steps to sign the image by using Docker content trust on the build system.

1. Generate the key and certificate pair by using OpenSSL. Change the key.pem file permission to 400. We will use this key as the image signing key.

openssl req -newkey rsa:2048 -nodes -keyout key.pem -x509 -days 365 -out certificate.pem
chmod 400 key.pem

2. Load the key to the local Docker trust repository. When you run the command to load the key, you are prompted to enter a passphrase to protect the key. The passphrase is required when you execute other Docker content trust commands to access the key.

docker trust key load key.pem —name helloapp-signer

3. Log in to docker.io and pull the hello docker image. Retag the image with the artifactory repository name.

docker pull docker.io/nginxdemos/hello
docker tag nginxdemos/hello:latest hyc-cloud-private-docker-local.artifactory.swg-devops.com/integrate/hello-app:latest

4. Enable Docker content trust and point the trust server to the IBM Cloud Private notary service.

export DOCKER_CONTENT_TRUST=1
export DOCKER_CONTENT_TRUST_SERVER=https://9.1.2.3:8443/notary

5. Register the certificate generated in step 1 with the image repository. The certificate name specified in the command must match the key name specified in the command in step 2.

docker trust signer add --key certificate.pem helloapp-signer hyc-cloud-private-docker-local.artifactory.swg-devops.com/integrate/hello-app

If you are running this command for the first time, you will see message: “You are about to create a new root signing key passphrase. This passphrase will be used to protect the most sensitive key in your signing system…”.

The message occurs because the command creates an offline key as the root for the Docker content trust. This is a one-time operation. The next time that you run the command, it will prompt you for the root key passphrase. If you are using the certificate to add this image repository for the first time, the repository entry will be created on the Docker system.

You might see error, “ Error: error contacting notary server: x509: certificate signed by unknown authority”. The error means that the build system does not trust the notary server signer certificate. You need to add the notary certificate to the build system truststore.

6. Log in to the artifactory registry from the build system, sign, and push the image. The Docker sign operation signs the image and pushes the trust metadata to the notary server. It prompts you to enter the signer key passphrase. The Docker push operation pushes the image to the registry.

docker login hyc-cloud-private-docker-local.artifactory.swg-devops.com
docker trust sign hyc-cloud-private-docker-local.artifactory.swg-devops.com/integrate/hello-app:latest
docker push hyc-cloud-private-docker-local.artifactory.swg-devops.com/integrate/hello-app:latest

Deploy the signed image

Next, on the target system, in the default namespace, set up the image policy to enforce image signing.

1. Create an imagePullSecret called imagepull-secret. The secret stores the credentials to pull the image from the Docker registry.

kubectl create secret docker-registry imagepull-secret --docker-server=hyc-cloud-private-docker-local.artifactory.swg-devops.com  --docker-username=<username> --docker-password=<password or API key> --docker-email=<email> -n default

2. Create a signer secret called signer-secret. The name in the secret must match the signing key name. The publicKey in the secret is the certificate file of the signing key.

kubectl create secret generic signer-secret --from-literal=name=helloapp-signer --from-file=publicKey=certificate.pem -n default

3. Create the image policy and deploy it to the default namespace. IBM Cloud Private supports cluster image policy and image policy for a namespace. It comes with a default cluster image policy that whitelists image registries. An image policy defined for a namespace overrides the cluster image policy. The trust section in the policy for a repository defines whether the trust is enabled, and what the signer secret and trust server are. When the trust is enabled, only the signed image which is successfully verified by the image signing can be deployed on the cluster. In the image policy file, make sure you put the secret created in step 2 in the signerSecrets.

apiVersion: securityenforcement.admission.cloud.ibm.com/v1beta1
kind: ImagePolicy
metadata:
  name: demo-image-policy
spec:
  repositories:
  - name: hyc-cloud-private-docker-local.artifactory.swg-devops.com/integrate/*
    policy:
      trust:
        enabled: true 
        signerSecrets:
        - name: signer-secret
        trustServer: https://9.1.2.3:8443/notary

4. Create the deployment file for the hello-app application and deploy it to the default namespace. Make sure that you put the secret created in step 1 to imagePullSecrets in the deployment file. As the image signature is verified, the pod with the container is created successfully during deployment.

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: helloapp-signed
spec:
  replicas: 1 
  template:
    spec:
      containers:
      - image: hyc-cloud-private-docker-local.artifactory.swg-devops.com/integrate/hello-app:latest
        name: helloapp-signed
        ports:
        - containerPort: 80
      imagePullSecrets:
      - name: imagepull-secret
      restartPolicy: Always

Conclusion

With the image signing feature, you can ensure that the workload application image that is deployed to your private cloud environment is trusted. And it is not tampered with in the registry or on the way to the deployment. Enterprises can also integrate the feature into their CICD pipeline automation to ensure a greater level of security.

References

• Using a notary for image signing
• Image signing support for image policies

Acknowledgement

The author wants to acknowledge Jayashree Ramanathan, Suresh Kumar, Shrinath Sanjay Thube, Kyle Oldham and Lesia Cox from IBM Cloud Private team for their contribution to this article.