Using Cert-Manager in OpenShift/OKD. Part 2— configuration and SSL certificate installation.

The article was initially published here: https://epam.github.io/edp-install/operator-guide/ssl-automation-okd/

Part 1 of this guide covers the cert-manager installation in OpenShift and integration with AWS Route53.

Part 2 consists of the cert-manager configuration and SSL certificate installation.

Part 3 describes certificates troubleshooting , renewals, and updating the kubeconfig.

Now that the cert-manager is installed and integrated with AWS Route53, let’s configure ClusterIssuers and Certificates.

Configure ClusterIssuers

There are two custom resources an Issuer and a ClusterIssuer. The difference between an Issuer and a ClusterIssuer is that Issuer is a namespaced resource, in our case, we need an issuer that will be available on the whole cluster.

1. Create the ClusterIssuer resource for Let’s Encrypt Staging and Prod environments that signs a Certificate using cert-manager.

Let’s Encrypt has a limit of duplicate certificates in the Prod environment. Therefore, we will also create a ClusterIssuer for Let's Encrypt Staging environment. By default, Let's Encrypt Staging certificates will not be trusted in your browser. The certificate validation cannot be tested in the Let's Encrypt Staging environment.

• Change user@example.com with your contact email.
• Replace hostedZoneID XXXXXXXXXXX with the DNS Hosted zone ID in AWS for your domain.
• Replace the region value ${region}.
• The secret under privateKeySecretRef will be created automatically by the cert-manager operator.

Contact emails from domain @example.com are forbidden, so change user@example.com with your real email — Let’s Encrypt will use this to contact you about expiring certificates, and issues related to your account.

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
spec:
  acme:
    email: user@example.com
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: letsencrypt-staging-issuer-account-key
    solvers:
    - dns01:
        route53:
          region: ${region}
          hostedZoneID: XXXXXXXXXXX

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    email: user@example.com
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: letsencrypt-prod-issuer-account-key
    solvers:
    - dns01:
        route53:
          region: ${region}
          hostedZoneID: XXXXXXXXXXX

2. Check the ClusterIssuer status:

oc describe clusterissuer letsencrypt-prod
oc describe clusterissuer letsencrypt-staging

3. If the ClusterIssuer state is not ready, investigate cert-manager controller pod logs:

oc get pod -n openshift-operators | grep 'cert-manager'
oc logs -f cert-manager-${replica_set}-${random_string} -n openshift-operators

Configure Certificates

1. Create a Certificate resource for the OpenShift Router (Ingress controller for OpenShift) and for the OpenShift APIServer in two different namespaces.

OpenShift Router supports a single wildcard certificate for Ingress/Route resources in different namespaces (so called, default SSL certificate). The Ingress controller expects the certificates in a Secret to be created in the openshift-ingress namespace; the API Server, in the openshift-config namespace. The cert-manager operator will automatically create these secrets from the Certificate resource.

Replace ${DOMAIN} with your domain name. It can be checked with oc whoami --show-server. Put domain names in quotes.

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: router-certs
  namespace: openshift-ingress
  labels:
    app: cert-manager
spec:
  secretName: router-certs
  secretTemplate:
    labels:
      app: cert-manager
  duration: 2160h # 90d
  renewBefore: 360h # 15d
  subject:
    organizations:
      - Org Name
  commonName: '*.${DOMAIN}'
  privateKey:
    algorithm: RSA
    encoding: PKCS1
    size: 2048
    rotationPolicy: Always
  usages:
    - server auth
    - client auth
  dnsNames:
    - '*.${DOMAIN}'
    - '*.apps.${DOMAIN}'
  issuerRef:
    name: letsencrypt-staging
    kind: ClusterIssuer

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: api-certs
  namespace: openshift-config
  labels:
    app: cert-manager
spec:
  secretName: api-certs
  secretTemplate:
    labels:
      app: cert-manager
  duration: 2160h # 90d
  renewBefore: 360h # 15d
  subject:
    organizations:
      - Org Name
  commonName: '*.${DOMAIN}'
  privateKey:
    algorithm: RSA
    encoding: PKCS1
    size: 2048
    rotationPolicy: Always
  usages:
    - server auth
    - client auth
  dnsNames:
    - '*.${DOMAIN}'
    - '*.apps.${DOMAIN}'
  issuerRef:
    name: letsencrypt-staging
    kind: ClusterIssuer

cert-manager supports ECDSA key pairs in the Certificate resource. To use it, change RSA privateKey to ECDSA:

privateKey:
  algorithm: ECDSA
  encoding: PKCS1
  size: 256
  rotationPolicy: Always

rotationPolicy: Always is highly recommended since cert-manager does not rotate private keys by default.

Full Certificate spec is described in the cert-manager API documentation.

2. Check that the certificates in the corresponding namespaces are ready:

3. Check the details of the certificates via CLI:

oc describe certificate api-certs -n openshift-config
oc describe certificate router-certs -n openshift-ingress

4. Check the cert-manager controller pod logs if the Staging Certificate condition is not ready for more than 10 minutes:

oc get pod -n openshift-operators | grep 'cert-manager'
oc logs -f cert-manager-${replica_set}-${random_string} -n openshift-operators

5. When the certificate is issued, the certificate chain and its private key will be put into the OpenShift Secret in the namespace indicated in the Certificate resource:

oc describe secret api-certs -n openshift-config
oc describe secret router-certs -n openshift-ingress

Modify OpenShift Router and API Server Custom Resources

1. Update the Custom Resource of your Router (Ingress controller). Patch the defaultCertificate object value with { "name": "router-certs" }:

oc patch ingresscontroller default -n openshift-ingress-operator --type=merge --patch='{"spec": { "defaultCertificate": { "name": "router-certs" }}}' --insecure-skip-tls-verify

After updating the IngressController object, the OpenShift Ingress operator redeploys the router.

2. Update the Custom Resource for the OpenShift API Server:

• Export the name of APIServer:

export OKD_API=$(oc whoami --show-server --insecure-skip-tls-verify | cut -f 2 -d ':' | cut -f 3 -d '/' | sed 's/-api././')

• Patch the servingCertificate object value with { "name": "api-certs" }:

oc patch apiserver cluster --type merge --patch="{\"spec\": {\"servingCerts\": {\"namedCertificates\": [ { \"names\": [  \"$OKD_API\"  ], \"servingCertificate\": {\"name\": \"api-certs\" }}]}}}" --insecure-skip-tls-verify

Move From Let’s Encrypt Staging Environment to Prod

1. Test the Staging certificate on the OpenShift Admin Console. The --insecure flag is used because Let's Encrypt Staging certificates are not trusted in browsers by default:

curl -v --insecure https://console-openshift-console.apps.${DOMAIN}

2. Change issuerRef to letsencrypt-prod in both Certificate resources:

oc edit certificate api-certs -n openshift-config
oc edit certificate router-certs -n openshift-ingress

issuerRef:
  name: letsencrypt-prod
  kind: ClusterIssuer

In case the certificate reissue is not triggered after that, try to force the certificate renewal with the cmctl tool:

cmctl renew router-certs -n openshift-ingress
cmctl renew api-certs -n openshift-config

If this won’t work, delete the api-certs and router-certs secrets. It should trigger the Prod certificates issuance:

oc delete secret router-certs -n openshift-ingress
oc delete secret api-certs -n openshift-config

Please note that these actions will lead to logging your account out of the OpenShift Admin Console, since certificates will be deleted. Accept the certificate warning in the browser and log in again after that.

3. Check the status of the Prod certificates:

oc describe certificate api-certs -n openshift-config
oc describe certificate router-certs -n openshift-ingress

or using the cmctl tool:

cmctl status certificate api-certs -n openshift-config
cmctl status certificate router-certs -n openshift-ingress

Certificates from Let’s Encrypt Prod should be issued within 5–7 min. In case of the DNS propagation — about 15–20 min.

4. Check the web console and make sure it has secure connection:

curl -v https://console-openshift-console.apps.${DOMAIN}

The certificate troubleshooting , renewals, and updating the kubeconfig are described in Part 3.