Using Cert-Manager in OpenShift/OKD. Part 1 — installation and integration with AWS Route53.

The article was initially published here: https://epam.github.io/edp-install/operator-guide/ssl-automation-okd/

Part 1 of this guide covers the cert-manager installation in OpenShift and integration with AWS Route53.

Part 2 consists of the cert-manager configuration and SSL certificate installation.

Part 3 describes certificates troubleshooting , renewals, and updating the kubeconfig.

The current approach was tested on OKD 4.10 and 4.12.

The following material covers Let’s Encrypt certificate automation with cert-manager using AWS Route53.

cert-manager is a complex Kubernetes/OpenShift operator that allows to issue, automatically renew SSL certificates, and manage PKI infrastructure.
It consists of cert-manager controller, webhook, and ca-injector.

With cert-manager you can secure DNS Name, URI, or IP. We will secure DNS Name in our example.

The guide describes how to automatically issue and install wildcard certificates on OpenShift Ingress Controller and API Server covering all cluster Routes. To secure separate OpenShift Routes, please refer to the OpenShift Route Support project for cert-manager.

Prerequisites

• OpenShift cluster v4.7 — v4.13;
• AWS Route53;
• Enabled AWS IRSA;
• The latest oc utility. The kubectl tool can also be used for most of the commands.

Install Cert-Manager Operator

Install the cert-manager operator via OpenShift OperatorHub that uses Operator Lifecycle Manager (OLM) under the hood.

1. Go to the OpenShift Admin Console → OperatorHub, search for the cert-manager, and click Install:

2. Modify the ClusterServiceVersion OLM resource, by selecting the Update approval → Manual.

If you select Update approval → Automatic, some parameters in the ClusterServiceVersion may be reset to defaults after the operator update and we do not want that.

However, installing an operator with Manual approval causes all operators installed in namespace openshift-operators to function as manual approval strategy.

After you select Manual approval, review the manual installation plan and approve it.

3. Navigate to Operators → Installed Operators and check the operator status to be Succeeded:

4. In case of errors, troubleshoot the Operator issues:

oc describe operator cert-manager -n openshift-operators
oc describe sub cert-manager -n openshift-operators

Create AWS Role for Route53

Since we need wildcard certificates, the only possible certificate validation method in our case is DNS-based. We will be configuring cert-manager to validate certificates using this validation method.

1. Check the DNS Hosted zone ID in AWS Route53 for your domain:

2. Create Route53 Permissions policy in AWS for cert-manager to be able to create DNS TXT records for the certificate validation. In this example, cert-manager permissions are given for a particular DNS zone only. Replace Hosted zone ID XXXXXXXX in the "Resource": "arn:aws:route53:::hostedzone/XXXXXXXXXXXX".

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "route53:GetChange",
      "Resource": "arn:aws:route53:::change/*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "route53:ChangeResourceRecordSets",
        "route53:ListResourceRecordSets"
      ],
      "Resource": "arn:aws:route53:::hostedzone/XXXXXXXXXXXX"
    }
  ]
}

3. Create an AWS Role with Custom trust policy for the cert-manager service account to use the AWS IRSA feature and then attach the created policy. Replace the following:

${aws-account-id} with the AWS account ID of the EKS cluster.

${aws-region} with the region where the EKS cluster is located.

${eks-hash} with the hash in the EKS API URL; this will be a random 32 character hex string, for example, 45DABD88EEE3A227AF0FA468BE4EF0B5.

${namespace} with the namespace where cert-manager is running.

${service-account-name} with the name of the ServiceAccount object created by cert-manager.

By default, it is “system:serviceaccount:openshift-operators:cert-manager” if cert-manager is installed via OperatorHub.

Attach the created Permission policy for Route53 to the Role.

Optionally, add Permissions boundary to the Role.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Principal": {
        "Federated": "arn:aws:iam::* ${aws-account-id}:oidc-provider/oidc.eks.${aws-region}.amazonaws.com/id/${eks-hash}"
      },
      "Condition": {
        "StringEquals": {
          "oidc.eks.${aws-region}.amazonaws.com/id/${eks-hash}:sub": "system:serviceaccount:${namespace}:${service-account-name}"
        }
      }
    }
  ]
}

4. Copy the created Role ARN.

Configure Cert-Manager Integration With AWS Route53

1. Annotate the ServiceAccount created by cert-manager (required for AWS IRSA), and restart the cert-manager pod.

Replace the eks.amazonaws.com/role-arn annotation value with your own Role ARN.

oc edit sa cert-manager -n openshift-operators

apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::XXXXXXXXXXXX:role/cert-manager

2. In case the ServiceAccount token cannot be read and the operator is installed using the OperatorHub, add fsGroup: 1001 via OpenShift ClusterServiceVersion OLM resource. It should be added to a cert-manager controller spec. These actions may not be required for OpenShift v4.10 and higher.

oc get csv
oc edit csv cert-manager.${VERSION}

spec:
  template:
    spec:
      securityContext:
        fsGroup: 1001
      serviceAccountName: cert-manager

A mutating admission controller will automatically modify all pods running with the service account. Example of how pods will look like:

apiVersion: apps/v1
kind: Pod
# ...
spec:
  # ...
  serviceAccountName: cert-manager
  serviceAccount: cert-manager
  containers:
  - name: ...
    # ...
    env:
      - name: AWS_ROLE_ARN
        value: >-
          arn:aws:iam::XXXXXXXXXXX:role/cert-manager
      - name: AWS_WEB_IDENTITY_TOKEN_FILE
        value: /var/run/secrets/eks.amazonaws.com/serviceaccount/token
    volumeMounts:
      - name: aws-iam-token
        readOnly: true
        mountPath: /var/run/secrets/eks.amazonaws.com/serviceaccount
  volumes:
    - name: aws-iam-token
      projected:
        sources:
          - serviceAccountToken:
              audience: sts.amazonaws.com
              expirationSeconds: 86400
              path: token
        defaultMode: 420

3. If you have separate public and private DNS zones for the same domain (split-horizon DNS), modify the cert-manager Deployment in order to validate DNS TXT records via public recursive nameservers.

Otherwise, you will be getting an error during a record validation:

Waiting for DNS-01 challenge propagation: NS ns-123.awsdns-00.net.:53 returned REFUSED for _acme-challenge.

To avoid the error, add --dns01-recursive-nameservers-only --dns01-recursive-nameservers=8.8.8.8:53,1.1.1.1:53 as ARGs to the cert-manager controller Deployment.

oc get csv
oc edit csv cert-manager.${VERSION}

  labels:
    app: cert-manager
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: cert-manager
    app.kubernetes.io/name: cert-manager
    app.kubernetes.io/version: v1.9.1
spec:
  containers:
    - args:
        - '--v=2'
        - '--cluster-resource-namespace=$(POD_NAMESPACE)'
        - '--leader-election-namespace=kube-system'
        - '--dns01-recursive-nameservers-only'
        - '--dns01-recursive-nameservers=8.8.8.8:53,1.1.1.1:53'

The Deployment must be modified via OpenShift ClusterServiceVersion OLM resource if the operator was installed using the OperatorHub. The OpenShift ClusterServiceVersion OLM resource includes several Deployments, and the ARGs must be modified only for the cert-manager controller.

Save the resource. After that, OLM will try to reload the resource automatically and save it to the YAML file. If OLM resets the config file, double-check the entered values:

The cert-manager configuration and SSL certificate installation are in the Part 2 of this article.