Using Cert-Manager in OpenShift/OKD. Part 1 — installation and integration with AWS Route53.
The article was initially published here: https://epam.github.io/edp-install/operator-guide/ssl-automation-okd/
Part 1 of this guide covers the cert-manager installation in OpenShift and integration with AWS Route53.
Part 2 consists of the cert-manager configuration and SSL certificate installation.
Part 3 describes certificates troubleshooting , renewals, and updating the kubeconfig.
The current approach was tested on OKD 4.10 and 4.12.
The following material covers Let’s Encrypt certificate automation with cert-manager using AWS Route53.
cert-manager is a complex Kubernetes/OpenShift operator that allows to issue, automatically renew SSL certificates, and manage PKI infrastructure.
It consists of cert-manager controller, webhook, and ca-injector.
With cert-manager you can secure DNS Name, URI, or IP. We will secure DNS Name in our example.
The guide describes how to automatically issue and install wildcard certificates on OpenShift Ingress Controller and API Server covering all cluster Routes. To secure separate OpenShift Routes, please refer to the OpenShift Route Support project for cert-manager.
Prerequisites
- OpenShift cluster v4.7 — v4.13;
- AWS Route53;
- Enabled AWS IRSA;
- The latest oc utility. The kubectl tool can also be used for most of the commands.
Install Cert-Manager Operator
Install the cert-manager operator via OpenShift OperatorHub that uses Operator Lifecycle Manager (OLM) under the hood.
- Go to the OpenShift Admin Console → OperatorHub, search for the cert-manager, and click Install:
2. Modify the ClusterServiceVersion
OLM resource, by selecting the Update approval → Manual.
If you select Update approval → Automatic, some parameters in the
ClusterServiceVersion
may be reset to defaults after the operator update and we do not want that.However, installing an operator with Manual approval causes all operators installed in namespace
openshift-operators
to function as manual approval strategy.
After you select Manual approval, review the manual installation plan and approve it.
3. Navigate to Operators → Installed Operators and check the operator status to be Succeeded:
4. In case of errors, troubleshoot the Operator issues:
oc describe operator cert-manager -n openshift-operators
oc describe sub cert-manager -n openshift-operators
Create AWS Role for Route53
Since we need wildcard certificates, the only possible certificate validation method in our case is DNS-based. We will be configuring cert-manager to validate certificates using this validation method.
- Check the DNS Hosted zone ID in AWS Route53 for your domain:
2. Create Route53 Permissions policy in AWS for cert-manager
to be able to create DNS TXT records for the certificate validation. In this example, cert-manager
permissions are given for a particular DNS zone only. Replace Hosted zone ID XXXXXXXX in the "Resource": "arn:aws:route53:::hostedzone/XXXXXXXXXXXX".
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "route53:GetChange",
"Resource": "arn:aws:route53:::change/*"
},
{
"Effect": "Allow",
"Action": [
"route53:ChangeResourceRecordSets",
"route53:ListResourceRecordSets"
],
"Resource": "arn:aws:route53:::hostedzone/XXXXXXXXXXXX"
}
]
}
3. Create an AWS Role with Custom trust policy for the cert-manager
service account to use the AWS IRSA feature and then attach the created policy. Replace the following:
${aws-account-id}
with the AWS account ID of the EKS cluster.
${aws-region}
with the region where the EKS cluster is located.
${eks-hash}
with the hash in the EKS API URL; this will be a random 32 character hex string, for example, 45DABD88EEE3A227AF0FA468BE4EF0B5.
${namespace}
with the namespace where cert-manager is running.
${service-account-name}
with the name of the ServiceAccount object created by cert-manager.By default, it is “system:serviceaccount:openshift-operators:cert-manager” if
cert-manager
is installed via OperatorHub.Attach the created Permission policy for Route53 to the Role.
Optionally, add Permissions boundary to the Role.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "sts:AssumeRoleWithWebIdentity",
"Principal": {
"Federated": "arn:aws:iam::* ${aws-account-id}:oidc-provider/oidc.eks.${aws-region}.amazonaws.com/id/${eks-hash}"
},
"Condition": {
"StringEquals": {
"oidc.eks.${aws-region}.amazonaws.com/id/${eks-hash}:sub": "system:serviceaccount:${namespace}:${service-account-name}"
}
}
}
]
}
4. Copy the created Role ARN.
Configure Cert-Manager Integration With AWS Route53
- Annotate the
ServiceAccount
created by cert-manager (required for AWS IRSA), and restart the cert-manager pod.
Replace the
eks.amazonaws.com/role-arn
annotation value with your own Role ARN.
oc edit sa cert-manager -n openshift-operators
apiVersion: v1
kind: ServiceAccount
metadata:
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::XXXXXXXXXXXX:role/cert-manager
2. In case the ServiceAccount
token cannot be read and the operator is installed using the OperatorHub, add fsGroup: 1001
via OpenShift ClusterServiceVersion OLM resource. It should be added to a cert-manager controller spec. These actions may not be required for OpenShift v4.10 and higher.
oc get csv
oc edit csv cert-manager.${VERSION}
spec:
template:
spec:
securityContext:
fsGroup: 1001
serviceAccountName: cert-manager
A mutating admission controller will automatically modify all pods running with the service account. Example of how pods will look like:
apiVersion: apps/v1
kind: Pod
# ...
spec:
# ...
serviceAccountName: cert-manager
serviceAccount: cert-manager
containers:
- name: ...
# ...
env:
- name: AWS_ROLE_ARN
value: >-
arn:aws:iam::XXXXXXXXXXX:role/cert-manager
- name: AWS_WEB_IDENTITY_TOKEN_FILE
value: /var/run/secrets/eks.amazonaws.com/serviceaccount/token
volumeMounts:
- name: aws-iam-token
readOnly: true
mountPath: /var/run/secrets/eks.amazonaws.com/serviceaccount
volumes:
- name: aws-iam-token
projected:
sources:
- serviceAccountToken:
audience: sts.amazonaws.com
expirationSeconds: 86400
path: token
defaultMode: 420
3. If you have separate public and private DNS zones for the same domain (split-horizon DNS), modify the cert-manager Deployment
in order to validate DNS TXT records via public recursive nameservers.
Otherwise, you will be getting an error during a record validation:
Waiting for DNS-01 challenge propagation: NS ns-123.awsdns-00.net.:53 returned REFUSED for _acme-challenge.
To avoid the error, add
--dns01-recursive-nameservers-only --dns01-recursive-nameservers=8.8.8.8:53,1.1.1.1:53
as ARGs to the cert-manager controllerDeployment
.
oc get csv
oc edit csv cert-manager.${VERSION}
labels:
app: cert-manager
app.kubernetes.io/component: controller
app.kubernetes.io/instance: cert-manager
app.kubernetes.io/name: cert-manager
app.kubernetes.io/version: v1.9.1
spec:
containers:
- args:
- '--v=2'
- '--cluster-resource-namespace=$(POD_NAMESPACE)'
- '--leader-election-namespace=kube-system'
- '--dns01-recursive-nameservers-only'
- '--dns01-recursive-nameservers=8.8.8.8:53,1.1.1.1:53'
The
Deployment
must be modified via OpenShift ClusterServiceVersion OLM resource if the operator was installed using the OperatorHub. The OpenShiftClusterServiceVersion
OLM resource includes several Deployments, and the ARGs must be modified only for the cert-manager controller.
Save the resource. After that, OLM will try to reload the resource automatically and save it to the YAML file. If OLM resets the config file, double-check the entered values:
The cert-manager configuration and SSL certificate installation are in the Part 2 of this article.