Using Cert-Manager in OpenShift/OKD. Part 2— configuration and SSL certificate installation.
The article was initially published here: https://epam.github.io/edp-install/operator-guide/ssl-automation-okd/
Part 1 of this guide covers the cert-manager installation in OpenShift and integration with AWS Route53.
Part 2 consists of the cert-manager configuration and SSL certificate installation.
Part 3 describes certificates troubleshooting , renewals, and updating the kubeconfig.
Now that the cert-manager is installed and integrated with AWS Route53, let’s configure ClusterIssuers and Certificates.
Configure ClusterIssuers
There are two custom resources an Issuer
and a ClusterIssuer
. The difference between an Issuer
and a ClusterIssuer
is that Issuer
is a namespaced resource, in our case, we need an issuer that will be available on the whole cluster.
- Create the ClusterIssuer resource for Let’s Encrypt Staging and Prod environments that signs a Certificate using cert-manager.
Let’s Encrypt has a limit of duplicate certificates in the Prod environment. Therefore, we will also create a
ClusterIssuer
for Let's Encrypt Staging environment. By default, Let's Encrypt Staging certificates will not be trusted in your browser. The certificate validation cannot be tested in the Let's Encrypt Staging environment.
- Change
user@example.com
with your contact email. - Replace
hostedZoneID
XXXXXXXXXXX with the DNS Hosted zone ID in AWS for your domain. - Replace the region value
${region}
. - The secret under
privateKeySecretRef
will be created automatically by thecert-manager
operator.
Contact emails from domain @example.com are forbidden, so change user@example.com with your real email — Let’s Encrypt will use this to contact you about expiring certificates, and issues related to your account.
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-staging
spec:
acme:
email: user@example.com
server: https://acme-staging-v02.api.letsencrypt.org/directory
privateKeySecretRef:
name: letsencrypt-staging-issuer-account-key
solvers:
- dns01:
route53:
region: ${region}
hostedZoneID: XXXXXXXXXXX
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-prod
spec:
acme:
email: user@example.com
server: https://acme-v02.api.letsencrypt.org/directory
privateKeySecretRef:
name: letsencrypt-prod-issuer-account-key
solvers:
- dns01:
route53:
region: ${region}
hostedZoneID: XXXXXXXXXXX
2. Check the ClusterIssuer
status:
oc describe clusterissuer letsencrypt-prod
oc describe clusterissuer letsencrypt-staging
3. If the ClusterIssuer
state is not ready, investigate cert-manager
controller pod logs:
oc get pod -n openshift-operators | grep 'cert-manager'
oc logs -f cert-manager-${replica_set}-${random_string} -n openshift-operators
Configure Certificates
- Create a Certificate resource for the OpenShift Router (Ingress controller for OpenShift) and for the OpenShift APIServer in two different namespaces.
OpenShift Router supports a single wildcard certificate for Ingress/Route resources in different namespaces (so called, default SSL certificate). The Ingress controller expects the certificates in a
Secret
to be created in theopenshift-ingress
namespace; the API Server, in theopenshift-config
namespace. Thecert-manager
operator will automatically create these secrets from theCertificate
resource.
Replace ${DOMAIN}
with your domain name. It can be checked with oc whoami --show-server
. Put domain names in quotes.
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: router-certs
namespace: openshift-ingress
labels:
app: cert-manager
spec:
secretName: router-certs
secretTemplate:
labels:
app: cert-manager
duration: 2160h # 90d
renewBefore: 360h # 15d
subject:
organizations:
- Org Name
commonName: '*.${DOMAIN}'
privateKey:
algorithm: RSA
encoding: PKCS1
size: 2048
rotationPolicy: Always
usages:
- server auth
- client auth
dnsNames:
- '*.${DOMAIN}'
- '*.apps.${DOMAIN}'
issuerRef:
name: letsencrypt-staging
kind: ClusterIssuer
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: api-certs
namespace: openshift-config
labels:
app: cert-manager
spec:
secretName: api-certs
secretTemplate:
labels:
app: cert-manager
duration: 2160h # 90d
renewBefore: 360h # 15d
subject:
organizations:
- Org Name
commonName: '*.${DOMAIN}'
privateKey:
algorithm: RSA
encoding: PKCS1
size: 2048
rotationPolicy: Always
usages:
- server auth
- client auth
dnsNames:
- '*.${DOMAIN}'
- '*.apps.${DOMAIN}'
issuerRef:
name: letsencrypt-staging
kind: ClusterIssuer
cert-manager supports ECDSA key pairs in the
Certificate
resource. To use it, change RSAprivateKey
to ECDSA:
privateKey:
algorithm: ECDSA
encoding: PKCS1
size: 256
rotationPolicy: Always
rotationPolicy: Always
is highly recommended since cert-manager does not rotate private keys by default.Full
Certificate
spec is described in the cert-manager API documentation.
2. Check that the certificates in the corresponding namespaces are ready:
3. Check the details of the certificates via CLI:
oc describe certificate api-certs -n openshift-config
oc describe certificate router-certs -n openshift-ingress
4. Check the cert-manager controller pod logs if the Staging Certificate condition is not ready for more than 10 minutes:
oc get pod -n openshift-operators | grep 'cert-manager'
oc logs -f cert-manager-${replica_set}-${random_string} -n openshift-operators
5. When the certificate is issued, the certificate chain and its private key will be put into the OpenShift Secret
in the namespace indicated in the Certificate
resource:
oc describe secret api-certs -n openshift-config
oc describe secret router-certs -n openshift-ingress
Modify OpenShift Router and API Server Custom Resources
- Update the Custom Resource of your Router (Ingress controller). Patch the
defaultCertificate
object value with{ "name": "router-certs" }
:
oc patch ingresscontroller default -n openshift-ingress-operator --type=merge --patch='{"spec": { "defaultCertificate": { "name": "router-certs" }}}' --insecure-skip-tls-verify
After updating the
IngressController
object, the OpenShift Ingress operator redeploys the router.
2. Update the Custom Resource for the OpenShift API Server:
- Export the name of
APIServer
:
export OKD_API=$(oc whoami --show-server --insecure-skip-tls-verify | cut -f 2 -d ':' | cut -f 3 -d '/' | sed 's/-api././')
- Patch the
servingCertificate
object value with{ "name": "api-certs" }
:
oc patch apiserver cluster --type merge --patch="{\"spec\": {\"servingCerts\": {\"namedCertificates\": [ { \"names\": [ \"$OKD_API\" ], \"servingCertificate\": {\"name\": \"api-certs\" }}]}}}" --insecure-skip-tls-verify
Move From Let’s Encrypt Staging Environment to Prod
- Test the Staging certificate on the OpenShift Admin Console. The
--insecure
flag is used because Let's Encrypt Staging certificates are not trusted in browsers by default:
curl -v --insecure https://console-openshift-console.apps.${DOMAIN}
2. Change issuerRef
to letsencrypt-prod
in both Certificate
resources:
oc edit certificate api-certs -n openshift-config
oc edit certificate router-certs -n openshift-ingress
issuerRef:
name: letsencrypt-prod
kind: ClusterIssuer
In case the certificate reissue is not triggered after that, try to force the certificate renewal with the cmctl tool:
cmctl renew router-certs -n openshift-ingress
cmctl renew api-certs -n openshift-config
If this won’t work, delete the
api-certs
androuter-certs
secrets. It should trigger the Prod certificates issuance:
oc delete secret router-certs -n openshift-ingress
oc delete secret api-certs -n openshift-config
Please note that these actions will lead to logging your account out of the OpenShift Admin Console, since certificates will be deleted. Accept the certificate warning in the browser and log in again after that.
3. Check the status of the Prod certificates:
oc describe certificate api-certs -n openshift-config
oc describe certificate router-certs -n openshift-ingress
or using the cmctl tool:
cmctl status certificate api-certs -n openshift-config
cmctl status certificate router-certs -n openshift-ingress
Certificates from Let’s Encrypt Prod should be issued within 5–7 min. In case of the DNS propagation — about 15–20 min.
4. Check the web console and make sure it has secure connection:
curl -v https://console-openshift-console.apps.${DOMAIN}
The certificate troubleshooting , renewals, and updating the kubeconfig are described in Part 3.