Cert-Manager Issuer for Cross-Account Route 53 [ EKS ]
Cert-Manager is very powerful tool when we talk about managing TLS certificates & issuer and no other tool comes near the Cert-Manager for kubernetes in terms of open source, visibility, documentation, installation option, integration and many more. Even with same account or cross-account option, there is direct integration option provided by cert-manager CRDs. This will lead to ease of setting of certificates and managing those created certificates.
ASSUMPTION
For this session/blog, we are going to use ACME certificates [or Let’s encrypt certificates] using DNS01 challenger.
Before setting-up we need to have the clarity of account and their functionality.
ACCOUNT-X — EKS SETUP
ACCOUNT Y — ROUTE 53
NOTE: For same account, you can use serviceaccount to make a call through OIDC To AWS IAM Role.
SETUP
IAM User credentials
Under Account Y, Create IAM user and provide privileged [Administrator] access or you can use following IAM policy to provide limited access to Route 53. This permission required to change or read information of the Hosted zone.
Once created. Copy ACCESS KEY & SECRET KEY of that IAM user. This is required further when we will configure Issuer.
ACCESS-KEY - AKIAXXXXXXXXXXXXXXX
SECRET-ACCESS-KEY - 5CE3sXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Get Hosted Zone ID
This hosted zone we need to copy from Route 53 of Account Y, dsf
Hosted Zone ID - ZO61XXXXXXXXXXXXX
Cert-Manager Setup
helm install cert-manager jetstack/cert-manager --namespace cert-manager --create-namespace --version v1.9.1 --set installCRDs=true
Check This Link for detailed installation of Cert-Manager.
BASE64 encoded secret access key
First, copy the secret access key that we created for IAM User for Account Y.
echo "5CE3sXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX" | base64
It will give base 64 encoded output which we will use to create a kubernetes secret for secretAccessKeySecretRef reference.
NUNFMXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX=
AWS Secret key Secret creation
Now, we need to create that secret before applying the changes.
and apply the following
kubectl apply -f accounty-secret-key.yaml
Check if the secret is created or not.
kubectl get secret -n cert-manager
ClusterIssuer Breakdown
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: cert-manager-staging-test
The above information is related to CRD of cert-manager. This includes apiVersion, kind & metadata
spec:
acme:
email: xxxxx-xxxxx@xxxxx.xxx
server: https://acme-staging-v02.api.letsencrypt.org/directory
privateKeySecretRef:
name: cert-manager-staging-test-secret
ACME BREAKDOWN:
- email [Require for general information for certification creation]
- server [This specifies what kind of certificates user want’s to create Like staging or prod certificates]
- privateKeySecretRef [Name of the secret]
solvers:
- selector:
dnsZones:
- "*.example.example"
- "example.com"
Solvers BREAKDOWN:
- Selectors [Selector specifies the configuration, For let’s encrypt, we need to specify dnsZones]
- dnsZones [dnsZones specifies the domain & wildcard information]
dns01:
route53:
region: XX-XXXX-X
hostedZoneID: ZO61XXXXXXXXXXXXX
accessKeyID: AKIAXXXXXXXXXXXXXX
secretAccessKeySecretRef:
name: awssecretkey
key: accounty-secret-key
dns01 [This specifies the actual information related to the route53 like region, hostedZoneId, access keys, role id]
ClusterIssuer file
Use below command to apply the clusterissuer configuration.
kubectl apply -f issuer.yaml
Once you apply the changes, you can check the cert-manager controller pod logs to check the behaviour or validate the logs.
LOGS:
I0817 19:16:29.779442 1 setup.go:111] cert-manager/clusterissuers "msg"="generating acme account private key" "related_resource_kind"="Secret" "related_resource_name"="cert-manager-staging-test-secret" "related_resource_namespace"="cert-manager" "resource_kind"="ClusterIssuer" "resource_name"="cert-manager-staging-test" "resource_namespace"="" "resource_version"="v1"I0817 19:16:29.906133 1 setup.go:219] cert-manager/clusterissuers "msg"="ACME server URL host and ACME private key registration host differ. Re-checking ACME account registration" "related_resource_kind"="Secret" "related_resource_name"="cert-manager-staging-test-secret" "related_resource_namespace"="cert-manager" "resource_kind"="ClusterIssuer" "resource_name"="cert-manager-staging-test" "resource_namespace"="" "resource_version"="v1"I0817 19:16:30.770143 1 setup.go:309] cert-manager/clusterissuers "msg"="verified existing registration with ACME server" "related_resource_kind"="Secret" "related_resource_name"="cert-manager-staging-test-secret" "related_resource_namespace"="cert-manager" "resource_kind"="ClusterIssuer" "resource_name"="cert-manager-staging-test" "resource_namespace"="" "resource_version"="v1"I0817 19:16:30.770174 1 conditions.go:95] Setting lastTransitionTime for Issuer "cert-manager-staging-test" condition "Ready" to 2022-08-17 19:16:30.770167831 +0000 UTC m=+2333.520794129I0817 19:16:30.783535 1 setup.go:202] cert-manager/clusterissuers "msg"="skipping re-verifying ACME account as cached registration details look sufficient" "related_resource_kind"="Secret" "related_resource_name"="cert-manager-staging-test-secret" "related_resource_namespace"="cert-manager" "resource_kind"="ClusterIssuer" "resource_name"="cert-manager-staging-test" "resource_namespace"="" "resource_version"="v1"I0817 19:16:34.907590 1 setup.go:202] cert-manager/clusterissuers "msg"="skipping re-verifying ACME account as cached registration details look sufficient" "related_resource_kind"="Secret" "related_resource_name"="cert-manager-staging-test-secret" "related_resource_namespace"="cert-manager" "resource_kind"="ClusterIssuer" "resource_name"="cert-manager-staging-test" "resource_namespace"="" "resource_version"="v1"
You can also validate the changes by using below command to check READY status. If it is showing True, it means it is ready to serve.
NOTE: If it is not showing True, inspect cert-manager controller logs to rectify the issue.
kubectl get clusterissuer
Certificate creation
Once, you create the clusterissuer with True ready status. You need to create certificates which responsible for creating secrets containing certificates.
Once you created manifest file containing certificates changes. Apply the above changes
kubectl apply -f cert-test.yaml
Once, you create, you can check Route 53 [On account Y] to validate the TXT record created by cert-manager.
_acme-challenge.example.com.xyz TXT Multivalue answer-"QCGSHgXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
Once, you check or validate, you can use below command to validate whether certificates are ready to use or not.
kubectl get certificates -n test
You can also describe or check content secret created by certificate CR.
kubectl describe secret cert-manager-staging-test-staging-secret -n test
OUTPUT:
Data
====
tls.key: 1675 bytes
tls.crt: 5753 bytes
Cert manager Certificate logs
I0817 19:21:19.993429 1 conditions.go:201] Setting lastTransitionTime for Certificate "cert-manager-staging-test-cert" condition "Ready" to 2022-08-17 19:21:19.993418389 +0000 UTC m=+2622.744044690I0817 19:21:19.994199 1 trigger_controller.go:200] cert-manager/certificates-trigger "msg"="Certificate must be re-issued" "key"="test/cert-manager-staging-test-cert" "message"="Issuing certificate as Secret does not exist" "reason"="DoesNotExist"I0817 19:21:19.994220 1 conditions.go:201] Setting lastTransitionTime for Certificate "cert-manager-staging-test-cert" condition "Issuing" to 2022-08-17 19:21:19.994216003 +0000 UTC m=+2622.744842295I0817 19:21:20.073511 1 controller.go:161] cert-manager/certificates-readiness "msg"="re-queuing item due to optimistic locking on resource" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"cert-manager-staging-test-cert\": the object has been modified; please apply your changes to the latest version and try again" "key"="test/cert-manager-staging-test-cert"I0817 19:21:20.073583 1 conditions.go:201] Setting lastTransitionTime for Certificate "cert-manager-staging-test-cert" condition "Ready" to 2022-08-17 19:21:20.073575848 +0000 UTC m=+2622.824202143I0817 19:21:20.358802 1 controller.go:161] cert-manager/certificates-key-manager "msg"="re-queuing item due to optimistic locking on resource" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"cert-manager-staging-test-cert\": the object has been modified; please apply your changes to the latest version and try again" "key"="test/cert-manager-staging-test-cert"I0817 19:21:20.388505 1 conditions.go:261] Setting lastTransitionTime for CertificateRequest "cert-manager-staging-test-cert-qd447" condition "Approved" to 2022-08-17 19:21:20.388478055 +0000 UTC m=+2623.139104353I0817 19:21:20.439792 1 conditions.go:261] Setting lastTransitionTime for CertificateRequest "cert-manager-staging-test-cert-qd447" condition "Ready" to 2022-08-17 19:21:20.439781619 +0000 UTC m=+2623.190407917I0817 19:21:20.455394 1 conditions.go:261] Setting lastTransitionTime for CertificateRequest "cert-manager-staging-test-cert-qd447" condition "Ready" to 2022-08-17 19:21:20.455383292 +0000 UTC m=+2623.206009583I0817 19:21:20.470152 1 controller.go:161] cert-manager/certificaterequests-issuer-acme "msg"="re-queuing item due to optimistic locking on resource" "error"="Operation cannot be fulfilled on certificaterequests.cert-manager.io \"cert-manager-staging-test-cert-qd447\": the object has been modified; please apply your changes to the latest version and try again" "key"="test/cert-manager-staging-test-cert-qd447"
Now, you can use above certificates in any kind of resources like ingress controller, service mesh etc.
NOTE: To know more about certificates & clusterissuer, visit official documentation provided by cert-manager.
How-To
Now, these certificates can use anywhere where you specify these certificates. But for this section, we will only focus ingress object.
Check below ingress manifest containg rules and it is also contain the secret that created through cert-manager certificate.
Once apply, check brower and check TLS
Check certificate details, you will see staging certificate issued by Let’s encrypt.
The above certificate is just to validate for testing purpose or only for staging purpose.
To implement for production environment, use following server URL in clusterIssuer YAML.
https://acme-v02.api.letsencrypt.org/directory
YAML:
Once, you added and repeat above steps, you will get valid certificates.
REFERENCES
- https://unsplash.com/photos/0vGohk5aw6E
- https://cert-manager.io/docs/
- https://kubernetes.io/docs/concepts/services-networking/ingress/