Cert-Manager Issuer for Cross-Account Route 53 [ EKS ]

7 min readMar 9, 2023

Cert-Manager is very powerful tool when we talk about managing TLS certificates & issuer and no other tool comes near the Cert-Manager for kubernetes in terms of open source, visibility, documentation, installation option, integration and many more. Even with same account or cross-account option, there is direct integration option provided by cert-manager CRDs. This will lead to ease of setting of certificates and managing those created certificates.

ASSUMPTION

For this session/blog, we are going to use ACME certificates [or Let’s encrypt certificates] using DNS01 challenger.

Before setting-up we need to have the clarity of account and their functionality.

ACCOUNT-X — EKS SETUP

ACCOUNT Y — ROUTE 53

NOTE: For same account, you can use serviceaccount to make a call through OIDC To AWS IAM Role.

SETUP

IAM User credentials

Under Account Y, Create IAM user and provide privileged [Administrator] access or you can use following IAM policy to provide limited access to Route 53. This permission required to change or read information of the Hosted zone.

Once created. Copy ACCESS KEY & SECRET KEY of that IAM user. This is required further when we will configure Issuer.

ACCESS-KEY - AKIAXXXXXXXXXXXXXXX
SECRET-ACCESS-KEY - 5CE3sXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

Get Hosted Zone ID

This hosted zone we need to copy from Route 53 of Account Y, dsf

Hosted Zone ID - ZO61XXXXXXXXXXXXX

Cert-Manager Setup

helm install cert-manager jetstack/cert-manager --namespace cert-manager --create-namespace --version v1.9.1 --set installCRDs=true

Check This Link for detailed installation of Cert-Manager.

BASE64 encoded secret access key

First, copy the secret access key that we created for IAM User for Account Y.

echo "5CE3sXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX" | base64

It will give base 64 encoded output which we will use to create a kubernetes secret for secretAccessKeySecretRef reference.

NUNFMXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX=

AWS Secret key Secret creation

Now, we need to create that secret before applying the changes.

and apply the following

kubectl apply -f accounty-secret-key.yaml

Check if the secret is created or not.

kubectl get secret -n cert-manager

ClusterIssuer Breakdown

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: cert-manager-staging-test

The above information is related to CRD of cert-manager. This includes apiVersion, kind & metadata

spec:
  acme:
    email: xxxxx-xxxxx@xxxxx.xxx
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: cert-manager-staging-test-secret

ACME BREAKDOWN:

email [Require for general information for certification creation]
server [This specifies what kind of certificates user want’s to create Like staging or prod certificates]
privateKeySecretRef [Name of the secret]

solvers:
    - selector:
        dnsZones:
          - "*.example.example"
          - "example.com"

Solvers BREAKDOWN:

Selectors [Selector specifies the configuration, For let’s encrypt, we need to specify dnsZones]
dnsZones [dnsZones specifies the domain & wildcard information]

dns01:
        route53:
          region: XX-XXXX-X
          hostedZoneID: ZO61XXXXXXXXXXXXX
          accessKeyID: AKIAXXXXXXXXXXXXXX
          secretAccessKeySecretRef:
            name: awssecretkey
            key: accounty-secret-key

dns01 [This specifies the actual information related to the route53 like region, hostedZoneId, access keys, role id]

ClusterIssuer file

Use below command to apply the clusterissuer configuration.

kubectl apply -f issuer.yaml

Once you apply the changes, you can check the cert-manager controller pod logs to check the behaviour or validate the logs.

LOGS:

I0817 19:16:29.779442       1 setup.go:111] cert-manager/clusterissuers "msg"="generating acme account private key" "related_resource_kind"="Secret" "related_resource_name"="cert-manager-staging-test-secret" "related_resource_namespace"="cert-manager" "resource_kind"="ClusterIssuer" "resource_name"="cert-manager-staging-test" "resource_namespace"="" "resource_version"="v1"I0817 19:16:29.906133       1 setup.go:219] cert-manager/clusterissuers "msg"="ACME server URL host and ACME private key registration host differ. Re-checking ACME account registration" "related_resource_kind"="Secret" "related_resource_name"="cert-manager-staging-test-secret" "related_resource_namespace"="cert-manager" "resource_kind"="ClusterIssuer" "resource_name"="cert-manager-staging-test" "resource_namespace"="" "resource_version"="v1"I0817 19:16:30.770143       1 setup.go:309] cert-manager/clusterissuers "msg"="verified existing registration with ACME server" "related_resource_kind"="Secret" "related_resource_name"="cert-manager-staging-test-secret" "related_resource_namespace"="cert-manager" "resource_kind"="ClusterIssuer" "resource_name"="cert-manager-staging-test" "resource_namespace"="" "resource_version"="v1"I0817 19:16:30.770174       1 conditions.go:95] Setting lastTransitionTime for Issuer "cert-manager-staging-test" condition "Ready" to 2022-08-17 19:16:30.770167831 +0000 UTC m=+2333.520794129I0817 19:16:30.783535       1 setup.go:202] cert-manager/clusterissuers "msg"="skipping re-verifying ACME account as cached registration details look sufficient" "related_resource_kind"="Secret" "related_resource_name"="cert-manager-staging-test-secret" "related_resource_namespace"="cert-manager" "resource_kind"="ClusterIssuer" "resource_name"="cert-manager-staging-test" "resource_namespace"="" "resource_version"="v1"I0817 19:16:34.907590       1 setup.go:202] cert-manager/clusterissuers "msg"="skipping re-verifying ACME account as cached registration details look sufficient" "related_resource_kind"="Secret" "related_resource_name"="cert-manager-staging-test-secret" "related_resource_namespace"="cert-manager" "resource_kind"="ClusterIssuer" "resource_name"="cert-manager-staging-test" "resource_namespace"="" "resource_version"="v1"

You can also validate the changes by using below command to check READY status. If it is showing True, it means it is ready to serve.

NOTE: If it is not showing True, inspect cert-manager controller logs to rectify the issue.

kubectl get clusterissuer

Certificate creation

Once, you create the clusterissuer with True ready status. You need to create certificates which responsible for creating secrets containing certificates.

Once you created manifest file containing certificates changes. Apply the above changes

kubectl apply -f cert-test.yaml

Once, you create, you can check Route 53 [On account Y] to validate the TXT record created by cert-manager.

_acme-challenge.example.com.xyz TXT Multivalue answer-"QCGSHgXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"

Once, you check or validate, you can use below command to validate whether certificates are ready to use or not.

kubectl get certificates -n test

You can also describe or check content secret created by certificate CR.

kubectl describe secret cert-manager-staging-test-staging-secret -n test

OUTPUT:

Data
====
tls.key:  1675 bytes
tls.crt:  5753 bytes

Cert manager Certificate logs

I0817 19:21:19.993429       1 conditions.go:201] Setting lastTransitionTime for Certificate "cert-manager-staging-test-cert" condition "Ready" to 2022-08-17 19:21:19.993418389 +0000 UTC m=+2622.744044690I0817 19:21:19.994199       1 trigger_controller.go:200] cert-manager/certificates-trigger "msg"="Certificate must be re-issued" "key"="test/cert-manager-staging-test-cert" "message"="Issuing certificate as Secret does not exist" "reason"="DoesNotExist"I0817 19:21:19.994220       1 conditions.go:201] Setting lastTransitionTime for Certificate "cert-manager-staging-test-cert" condition "Issuing" to 2022-08-17 19:21:19.994216003 +0000 UTC m=+2622.744842295I0817 19:21:20.073511       1 controller.go:161] cert-manager/certificates-readiness "msg"="re-queuing item due to optimistic locking on resource" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"cert-manager-staging-test-cert\": the object has been modified; please apply your changes to the latest version and try again" "key"="test/cert-manager-staging-test-cert"I0817 19:21:20.073583       1 conditions.go:201] Setting lastTransitionTime for Certificate "cert-manager-staging-test-cert" condition "Ready" to 2022-08-17 19:21:20.073575848 +0000 UTC m=+2622.824202143I0817 19:21:20.358802       1 controller.go:161] cert-manager/certificates-key-manager "msg"="re-queuing item due to optimistic locking on resource" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"cert-manager-staging-test-cert\": the object has been modified; please apply your changes to the latest version and try again" "key"="test/cert-manager-staging-test-cert"I0817 19:21:20.388505       1 conditions.go:261] Setting lastTransitionTime for CertificateRequest "cert-manager-staging-test-cert-qd447" condition "Approved" to 2022-08-17 19:21:20.388478055 +0000 UTC m=+2623.139104353I0817 19:21:20.439792       1 conditions.go:261] Setting lastTransitionTime for CertificateRequest "cert-manager-staging-test-cert-qd447" condition "Ready" to 2022-08-17 19:21:20.439781619 +0000 UTC m=+2623.190407917I0817 19:21:20.455394       1 conditions.go:261] Setting lastTransitionTime for CertificateRequest "cert-manager-staging-test-cert-qd447" condition "Ready" to 2022-08-17 19:21:20.455383292 +0000 UTC m=+2623.206009583I0817 19:21:20.470152       1 controller.go:161] cert-manager/certificaterequests-issuer-acme "msg"="re-queuing item due to optimistic locking on resource" "error"="Operation cannot be fulfilled on certificaterequests.cert-manager.io \"cert-manager-staging-test-cert-qd447\": the object has been modified; please apply your changes to the latest version and try again" "key"="test/cert-manager-staging-test-cert-qd447"

Now, you can use above certificates in any kind of resources like ingress controller, service mesh etc.

NOTE: To know more about certificates & clusterissuer, visit official documentation provided by cert-manager.

How-To

Now, these certificates can use anywhere where you specify these certificates. But for this section, we will only focus ingress object.

Check below ingress manifest containg rules and it is also contain the secret that created through cert-manager certificate.

Once apply, check brower and check TLS

Check certificate details, you will see staging certificate issued by Let’s encrypt.

The above certificate is just to validate for testing purpose or only for staging purpose.

To implement for production environment, use following server URL in clusterIssuer YAML.

https://acme-v02.api.letsencrypt.org/directory

YAML:

Once, you added and repeat above steps, you will get valid certificates.