How-To: Automatic SSL Certificate Management for your Kubernetes Application Deployment

Jaroslav Pantsjoha
Oct 24, 2019 · 12 min read

A bit of the Same, But Different. We deployed the app, but Let’s ensure our SSL Certificate is managed automatically for our Application Deployment.

Image for post
Image for post

Welcome back, or welcome for the very first time. This Security themed post will take the example of the previous blog post, “How-to: Kubernetes Application Deployment with DNS management” — which exemplifies the relative simplicity in application deployment with automatic DNS management, and takes it to the next “production-closer” level.

Let’s ensure such an application deployment process is matched with the SSL certificate. This post will lead it’s way into the GitOps deployment methodology again. The purpose of which is to codify your infrastructure, store it in Git and let the likes of Weaveworks Flux ( deploy it automatically, and in full.

Your System Requirements

This may be obvious, but you will need , and installed on whichever system you are running this.

As I do intend the hands-on number of your readers to be able to duplicate the success, so do ensure you read the First Blog Post, which outlines the Google DNS zone requirement and the follow-up application configuration, from here. The full code example with yet-another DNS zone, but the same service configuration, with yet full working code example, is found on my own bitbucket.

There is a number of dependencies to get this off the ground, to reproduce this, you will require to ensure the following checklist is in order;

  • Your Google Cloud DNS zone is setup
  • Kubernetes Cluster is created
  • Kubernetes Demo Application is running from the previous post.
  • Get ready to start pasting more…

Where were we

Before getting into the meat of this, let’s fast forward this dependency prep formality.

Image for post
Image for post

DNS ZONE is the same as before.

Image for post
Image for post

I’ve deployed the same Kubernetes cluster, via Google Cloud gcloud cli utility.


gcloud beta container --project "$PROJECT" clusters create "$CLUSTER_NAME" \
--zone "us-central1-a" --no-enable-basic-auth \
--cluster-version "1.14.6-gke.2" --machine-type "n1-standard-2" \
--image-type "COS" --disk-type "pd-ssd" --disk-size "50" \
--metadata disable-legacy-endpoints=true \
--scopes "" \
--preemptible --num-nodes "2" \
--enable-cloud-logging \
--enable-stackdriver-kubernetes --enable-ip-alias \
--network "projects/$PROJECT/global/networks/default" \
--subnetwork "projects/$PROJECT/regions/us-central1/subnetworks/default" \
--default-max-pods-per-node "110" --addons HorizontalPodAutoscaling,HttpLoadBalancing \
--enable-autoupgrade \


kubeconfig entry generated for jpworks-cluster.NAME             LOCATION       MASTER_VERSION  MASTER_IP       MACHINE_TYPE   NODE_VERSION  NUM_NODES  STATUSjpworks-cluster  us-central1-a  1.14.6-gke.2  n1-standard-2  1.14.6-gke.2  2          RUNNING

Deploy the very same Demo Nginx App, Ingress, and the DNS Manager;

k apply -f external-dns
deployment.extensions/external-dns created
serviceaccount/external-dns created created created
k apply -f nginx-app/
configmap/nginx-configuration created
deployment.extensions/nginx-app created
service/nginx-service created
k apply -f ingress.yaml
ingress.extensions/nginx-ingress-default created

Verify. Same old, same old.

k get po,ing,svc
pod/external-dns-8947bd5b9-zx98s 1/1 Running 0 6m20s
pod/nginx-app-6979bdd88f-bqtmb 1/1 Running 0 6m12s
ingress.extensions/nginx-ingress-default 80 5m47s
service/kubernetes ClusterIP <none> 443/TCP 12m
service/nginx-service LoadBalancer 80:31338/TCP 6m13s
Image for post
Image for post

This does get some (little) amount of time to get DNS records propagated throughout, — taken me closer to 10 minutes to finally see it work here.

$ curl -I
HTTP/1.1 200 OK
Server: nginx
Date: Thu, 24 Oct 2019 11:52:18 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 55
Last-Modified: Thu, 24 Oct 2019 11:41:44 GMT
ETag: "5db18df8-37"
Accept-Ranges: bytes
Via: 1.1 google
You have new mail in /var/mail/jp
Jaroslavs-MacBook-Pro:external-dns-demo jp$ curl
<h1>hello world</h1><p>flux-kubernetes-demos/nginx</p>

SSL Manage This

Now that the environment setup and formalities are out of the way with the prereqs, let’s get to the bit what you came here for — Let’s get this under SSL Certificate and auto-manage this toil! This is a SSL Certificate for GKE Ingress — a GCP Cloud Provider-based Demo.

There are key points worthy to go-over for this particular cert-manager demonstration;

  • I am using the Let’s Encrypt Certificate Authority for this demo. There are a few to choose from.
  • I am using the DNS01 validation, there are other methods like http01 possible
  • My demo is based on the GCP platform and the relevant service-account access control setup is required.

The documentation on the cert-manager website is quite handy, but it is the usual case of piecing-up all the components together to get it to work, particularly

Go Time. Let’s get our public-facing Nginx App Certifi’ca’ted with SSL certs.

Apply the cert-manager YAML to your cluster. You need is to deploy the latest-greatest cert-manager latest, straight into Kubernetes environment. (


kubectl apply -f

TO created created created created created created
namespace/cert-manager created
serviceaccount/cert-manager-cainjector created
serviceaccount/cert-manager created
serviceaccount/cert-manager-webhook created created created created created created created created created created created created created created created created created created created created created created
service/cert-manager created
service/cert-manager-webhook created
deployment.apps/cert-manager-cainjector created
deployment.apps/cert-manager created
deployment.apps/cert-manager-webhook created created created created

This will create the namespace with all the dependencies ready to go. Now, as you’d expect we have to get a few things in order.

HouseKeeping Vars

# Global VARS to work with

Setup the GCP Service Account & Secret KEY
This will address the Kubernetes-context Cert-Manager managing that DNS zone, as that would work via the dedicated service account.

gcloud iam service-accounts create dns01-solver# Create SA to access GCP dns services with 
gcloud iam service-accounts keys create key.json \
— iam-account $CLOUD_DNS_SA@$
# Bind the role dns.admin to this service account, so it can be used to support the ACME DNS01 challenge.gcloud projects add-iam-policy-binding $PROJECT_ID \
— member serviceAccount:$CLOUD_DNS_SA \
— role roles/dns.admin

Create that K8 context Service Account secret to be used with the Cert-Manager.

kubectl create secret generic clouddns-$CLOUD_DNS_SA-svc-acct — from-file=key.json

This will create something along the lines of

ApiVersion: v1
kind: Secret
namespace: cert-manager
name: clouddns-dns01-solver-svc-acct
type: Opaque
key.json: blah-the-long-and-boring-line-of-encoded-key.json-contents-reference=

Create Cluster Issuer

These represent a certificate authority from which signed x509 certificates can be obtained, such as Let’s Encrypt, or your own signing key pair stored in a Kubernetes Secret resource. They are referenced by Certificate resources in order to request certificates from them. (

I have gone for the ClusterIssuer type so my certificates are managed for the global scope of the Kubernetes Cluster, not a particular namespace if it were a mere alone. (

ClusterIssuers are a resource type similar to Issuers. They are specified in exactly the same way, but they do not belong to a single namespace and can be referenced by Certificate resources from multiple different namespaces.


kind: ClusterIssuer
name: contino-cluster-cert-issuer
namespace: default
name: jpworks-contino-cert-account-key
- name: cloud-dns-provider
project: jaroslav-pantsjoha-contino
name: clouddns-dns01-solver-svc-acct
key: key.json

TO createdk describe
Name: contino-cluster-cert-issuer
Labels: <none>
Project: jaroslav-pantsjoha-contino
Service Account Secret Ref:
Key: key.json
Name: clouddns-dns01-solver-svc-acct
Name: cloud-dns-provider
Private Key Secret Ref:
Name: jpworks-contino-cert-account-key
Last Registered Email:
Last Transition Time: 2019-10-24T12:28:05Z
Message: The ACME account was registered with the ACME server
Reason: ACMEAccountRegistered
Status: True # << --- This bit
Type: Ready # << ---- We're good to go!
Events: <none>

Create your Initial Certificate

As the name suggests, let’s request that certificate. Here we essentially have a CSR ‘form’, with the validation type. DNS01 is the validation method of choice. Coupled with the service we have running on this Kubernetes cluster, this method should provide the automated Certificate setup and on-going renewal management.


kind: Certificate
name: jpworks-squadzero-io-certificate
secretName: jpworks-squadzero-io-certificate
# The certificate common name, use one from your domains.
# we can do wildcard here to catch all zone domans
commonName: "*"
kind: ClusterIssuer
name: contino-cluster-cert-issuer
- dns01:
provider: cloud-dns-provider
# certificate wildcards or FQDN
- "*"
# Provide same list as `domains` section.
- "*"


$ k describe
Name: jpworks-squadzero-io-certificate
Namespace: default
Labels: <none>
Provider: cloud-dns-provider
Common Name: *
Dns Names:
Issuer Ref:
Kind: ClusterIssuer
Name: contino-cluster-cert-issuer
Secret Name: jpworks-squadzero-io-certificate
Last Transition Time: 2019-10-24T12:30:37Z
Message: Certificate issuance in progress. Temporary certificate issued.
Reason: TemporaryCertificate
Status: False
Type: Ready
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Generated 2m15s cert-manager Generated new private key
Normal GenerateSelfSigned 2m15s cert-manager Generated temporary self signed certificate
Normal OrderCreated 2m15s cert-manager Created Order resource "jpworks-squadzero-io-certificate-858028746"

Create Ingress to use that certificate, or update existing ones.

This is down to choice, we can update the existing ingress and have the annotation ` true` to ensure it’s backward compatible.


apiVersion: extensions/v1beta1
kind: Ingress
annotations: contino-cluster-cert-issuer "true"
name: secure-ingress
namespace: default
- hosts:
secretName: jpworks-squadzero-io-certificate

- host:
- backend:
serviceName: nginx-service
servicePort: 80
path: /


$ k get ingress
nginx-ingress-default 80 54m
secure-ingress 80, 443 2m14s
$ k describe ing secure-ingress
Name: secure-ingress
Namespace: default
Default backend: default-http-backend:80 (
jpworks-squadzero-io-certificate terminates
Host Path Backends
---- ---- --------
/ nginx-service:80 (
Annotations: k8s-ssl-6d9affb27c1fd9d1-262a5bd4122769ef--d8e364bffd9eccef k8s-fw-default-secure-ingress--d8e364bffd9eccef contino-cluster-cert-issuer {"k8s-be-30875--d8e364bffd9eccef":"HEALTHY","k8s-be-31338--d8e364bffd9eccef":"HEALTHY"}
.... true
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ADD 4m17s loadbalancer-controller default/secure-ingress
Normal CREATE 3m38s loadbalancer-controller ip:

Our Services and Ingress Console view, after all these configurations, should look like this;

Image for post
Image for post

Waiting Game commences. Despite the seemingly complete configuration, the immediate result is that the generated a self-signed certificate. This is a temporary certificate until it gets signed by the CA (Let’s Encrypt)

openssl s_client -showcerts -connect
depth=0 O = cert-manager, CN = *
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 O = cert-manager, CN = *
verify error:num=21:unable to verify the first certificate
verify return:1
Certificate chain
0 s:/O=cert-manager/CN=*
i:/O=cert-manager/CN=cert-manager.local ### <---

We can tail the log from the cert-manager, as well as view the zone record.

k -n cert-manager logs -f cert-manager-84fc69dbdf-jbl9w....
E1024 12:51:46.465810 1 sync.go:183] cert-manager/controller/challenges "msg"="propagation check failed" "error"="DNS record for \"\" not yet propagated" "dnsName"="" "resource_kind"="Challenge" "resource_name"="jpworks-squadzero-io-certificate-858028746-0" "resource_namespace"="default" "type"="dns-01"
Image for post
Image for post
Waiting for the auto-generated DNS01 challenge to be approved — DNS propagation wait

Now, Wait ….a few more minutes

What is happening there?
Cert-Manager will discover the certificate requirement across the cluster (from ingress definitions)
- Cert-Manager will the automatically create the DNS challenge (you can view your DNS zone)
- Cert-Manager will then match/self-solve that DNS challenge, and issue a temp certificate
- Cert-Manager will get the designated CA (Let’s Encrypt) to sign it, to make it all official and ‘permanent’
- Cert-Manager will repeat this on every renewal date (every 3 months with Let’s Encrypt)

Short Time Later

k describe cert jpworks-squadzero-io-certificate
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Generated 30m cert-manager Generated new private key
Normal GenerateSelfSigned 30m cert-manager Generated temporary self signed certificate
Normal OrderCreated 30m cert-manager Created Order resource "jpworks-squadzero-io-certificate-858028746"
Normal OrderComplete 7m12s cert-manager Order "jpworks-squadzero-io-certificate-858028746" completed successfully
Normal CertIssued 7m12s cert-manager Certificate issued successfully
k -n cert-manager logs -f cert-manager-84fc69dbdf-jbl9w"related_resource_kind"="Certificate" "related_resource_name"="jpworks-squadzero-io-certificate" "related_resource_namespace"="default" "resource_kind"="Ingress" "resource_name"="secure-ingress" "resource_namespace"="default"
I1024 12:56:27.509017 1 sync.go:208] cert-manager/controller/ingress-shim "level"=0 "msg"="certificate resource has no owner. refusing to update non-owned certificate resource for ingress" "related_resource_kind"="Certificate" "related_resource_name"="jpworks-squadzero-io-certificate" "related_resource_namespace"="default" "resource_kind"="Ingress" "resource_name"="secure-ingress" "resource_namespace"="default"
I1024 12:56:27.509144 1 controller.go:135] cert-manager/controller/ingress-shim "level"=0 "msg"="finished processing work item" "key"="default/secure-ingress"

Trust but Verify

Curl and OpenSSL validate the results, with;

$ curl -vI
* Trying
* Connected to ( port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
* CAfile: /etc/ssl/cert.pem
CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-CHACHA20-POLY1305
* ALPN, server accepted to use h2
* Server certificate:
* subject: CN=*
* start date: Oct 24 11:53:51 2019 GMT
* expire date: Jan 22 11:53:51 2020 GMT
* subjectAltName: host "" matched cert's "*"
* issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
# OPENSSL check that$ openssl s_client -showcerts -connect
depth=2 O = Digital Signature Trust Co., CN = DST Root CA X3
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
verify return:1
depth=0 CN = *
verify return:1
Certificate chain
0 s:/CN=*
i:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3

Curl-tested, OpenSSL-reviewed, let’s get that screenshot as well, shall we;

Image for post
Image for post

Saved you a lot of manual effort and your application communication is secure. Time for Beers?

If you enjoyed this post and want to see more end-to-end Kubernetes demos, please clap and share the post

See you in the next post,

Image for post
Image for post

PS There is a host of exciting Kubernetes projects taking place at Contino. If you are looking to work on the latest-greatest infrastructure stack or looking for a challenge, — Get in touch! We’re hiring!

We’re looking for bright minds at every level. At Contino, we pride ourselves on delivering the best practices cloud transformation projects, for medium-sized businesses to large enterprises.


By the way, 👏🏻 *clap* 👏🏻 your hands (up to 50x) if you enjoyed this post. It encourages me to keep writing and help other people finding it :)

Contino Engineering

Opinions from Contino Engineering

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store