How-To: Automatic SSL Certificate Management for your Kubernetes Application Deployment
A bit of the Same, But Different. We deployed the app, but Let’s ensure our SSL Certificate is managed automatically for our Application Deployment.
Welcome back, or welcome for the very first time. This Security themed post will take the example of the previous blog post, “How-to: Kubernetes Application Deployment with DNS management” — which exemplifies the relative simplicity in application deployment with automatic DNS management, and takes it to the next “production-closer” level.
Let’s ensure such an application deployment process is matched with the SSL certificate. This post will lead it’s way into the GitOps deployment methodology again. The purpose of which is to codify your infrastructure, store it in Git and let the likes of Weaveworks Flux (https://github.com/fluxcd/flux) deploy it automatically, and in full.
Your System Requirements
This may be obvious, but you will need git, google-cloud-sdk and kubectl installed on whichever system you are running this.
As I do intend the hands-on number of your readers to be able to duplicate the success, so do ensure you read the First Blog Post, which outlines the Google DNS zone requirement and the follow-up application configuration, from here. The full code example with yet-another DNS zone, but the same service configuration, with yet full working code example, is found on my own bitbucket.
There is a number of dependencies to get this off the ground, to reproduce this, you will require to ensure the following checklist is in order;
- Your Google Cloud DNS zone is setup
- Kubernetes Cluster is created
- Kubernetes Demo Application is running from the previous post.
- Get ready to start pasting more…
Where were we
Before getting into the meat of this, let’s fast forward this dependency prep formality.
DNS ZONE is the same as before.
I’ve deployed the same Kubernetes cluster, via Google Cloud gcloud cli utility.
FROM
PROJECT=jaroslav-pantsjoha-contino
CLUSTER_NAME=jpworks-clustergcloud beta container --project "$PROJECT" clusters create "$CLUSTER_NAME" \
--zone "us-central1-a" --no-enable-basic-auth \
--cluster-version "1.14.6-gke.2" --machine-type "n1-standard-2" \
--image-type "COS" --disk-type "pd-ssd" --disk-size "50" \
--metadata disable-legacy-endpoints=true \
--scopes "https://www.googleapis.com/auth/cloud-platform" \
--preemptible --num-nodes "2" \
--enable-cloud-logging \
--enable-stackdriver-kubernetes --enable-ip-alias \
--network "projects/$PROJECT/global/networks/default" \
--subnetwork "projects/$PROJECT/regions/us-central1/subnetworks/default" \
--default-max-pods-per-node "110" --addons HorizontalPodAutoscaling,HttpLoadBalancing \
--enable-autoupgrade \
--enable-autorepair
TO
kubeconfig entry generated for jpworks-cluster.NAME LOCATION MASTER_VERSION MASTER_IP MACHINE_TYPE NODE_VERSION NUM_NODES STATUSjpworks-cluster us-central1-a 1.14.6-gke.2 35.226.209.236 n1-standard-2 1.14.6-gke.2 2 RUNNING
Deploy the very same Demo Nginx App, Ingress, and the DNS Manager;
k apply -f external-dns
deployment.extensions/external-dns created
serviceaccount/external-dns created
clusterrole.rbac.authorization.k8s.io/external-dns created
clusterrolebinding.rbac.authorization.k8s.io/external-dns-viewer createdk apply -f nginx-app/
configmap/nginx-configuration created
deployment.extensions/nginx-app created
service/nginx-service createdk apply -f ingress.yaml
ingress.extensions/nginx-ingress-default created
Verify. Same old, same old.
k get po,ing,svc
NAME READY STATUS RESTARTS AGE
pod/external-dns-8947bd5b9-zx98s 1/1 Running 0 6m20s
pod/nginx-app-6979bdd88f-bqtmb 1/1 Running 0 6m12sNAME HOSTS ADDRESS PORTS AGE
ingress.extensions/nginx-ingress-default nginx.jpworks.squadzero.io 35.241.22.96 80 5m47sNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.0.0.1 <none> 443/TCP 12m
service/nginx-service LoadBalancer 10.0.1.23 10.128.0.62 80:31338/TCP 6m13s
This does get some (little) amount of time to get DNS records propagated throughout, — taken me closer to 10 minutes to finally see it work here.
$ curl -I nginx.jpworks.squadzero.io
HTTP/1.1 200 OK
Server: nginx
Date: Thu, 24 Oct 2019 11:52:18 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 55
Last-Modified: Thu, 24 Oct 2019 11:41:44 GMT
ETag: "5db18df8-37"
Accept-Ranges: bytes
Via: 1.1 googleYou have new mail in /var/mail/jp
Jaroslavs-MacBook-Pro:external-dns-demo jp$ curl nginx.jpworks.squadzero.io
<h1>hello world</h1><p>flux-kubernetes-demos/nginx</p>
SSL Manage This
Now that the environment setup and formalities are out of the way with the prereqs, let’s get to the bit what you came here for — Let’s get this under SSL Certificate and auto-manage this toil! This is a SSL Certificate for GKE Ingress — a GCP Cloud Provider-based Demo.
There are key points worthy to go-over for this particular cert-manager demonstration;
- I am using the Let’s Encrypt Certificate Authority for this demo. There are a few to choose from.
- I am using the DNS01 validation, there are other methods like http01 possible
- My demo is based on the GCP platform and the relevant service-account access control setup is required.
The documentation on the cert-manager website is quite handy, but it is the usual case of piecing-up all the components together to get it to work, particularly https://docs.cert-manager.io/en/latest/tutorials/acme/dns-validation.html
Go Time. Let’s get our public-facing Nginx App Certifi’ca’ted with SSL certs.
Apply the cert-manager YAML to your cluster. You need is to deploy the latest-greatest cert-manager latest, straight into Kubernetes environment. (https://docs.cert-manager.io/en/latest/tutorials/venafi/securing-ingress.html#installing-cert-manager)
FROM
kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v0.10.1/cert-manager.yamlTO
customresourcedefinition.apiextensions.k8s.io/certificaterequests.certmanager.k8s.io created
customresourcedefinition.apiextensions.k8s.io/certificates.certmanager.k8s.io created
customresourcedefinition.apiextensions.k8s.io/challenges.certmanager.k8s.io created
customresourcedefinition.apiextensions.k8s.io/clusterissuers.certmanager.k8s.io created
customresourcedefinition.apiextensions.k8s.io/issuers.certmanager.k8s.io created
customresourcedefinition.apiextensions.k8s.io/orders.certmanager.k8s.io created
namespace/cert-manager created
serviceaccount/cert-manager-cainjector created
serviceaccount/cert-manager created
serviceaccount/cert-manager-webhook created
clusterrole.rbac.authorization.k8s.io/cert-manager-cainjector created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-cainjector created
clusterrole.rbac.authorization.k8s.io/cert-manager-leaderelection created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-issuers created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-clusterissuers created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-certificates created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-orders created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-challenges created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-ingress-shim created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-leaderelection created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-issuers created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-clusterissuers created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-certificates created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-orders created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-challenges created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-ingress-shim created
clusterrole.rbac.authorization.k8s.io/cert-manager-view created
clusterrole.rbac.authorization.k8s.io/cert-manager-edit created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-webhook:auth-delegator created
rolebinding.rbac.authorization.k8s.io/cert-manager-webhook:webhook-authentication-reader created
clusterrole.rbac.authorization.k8s.io/cert-manager-webhook:webhook-requester created
service/cert-manager created
service/cert-manager-webhook created
deployment.apps/cert-manager-cainjector created
deployment.apps/cert-manager created
deployment.apps/cert-manager-webhook created
apiservice.apiregistration.k8s.io/v1beta1.webhook.certmanager.k8s.io created
mutatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook created
validatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook createdThis will create the namespace with all the dependencies ready to go. Now, as you’d expect we have to get a few things in order.
HouseKeeping Vars
# Global VARS to work with
PROJECT_ID=jaroslav-pantsjoha-contino
CLOUD_DNS_SA=dns01-solverSetup the GCP Service Account & Secret KEY
This will address the Kubernetes-context Cert-Manager managing that DNS zone, as that would work via the dedicated service account.
gcloud iam service-accounts create dns01-solver# Create SA to access GCP dns services with
gcloud iam service-accounts keys create key.json \
— iam-account $CLOUD_DNS_SA@$PROJECT_ID.iam.gserviceaccount.com# Bind the role dns.admin to this service account, so it can be used to support the ACME DNS01 challenge.gcloud projects add-iam-policy-binding $PROJECT_ID \
— member serviceAccount:$CLOUD_DNS_SA \
— role roles/dns.admin
Create that K8 context Service Account secret to be used with the Cert-Manager.
kubectl create secret generic clouddns-$CLOUD_DNS_SA-svc-acct — from-file=key.jsonThis will create something along the lines of
ApiVersion: v1
kind: Secret
metadata:
namespace: cert-manager
name: clouddns-dns01-solver-svc-acct
type: Opaque
data:
key.json: blah-the-long-and-boring-line-of-encoded-key.json-contents-reference=Create Cluster Issuer
These represent a certificate authority from which signed x509 certificates can be obtained, such as Let’s Encrypt, or your own signing key pair stored in a Kubernetes Secret resource. They are referenced by Certificate resources in order to request certificates from them. (https://docs.cert-manager.io/en/latest/tasks/issuers/index.html)
I have gone for the ClusterIssuer type so my certificates are managed for the global scope of the Kubernetes Cluster, not a particular namespace if it were a mere Issueralone. (https://docs.cert-manager.io/en/latest/reference/clusterissuers.html)
ClusterIssuers are a resource type similar to Issuers. They are specified in exactly the same way, but they do not belong to a single namespace and can be referenced by Certificate resources from multiple different namespaces.
FROM
apiVersion: certmanager.k8s.io/v1alpha1
kind: ClusterIssuer
metadata:
name: contino-cluster-cert-issuer
namespace: default
spec:
acme:
server: https://acme-v02.api.letsencrypt.org/directory
email: jaroslav.pantsjoha@contino.io
privateKeySecretRef:
name: jpworks-contino-cert-account-key
dns01:
providers:
- name: cloud-dns-provider
clouddns:
project: jaroslav-pantsjoha-contino
serviceAccountSecretRef:
name: clouddns-dns01-solver-svc-acct
key: key.jsonTO
clusterissuer.certmanager.k8s.io/contino-cluster-cert-issuer createdk describe clusterissuer.certmanager.k8s.io/contino-cluster-cert-issuer
Name: contino-cluster-cert-issuer
Namespace:
Labels: <none>
Annotations: kubectl.kubernetes.io/last-applied-configuration:
.....
Spec:
Acme:
dns01:
Providers:
Clouddns:
Project: jaroslav-pantsjoha-contino
Service Account Secret Ref:
Key: key.json
Name: clouddns-dns01-solver-svc-acct
Name: cloud-dns-provider
Email: jaroslav.pantsjoha@contino.io
Private Key Secret Ref:
Name: jpworks-contino-cert-account-key
Server: https://acme-v02.api.letsencrypt.org/directory
Status:
Acme:
Last Registered Email: jaroslav.pantsjoha@contino.io
Uri: https://acme-v02.api.letsencrypt.org/acme/acct/<omitted>
Conditions:
Last Transition Time: 2019-10-24T12:28:05Z
Message: The ACME account was registered with the ACME server
Reason: ACMEAccountRegistered
Status: True # << --- This bit
Type: Ready # << ---- We're good to go!
Events: <none>
Create your Initial Certificate
As the name suggests, let’s request that certificate. Here we essentially have a CSR ‘form’, with the validation type. DNS01 is the validation method of choice. Coupled with the External-DNS service we have running on this Kubernetes cluster, this method should provide the automated Certificate setup and on-going renewal management.
FROM
apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
name: jpworks-squadzero-io-certificate
spec:
secretName: jpworks-squadzero-io-certificate
# The certificate common name, use one from your domains.
# we can do wildcard here to catch all zone domans
commonName: "*.jpworks.squadzero.io"
issuerRef:
kind: ClusterIssuer
name: contino-cluster-cert-issuer
acme:
config:
- dns01:
provider: cloud-dns-provider
domains:
# certificate wildcards or FQDN
- "*.jpworks.squadzero.io"
dnsNames:
# Provide same list as `domains` section.
- "*.jpworks.squadzero.io"TO
$ k describe certificate.certmanager.k8s.io/jpworks-squadzero-io-certificate
Name: jpworks-squadzero-io-certificate
Namespace: default
Labels: <none>
Annotations: kubectl.kubernetes.io/last-applied-configuration:
......
Spec:
Acme:
Config:
dns01:
Provider: cloud-dns-provider
Domains:
*.jpworks.squadzero.io
Common Name: *.jpworks.squadzero.io
Dns Names:
*.jpworks.squadzero.io
Issuer Ref:
Kind: ClusterIssuer
Name: contino-cluster-cert-issuer
Secret Name: jpworks-squadzero-io-certificate
Status:
Conditions:
Last Transition Time: 2019-10-24T12:30:37Z
Message: Certificate issuance in progress. Temporary certificate issued.
Reason: TemporaryCertificate
Status: False
Type: Ready
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Generated 2m15s cert-manager Generated new private key
Normal GenerateSelfSigned 2m15s cert-manager Generated temporary self signed certificate
Normal OrderCreated 2m15s cert-manager Created Order resource "jpworks-squadzero-io-certificate-858028746"Create Ingress to use that certificate, or update existing ones.
This is down to choice, we can update the existing ingress and have the annotation `kubernetes.io/ingress.allow-http: true` to ensure it’s backward compatible.
FROM
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
certmanager.k8s.io/cluster-issuer: contino-cluster-cert-issuer
kubernetes.io/ingress.allow-http: "true"
name: secure-ingress
namespace: default
spec:
tls:
- hosts:
- secure.jpworks.squadzero.io
secretName: jpworks-squadzero-io-certificate
rules:
- host: secure.jpworks.squadzero.io
http:
paths:
- backend:
serviceName: nginx-service
servicePort: 80
path: /TO
$ k get ingress
NAME HOSTS ADDRESS PORTS AGE
nginx-ingress-default nginx.jpworks.squadzero.io 35.241.22.96 80 54m
secure-ingress secure.jpworks.squadzero.io 34.98.102.173 80, 443 2m14s$ k describe ing secure-ingress
Name: secure-ingress
Namespace: default
Address: 34.98.102.173
Default backend: default-http-backend:80 (10.40.1.6:8080)
TLS:
jpworks-squadzero-io-certificate terminates secure.jpworks.squadzero.io
Rules:
Host Path Backends
---- ---- --------
secure.jpworks.squadzero.io
/ nginx-service:80 (10.40.0.7:80)
Annotations:
ingress.kubernetes.io/ssl-cert: k8s-ssl-6d9affb27c1fd9d1-262a5bd4122769ef--d8e364bffd9eccef
ingress.kubernetes.io/static-ip: k8s-fw-default-secure-ingress--d8e364bffd9eccef.....certmanager.k8s.io/cluster-issuer: contino-cluster-cert-issuer
ingress.kubernetes.io/backends: {"k8s-be-30875--d8e364bffd9eccef":"HEALTHY","k8s-be-31338--d8e364bffd9eccef":"HEALTHY"}
....
kubernetes.io/ingress.allow-http: true
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ADD 4m17s loadbalancer-controller default/secure-ingress
Normal CREATE 3m38s loadbalancer-controller ip: 34.98.102.173
Our Services and Ingress Console view, after all these configurations, should look like this;
Waiting Game commences. Despite the seemingly complete configuration, the immediate result is that the cert-manager generated a self-signed certificate. This is a temporary certificate until it gets signed by the CA (Let’s Encrypt)
openssl s_client -showcerts -connect secure.jpworks.squadzero.io:443
CONNECTED(00000005)
depth=0 O = cert-manager, CN = *.jpworks.squadzero.io
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 O = cert-manager, CN = *.jpworks.squadzero.io
verify error:num=21:unable to verify the first certificate
verify return:1
---
Certificate chain
0 s:/O=cert-manager/CN=*.jpworks.squadzero.io
i:/O=cert-manager/CN=cert-manager.local ### <---
.....We can tail the log from the cert-manager, as well as view the zone record.
k -n cert-manager logs -f cert-manager-84fc69dbdf-jbl9w....
E1024 12:51:46.465810 1 sync.go:183] cert-manager/controller/challenges "msg"="propagation check failed" "error"="DNS record for \"jpworks.squadzero.io\" not yet propagated" "dnsName"="jpworks.squadzero.io" "resource_kind"="Challenge" "resource_name"="jpworks-squadzero-io-certificate-858028746-0" "resource_namespace"="default" "type"="dns-01"
Now, Wait ….a few more minutes
What is happening there?
- Cert-Manager will discover the certificate requirement across the cluster (from ingress definitions)
- Cert-Manager will the automatically create the DNS challenge (you can view your DNS zone)
- Cert-Manager will then match/self-solve that DNS challenge, and issue a temp certificate
- Cert-Manager will get the designated CA (Let’s Encrypt) to sign it, to make it all official and ‘permanent’
- Cert-Manager will repeat this on every renewal date (every 3 months with Let’s Encrypt)
Short Time Later
k describe cert jpworks-squadzero-io-certificate
.....
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Generated 30m cert-manager Generated new private key
Normal GenerateSelfSigned 30m cert-manager Generated temporary self signed certificate
Normal OrderCreated 30m cert-manager Created Order resource "jpworks-squadzero-io-certificate-858028746"
Normal OrderComplete 7m12s cert-manager Order "jpworks-squadzero-io-certificate-858028746" completed successfully
Normal CertIssued 7m12s cert-manager Certificate issued successfullyk -n cert-manager logs -f cert-manager-84fc69dbdf-jbl9w"related_resource_kind"="Certificate" "related_resource_name"="jpworks-squadzero-io-certificate" "related_resource_namespace"="default" "resource_kind"="Ingress" "resource_name"="secure-ingress" "resource_namespace"="default"
I1024 12:56:27.509017 1 sync.go:208] cert-manager/controller/ingress-shim "level"=0 "msg"="certificate resource has no owner. refusing to update non-owned certificate resource for ingress" "related_resource_kind"="Certificate" "related_resource_name"="jpworks-squadzero-io-certificate" "related_resource_namespace"="default" "resource_kind"="Ingress" "resource_name"="secure-ingress" "resource_namespace"="default"
I1024 12:56:27.509144 1 controller.go:135] cert-manager/controller/ingress-shim "level"=0 "msg"="finished processing work item" "key"="default/secure-ingress"
Trust but Verify
Curl and OpenSSL validate the results, with;
$ curl -vI https://secure.jpworks.squadzero.io/
* Trying 34.98.102.173...
* TCP_NODELAY set
* Connected to secure.jpworks.squadzero.io (34.98.102.173) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
* CAfile: /etc/ssl/cert.pem
CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-CHACHA20-POLY1305
* ALPN, server accepted to use h2
* Server certificate:
* subject: CN=*.jpworks.squadzero.io
* start date: Oct 24 11:53:51 2019 GMT
* expire date: Jan 22 11:53:51 2020 GMT
* subjectAltName: host "secure.jpworks.squadzero.io" matched cert's "*.jpworks.squadzero.io"
* issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use# OPENSSL check that$ openssl s_client -showcerts -connect secure.jpworks.squadzero.io:443
CONNECTED(00000005)
depth=2 O = Digital Signature Trust Co., CN = DST Root CA X3
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
verify return:1
depth=0 CN = *.jpworks.squadzero.io
verify return:1
---
Certificate chain
0 s:/CN=*.jpworks.squadzero.io
i:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
-----BEGIN CERTIFICATE-----
MIIFZTCCBE2gAwIBAgISA2Nob1YBGrHPmTnzq3a62WAoMA0GCSqGSIb3DQEBCwUA
MEoxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MSMwIQYDVQQD
ExpMZXQncyBFbmNyeXB0IEF1dGhvcml0eSBYMzAeFw0xOTEwMjQxMTUzNTFaFw0y
Curl-tested, OpenSSL-reviewed, let’s get that screenshot as well, shall we;
There.
Saved you a lot of manual effort and your application communication is secure. Time for Beers?
If you enjoyed this post and want to see more end-to-end Kubernetes demos, please clap and share the post
See you in the next post,
JP
PS There is a host of exciting Kubernetes projects taking place at Contino. If you are looking to work on the latest-greatest infrastructure stack or looking for a challenge, — Get in touch! We’re hiring!
We’re looking for bright minds at every level. At Contino, we pride ourselves on delivering the best practices cloud transformation projects, for medium-sized businesses to large enterprises.
