Manage SSL certificates for local Kubernetes clusters with cert-manager

I use local Kubernetes clusters extensively and often need to browse websites or call APIs hosted in my local cluster.

While HTTP is not a problem, HTTPS requires that the server has an SSL certificate configured and the client trusts that certificate.

This is not only true when calling from the host, but can also be a concern for services running inside the cluster that need to talk to another service over HTTPS.

In a previous story I wrote about securing the Kubernetes api server with an OIDC compatible identity provider (Keycloak).

Kind, Keycloak — Securing Kubernetes api server with OIDC

The api server was configured to talk with a Keycloak instance running inside the local cluster and the communication was required to use HTTPS.

I used a self signed certificate and configured the api server to trust the manually generated certificate.

In this story, I will show a simpler and more extensible solution, using cert-manager to automate the creation of our Keycloak instance certificate, trust the root certificate on the host system, mount it on cluster nodes, and import it in chrome to allow browsing HTTPS websites seamlessly.

Create the root certificate

The root certificate is the one that will be used as a certificate authority. This is the certificate that we will use to sign other certificates, and the one that needs to be trusted when establishing a secure communication.

It basically boils down to a private key and a certificate, we can create both easily with openssl:

# create a folder to store certificate files
mkdir -p .ssl
# generate an rsa key
openssl genrsa -out .ssl/root-ca-key.pem 2048
# generate root certificate
openssl req -x509 -new -nodes -key .ssl/root-ca-key.pem \
  -days 3650 -sha256 -out .ssl/root-ca.pem -subj "/CN=kube-ca"

Now we have a root certificate, we will use it to act like a certificate authority, we will use it to issue and sign other certificates.

The challenging part is to get this certificate trusted though, that’s what makes certificates secure, we could sign a certificate for google.com with our root certificate… but nobody would trust it.

Create a local Kubernetes cluster

Linux stores the list of trusted certificates in the /etc/ssl/certs folder but mounting our root certificate here could be a security issue, we will mount it in a separate folder on cluster nodes instead,/opt/ca-certificates for example.

Adding the root certificate in cluster nodes will allow workloads to mount it when needed.

So, let’s create a local cluster with Kind and mount our root certificate in the /opt/ca-certificates folder on cluster nodes:

kind create cluster --image kindest/node:v1.23.1 --config - <<EOF
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  # mount our root certificate in a separate folder
  extraMounts:
  - hostPath: $PWD/.ssl/root-ca.pem
    containerPath: /opt/ca-certificates/root-ca.pem
    readOnly: true
- role: worker
  # mount our root certificate in a separate folder
  extraMounts:
  - hostPath: $PWD/.ssl/root-ca.pem
    containerPath: /opt/ca-certificates/root-ca.pem
    readOnly: true
EOF

Deploy MetalLB, ingress-nginx and dnsmasq

I already wrote about MetalLB, ingress-nginx and dnsmasq in previous stories. Please refer to those stories if you need more details about why and how to set them up.

Kind, Cilium, MetalLB, and still no kube-proxy

Using dnsmasq for local Kind clusters

Or just run the script below if you want to get them installed quickly.

# install metallb
KIND_NET_CIDR=$(docker network inspect kind \
  -f '{{(index .IPAM.Config 0).Subnet}}')
METALLB_IP_START=$(echo ${KIND_NET_CIDR} | sed "s@0.0/16@255.200@")
METALLB_IP_END=$(echo ${KIND_NET_CIDR} | sed "s@0.0/16@255.250@")
METALLB_IP_RANGE="${METALLB_IP_START}-${METALLB_IP_END}"
helm upgrade --install --wait --timeout 15m \
  --namespace metallb-system --create-namespace \
  --repo https://metallb.github.io/metallb metallb metallb \
  --values - <<EOF
configInline:
  address-pools:
  - name: default
    protocol: layer2
    addresses:
    - ${METALLB_IP_RANGE}
EOF
# install ingress-nginx
helm upgrade --install --wait --timeout 15m \
  --namespace ingress-nginx --create-namespace \
  --repo https://kubernetes.github.io/ingress-nginx \
  ingress-nginx ingress-nginx \
  --values - <<EOF
defaultBackend:
  enabled: true
EOF
# retrieve local load balancer IP address
LB_IP=$(kubectl get svc -n ingress-nginx ingress-nginx-controller \
  -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
# point kind.cluster domain (and subdomains) to our load balancer
echo "address=/kind.cluster/$LB_IP" | \
  sudo tee /etc/dnsmasq.d/kind.k8s.conf
# restart dnsmasq
sudo systemctl restart dnsmasq

We now have an ingress controller running, served by a local load balancer and a kind.cluster dns domain that points to the load balancer.

Next we will deploy and configure cert-manager to generate SSL certificates automatically for registered ingresses, based on our root certificate.

Deploy cert-manager

First, we need to deploy cert-manager with Helm:

helm upgrade --install --wait --timeout 15m \
  --namespace cert-manager --create-namespace \
  --repo https://charts.jetstack.io cert-manager cert-manager \
  --values - <<EOF
installCRDs: true
EOF

After a while, cert-manager should be up and running.

We can configure a ClusterIssuer to generate self signed certificates and act as a certificate authority but we will need to create a secret with our root certificate first (so that cert-manager can sign generated certificates with it):

# create secret in the cert-manager namespace
# holding our root certificate and private key
kubectl create secret tls -n cert-manager root-ca \
  --cert=.ssl/root-ca.pem \
  --key=.ssl/root-ca-key.pem

Finally we can create our ClusterIssuer:

# create cert-manager cluster issuer using the secret
# containing our root certificate created earlier
kubectl apply -n cert-manager -f - <<EOF
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: ca-issuer
spec:
  ca:
    secretName: root-ca
EOF

At this point, every certificate generated by cert-manager through the ClusterIssuer declared above will be signed with our root certificate.

If we get our root certificate to be trusted it will be all good.

Next we will deploy Keycloak and expose it with an Ingress, asking cert-manager to generate the SSL certificate for us.

Deploy Keycloak

Just like in the previous story, we will use Helm to deploy Keycloak.

The only difference is that we will add the cert-manager.io/cluster-issuer annotation to the ingress definition in order to instruct cert-manager to generate an SSL certificate.

Please note that TLS termination will happen in the ingress controller, not in the target pod itself. The communication will be encrypted between the client and the ingress controller, but the traffic between the ingress controller and the pod will stay in clear text.

Run the command below to deploy Keycloak:

helm upgrade --install --wait --timeout 15m \
  --namespace keycloak --create-namespace \
  --repo https://charts.bitnami.com/bitnami keycloak keycloak \
  --reuse-values --values - <<EOF
proxyAddressForwarding: true
ingress:
  enabled: true
  hostname: keycloak.kind.cluster
  annotations:
    kubernetes.io/ingress.class: nginx
    cert-manager.io/cluster-issuer: ca-issuer
  tls: true
EOF

Once deployed, trying to browse http://keycloak.kind.cluster should trigger a redirection to https://keycloak.kind.cluster (HTTP to HTTPS), but it should still appear unsafe in your browser.

HTTPS is ok but root certificate not trusted

This because the exercised SSL certificate is signed by our root certificate but your browser probably does not trust it yet.

Configure browser to trust root certificate

Settings to configure a browser to trust certificates vary per browser, I will be assuming Chrome here.

1. Open a browser at chrome://settings/certificates
2. Click the Authorities tab and click Import then select the root-ca.pem file generated in the first step of this story
3. Check Trust this certificate for identifying websites and validate

Now, browsing https://keycloak.kind.cluster should show as safe and exhibit the small lock icon 🎉

HTTPS and trusted root certificate !

Configure host to trust root certificate

Our browser now trusts the root certificate but how about tools like curl ?

$ curl https://keycloak.kind.cluster
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html
curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

This is because a browser and the local system usually need to be configured differently.

To configure the host system to trust our root certificate, we need to run the script below:

# copy our root certificate
sudo cp .ssl/root-ca.pem /usr/local/share/ca-certificates/ca.crt
# update ca certificates
sudo update-ca-certificates

Now, curl will work fine 🎉

$ curl https://keycloak.kind.cluster
<!--
  ~ Copyright 2016 Red Hat, Inc. and/or its affiliates
  ~ and other contributors as indicated by the @author tags.
  ~
  ~ Licensed under the Apache License, Version 2.0 (the "License");
  ~ you may not use this file except in compliance with the License.
  ~ You may obtain a copy of the License at
  ~
  ~ http://www.apache.org/licenses/LICENSE-2.0
  ~
  ~ Unless required by applicable law or agreed to in writing, software
  ~ distributed under the License is distributed on an "AS IS" BASIS,
  ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  ~ See the License for the specific language governing permissions and
  ~ limitations under the License.
  -->
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
    <meta http-equiv="refresh" content="0; url=/auth/" />
    <meta name="robots" content="noindex, nofollow">
    <script type="text/javascript">
        window.location.href = "/auth/"
    </script>
</head>
<body>
    If you are not redirected automatically, follow this <a href='/auth'>link</a>.
</body>
</html>

Wrapping it up

In the end, working with SSL certificates and local clusters is a bit of work.

Somethimes things need to be adapted, especially workloads might need to mount the certificate from the host.

It’s really worth the effort though, as being able to work with HTTPS locally helps troubleshooting things like mixed content that would be blocked in production or unexpected insecure communication issues.

You can find all the code provided in this article (and more) in my GitHub repository:

GitHub - eddycharly/kind-playground: Playing with local Kubernetes cluster with Kind, Cilium…