Using a wildcard certificate within your Kubernetes cluster

In a recent article I wrote about how to add TLS connectivity to MinIO using a Let’s Encrypt wildcard certificate. I realised that that was a very specific use case and that you may be interested in knowing how to use your wildcard certificate across any of your Kubernetes Services.

Using wildcard certificates with coreDNS

Wildcard Certificates

Certificates are normally issued for a specific domain, eg: requillion-solutions.com.au. This is known as the certificates common name. It means that if you access a website, such as requillion-solutions.com.au, the browser can check that the certificate really represents the site you visited. It checks the common name against the host name it connected to. If they match — great. If not, you will probably get an error message.

Wildcard certificates are like any single domain certificate apart from their common name having a *. at the start. The * (called the wildcard) represents any single layer of subdomain on top of the base domain, eg: *.requillion-solutions.com.au will match the subdomain www.requillion-solutions.com.au.

When matching a wildcard certificate, it allows the upper most subdomain to be anything. So, for a wildcard certificate of *.requillion-solutions.com.au, the following will match:

• www.requillion-solutions.com.au
• pg.requillion-solutions.com.au
• kong.requillion-solutions.com.au

But it will not match:

• requillion-solutions.com.au
• my.site.requillion-solutions.com.au
• api.kong.requillion-solutions.com.au

Remember that the hostname must match the certificates, wildcarded common name.

Note: It is not possible to have a wildcard certificate with two levels of wildcard, eg: *.*.requillion-solutions.com.au.

Applying wildcard certificates to a Kubernetes Service

Kubernetes creates internal DNS entries for its services. For example, assuming a default deployment of Kubernetes, a Service called pg in namespace database would be accessible via:

• pg
• pg.database
• pg.database.svc
• pg.database.svc.cluster.local

Now, if the database connection is secured with a wildcard certificate (eg: *.requillion-solutions.com.au), you need to use that domain in the connection string, not the svc.cluster.local domain as that will not match the certificate that is returned.

So, if you use the Kubernetes DNS entry (eg: pg.database.svc.cluster.local) to connect to your database, the certificate will not match and the connection will fail. If you use, say, pg.requillion-solutions.com.au, the certificate would match but you will not reach the service to get the certificate, so that would fail too!

coreDNS configuration

The solution is to configure the DNS within the Kubernetes cluster to map the wildcarded domain to the internal Service entry.

From v1.21, Kubernetes has used coreDNS to provide DNS services for the cluster. Like any resource, Kubernetes allows you to alter the configuration. One of the changes you can make is to add a mapping between the domains (this is the equivalent to a CNAME entry in a public DNS configuration).

With the mapping you can map from your wildcarded domain to your internal one, eg:

pg.requillion-solution.com.au -> pg.database.svc.cluster.local

Now, when you connect to pg.requillion-solution.com.au you will be pointed to the service at pg.database.svc.cluster.local, so you will connect. You will then get back the wildcard certificate with a common name of *.requillion.solutions.com.au, which will match your request and the connection will be made.

Using this mapping, you now have a solution. All that remains is add this mapping in for every Service you create that needs a TLS connection.

Regex Mapping

Adjusting the coreDNS configuration each time you add a Service or namespace is not very efficient. But the good thing is, coreDNS accepts regex expressions.

Take the following:

(.*)-ns-(.*)\.requillion-solutions\.com.au\.$ ->  {1}.{2}.svc.cluster.local

This mapping will allow you to define your service and namespace and still match the wildcard certificate. It takes a name, such as:

<service name>-ns-<namespace>.requillion-solutions.com.au

eg:

my-pgsql-cluster-rw-ns-pg.requillion-solutions.com.au

And then maps it to:

<service name>.<namespace>.svc.cluster.local

Or:

my-pgsql-cluster-rw.pg.svc.cluster.local

By bringing the service name and namespace into a single subdomain (<service name>-ns-<namespace>), it can now be matched against a wildcarded certificate.

Reverse mapping

One of the problems with using regex to map the domain entries like we did above (called mapping the question in DNS terms) is that when a reverse look up is performed (IP address to domain name), it returns the internal DNS entry. This can cause some security checks to fail.

This can be fixed by including a reverse mapping, like this:

/(.*)\.(.*)\.svc\.cluster\.local -> {1}-ns-{2}.requillion-solutions.com.au

Configuring coreDNS

Let’s make these changes to the coreDNS. Execute the following from wherever you run kubectl (I run this on k8s-master):

export KUBE_EDITOR=nano
kubectl -n kube-system edit configmap/coredns

The first line tells kubectl to use nano as an editor rather than vi. The second line calls up the editor to change the config for coreDNS.

Now add the following above the ready line:

        rewrite stop {
          name regex (.*)-ns-(.*)\.requillion-solutions\.com\.au\.$ {1}.{2}.svc.cluster.local
          answer name (.*)\.(.*)\.svc\.cluster\.local  {1}-ns-{2}.requillion-solutions.com.au.
        }

Of course, for your use case, change the requillion-solutions.com.au on both lines. Note that the dots need to be escaped with \ in the first line.

Once you save and exit the editor, coreDNS should detect the change and load it. If it does not, you may need to restart the coreDNS pods.

Fully Qualified Domain Name (FQDN)

When using this rewrite configuration, it is important to understand how Fully Qualified Domain Names (FQDN) work.

Let’s take an example, such as using curl to access a service called my-svc:

curl my-svc

Because my-svc does not have a terminating ‘.’ it means that curl will use search options to find the IP address. In a unix Pod, these search options are found in /etc/resolve.conf and look like this:

search default.svc.cluster.local svc.cluster.local cluster.local

In this case, default refers to the namespace of the Pod. You can see that there are three search options. This means that curl will try:

my-svc.default.svc.cluster.local
my-svc.svc.cluster.local
my-svc.cluster.local
my-svc

It does this because my-svc is not fully qualified, leaving the option to search for the right domain.

With the rewrite, the search options interfere with the rewrite rules. One option is to make sure that all calls to the service use Fully Qualified Domain Names (FQDN), such as:

curl my-svc.default.requillion-solutions.com.au.

The final ‘.’, says this is an FQDN and should not have search options appended. This reliance on a FQDN can cause hard to diagnose problems.

Instead, we only rewrite for FQDNs and so the final search option will be matched and rewritten correctly. To ensure this, we add \.$ to the end of the regex to match only domain names with ‘.’ at the end.

There is more to this story with regards to the ndots option but for this article, we do not have to consider this.

Testing the configuration

We can check that the rewrite is working. If you have been following me, you will know that I use a simple Linux web service for various tests. This test service is based on a Linux distribution that also allows you to log in to its Pod and use curl for testing.

If you have such a Pod (eg: I have hello-world-1–969c488cc-7grzm running in my default namespace), you can log in to it with:

kubectl exec -it <pod name> -n <namespace> -- /bin/bash

If you have followed the installation of my test service, you may need to add DNS utilities:

apt install dnsutils -y

Now you can query the DNS with:

nslookup my-pgsql-cluster-rw-ns-pg.requillion-solutions.com.au 

Again, replace the requillion-solutions.com.au domain with your own. For me this gives:

Server:  10.96.0.10
Address: 10.96.0.10#53

Name: my-pgsql-cluster-rw-ns-pg.requillion-solutions.com.au
Address: 10.104.110.160

If the rewrite is working correctly, you should not see any errors.

Debugging errors

If you have errors, the best place to start is the coreDNS logs. To enable these logs, edit the configuration again with:

kubectl -n kube-system edit configmap/coredns

Add in log here:

apiVersion: v1
data:
  Corefile: |
    .:53 {
        log
        errors 

Now look for the coreDNS pod:

kubectl get pods -n kube-system

Select one of the coreDNS servers and look at the logs:

kubectl logs coredns-5dd5756b68-48pmm  -n kube-system -f

As this is a busy log, you may want to grep for something connected to your rewrite, eg: grep requillion in my case.

The -f selects the follow mode. In another terminal session you can now adjust the configuration and save it. The coreDNS Pod should reload the configuration and you should see this in the log.

[INFO] plugin/reload: Running configuration SHA512 = 096b6d0359815ab...

It can take a minute or two to reload. Alternatively, you can delete both coreDNS Pods.

You can look at the errors to help determine what is wrong. Lines with NXDOMAIN are basically saying that the domain could not be found. If you see something like this, you can see that the search options are being added to the domain:

[INFO] 192.168.183.185:50653 - 8406 "A IN minio.requillion-solutions.cloud.cluster.local. udp 75 false 1232" NXDOMAIN qr,aa,rd 157 0.00038092s

Check for lines that have NOERROR:

[INFO] 192.168.183.178:45062 - 3435 "A IN my-pgsql-cluster-rw-ns-pg.requillion-solutions.cloud. udp 70 false 512" NOERROR qr,aa,rd 138 0.00015082s

Thoughts on namespaces

Namespaces are designed to separate concerns within the Kubernetes cluster. Whilst I have presented a way to use a wildcard certificate for any namespace, you may want to consider creating a wildcard certificate for each namespace.

Summary

In this article we looked at how we can use the coreDNS rewrite configuration to allow us to access internal Services using domains that match our wildcard certificates.

First, we looked at what a wildcard certificate is and how it works. We saw that you can only wildcard one level of domain and that this presents problems when working with internal Kubernetes Services.

We used a regex rewrite rule to split a subdomain into separate subdomains to allow us to map the wildcard compatible domain into the structured domains required by Kubernetes.

Finally, we tested the mapping by accessing a service using the certificate domain.

If you found this article of interest, please give me a clap as that helps me identify what people find useful and what future articles I should write. If you have any suggestions, please add them in the comments section.