Adventures in Kubernetes

“For Want of a Nail”

I have a — let’s not go there — need to alias a Kubernetes Service to a fully-qualified domain name. The hack to get this done took me on a learning opportunity with CoreDNS. CoreDNS is an alternative to kube-dns (link).


I need to temporarily alias [release-name]-[chart-name]-orderer to to progress (resolve!?) a Helm Chart for Hyperledger Fabric on Kubernetes.

One Solution?

Success is defined to be that, when Pods call, the name is resolved to the aforementioned Service. It is possible to configure the cluster’s kube-dns with so-called “stub domains” (configured with a ConfigMap) such that, DNS requests for (in this case) * are resolved by a DNS service that I control. One based on CoreDNS.


Let’s test it locally.

I’m going to retain CoreDNS’s example of (dot-org) as a known-working configuration and will add my (dot-com):

NB The magic happens in line #13 where orderer is aliased to [[RELEASE-NAME]]-[[CHART-NAME]]-orderer. Do not forget the terminating “.”. I will replace RELEASE-NAME and CHART-NAME with actual values.

We need a configuration file for CoreDNS too. By default this is called Corefile:

NB this defines 2 DNS domains ( and Both comprise a pipeline of plugins beginning with file. The value after file is a parameter and references the above e.g. file.

Let’s run the Docker image:

docker run \
--interactive \
--tty \
--publish=1053:1053/tcp \
--publish=1053:1053/udp \
--volume=$PWD:/tmp \
coredns/coredns \
-conf /tmp/Corefile \
-dns.port 1053
NB The double-publish on port 1053 is because DNS uses both TCP and UDP. I’m assuming the above files are in the current (${PWD}) directory. This directory is mapped to /tmp in the container. The container will find Corefile in this directory and it references and that should be in the same (/tmp) directory.

This should output something similar to:
2018/08/22 16:48:32 [INFO] CoreDNS-1.2.0
2018/08/22 16:48:32 [INFO] linux/amd64, go1.10.3, 2e322f6
linux/amd64, go1.10.3, 2e322f6

From another terminal try:

nslookup -port 1053 localhost

And this should produce:

Server:  localhost
Address: ::1#1053 canonical name = x-hyperledger-fabric-orderer.
** server can't find x-hyperledger-fabric-orderer: REFUSED
NB This is correctly wrong ;-) CoreDNS has aliased to x-hyperledger-fabric-orderer. That’s correct. There is currently nothing on that endpoint, so we receive a “can’t find” error (correctly).

CoreDNS’ documentation uses dig instead of nslookup…. your preference:

dig @localhost -p 1053 a

Container-Optimized OS (COS)

If you have a container and you’d like to run it on Google Cloud Platform, Kubernetes is a good, default choice. In this case, I’m going to run CoreDNS on a COS instance in the same project as my Kubernetes cluster.

To make life straightforward, after provisioning a COS instance, I’m going to run the CoreDNS command manually. COS provides a read-writable filesystem on /tmp which is why I chose this directory previously.

Before running the container, let’s copy the CoreDNS configuration files to the instance:

for FILE in Corefile
gcloud compute scp \
${FILE} \
${INSTANCE}:/tmp \
NB COS uses systemd-resolved and COS defaults to DNSStubListener=udp . To run CoreDNS you’ll need to disable this which you can do (temporarily) with sudo systemctl stop systemd-resolved.

Then gcloud compute ssh and run the Docker command as before but (1) change the local directory to /tmp and (2) change the DNS port to its default (53):

docker run \
--interactive \
--tty \
--publish=53:53/tcp \
--publish=53:53/udp \
--volume=/tmp:/tmp \
coredns/coredns \
-conf /tmp/Corefile \
-dns.port 53
Issue #1: I would prefer to not open firewall ports unncessarily. However, I’ve been unable to connect to the CoreDNS instance from the Kubernetes cluster using its internal IP. Investigating. The CoreDNS’ instance’s internal IP is available:
gcloud compute instances describe ${INSTANCE} \
--project=${PROJECT} \
) && echo ${DNS}

The public IP (which works) is available:

gcloud compute instances describe ${INSTANCE} \
--project=${PROJECT} \
) && echo ${DNS}

For the public IP, you’ll need to punch a hole in the firewall. I’ll leave this to your discretion; this opens the instance’s ports to the internet:

gcloud compute instances add-tags ${INSTANCE} \
--tags=coredns \
gcloud compute firewall-rules create temp-test-coredns \
--action=ALLOW \
--rules=tcp:53,udp:53 \
--target-tags=coredns \

And then, from another instance in the cluster:

nslookup ${DNS}
Address: canonical name = x-hyperledger-fabric-orderer.

Or, in my case, because I’m running from another COS instance:

docker run \
--net=host \
busybox \
nslookup ${DNS}
Address: canonical name = x-hyperledger-fabric-orderer
*** Can't find No answer
NB busybox’s nslookup has slightly different behavior. It reports the correct alias but confirms it’s unable to find (rather than x-hyperledger-fabric-orderer).


All working well to this point, let’s revise Kubernetes’ kube-dns configuration by applying a ConfigMap that defines the DNS server:

NB You must replace ${DNS} with its value. The result must be enclosed in quotes. The manifest explicitly references the kube-system namespace.


kubectl apply --filename=kube-dns.yaml
Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply
configmap/kube-dns configured
NB Ignore the “Warning”.
Kubernetes Console showing system objects

And, let’s test it!


Issues #2: My attempt to resolve to the Kubernetes Service name is not working. I do not know why. Investigating.

This (CNAME) does not work:

Instead, I determined the Service’s IP address and reconfigured the CoreDNS file to A(lias) to that:

kubectl get services \
--selector=component=orderer \
--namespace=${NAMESPACE} \
) && echo ${ORDERER}

And then:

NB You must replace ${ORDERER} with its value.

Now, from a Pod within the cluster, nslookup will resolve

kubectl run test \
--namespace=${NAMESPACE} \
--image=busybox \
--stdin \
If you don't see a command prompt, try pressing enter.
/ # nslookup
Address 1: x-hyperledger-fabric-orderer.ytterbium.svc.cluster.local
/ #


Outstanding Issues

  • Why am I unable to use internal IPs?
  • How (!) can I write the to resolve to CNAME (or SRV?)?


It’s (much easier than I made it look) to run your own DNS service using CoreDNS.

If you wish to complement a Kubernetes cluster’s kube-dns with a DNS service that you control that’s easy too.

While I achieved my goal and and am now able to test my Fabric on Helm deployment with the Pods able to resolve to Kubernetes Service names (and prove a debugging point), the journey was longer than I would have liked :-)

That’s all!