CoreDNS
Adventures in Kubernetes
“For Want of a Nail”
I have a — let’s not go there — need to alias a Kubernetes Service to a fully-qualified domain name. The hack to get this done took me on a learning opportunity with CoreDNS. CoreDNS is an alternative to kube-dns (link).
Problem
I need to temporarily alias [release-name]-[chart-name]-orderer
to orderer.example.com
to progress (resolve!?) a Helm Chart for Hyperledger Fabric on Kubernetes.
One Solution?
Success is defined to be that, when Pods call orderer.example.com
, the name is resolved to the aforementioned Service. It is possible to configure the cluster’s kube-dns with so-called “stub domains” (configured with a ConfigMap) such that, DNS requests for (in this case) *.example.com
are resolved by a DNS service that I control. One based on CoreDNS.
CoreDNS
Let’s test it locally.
I’m going to retain CoreDNS’s example of example.org
(dot-org) as a known-working configuration and will add my ownexample.com
(dot-com):
NB The magic happens in line #13 where
orderer
is aliased to[[RELEASE-NAME]]-[[CHART-NAME]]-orderer.
Do not forget the terminating “.”. I will replaceRELEASE-NAME
andCHART-NAME
with actual values.
We need a configuration file for CoreDNS too. By default this is called Corefile
:
NB this defines 2 DNS domains (
example.com
andexample.org
). Both comprise a pipeline of plugins beginning withfile
. The value afterfile
is a parameter and references the above e.g.example.com
file.
Let’s run the Docker image:
docker run \
--interactive \
--tty \
--publish=1053:1053/tcp \
--publish=1053:1053/udp \
--volume=$PWD:/tmp \
coredns/coredns \
-conf /tmp/Corefile \
-dns.port 1053
NB The double-publish on port
1053
is because DNS uses both TCP and UDP. I’m assuming the above files are in the current (${PWD}
) directory. This directory is mapped to/tmp
in the container. The container will findCorefile
in this directory and it referencesexample.org
andexample.com
that should be in the same (/tmp
) directory.
This should output something similar to:
example.com.:1053
example.org.:1053
2018/08/22 16:48:32 [INFO] CoreDNS-1.2.0
2018/08/22 16:48:32 [INFO] linux/amd64, go1.10.3, 2e322f6
CoreDNS-1.2.0
linux/amd64, go1.10.3, 2e322f6
From another terminal try:
nslookup orderer.example.com -port 1053 localhost
And this should produce:
Server: localhost
Address: ::1#1053orderer.example.com canonical name = x-hyperledger-fabric-orderer.
** server can't find x-hyperledger-fabric-orderer: REFUSED
NB This is correctly wrong ;-) CoreDNS has aliased
orderer.example.com
tox-hyperledger-fabric-orderer
. That’s correct. There is currently nothing on that endpoint, so we receive a “can’t find” error (correctly).
CoreDNS’ documentation uses dig instead of nslookup…. your preference:
dig @localhost -p 1053 a orderer.example.com
Container-Optimized OS (COS)
If you have a container and you’d like to run it on Google Cloud Platform, Kubernetes is a good, default choice. In this case, I’m going to run CoreDNS on a COS instance in the same project as my Kubernetes cluster.
To make life straightforward, after provisioning a COS instance, I’m going to run the CoreDNS command manually. COS provides a read-writable filesystem on /tmp
which is why I chose this directory previously.
Before running the container, let’s copy the CoreDNS configuration files to the instance:
PROJECT=[[YOUR-PROJECT]]
INSTANCE=[[YOUR-INSTANCE]]for FILE in Corefile example.org example.com
do
gcloud compute scp \
${FILE} \
${INSTANCE}:/tmp \
--project=${PROJECT}
done
NB COS uses systemd-resolved and COS defaults to
DNSStubListener=udp
. To run CoreDNS you’ll need to disable this which you can do (temporarily) withsudo systemctl stop systemd-resolved
.
Then gcloud compute ssh
and run the Docker command as before but (1) change the local directory to /tmp
and (2) change the DNS port to its default (53
):
docker run \
--interactive \
--tty \
--publish=53:53/tcp \
--publish=53:53/udp \
--volume=/tmp:/tmp \
coredns/coredns \
-conf /tmp/Corefile \
-dns.port 53
Issue #1: I would prefer to not open firewall ports unncessarily. However, I’ve been unable to connect to the CoreDNS instance from the Kubernetes cluster using its internal IP. Investigating. The CoreDNS’ instance’s internal IP is available:
DNS=$(\
gcloud compute instances describe ${INSTANCE} \
--project=${PROJECT} \
--format="value(networkInterfaces[0].networkIP)"\
) && echo ${DNS}
The public IP (which works) is available:
DNS=$(\
gcloud compute instances describe ${INSTANCE} \
--project=${PROJECT} \
--format="value(networkInterfaces[0].accessConfigs[0].natIP)"\
) && echo ${DNS}
For the public IP, you’ll need to punch a hole in the firewall. I’ll leave this to your discretion; this opens the instance’s ports to the internet:
gcloud compute instances add-tags ${INSTANCE} \
--tags=coredns \
--project=${PROJECT}gcloud compute firewall-rules create temp-test-coredns \
--action=ALLOW \
--rules=tcp:53,udp:53 \
--target-tags=coredns \
--project=${PROJECT}
And then, from another instance in the cluster:
nslookup orderer.example.com ${DNS}Server: 10.138.0.5
Address: 10.138.0.5#53orderer.example.com canonical name = x-hyperledger-fabric-orderer.
Or, in my case, because I’m running from another COS instance:
docker run \
--net=host \
busybox \
nslookup orderer.example.com ${DNS}Server: 10.138.0.5
Address: 10.138.0.5:53orderer.example.com canonical name = x-hyperledger-fabric-orderer*** Can't find orderer.example.com: No answer
NB busybox’s nslookup has slightly different behavior. It reports the correct alias but confirms it’s unable to find
orderer.example.com
(rather thanx-hyperledger-fabric-orderer
).
Kubernetes
All working well to this point, let’s revise Kubernetes’ kube-dns configuration by applying a ConfigMap that defines the DNS server:
NB You must replace
${DNS}
with its value. The result must be enclosed in quotes. The manifest explicitly references thekube-system
namespace.
Then:
kubectl apply --filename=kube-dns.yaml
Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply
configmap/kube-dns configured
NB Ignore the “Warning”.
And, let’s test it!
Testing
Issues #2: My attempt to resolve
orderer.example.com
to the Kubernetes Service name is not working. I do not know why. Investigating.
This (CNAME
) does not work:
Instead, I determined the Service’s IP address and reconfigured the CoreDNS example.com
file to A
(lias) to that:
ORDERER=$(\
kubectl get services \
--selector=component=orderer \
--namespace=${NAMESPACE} \
--output=jsonpath="{.items[0].spec.clusterIP}"\
) && echo ${ORDERER}
And then:
NB You must replace
${ORDERER}
with its value.
Now, from a Pod within the cluster, nslookup
will resolve orderer.example.com
:
kubectl run test \
--namespace=${NAMESPACE} \
--image=busybox \
--stdin \
--tty
If you don't see a command prompt, try pressing enter.
/ # nslookup orderer.example.com
Name: orderer.example.com
Address 1: 10.27.253.203 x-hyperledger-fabric-orderer.ytterbium.svc.cluster.local
/ #
Bam!
Outstanding Issues
- Why am I unable to use internal IPs?
- How (!) can I write the example.com to resolve to
CNAME
(orSRV
?)?
Conclusion
It’s (much easier than I made it look) to run your own DNS service using CoreDNS.
If you wish to complement a Kubernetes cluster’s kube-dns with a DNS service that you control that’s easy too.
While I achieved my goal and and am now able to test my Fabric on Helm deployment with the Pods able to resolve orderer.example.com
to Kubernetes Service names (and prove a debugging point), the journey was longer than I would have liked :-)
That’s all!