Kubernetes multi-cluster networking with Cilium and kops
Why use multiple clusters
At Sportradar we prefer multiple single-tenant Kubernetes clusters over a single multi-tenant cluster. Sportradar has a distributed development organisation and having multiple clusters makes it easier to ensure there are no global locks on cluster operations. Our current strategy is to have one business/service unit have their own cluster that they use for all their applications.
Since there is a huge amount of data exchange between units, the challenge with having multiple clusters quickly appears. For many use cases, exposing services out of the cluster in common ways like
LoadBalancer works just fine. But there are also many cases where you gain very little from the additional hops the traffic takes compared to the overhead. Some services also expose their data as Kafka topics for quick processes, which cannot be exposed out of the cluster using any of these mechanisms.
As we felt exposing services between clusters should be easier than a multi-tenant cluster, we began looking at various ways connecting pods in different clusters directly. This isn’t more magical than ensuring routes exists between various cluster networks. The question is just how to configure this in the easiest way possible given that we would have a lot of cross-cluster communication.
The most common way seems to be using BGP in some form. This shouldn’t really be a surprise to anyone.Both Kube-router and Calico CNI base their cluster networking on BGP. And especially the former has pretty good support for configuring BGP peers to networks outside of a cluster. But even though BGP isn’t that hard, it is still far from as intuitive as we’d expect in the Kubernetes world. And the amount of configuration per cluster is substantial as well.
As luck has it, Cilium announced the ClusterMesh feature in 1.2 just before we were about the get our hands dirty with one of the other CNIs. Looking at the documentation for ClusterMesh this was in a whole different ballpark compared to some of the examples I have seen with other CNIs. We had no doubt this was the option for us.
One thing that is worth mentioning though: Cilium allows you to set up mesh between clusters, but it cannot talk to any other networks. For that you need to combine Cilium with something using BGP.
Setting up the mesh
The Cilium documentation gives a rough overview of the process, but as we configured almost everything from scratch, I thought I would share how we set up two clusters and connected them together.
Creating a couple of Kubernetes clusters
We use kops for operating our Kubernetes clusters. For the clusters using cilium we have a shared subnet with shared routing tables. This just makes it easier to manage routes to other VPCs and to our on-premise sites. For our clusterMesh, this means we don’t have to do much configuration to get node connectivity. All we do is create a Security Group that allows nodes to talk to other members of the security group. The kops documentation details how to configure such a setup.
I won’t go into any details about how to use kops here. Just the key elements of a cluster configuration necessary to get ClusterMesh working. I prefer creating a cluster configuration file and then add/edit the changes necessary.
As Cilium requires etcd3, ensure that you have explicitly set an etcd3 version as follows:
Naturally, we cannot have conflicting routes, so ensure that the internal cluster network CIDRs are not overlapping. You can just split the default
100.64.0.0/10into multiple CIDRs of whatever size you find appropriate. Also add the
networking part stating we want to use Cilium and that we want version 1.2.2:
nonMasqueradeCIDR: 100.<63 + cluster-id>.0.0/16
For every instance group, ensure you are using a Stretch image (or CoreOS if you prefer):
Create a couple of cluster configuration files with the changes above and then run
kops create -f <cluster>.yaml for each of them.
Once you have a couple of clusters up, you need to upgrade the Cilium
DaemonSet and RBAC resources. The
DaemonSet spec currently used by kops does not set the
cluster-name configuration, nor does it mount the ClusterMesh configuration.
kubectl apply -f https://raw.githubusercontent.com/cilium/cilium/HEAD/examples/kubernetes/1.10/cilium-rbac.yaml
kubectl apply -f https://raw.githubusercontent.com/cilium/cilium/HEAD/examples/kubernetes/1.10/cilium-ds.yaml
kubectl set image daemonset/cilium -n kube-system cilium-agent=docker.io/cilium/cilium:v1.2.2
Ensure you are using at least version 1.2.2 as it contains some essential bugfixes required to get ClusterMesh working.
Configuring the Cilium agents
Next up is adding the
cluster-name to each of the clusters. On all your clusters run
kubectl -n kube-system edit cm cilium-config and add the following:
Ensure that the
<id> is unique across the clusters.
While editing the files above, also take note of the
etcd-cluster literal. Create a file with same name as
cluster-name specified above containing the etcd-cluster literal. Then create a secret containing these etcd configs:
kubectl create secret generic cilium-cluster-mesh --from-file=./cluster-name1 --from-file=./cluster-name2 ... --from-file=./cluster-nameN
The Cilium agents will recognize and ignore references to its local etcd-cluster, so it is perfectly safe to run the identical command across all clusters.
After creating the secrets, restart the Cilium agent on all clusters:
kubectl -n kube-system delete pods -l k8s-app=cilium
Hopefully you should be able to see remote nodes coming in when running
kubectl -n kube-system exec -ti cilium-12345 cilium node list It should look something like this:
Name IPv4 Address Endpoint CIDR
cluster1/ip-10-0-53-242 10.0.53.242 100.64.129.0/24
cluster1/ip-10-0-53-247 10.0.53.247 100.64.131.0/24
cluster1/ip-10-0-54-163 10.0.54.163 100.64.128.0/24
cluster1/ip-10-0-54-169 10.0.54.169 100.64.133.0/24
cluster1/ip-10-0-55-61 10.0.55.61 100.64.130.0/24
cluster1/ip-10-0-55-86 10.0.55.86 100.64.132.0/24
cluster2/ip-10-0-53-161 10.0.53.161 100.65.133.0/24
cluster2/ip-10-0-53-209 10.0.53.209 100.65.128.0/24
cluster2/ip-10-0-54-229 10.0.54.229 100.65.131.0/24
cluster2/ip-10-0-54-57 10.0.54.57 100.65.129.0/24
cluster2/ip-10-0-55-100 10.0.55.100 100.65.130.0/24
cluster2/ip-10-0-55-111 10.0.55.111 100.65.132.0/24
You can also see connectivity status by running
kubectl -n kube-system exec -ti cilium-12345 cilium status
And of course, try something like running
curl or something against a remote pod.