Service Mesh Communication Across Kubernetes Clusters
You can secure service-to-service communication across multiple Kubernetes clusters with Consul’s mesh gateway feature. Mesh gateways enable you to secure cross-data center communication that may be sent over the public internet with mTLS.
In this tutorial, you will deploy two Consul datacenters on separate Kubernetes clusters with Consul’s service mesh, WAN federation, and mesh gateways configured. You will then deploy two services into the service mesh, one in each Consul datacenter.
To securely connect the two services, you will configure the service sidecar proxies to route communication through the mesh gateways. Finally, you will test that the services can communicate.
Prerequisites
To complete this tutorial, you will need the following environments provisioned.
- Two Kubernetes clusters running in environments that support external load balancers.
kubectl
installed locally with two contexts, one for each Kubernetes cluster.- Helm 3 installed locally.
Configure Consul
You will use Helm 3 to install Consul on your existing Kubernetes clusters. In the Helm chart, you will need to configure your Consul servers, the Consul datacenter, and the following features:
- Consul service mesh
- mesh gateways
- WAN federation
You will learn how to configure each component section of the Helm values file, and then examine and apply the complete configuration for each data center.
Add The HashiCorp Helm Chart Repository
For this tutorial, Consul is being installed into the Kubernetes default namespace. To install into another namespace, add the -n
flag to the kubectl
and helm
commands.
First, you will need to add the HashiCorp Helm chart.
$ helm repo add hashicorp https://helm.releases.hashicorp.com
"hashicorp" has been added to your repositories
Configure Consul service mesh
By default, all Consul agents will be added to the Consul service mesh and catalog. However, your Kubernetes services will still need sidecar proxies to secure communication.
The first step to enable Consul to automatically add a sidecar proxy to all the service pods is to install the resources necessary on the Kubernetes node. That is achieved by adding the following stanza to the values yaml files that will be applied to your data centers.
connectInject:
enabled: true
Note, you will still need to configure the service to automatically deploy sidecar proxies at deployment time.
Configure mesh gateways
Next, enable a mesh gateway. A mesh gateway is a proxy that provides an accessible IP address that other datacenters can reach. This also resolves issues with pod IP address ranges overlapping between datacenters. This stanza will also need to be included in the values yaml files that will be applied to your data centers.
meshGateway:
enabled: true
Configure WAN federation
WAN federation connects Consul servers from multiple datacenters into the same WAN gossip pool. WAN federation enables services to discover each other across data centers. The WAN federation configuration is slightly different for primary versus secondary datacenters. This yaml snippet shows how to configure the primary data center. Notice the createFederationSecret entry. This should only be set in a primary data center. Later in this tutorial, you will export the secret from the primary data center and inject it into the secondary data center. This will allow the secondary data center to automatically negotiate WAN federation with the primary.
Note mesh gateways also require TLS encryption.
federation:
enabled: true
createFederationSecret: true
tls:
enabled: true
Deploy Consul datacenter dc1
Finish configuring dc1
and the Consul agents. Below is a complete Consul datacenter configuration file, helm-consul-dc1-values.yaml
.
global:
name: consul
datacenter: dc1
federation:
enabled: true
createFederationSecret: true
tls:
enabled: true
server:
replicas: 1
ui:
enabled: true
meshGateway:
enabled: true
replicas: 1
connectInject:
enabled: true
controller:
enabled: true
Your kubectl
context should be connected to the Kubernetes cluster where you are deploying Consul datacenter dc1
.
$ kubectl config use-context dc1
Switched to context "dc1".
Use helm
to install Consul with the hashicorp/consul
chart.
$ helm install -f helm-consul-values.yaml consul hashicorp/consul --wait
This command will wait until Consul is completely installed, which may take a few minutes depending on your environment.
Set mesh gateway mode
Mesh gateways can be configured per data center in one of three modes: local
, remote
, or none
. The official documentation has full details on the implications of running in each mode.
For this tutorial, create a file name proxy-defaults.yaml
that contains a CRD specification that globally configures all proxies to run in local
mode.
apiVersion: consul.hashicorp.com/v1alpha1
kind: ProxyDefaults
metadata:
name: global
spec:
meshGateway:
mode: local
Use kubectl
to apply the CRD.
$ kubectl apply -f proxy-defaults.yaml
proxydefaults.consul.hashicorp.com/global created
Export secrets
You will need to export the federation secret created with the Consul datacenter dc1
to use with Consul datacenter dc2
.
$ kubectl get secret consul-federation -o yaml > consul-federation-secret.yaml
Deploy Consul datacenter dc2
Now that you have deployed the Consul datacenter dc1
, you can configure and deploy dc2
. Connect your kubectl
context to dc2
.
$ kubectl config use-context dc2
Switched to context "dc2".
Create the federation secret in dc2
.
$ kubectl apply -f consul-federation-secret.yaml
secret/consul-federation created
Now that you have prepared your Kubernetes cluster, finish configuring dc2
and the Consul agents. The dc2
datacenter will need the following additional options configured:
caCert
- contains the certificate of the CA to use for TLS communication, which is the Consul federation secret exported from Consul datacenterdc1
.caKey
- contains the private key of the CA to use for TLS communication, which is the Consul federation secret exported from Consul datacenterdc1
.server
config - contains the server information from Consul datacenterdc1
necessary to configure federation.
Create the customized chart helm-consul-dc2-values.yaml
.
global:
name: consul
datacenter: dc2
tls:
enabled: true
caCert:
secretName: consul-federation
secretKey: caCert
caKey:
secretName: consul-federation
secretKey: caKey
federation:
enabled: true
server:
replicas: 1
extraVolumes:
- type: secret
name: consul-federation
items:
- key: serverConfigJSON
path: config.json
load: true
connectInject:
enabled: true
controller:
enabled: true
meshGateway:
enabled: true
replicas: 1
Finally, use helm
to install Consul.
$ helm install -f helm-consul-dc2-values.yaml hashicorp hashicorp/consul --wait
Set mesh gateway mode
In dc2
, you must also deploy a ProxyDefaults
CRD that sets the mesh gateway mode. Use kubectl
and the proxy-defaults.yaml
the file you created earlier to apply the CRD to dc2
.
$ kubectl apply -f proxy-defaults.yaml
proxydefaults.consul.hashicorp.com/global created
Verify the datacenters are connected
To verify that the Consul datacenters are connected and WAN federated, you can use kubectl
to execute a Consul CLI command to query for a list of servers in the WAN gossip pool. All the servers, from both data centers, should be listed.
$ kubectl exec statefulset/consul-server -- consul members -wan
Node Address Status Type Build Protocol DC Segment
consul-server-0.dc1 10.0.5.162:8302 alive server 1.9.4 2 dc1 <all>
consul-server-0.dc2 10.0.6.93:8302 alive server 1.9.4 2 dc2 <all>
Deploy microservices
Now that you have two connected Consul datacenters, you can deploy a service in each using kubectl
.
Deploy the static-client
service
The static-client
service in this tutorial represents a frontend service, for example, a website.
Change contexts to communicate with Consul datacenter dc1
.
$ kubectl config use-context dc1
Switched to context "dc1".
The service definition includes two Consul specific annotations:
"consul.hashicorp.com/connect-inject": "true"
- ensures that the service is deployed into the Consul service mesh with a sidecar proxy and automatically registered in the Consul catalog."consul.hashicorp.com/connect-service-upstreams": "static-server:1234:dc2"
- explicitly declares that the upstream service is thestatic-server
service. Note that the annotation value contains a segment that matches the upstream global datacenter name configuration entry.
First, create a yaml file, static-client.yaml
, to define the static-client
service.
apiVersion: v1
kind: ServiceAccount
metadata:
name: static-client
---
apiVersion: v1
kind: Service
metadata:
name: static-client
spec:
selector:
app: static-client
ports:
- port: 4321
targetPort: 4321
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: static-client
name: static-client
spec:
replicas: 1
selector:
matchLabels:
app: static-client
template:
metadata:
annotations:
consul.hashicorp.com/connect-inject: 'true'
consul.hashicorp.com/connect-service-upstreams: 'static-server:1234:dc2'
labels:
app: static-client
spec:
containers:
- name: static-client
image: tutum/curl:latest
ports:
- containerPort: 4321
command: ['/bin/sh', '-c', '--']
args: ['while true; do sleep 30; done;']
The "consul.hashicorp.com/connect-inject": "true"
annotation causes Consul to deploy a sidecar proxy alongside the static-client
service. The sidecar proxy can both accept and establish connections using Consul.
Now, deploy the static-client
service into the Consul datacenter dc1
.
$ kubectl apply -f static-client.yaml
serviceaccount/static-client created
service/static-client created
deployment.apps/static-client created
Use kubectl
to check that the pods were deployed and are running successfully.
$ kubectl get deploy | grep static-client
kubectl get deploy | grep static-client
Deploy the static-server
service
The static-server
service in this tutorial represents a backend service, for example, a database.
Change contexts to communicate with Consul datacenter dc2
.
$ kubectl config use-context dc2
Switched to context "dc2".
First, create a yaml file, static-server.yaml
, to define the static-server
service. Note, the static-server
service also includes the consul.hashicorp.com/connect-inject
annotation.
apiVersion: v1
kind: ServiceAccount
metadata:
name: static-server
---
apiVersion: v1
kind: Service
metadata:
name: static-server
spec:
selector:
app: static-server
ports:
- port: 1234
targetPort: 8080
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: static-server
name: static-server
spec:
replicas: 1
selector:
matchLabels:
app: static-server
template:
metadata:
annotations:
consul.hashicorp.com/connect-inject: 'true'
labels:
app: static-server
spec:
containers:
- name: static-server
image: hashicorp/http-echo:latest
args:
- -text="hello world"
- -listen=:8080
ports:
- containerPort: 8080
name: http
Now, deploy the static-server
service.
$ kubectl apply -f static-server.yaml
serviceaccount/static-server created
service/static-server created
deployment.apps/static-server created
Use kubectl
to check that the pods were deployed and are running successfully.
$ kubectl get deploy | grep static-server
NAME READY UP-TO-DATE AVAILABLE AGE
static-server 1/1 1 1 2m59s
Discover services across Kubernetes clusters
Servers participate in WAN gossip to share membership information, which allows servers to perform cross-data center requests. To secure these requests, the WAN gossip is sent through the mesh gateways which encrypt the communication with mTLS. The requests include service queries.
To discover services across your Kubernetes clusters, you can use the Consul UI or CLI to query the available services.
Consul CLIConsul UI
First, connect to one of the servers in the Consul datacenter dc2
.
$ kubectl config use-context dc2
Switched to context "dc2"
Use kubectl
to execute a Consul CLI command that retrieves a list of all services in the other data center, dc1
. The output will confirm that you can request service information from the WAN-connected data center.
$ kubectl exec statefulset/consul-server -- consul catalog services -datacenter dc1
consul
mesh-gateway
static-client
static-client-sidecar-proxy
Route service communication across Kubernetes clusters
Finally, verify that communication is being routed through the mesh gateways. Use curl
to verify that the server dc2
can retrieve data from the client in dc1
.
Connect to the Kubernetes cluster where Consul datacenter dc1
is running.
$ kubectl config use-context dc1
Switched to context "dc1".
Using kubectl
to connect to the client and request data from the server.
$ kubectl exec deploy/static-client -c static-client -- curl -sS http://localhost:1234
“hello world”
👋 Join us today !!
If this post was helpful, please click the clap 👏 button below a few times to show your support! ⬇