HA Consul Cluster on GCP stateful MIG | Service discovery by DNS without running consul clients on all vm’s

Architecture of Consul Cluster in HA mode.

Introduction

The goal of this article is to deploy a stateful consul cluster in High Availibility mode, Which will retain the cluster information like cluster configuration, service discovery and KV stores using gcp stateful mig even in case of node failure/recreation.

Service discovery will be available by DNS to all VMs in GCP Network, mitigating the requirement to deploy Consul client on every VM. Instead resolution of the <dc>.consul domain will be via VM metadata and Cloud DNS forwarding.

Running Consul in a stateful managed instance group will allow for self healing in the event of VM failure, with the stateful config and KV data retained on VM recreation.

Terraform Code repository to setup consul cluster

GitHub - yashwantmahawar/ha_consul_cluster_on_gcp_stateful_mig…

List of required GCP resources to setup the cluster

1. Service Account
2. Stateful MIG + Instance Template
3. TCP/UDP (L4 Internal Load Balancer + Backends)
4. Private DNS Forwarding
5. Firewall Rules

Service Account

A specific service account is required for the Consul server infrastructure. This provides sufficient privileges for the instances to make required API calls for the cloud auto join process. Additional permissions can be added to allow upload of Consul backups to a GCS bucket in the same project.

Stateful MIG

The Stateful MIG provides an operationally robust manner to manage the VMs, which includes the ability to health check VMs and recover failed hosts in an automated manner to maintain the integrity of the cluster.

The Stateful MIG will retain the stateful resources during an instance update; the Consul data disk will be treated as stateful to preserve configuration, service discovery and KV store. The boot disk is treated as stateless allowing for OS and application upgrades to be rolled out whilst retaining the existing data.

Internal Load Balancer

Two ILBs are deployed to provide a backend to provide both TCP and UDP access to the Consul Cluster. Because DNS requests can be either TCP or UDP we required a shared forwarding rule to be defined. The backend service is the Stateful MIG. Consul client register request will be TCP.

To remove the need to specify every Consul VM IP in the Cloud DNS forwarding configuration DNS requests are sent via the ILB that fronts the Stateful MIG. These are then forwarded to the Consul VMs for resolution

Startup Script / Required VM Configuration

The local VMs will be configured by startup script to perform the actions listed below

1. Attach the data disk (mount the consul data persistent disk)

#bind disk
DISK_ID=/dev/disk/by-id/google-data-disk
MNT_DIR=/var/lib/consul
mkdir -p $MNT_DIR
if [[ $(lsblk $DISK_ID -no fstype) != 'ext4' ]]; then
          sudo mkfs.ext4 -m 0 -F -E lazy_itable_init=0,lazy_journal_init=0,discard $DISK_ID
else
    sudo e2fsck -fp $DISK_ID
    sudo resize2fs $DISK_ID
fi
if [[ ! $(grep -qs "$MNT_DIR " /proc/mounts) ]]; then
    if [[ ! $(grep -qs "$MNT_DIR " /etc/fstab) ]]; then
        UUID=$(blkid -s UUID -o value $DISK_ID)
        echo "UUID=$UUID $MNT_DIR ext4 discard,defaults,nofail 0 2" | sudo tee -a /etc/fstab
    fi
    systemctl daemon-reload
    sudo mount $MNT_DIR
fi

2. Configuration files for consul (update default.config)

{
    "bind_addr": "0.0.0.0",
    "client_addr": "0.0.0.0",
    "server": true,
    "ui": true,
    "datacenter": "dc1",
    "data_dir": "/var/lib/consul",
    "encrypt": "encryption_key",
    "log_level": "INFO",
    "enable_syslog": true,
    "leave_on_terminate": true,
    "bootstrap_expect" : 3,
    "retry_join": [ "provider=gce tag_value=consul-cluser-01 zone_pattern=asia-east1-.*" ]
}

3. DNS forwarding (listen for requests on port 53 and forward to Consul on port 8600).
Any tool can be used to perform port forwarding like iptables, here we are using the nftables.

echo '''
table ip dnsnat {
        chain prerouting {
                type nat hook prerouting priority dstnat; policy accept;
                udp dport 53 redirect to :8600
                tcp dport 53 redirect to :8600
        }
}''' | tee sudo /etc/nftables.conf

4. Cloud DNS response (setup configuration to write source for DNS)

Why to masq cloud DNS response: If DNS clients receive a response to a query from an IP address other than the one which they sent the request to, the request is dropped. This behaviour can be observed by using dig against the Consul ILB.

Therefore in order to ensure that the response packets are returned with the correct IP address they need to be rewritten. We cannot rewrite all DNS packets to the ILB as there may be legitimate requests to the Consul servers directly. However we know that Cloud DNS forwards requests from the IP range 35.199.192.0/19 which means we can construct a specific translation rule for packets returning to that range.

sudo nft list ruleset
sudo nft add table clouddns
sudo nft add chain clouddns postrouting {type filter hook postrouting priority 300\;}
sudo nft add rule clouddns postrouting ip saddr \
  $(curl -s -HMetadata-Flavor:Google metadata/computeMetadata/v1/instance/network-interfaces/0/ip) \
  udp sport 53 ip daddr 35.199.192.0/19 ip saddr set ${dns_static_ip}

Private DNS Forwarding

In a typical Consul service mesh setup every VM has a local Consul agent which can be used to perform service discovery using HTTP or gRPC. However when dealing with a heterogeneous environment DNS is often the lowest common denominator for service discovery.

In order to support DNS resolution we can leverage Cloud DNS forwarding, which will then allow us to expose services registered with Consul via DNS.

Firewall Rules

The following firewall rules are required in order to secure access to the Consul environment to expected traffic. This is primarily inter-cluster communication, client-server and external DNS requests

Conclusion

1. Above consul cluster setup is ready to face any node failure without impacting cluster performace.
2. Service discovery using dns without install consul client.
3. We can easily scale-in or scale-out consul cluster nodes.
4. Cluster data will persist on disk in case of node failure (node recreation).
5. Able to do service discovery via dns without updating resolve.conf file