Set up a Highly Available Kubernetes cluser using HAproxy and Keepalived the kubeadm way

Container Orchestration becomes a must when it comes to manage a significant load of containers. Kubernetes is the standard container orchestrator now in the market. In this tutorial, I will go through the steps of setting up a highly available kubernetes cluster.

A high availability cluster consists of multiple control plane nodes (master nodes), multiple worker nodes and multiple load balancing nodes and an external ETCD cluster.

The way we chose to achieve high availability of a Kubernetes cluster is explained below:

• Two Control plane nodes: If the main master node is down, the secondary master node can take over without impacting the functioning of the cluster.
• Two Worker nodes: To achieve the high availability of the application as it will be deployed as multiple instances across these two workers.
• Two HAProxy servers: It provides high performance load balancing, we will have two servers that hosts HAProxy to achieve its high availability.
• Two Keepalived servers: It provides a VRPP implementation and allows you to configure Linux machines for load balancing, preventing single points of failure.
• Virtual IP Address (VIP) : It is af loating IP address, that means in the event of node failures, the IP address can be passed between nodes allowing for failover.

The number of servers we need to achieve this architecture is 9 servers: 2 masters, 2 workers, 2 haproxy servers, 3 etcd members.

Set up HA and secure etcd cluster with TLS encryption

In this part, we are going to create an etcd cluster with three members and uses TLS certificates to encrypt the traffic between members (peer communication) and the traffic coming from a client.

We’ll start be Generating TLS certificates on our local workstation

1. Download required binaries

We use cfssl self-signed certificate as our CA root certificate

{
 wget -q - show-progress \
 https://storage.googleapis.com/kubernetes-the-hard-way/cfssl/1.4.1/linux/cfssl \
 https://storage.googleapis.com/kubernetes-the-hard-way/cfssl/1.4.1/linux/cfssljson
 
 chmod +x cfssl cfssljson
 sudo mv cfssl cfssljson /usr/local/bin/
}

2. Create a Certificate Authority (CA)

We then use this CA to create other TLS certificates

{

cat > ca-config.json <<EOF
{
    "signing": {
        "default": {
            "expiry": "8760h"
        },
        "profiles": {
            "etcd": {
                "expiry": "8760h",
                "usages": ["signing","key encipherment","server auth","client auth"]
            }
        }
    }
}
EOF

cat > ca-csr.json <<EOF
{
  "CN": "etcd cluster",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "MR",
      "L": "Morocco",
      "O": "Kubernetes",
      "OU": "ETCD-CA",
      "ST": "Casablanca"
    }
  ]
}
EOF

cfssl gencert -initca ca-csr.json | cfssljson -bare ca

}

3. Create TLS certificates

{

ETCD1_IP="<etcd1_ip_address>"
ETCD2_IP="<etcd2_ip_address>"
ETCD3_IP="<etcd3_ip_address>"

cat > etcd-csr.json <<EOF
{
  "CN": "etcd",
  "hosts": [
    "localhost",
    "127.0.0.1",
    "${ETCD1_IP}",
    "${ETCD2_IP}",
bash    "${ETCD3_IP}"
    
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "MR",
      "L": "Morocco",
      "O": "Kubernetes",
      "OU": "etcd",
      "ST": "Casablanca"
    }
  ]
}
EOF

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=etcd etcd-csr.json | cfssljson -bare etcd

}

4. Copy the certificates to etcd nodes

{
declare -a NODES=(<etcd1_ip_address> <etcd2_ip_address> <etcd3_ip_address>)
for node in ${NODES[@]}; do
 scp ca.pem etcd.pem etcd-key.pem sysadmin@$node: 
done
}

Now that we’ve copied the certificates to all etcd nodes, Let’s move on to configure etcd members.

1. Copy the certificates to a standard location

Perform all commands logged in as root user or prefix each command with sudo as appropriate

{
  mkdir -p /etc/etcd/pki
  mv ca.pem etcd.pem etcd-key.pem /etc/etcd/pki/ 
}

2. Download etcd & etcdctl binaries from Github

{
  ETCD_VER=v3.5.1
  wget -q --show-progress "https://github.com/etcd-io/etcd/releases/download/${ETCD_VER}/etcd-${ETCD_VER}-linux-amd64.tar.gz"
  tar zxf etcd-v3.5.1-linux-amd64.tar.gz
  mv etcd-v3.5.1-linux-amd64/etcd* /usr/local/bin/
  rm -rf etcd*
}

3. Create systemd unit file for etcd service

We are going to set NODE_IP to the correct IP of the machine where we are running this

{

NODE_IP="<etcd1_ip_address>"

ETCD_NAME=$(hostname -s)

ETCD1_IP="<etcd1_ip_address>"
ETCD2_IP="<etcd2_ip_address>"
ETCD3_IP="<etcd3_ip_address>"


cat <<EOF >/etc/systemd/system/etcd.service
[Unit]
Description=etcd

[Service]
Type=notify
ExecStart=/usr/local/bin/etcd \\
  --name ${ETCD_NAME} \\
  --cert-file=/etc/etcd/pki/etcd.pem \\
  --key-file=/etc/etcd/pki/etcd-key.pem \\
  --peer-cert-file=/etc/etcd/pki/etcd.pem \\
  --peer-key-file=/etc/etcd/pki/etcd-key.pem \\
  --trusted-ca-file=/etc/etcd/pki/ca.pem \\
  --peer-trusted-ca-file=/etc/etcd/pki/ca.pem \\
  --peer-client-cert-auth \\
  --client-cert-auth \\
  --initial-advertise-peer-urls https://${NODE_IP}:2380 \\
  --listen-peer-urls https://${NODE_IP}:2380 \\
  --advertise-client-urls https://${NODE_IP}:2379 \\
  --listen-client-urls https://${NODE_IP}:2379,https://127.0.0.1:2379 \\
  --initial-cluster-token etcd-cluster-0 \\
  --initial-cluster kubernetes-worker01=https://${ETCD1_IP}:2380,kubernetes-worker02=https://${ETCD2_IP}:2380,haproxy01=https://${ETCD3_IP}:2380 \\
  --initial-cluster-state new
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF

}

4. Enable and Start etcd service

{
  systemctl daemon-reload
  systemctl enable --now etcd
}

5. Export the following environment variables

export ETCDCTL_API=3 
export ETCDCTL_ENDPOINTS=https://<etcd1_ip_address>:2379,https://<etcd2_ip_address>:2379,https://<etcd3_ip_address>:2379
export ETCDCTL_CACERT=/etc/etcd/pki/ca.pem
export ETCDCTL_CERT=/etc/etcd/pki/etcd.pem
export ETCDCTL_KEY=/etc/etcd/pki/etcd-key.pem

6. Verify Etcd cluster status

In any one of the etcd nodes

etcdctl member list
etcdctl endpoint status
etcdctl endpoint health

Configure Highly available HAproxy with Keepalived

First, we need to install HAproxy and keepalived on both VMs by running the following command :

sudo apt install -y haproxy keepalived

Next, we are going to configure them.

HAproxy configuration

1. Go to HAproxy config file sudo nano /etc/haproxy/haproxy.cfg and add at the bottom of the file the following config.

frontend kube-apiserver
  bind *:6443
  mode tcp
  option tcplog
  default_backend kube-apiserver

backend kube-apiserver
    mode tcp
    option tcplog
    option tcp-check
    balance roundrobin
    default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100
    server kubernetes-master01 <master1_ip_address>:6443 check
    server kubernetes-master02 <master2_ip_address>:6443 check

The exact same config should be applied to the second HAproxy server.

2. Save the file and run the following command to restart HAproxy and enable it.

systemctl restart haproxy

systemctl enable haproxy

Keepalived configuration

1. Go to Keepalived config file sudo nano /etc/keepalived/keepalived.conf et modify it as following :

global_defs {
  notification_email {
  }
  router_id LVS_DEVEL
  vrrp_skip_check_adv_addr
  vrrp_garp_interval 0
  vrrp_gna_interval 0
}

vrrp_script chk_haproxy {
  script "killall -0 haproxy"
  interval 2
  weight 2
}

vrrp_instance haproxy-vip {
  state BACKUP
  priority 100
  interface ens192                       # Network card
  virtual_router_id 60
  advert_int 1
  authentication {
    auth_type PASS
    auth_pass 1111
  }
  unicast_src_ip <haproxy1>    # The IP address of this machine
  unicast_peer {
    <haproxy2>                       # The IP address of peer machines
  }

  virtual_ipaddress {
    <vip>/24                  # The VIP address
  }

  track_script {
    chk_haproxy
  }
}conf

The config of the second Keepalived is slightly different,

- For the interface field, we must provide your own network card information. In our case, it’s ens192

- The IP address provided for unicast_src_ip is the IP address of the current machine. For the field unicast_peer, we must provide the IP address of the second HAproxy/Keepalived server.

2. Save the file and run the following command to restart Keepalived and enable it.

systemctl restart keepalived

systemctl enable keepalived

Verify High availability

We can test the config we did above to make sure that we achieve high availability.

Now, the haproxy02 VM has the VIP.

Once we stop the HAproxy in this machine sudo systemctl stop haproxy as a way to simulate a failure on this node, we can notice that the floating get failed over to other machine:

Create Kubernetes cluster the Kubeadm way

This part is a guide that will walk you through the steps of installing and deploying Multi-node Highly available Kubernetes.

Step 1: Install Kubernetes prerequisites

We need to execute the following commands to install Kubernetes in all nodes ( Masters and Workers) :

sudo apt install apt-transport-https curl

Then we are going to add Kubernetes signing key and add the Kubernetes repository as a package source :

curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add
echo "deb https://apt.kubernetes.io/ kubernetes-xenial main" >> ~/kubernetes.list
sudo mv ~/kubernetes.list /etc/apt/sources.list.d

After that, we are going to update the nodes : sudo apt update

Finally, we are going to install kubernetes tools :

sudo apt-get -y kubelet kubeadm kubectl kubernetes-cni

Step 2: Disable Swap memory (all nodes)

Kubernetes fails to function in a system that is using swap memory. Hence, it must be disabled in the master node and all worker nodes. Execute the following command to disable swap memory:

sudo swapoff -a

We must ensure that the swap is disabled even after reboots, for this reason, we are going to comment out the /swapfile line in /etc/fstab

Step 3: Setting Unique Hostnames (all nodes)

All the hosts must have unique hostnames, we will execute this command in all servers :

sudo hostnamectl set-hostname <unique_hostname>

Step 4: Letting Iptables See Bridged Traffic (all nodes)

We must ensure that all nodes correctly see bridged traffic, for that we are going to set the value of net.bridge.bridge-nf-call-iptables to 1 by executing the following command:

sudo sysctl net.bridge.bridge-nf-call-iptables=1

Step 5: Changing Docker Cgroup Driver (all nodes)

By default, Docker installs with “cgroupfs” as the cgroup driver. However, Kubernetes recommends that Docker should run with “systemd” as the driver to avoid warnings during the cluster initialization.

sudo mkdir /etc/docker
cat <<EOF | sudo tee /etc/docker/daemon.json
{ "exec-opts": ["native.cgroupdriver=systemd"],
"log-driver": "json-file",
"log-opts":
{ "max-size": "100m" },
"storage-driver": "overlay2"
}
EOF

Then, execute the following commands to restart and enable Docker on system boot-up

sudo systemctl enable docker
sudo systemctl daemon-reload
sudo systemctl restart docker

Step 6: Copy etcd certificates to a standard location

{
  mkdir -p /etc/kubernetes/pki/etcd
  mv ca.pem etcd.pem etcd-key.pem /etc/kubernetes/pki/etcd/
}

Step 7: Create Cluster Configuration

We need this config file to specify that we are going to be using an external etcd cluster

{

ETCD1_IP="<etcd1_ip_address>"
ETCD2_IP="<etcd2_ip_address>"
ETCD3_IP="<etcd3_ip_address>"

cat <<EOF > kubeadm-config.yaml
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
controlPlaneEndpoint: "<vip>:6643" # change this (see below)
networking:
  podSubnet: "192.168.0.0/16"
etcd:
    external:
        endpoints:
        - https://${ETCD1_IP}:2379
        - https://${ETCD2_IP}:2379
        - https://${ETCD3_IP}:2379
        caFile: /etc/kubernetes/pki/etcd/ca.pem
        certFile: /etc/kubernetes/pki/etcd/etcd.pem
        keyFile: /etc/kubernetes/pki/etcd/etcd-key.pem

EOF

}

You can notice that the endpoint of the cluster should be the Virtual IP and the port 6443

Step 8: Initialize the Kubernetes Master Node ( Main Control Plane)

Starting from this step, we are going to initialize the Kubernetes cluster. On the first master, we are going to execute kubeadm init command :

sudo kubeadm init --config kubeadm-config.yaml --ignore-preflight-errors=all

Once the cluster initialization successfully completed, we can see our first node is added to the cluster :

Before starting to use the cluster, we are going to execute the following commands to bring Kubernetes cluster config to Home directory :

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Step 9: Deploying Calico network

We are going to deploy Calico network as a CNI provider for kubernetes using the following command:

kubectl create -f https://docs/projectcalico.org/v3.14/manifests/calico.yaml

We can verify if Kubernetes loaded up the Calico network by executing the command of displaying the pods running in all namespaces :

kubectl get pods -A

We can also view the health of the components using the get component status command:

kubectl get componentstatus

Step 10: Joining Second Master Node to the Kubernetes Cluster

Before joining the second master node, we need to copy all config files from the first master node. For this reason, we are going to write a script copy.sh as you can see and run it :

nano copy.sh

USER=sysadmin
# Set the control_plane_ips to all other master node ips or hostnames
CONTROL_PLANE_IPS="<master2_ip_address>"
for host in ${CONTROL_PLANE_IPS}; do
    scp /etc/kubernetes/pki/ca.crt "${USER}"@$host:
    scp /etc/kubernetes/pki/ca.key "${USER}"@$host:
    scp /etc/kubernetes/pki/sa.key "${USER}"@$host:
    scp /etc/kubernetes/pki/sa.pub "${USER}"@$host:
    scp /etc/kubernetes/pki/front-proxy-ca.crt "${USER}"@$host:
    scp /etc/kubernetes/pki/front-proxy-ca.key "${USER}"@$host:
    scp /etc/kubernetes/admin.conf "${USER}"@$host:
done

Then in the second master node, we are going to create another script move.sh to move those config files to /etc/kubernetes/pki/etcd:

nano move.sh

USER=sysadmin
mkdir -p /etc/kubernetes/pki/etcd
mv /home/${USER}/ca.crt /etc/kubernetes/pki/
mv /home/${USER}/ca.key /etc/kubernetes/pki/
mv /home/${USER}/sa.pub /etc/kubernetes/pki/
mv /home/${USER}/sa.key /etc/kubernetes/pki/
mv /home/${USER}/front-proxy-ca.crt /etc/kubernetes/pki/
mv /home/${USER}/front-proxy-ca.key /etc/kubernetes/pki/
mv /home/${USER}/admin.conf /etc/kubernetes/admin.conf

After that, we are going to generate the cluster join command:

kubeadm token create --print-join-command

Finally, we are going to execute the join command printed and add the prefix --control-plane at the end

 sudo kubeadm join <vip>:6443 --token xxxxxx \
 --discovery-token-ca-cert-hash sha256:xxxxxxxx \
 --control-plane

Step 11: Joining Worker Nodes to the Kubernetes Cluster (worker nodes)

At the end, we are going to join each worker to the cluster with the join command generated above:

sudo kubeadm join <vip>:6443 --token xxxxx \
 --discovery-token-ca-cert-hash sha256:xxxxxxx

Once the joind command successfuly executed, we can view that all nodes are added to the cluster :