Provisioning HA unmanaged K8S Cluster using Ansible and KubeSpray on EC2 instances.

Hola again and Welcome back!

Today we are going to see how to provision a highly available multi-master Kubernetes cluster using Kubespray on EC2 instances. We will create 3 nodes that will act as both master and slave. However, this architecture, wherein a node acts like both master and slave, is highly unrecommended for running a production environment. This just only provides a cost-effective solution to run your application on nodes that can act as both masters and slaves.

ProTip: Always go for creating a separate masters and slaves node in a production environment.

The HA Architecture

Here is what we are trying to achieve at a top-level. There is a lot of stuff that will happen behind the scenes and will be taken care of by Kubespray.

(Image Source: Kubernetes Official Documentation)

System Requirement

• Number of nodes: 4
• CPU: 2
• RAM: 2GB
• OS: Ubuntu 18.04

Among the 4 nodes, one node will be your Ansible node, and the remaining three nodes will act as master and worker both. On the Ansible node, you will setup ansible and it will be your control node which will control the other 3 nodes. Using this control node, you will set up an HA cluster by running the ansible-playbook of Kubespray.

Let’s set up and get our hands dirty and feet wet

Note: Perform the below steps from 1 to 4 on all the four nodes.

1. Changing the hostnames

Provide a meaningful hostname to all the nodes before proceeding any further. For time being I’ll make use of hostnames such as ansible, node1, node2, node3 for the four nodes, respectively.

$ sudo hostnamectl set-hostname ansible
$ exec bash

Using the above commands set up the hostname for remaining nodes as node1, node2, and node3.

2. Changing the authentication methods inside the sshd_config file

Run the below command in the terminal to get inside the sshd_config file.

$ vi /etc/ssh/sshd_config

Uncomment and set PermitRootLogin, PubkeyAuthentication, PasswordAuthentication to yes as given below.

PermitRootLogin yes
PubkeyAuthentication yes
PasswordAuthentication yes

3. Configuring the local DNS by editing the /etc/hosts file

Run the below command in the terminal to get inside the hosts file of the OS.

$ vi /etc/hosts

Place your host's private IP along with the hostname of the nodes that we gave to all nodes in step 1. Below is the example of how your file should look like after updating.

127.0.0.1 localhost
10.x.x.x  ansible
10.x.x.x  node1
10.x.x.x  node2
10.x.x.x  node3

4. Set the password for the root user

In each of the nodes, set the password for the root account by using the below commands.

$ sudo su
$ passwd root
Enter the password: <type_a_pass>
Re-enter the password: <retype_the_pass>
$ systemctl reload sshd

Note: Perform the below steps from 5 to X only on the Ansible node.

5. Setting up the SSH

Now, you have permitted the root login to all the nodes by setting the PermitRootLogin flag to yes and other necessary flags to yes in Step 2.
Now, It’s time to establish a Passwordless SSH connection from the root account of your ansible node to the root account of all the other remaining nodes.

Run this below command from the ansible node to achieve root login, from the ansible node to all the remaining nodes.

$ sudo su
$ cd /root/.ssh
$ ssh-keygen
$ ssh-copy-id root@node1
$ ssh-copy-id root@node2
$ ssh-copy-id root@node3

6. Setting up HA Proxy Server

HAProxy is free, open-source software that provides a high availability load balancer and proxy server for TCP and HTTP-based applications that spread requests across multiple servers.

The HA proxy here acts as an ‘Internal Load Balancer’ when a user hits the IP, HA proxy does a health check of the master nodes and selects one out of many (out of three in our case) in a Round Robin fashion. The scheme(round-robin, random, etc) you want the HA proxy to select nodes can be defined in the configuration file.

In this tutorial, we’ll set up the HA proxy on the ansible node. However, if you wish to use other nodes, you are free to use as well.

Follow the steps given underneath to set up the HA on the ansible node.

• Add the DNS name for the load balancer and the IP of your ansible node in the hosts file

$ sudo su 
$ vi /etc/hosts

• Add the following line at the end of the file.

10.x.x.x elb.example.com

Note: Here 10.x.x.x is the public IP address of my Ansible node. Replace 10.x.x.x with the public IP of your Ansible node.

• Finally, install the HAproxy server on the Ansible node

$ sudo su 
$ apt install haproxy

• Configuring the HA Proxy

$ vi /etc/haproxy/haproxy.cfg 

• Add the following lines at the end of the script.

listen kubernetes-apiserver-https
bind <AnsibleNodeIP>:8383
mode tcp
option log-health-checks
timeout client 3h
timeout server 3h
server master1 <Master1IP>:6443 check check-ssl verify none inter 10000
server master2 <Master2IP>:6443 check check-ssl verify none inter 10000
server master3 <Master3IP>:6443 check check-ssl verify none inter 10000
balance roundrobin

Note: Replace the AnsibleNodeIP, Master1IP, Master2IP, and Master3IP with the public IP address of your ansible, node1, node2, and node3 respectively in the below given code

• Restart the haproxy service.

$ systemctl restart haproxy

7. Configuring Ansible node For HA Cluster

Now we are just a few steps behind our target. Let’s do little more configurations and ultimately start provisioning our cluster.

• Clone the ansible-playbook git repository:

$ sudo su
$ cd ~
$ mkdir HA
$ cd HA
$ git clone https://github.com/kubernetes-sigs/kubespray.git
$ cd kubespray

• Installing python3-pip and requirements from the requirements.txt file:

$ apt-get update && apt-get install -y python3-pip
$ sudo pip3 install -r requirements.txt

• Copy the “inventory/sample” as “inventory/mycluster”

$ cp -rfp inventory/sample inventory/mycluster

• Update Ansible inventory file with inventory builder

$ declare -a IPS=(Node1IP Node2IP Node3IP)
$ CONFIG_FILE=inventory/mycluster/inventory.ini python3 contrib/inventory_builder/inventory.py ${IPS[@]}

Note: Replace the Node1IP, Node2IP, and Node3IP with the node1, node2, and node3 Private IP respectively.

• You can review and change parameters according to your need under the “inventory/mycluster/group_vars”

$ cat inventory/mycluster/group_vars/all/all.yml
$ cat inventory/mycluster/group_vars/k8s-cluster/k8s-cluster.yml

• Configure the docker version.

When I first ran without hardcoding the docker version, I got an error:
“Docker client version 1.40 is too new. The maximum supported API version is 1.39”
To resolve this error hardcode the version inside the ubuntu-amd64.yml file and set the docker version in the k8s-cluster.yml file.

• Open the ubuntu-amd64.yml file using the vi editor.

$ vi roles/container-engine/docker/vars/ubuntu-amd64.yml

• Comment out all the other versions under docker_versioned_pkg: and add the below docker version.

‘19.09’: docker-ce=5:19.03.83–0ubuntu-{{ansible_distribution_release|lower}}

• Now, also hard-code the docker version inside the k8s-cluster.yml file.

$ vi inventory/mycluster/group_vars/k8s-cluster/k8s-cluster.yml

• Add anywhere the below line inside the k8s-cluster.yml file.

docker_version: “19.09”

• Copy inventory.ini to hosts.yaml

$ cp inventory/mycluster/inventory.ini inventory/mycluster/hosts.yaml

• Enabling the external load Balancer in the all.yaml file.

$ vi inventory/mycluster/group_vars/all/all.yaml

• Edit and uncomment the following lines

apiserver_loadbalancer_domain_name: <LoadBalancerDNS>
loadbalancer_apiserver:
address: <AnsibleNodeIP>
port: 8383
loadbalancer_apiserver_localhost: false

Note: Replace the LoadBalancerDNS with the name that you provided in the Step 6(elb.example.com in my case) and AnsibleNodeIP with the Private IP of the ansible node.

• Run the ansible-playbook.

$ ansible-playbook -i inventory/mycluster/hosts.yaml --user root cluster.yml

Congratulations, We did it!
Now, just wait for several minutes to let the ansible-playbook do the work for you. Usually, ansible-playbook will take around 10 to 20 minutes (depending on the speed of the internet on EC2 instances) and ultimately will provide you the cluster.

Deploying a sample application that contains both front-end and the back-end on the provisioned cluster

Navigate to the home directory inside any of your master node using the below command.

$ cd ~

Create a new folder named “sample-app” using the below command.

$ mkdir sample-app

Navigate to the folder which you just created using the below command.

$ cd sample-app

Define a file pv.yaml using the vi editor and save the below content in it.

pv.yaml

Define a Deployment in knote.yaml file for front-end application and save the below content in it.

knote.yaml

Define a Service in knote-svc.yaml file for front-end application and save the below content in it.

knote-svc.yaml

Define a Deployment in mongo.yaml file for back-end application and save the below content in it.

mongo.yaml

Make sure you have a knote.yaml, knote-svc.yaml, pv.yaml and mongo.yaml file inside the sample-app directory

$ tree .
sample-app/
├── knote.yaml
└── mongo.yaml

So far, so good. Everything is ready!
Submit your resource definitions to Kubernetes with the following command:

$ kubectl apply -f sample-app

That’s all, folks!

Hope you enjoyed coding with me. Leave a comment down below on how this tutorial is. If you faced any issue, leave a comment, I’ll help. Criticisms are also welcome. 😉

See you soon with another post!

Happy Coding!❤