Kubernetes on baremetal: kubespray-terraform Multimaster-HA , haproxy-API , Traefik and App’s with Horizontal Pod Autoscaling.

Deploy kubernetes on baremetal(VMware hypervisor) with terraform.
— HAPROXY to loadbalance k8s api-services.
— Traefik ingress for traffic http and https.
— metrics-servers for horizontal pod autoscaling and node metrics.

kubespray-terraform

I running Kubernetes with kubespray in Centos 7 using HAPROXY to loadbalancer Kubernetes_API,http and http traffic ports using Traefik ingress.

based in this documentation:

https://blog.openshift.com/haproxy-highly-available-keepalived/

https://github.com/kubernetes-sigs/kubespray/blob/master/docs/ha-mode.md

https://blog.inkubate.io/install-and-manage-automatically-a-kubernetes-cluster-on-vmware-vsphere-with-terraform-and-kubespray/

Regards!!!

Requirements:

• Terraform v0.11.11 ( provider.vsphere v1.9.0)
• Ansible 2.7.5
• Python Library (details in kubespray requirements.yml)
• Deploy HAPROXY HA.

Video:

MY GITHUB REPO :

nightmareze1/haproxy-ansible-kubernetes

nightmareze1/kubespray-terraform

1- Clone kubespray repository

git clone https://github.com/kubernetes-sigs/kubespray.git
cd kubespray
root@jenkins:/kubespray# pip install -r requirements.txt

2- Configure kubespray ansible inventory , you need copy sample directory to prd

root@jenkins:/kubespray# cp -rfp inventory/sample inventory/prod
root@jenkins:/kubespray# ls -la inventory/
total 12
drwxr-xr-x 3 root root 4096 Dec 16 17:04 .
drwxr-xr-x 12 root root 4096 Dec 26 17:11 ..
drwxr-xr-x 3 root root 4096 Dec 20 10:13 prod
root@jenkins:/kubespray# ls -la inventory/prod/
total 16
drwxr-xr-x 3 root root 4096 Dec 20 10:13 .
drwxr-xr-x 3 root root 4096 Dec 16 17:04 ..
drwxr-xr-x 4 root root 4096 Dec 16 19:40 group_vars
-rw-r — r — 1 root root 474 Dec 20 10:13 hosts.ini

3- Configure VIP HAPROXY in kubespray all.yml

root@jenkins:/kubespray# vi inventory/prod/group_vars/all/all.yml

## External LB example config

#apiserver_loadbalancer_domain_name: “elb.apps.stg.itshellws-k8s.com”
loadbalancer_apiserver:
 address: 172.16.250.150
 port: 6443

• Check git diff of principal kubespray repository and mylocal(you need change this for use mycentos 7 template).

git diff

4- Create a new directory outside kubespray and clone haproxy-ansible-kubernetes.

git clone https://github.com/nightmareze1/haproxy-ansible-kubernetes.git

5- Create VMware template with Centos7

— Install Centos7 minimal with 2 disk in LVM
— SO- disk0
— Docker_Volume- disk1

Later, running the all commands details in centos7_template.yml the final step is copy you ssh-keys :

ssh-copy-id root@centos7machine

test ssh conection, it’s if success convert the virtual machine in template.

ssh root@centos7machine

6- Follow this readme https://github.com/nightmareze1/haproxy-ansible-kubernetes/blob/master/README.md

# haproxy-ansible-kubernetes

This role install HAPROXY,KEEPALIVED,STATS AND HATOP

Launch ha-proxy ansible you need update the inventory with you haproxy ips.

- based in openshift and kubespray documentation:

https://blog.openshift.com/haproxy-highly-available-keepalived/

https://github.com/kubernetes-sigs/kubespray/blob/master/docs/ha-mode.md

# inventory example

root@jenkins:/haproxy-ansible-kubernetes/ansible# cat inventory/hosts.ini

[all]
haproxy1 ansible_host=172.16.250.151 ip=172.16.250.151 state=MASTER priority=100
haproxy2 ansible_host=172.16.250.152 ip=172.16.250.152 state=BACKUP priority=98
[haproxy]
haproxy1
haproxy2
[keepalived]
haproxy1
haproxy2

# Configure VIP (Loadbalancer IP for k8s)

root@jenkins:/haproxy-ansible-kubernetes/ansible# ls
inventory playbook.yml roles
root@jenkins:/haproxy-ansible-kubernetes/ansible# cat inventory/group_vars/keepalived/keepalived.yml

# VIP — LOADBALANCER IP

keepalived_loadbalancer_vip: '172.16.250.150'

# HAProxy — Frontend and Backends

In this example I have two kubernetes clusters configured in my haproxy.yml (openshift and k8s-native with kubespray)

2- Later, The complete step1 you are ready to launch ansible-playbook.(optional launch common and syslog).

---
- name: haproxy provision
 hosts: haproxy
 become: yes
 become_user: ‘root’
 become_method: ‘sudo’
roles:
 # — { role: common, tags: pkg-common }
 # — { role: rsyslog, tags: rsyslog }
 — { role: haproxy, tags: haproxy } 
 — { role: keepalived, tags: keepalived }
root@jenkins# sudo ansible-playbook -i inventory/hosts.ini playbook.yml — extra-vars=”install=True allow_restart=True” -vvvv

3- playbook results:

## Check ping VIP IP

## Check haproxy stats using HATOP (inside one haproxy execute)

hatop -s /var/lib/haproxy/stats

## Check haproxy stats using url

http://172.16.250.150:9000/haproxy_stats

Test that url publics with haproxy works.

7- haproxy is running correctly , you can advanced to next step.

# Launch terraform infraesctucture for k8s-cluster without HAPROXY.

• For running terraform you need configure you vcenter_cluster,datastore,template-name and the same nodes-ip’s that kube-spray inventory.

- Launch terraform plan

root@jenkins:/kube-spray/terraform# ./terraform plan
Plan: 5 to add, 0 to change, 0 to destroy.
 — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — 
Note: You didn’t specify an “-out” parameter to save this plan, so Terraform
can’t guarantee that exactly these actions will be performed if
“terraform apply” is subsequently run.
root@jenkins:/home/zz/kubernetes/kube-spray/terraform# ./terraform apply -auto-approve

• Later that Terraform plan is success you can running ansible kubespray.

root@jenkins:/kube-spray# cd kubespray/
root@jenkins:/kubespray# ls
ansible.cfg cluster.yml contrib Dockerfile extra_playbooks library Makefile OWNERS README.md remove-node.yml reset.yml scale.yml SECURITY_CONTACTS setup.py upgrade-cluster.yml
cluster.retry code-of-conduct.md CONTRIBUTING.md docs inventory LICENSE mitogen.yaml OWNERS_ALIASES RELEASE.md requirements.txt roles scripts setup.cfg tests Vagrantfile
root@jenkins:/kubespray# sudo ansible-playbook -i inventory/prod/hosts.ini -become -become-user=root cluster.yml -vvvv

• Connect to single master and look cluster-config in ~/.kube/config

Configure kubectl client with this config and run ./services.sh

Check traefik-ingress

• Open kubernetes dashboard and look the all resources.

kubernetes-dashboard with heapster-controller and metrics-servers.

ENJOY — Kubernetes HA-CLUSTER in Baremetal !!!