Kubernetes on baremetal: kubespray-terraform Multimaster-HA , haproxy-API , Traefik and App’s with Horizontal Pod Autoscaling.

Deploy kubernetes on baremetal(VMware hypervisor) with terraform.
 — HAPROXY to loadbalance k8s api-services.
 — Traefik ingress for traffic http and https.
 — metrics-servers for horizontal pod autoscaling and node metrics.

kubespray-terraform

I running Kubernetes with kubespray in Centos 7 using HAPROXY to loadbalancer Kubernetes_API,http and http traffic ports using Traefik ingress.

based in this documentation:

https://blog.openshift.com/haproxy-highly-available-keepalived/
https://github.com/kubernetes-sigs/kubespray/blob/master/docs/ha-mode.md
https://blog.inkubate.io/install-and-manage-automatically-a-kubernetes-cluster-on-vmware-vsphere-with-terraform-and-kubespray/

Regards!!!

Requirements:

  • Terraform v0.11.11 ( provider.vsphere v1.9.0)
  • Ansible 2.7.5
  • Python Library (details in kubespray requirements.yml)
  • Deploy HAPROXY HA.

Video:

MY GITHUB REPO :

1- Clone kubespray repository

git clone https://github.com/kubernetes-sigs/kubespray.git
cd kubespray
root@jenkins:/kubespray# pip install -r requirements.txt
2- Configure kubespray ansible inventory , you need copy sample directory to prd
root@jenkins:/kubespray# cp -rfp inventory/sample inventory/prod
root@jenkins:/kubespray# ls -la inventory/
total 12
drwxr-xr-x 3 root root 4096 Dec 16 17:04 .
drwxr-xr-x 12 root root 4096 Dec 26 17:11 ..
drwxr-xr-x 3 root root 4096 Dec 20 10:13 prod
root@jenkins:/kubespray# ls -la inventory/prod/
total 16
drwxr-xr-x 3 root root 4096 Dec 20 10:13 .
drwxr-xr-x 3 root root 4096 Dec 16 17:04 ..
drwxr-xr-x 4 root root 4096 Dec 16 19:40 group_vars
-rw-r — r — 1 root root 474 Dec 20 10:13 hosts.ini
3- Configure VIP HAPROXY in kubespray all.yml

root@jenkins:/kubespray# vi inventory/prod/group_vars/all/all.yml

## External LB example config


#apiserver_loadbalancer_domain_name: “elb.apps.stg.itshellws-k8s.com”
loadbalancer_apiserver:
address: 172.16.250.150
port: 6443

  • Check git diff of principal kubespray repository and mylocal(you need change this for use mycentos 7 template).
git diff
4- Create a new directory outside kubespray and clone haproxy-ansible-kubernetes.
git clone https://github.com/nightmareze1/haproxy-ansible-kubernetes.git
5- Create VMware template with Centos7

— Install Centos7 minimal with 2 disk in LVM
 — SO- disk0 
 — Docker_Volume- disk1
 
Later, running the all commands details in centos7_template.yml the final step is copy you ssh-keys :

ssh-copy-id root@centos7machine

test ssh conection, it’s if success convert the virtual machine in template.

ssh root@centos7machine
6- Follow this readme https://github.com/nightmareze1/haproxy-ansible-kubernetes/blob/master/README.md

# haproxy-ansible-kubernetes

This role install HAPROXY,KEEPALIVED,STATS AND HATOP

Launch ha-proxy ansible you need update the inventory with you haproxy ips.

- based in openshift and kubespray documentation:

https://blog.openshift.com/haproxy-highly-available-keepalived/

https://github.com/kubernetes-sigs/kubespray/blob/master/docs/ha-mode.md

# inventory example

root@jenkins:/haproxy-ansible-kubernetes/ansible# cat inventory/hosts.ini

[all]
haproxy1 ansible_host=172.16.250.151 ip=172.16.250.151 state=MASTER priority=100
haproxy2 ansible_host=172.16.250.152 ip=172.16.250.152 state=BACKUP priority=98
[haproxy]
haproxy1
haproxy2
[keepalived]
haproxy1
haproxy2

# Configure VIP (Loadbalancer IP for k8s)

root@jenkins:/haproxy-ansible-kubernetes/ansible# ls
inventory playbook.yml roles
root@jenkins:/haproxy-ansible-kubernetes/ansible# cat inventory/group_vars/keepalived/keepalived.yml

# VIP — LOADBALANCER IP


keepalived_loadbalancer_vip: '172.16.250.150'

# HAProxy — Frontend and Backends

In this example I have two kubernetes clusters configured in my haproxy.yml (openshift and k8s-native with kubespray)

2- Later, The complete step1 you are ready to launch ansible-playbook.(optional launch common and syslog).
---
- name: haproxy provision
hosts: haproxy
become: yes
become_user: ‘root’
become_method: ‘sudo’
roles:
# — { role: common, tags: pkg-common }
# — { role: rsyslog, tags: rsyslog }
— { role: haproxy, tags: haproxy }
— { role: keepalived, tags: keepalived }
root@jenkins# sudo ansible-playbook -i inventory/hosts.ini playbook.yml — extra-vars=”install=True allow_restart=True” -vvvv
3- playbook results:

## Check ping VIP IP

## Check haproxy stats using HATOP (inside one haproxy execute)

hatop -s /var/lib/haproxy/stats

## Check haproxy stats using url

http://172.16.250.150:9000/haproxy_stats

Test that url publics with haproxy works.
7- haproxy is running correctly , you can advanced to next step.

# Launch terraform infraesctucture for k8s-cluster without HAPROXY.

  • For running terraform you need configure you vcenter_cluster,datastore,template-name and the same nodes-ip’s that kube-spray inventory.

- Launch terraform plan

root@jenkins:/kube-spray/terraform# ./terraform plan
Plan: 5 to add, 0 to change, 0 to destroy.
 — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — 
Note: You didn’t specify an “-out” parameter to save this plan, so Terraform
can’t guarantee that exactly these actions will be performed if
“terraform apply” is subsequently run.
root@jenkins:/home/zz/kubernetes/kube-spray/terraform# ./terraform apply -auto-approve
  • Later that Terraform plan is success you can running ansible kubespray.
root@jenkins:/kube-spray# cd kubespray/
root@jenkins:/kubespray# ls
ansible.cfg cluster.yml contrib Dockerfile extra_playbooks library Makefile OWNERS README.md remove-node.yml reset.yml scale.yml SECURITY_CONTACTS setup.py upgrade-cluster.yml
cluster.retry code-of-conduct.md CONTRIBUTING.md docs inventory LICENSE mitogen.yaml OWNERS_ALIASES RELEASE.md requirements.txt roles scripts setup.cfg tests Vagrantfile
root@jenkins:/kubespray# sudo ansible-playbook -i inventory/prod/hosts.ini -become -become-user=root cluster.yml -vvvv
  • Connect to single master and look cluster-config in ~/.kube/config
Configure kubectl client with this config and run ./services.sh
Check traefik-ingress
  • Open kubernetes dashboard and look the all resources.
kubernetes-dashboard with heapster-controller and metrics-servers.
ENJOY — Kubernetes HA-CLUSTER in Baremetal !!!