Ezequiel Arielli
Dec 27, 2018 · 5 min read

Kubernetes on baremetal: kubespray-terraform Multimaster-HA , haproxy-API , Traefik and App’s with Horizontal Pod Autoscaling.

Deploy kubernetes on baremetal(VMware hypervisor) with terraform.
— HAPROXY to loadbalance k8s api-services.
— Traefik ingress for traffic http and https.
— metrics-servers for horizontal pod autoscaling and node metrics.

kubespray-terraform

I running Kubernetes with kubespray in Centos 7 using HAPROXY to loadbalancer Kubernetes_API,http and http traffic ports using Traefik ingress.

based in this documentation:

https://blog.openshift.com/haproxy-highly-available-keepalived/

https://github.com/kubernetes-sigs/kubespray/blob/master/docs/ha-mode.md

https://blog.inkubate.io/install-and-manage-automatically-a-kubernetes-cluster-on-vmware-vsphere-with-terraform-and-kubespray/

Regards!!!

Requirements:

  • Terraform v0.11.11 ( provider.vsphere v1.9.0)
  • Ansible 2.7.5
  • Python Library (details in kubespray requirements.yml)
  • Deploy HAPROXY HA.

Video:

MY GITHUB REPO :

1- Clone kubespray repository

git clone https://github.com/kubernetes-sigs/kubespray.gitcd kubespray
root@jenkins:/kubespray# pip install -r requirements.txt

2- Configure kubespray ansible inventory , you need copy sample directory to prd

root@jenkins:/kubespray# cp -rfp inventory/sample inventory/prodroot@jenkins:/kubespray# ls -la inventory/
total 12
drwxr-xr-x 3 root root 4096 Dec 16 17:04 .
drwxr-xr-x 12 root root 4096 Dec 26 17:11 ..
drwxr-xr-x 3 root root 4096 Dec 20 10:13 prod
root@jenkins:/kubespray# ls -la inventory/prod/
total 16
drwxr-xr-x 3 root root 4096 Dec 20 10:13 .
drwxr-xr-x 3 root root 4096 Dec 16 17:04 ..
drwxr-xr-x 4 root root 4096 Dec 16 19:40 group_vars
-rw-r — r — 1 root root 474 Dec 20 10:13 hosts.ini

3- Configure VIP HAPROXY in kubespray all.yml


root@jenkins:/kubespray# vi inventory/prod/group_vars/all/all.yml

## External LB example config


#apiserver_loadbalancer_domain_name: “elb.apps.stg.itshellws-k8s.com”
loadbalancer_apiserver:
address: 172.16.250.150
port: 6443
  • Check git diff of principal kubespray repository and mylocal(you need change this for use mycentos 7 template).
git diff

4- Create a new directory outside kubespray and clone haproxy-ansible-kubernetes.

git clone https://github.com/nightmareze1/haproxy-ansible-kubernetes.git

5- Create VMware template with Centos7

— Install Centos7 minimal with 2 disk in LVM
— SO- disk0
— Docker_Volume- disk1

Later, running the all commands details in centos7_template.yml the final step is copy you ssh-keys :

ssh-copy-id root@centos7machine

test ssh conection, it’s if success convert the virtual machine in template.

ssh root@centos7machine

6- Follow this readme https://github.com/nightmareze1/haproxy-ansible-kubernetes/blob/master/README.md

# haproxy-ansible-kubernetes

This role install HAPROXY,KEEPALIVED,STATS AND HATOP

Launch ha-proxy ansible you need update the inventory with you haproxy ips.

- based in openshift and kubespray documentation:

https://blog.openshift.com/haproxy-highly-available-keepalived/

https://github.com/kubernetes-sigs/kubespray/blob/master/docs/ha-mode.md

# inventory example

root@jenkins:/haproxy-ansible-kubernetes/ansible# cat inventory/hosts.ini

[all]
haproxy1 ansible_host=172.16.250.151 ip=172.16.250.151 state=MASTER priority=100
haproxy2 ansible_host=172.16.250.152 ip=172.16.250.152 state=BACKUP priority=98
[haproxy]
haproxy1
haproxy2
[keepalived]
haproxy1
haproxy2

# Configure VIP (Loadbalancer IP for k8s)

root@jenkins:/haproxy-ansible-kubernetes/ansible# ls
inventory playbook.yml roles
root@jenkins:/haproxy-ansible-kubernetes/ansible# cat inventory/group_vars/keepalived/keepalived.yml

# VIP — LOADBALANCER IP


keepalived_loadbalancer_vip: '172.16.250.150'

# HAProxy — Frontend and Backends

In this example I have two kubernetes clusters configured in my haproxy.yml (openshift and k8s-native with kubespray)

2- Later, The complete step1 you are ready to launch ansible-playbook.(optional launch common and syslog).

---
- name: haproxy provision
hosts: haproxy
become: yes
become_user: ‘root’
become_method: ‘sudo’
roles:
# — { role: common, tags: pkg-common }
# — { role: rsyslog, tags: rsyslog }
— { role: haproxy, tags: haproxy }
— { role: keepalived, tags: keepalived }
root@jenkins# sudo ansible-playbook -i inventory/hosts.ini playbook.yml — extra-vars=”install=True allow_restart=True” -vvvv

3- playbook results:

## Check ping VIP IP

## Check haproxy stats using HATOP (inside one haproxy execute)

hatop -s /var/lib/haproxy/stats

## Check haproxy stats using url

http://172.16.250.150:9000/haproxy_stats

Test that url publics with haproxy works.

7- haproxy is running correctly , you can advanced to next step.

# Launch terraform infraesctucture for k8s-cluster without HAPROXY.

  • For running terraform you need configure you vcenter_cluster,datastore,template-name and the same nodes-ip’s that kube-spray inventory.

- Launch terraform plan

root@jenkins:/kube-spray/terraform# ./terraform planPlan: 5 to add, 0 to change, 0 to destroy. — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — Note: You didn’t specify an “-out” parameter to save this plan, so Terraform
can’t guarantee that exactly these actions will be performed if
“terraform apply” is subsequently run.
root@jenkins:/home/zz/kubernetes/kube-spray/terraform# ./terraform apply -auto-approve
  • Later that Terraform plan is success you can running ansible kubespray.
root@jenkins:/kube-spray# cd kubespray/
root@jenkins:/kubespray# ls
ansible.cfg cluster.yml contrib Dockerfile extra_playbooks library Makefile OWNERS README.md remove-node.yml reset.yml scale.yml SECURITY_CONTACTS setup.py upgrade-cluster.yml
cluster.retry code-of-conduct.md CONTRIBUTING.md docs inventory LICENSE mitogen.yaml OWNERS_ALIASES RELEASE.md requirements.txt roles scripts setup.cfg tests Vagrantfile
root@jenkins:/kubespray# sudo ansible-playbook -i inventory/prod/hosts.ini -become -become-user=root cluster.yml -vvvv
  • Connect to single master and look cluster-config in ~/.kube/config
Configure kubectl client with this config and run ./services.sh
Check traefik-ingress
  • Open kubernetes dashboard and look the all resources.
kubernetes-dashboard with heapster-controller and metrics-servers.

ENJOY — Kubernetes HA-CLUSTER in Baremetal !!!

Ezequiel Arielli

Written by

DevOps SR | SRE @Miroculus.com San Francisco, CA www.itshellws.org

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade