Ezequiel Arielli
5 min readDec 27, 2018


Kubernetes on baremetal: kubespray-terraform Multimaster-HA , haproxy-API , Traefik and App’s with Horizontal Pod Autoscaling.

Deploy kubernetes on baremetal(VMware hypervisor) with terraform.
— HAPROXY to loadbalance k8s api-services.
— Traefik ingress for traffic http and https.
— metrics-servers for horizontal pod autoscaling and node metrics.


I running Kubernetes with kubespray in Centos 7 using HAPROXY to loadbalancer Kubernetes_API,http and http traffic ports using Traefik ingress.

based in this documentation:






  • Terraform v0.11.11 ( provider.vsphere v1.9.0)
  • Ansible 2.7.5
  • Python Library (details in kubespray requirements.yml)
  • Deploy HAPROXY HA.



1- Clone kubespray repository

git clone https://github.com/kubernetes-sigs/kubespray.gitcd kubespray
root@jenkins:/kubespray# pip install -r requirements.txt

2- Configure kubespray ansible inventory , you need copy sample directory to prd

root@jenkins:/kubespray# cp -rfp inventory/sample inventory/prodroot@jenkins:/kubespray# ls -la inventory/
total 12
drwxr-xr-x 3 root root 4096 Dec 16 17:04 .
drwxr-xr-x 12 root root 4096 Dec 26 17:11 ..
drwxr-xr-x 3 root root 4096 Dec 20 10:13 prod
root@jenkins:/kubespray# ls -la inventory/prod/
total 16
drwxr-xr-x 3 root root 4096 Dec 20 10:13 .
drwxr-xr-x 3 root root 4096 Dec 16 17:04 ..
drwxr-xr-x 4 root root 4096 Dec 16 19:40 group_vars
-rw-r — r — 1 root root 474 Dec 20 10:13 hosts.ini

3- Configure VIP HAPROXY in kubespray all.yml

root@jenkins:/kubespray# vi inventory/prod/group_vars/all/all.yml

## External LB example config

#apiserver_loadbalancer_domain_name: “elb.apps.stg.itshellws-k8s.com”
port: 6443
  • Check git diff of principal kubespray repository and mylocal(you need change this for use mycentos 7 template).
git diff

4- Create a new directory outside kubespray and clone haproxy-ansible-kubernetes.

git clone https://github.com/nightmareze1/haproxy-ansible-kubernetes.git

5- Create VMware template with Centos7

— Install Centos7 minimal with 2 disk in LVM
— SO- disk0
— Docker_Volume- disk1

Later, running the all commands details in centos7_template.yml the final step is copy you ssh-keys :

ssh-copy-id root@centos7machine

test ssh conection, it’s if success convert the virtual machine in template.

ssh root@centos7machine

6- Follow this readme https://github.com/nightmareze1/haproxy-ansible-kubernetes/blob/master/README.md

# haproxy-ansible-kubernetes


Launch ha-proxy ansible you need update the inventory with you haproxy ips.

- based in openshift and kubespray documentation:



# inventory example

root@jenkins:/haproxy-ansible-kubernetes/ansible# cat inventory/hosts.ini

haproxy1 ansible_host= ip= state=MASTER priority=100
haproxy2 ansible_host= ip= state=BACKUP priority=98

# Configure VIP (Loadbalancer IP for k8s)

root@jenkins:/haproxy-ansible-kubernetes/ansible# ls
inventory playbook.yml roles
root@jenkins:/haproxy-ansible-kubernetes/ansible# cat inventory/group_vars/keepalived/keepalived.yml


keepalived_loadbalancer_vip: ''

# HAProxy — Frontend and Backends

In this example I have two kubernetes clusters configured in my haproxy.yml (openshift and k8s-native with kubespray)

2- Later, The complete step1 you are ready to launch ansible-playbook.(optional launch common and syslog).

- name: haproxy provision
hosts: haproxy
become: yes
become_user: ‘root’
become_method: ‘sudo’
# — { role: common, tags: pkg-common }
# — { role: rsyslog, tags: rsyslog }
— { role: haproxy, tags: haproxy }
— { role: keepalived, tags: keepalived }
root@jenkins# sudo ansible-playbook -i inventory/hosts.ini playbook.yml — extra-vars=”install=True allow_restart=True” -vvvv

3- playbook results:

## Check ping VIP IP

## Check haproxy stats using HATOP (inside one haproxy execute)

hatop -s /var/lib/haproxy/stats

## Check haproxy stats using url

Test that url publics with haproxy works.

7- haproxy is running correctly , you can advanced to next step.

# Launch terraform infraesctucture for k8s-cluster without HAPROXY.

  • For running terraform you need configure you vcenter_cluster,datastore,template-name and the same nodes-ip’s that kube-spray inventory.

- Launch terraform plan

root@jenkins:/kube-spray/terraform# ./terraform planPlan: 5 to add, 0 to change, 0 to destroy. — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — Note: You didn’t specify an “-out” parameter to save this plan, so Terraform
can’t guarantee that exactly these actions will be performed if
“terraform apply” is subsequently run.
root@jenkins:/home/zz/kubernetes/kube-spray/terraform# ./terraform apply -auto-approve
  • Later that Terraform plan is success you can running ansible kubespray.
root@jenkins:/kube-spray# cd kubespray/
root@jenkins:/kubespray# ls
ansible.cfg cluster.yml contrib Dockerfile extra_playbooks library Makefile OWNERS README.md remove-node.yml reset.yml scale.yml SECURITY_CONTACTS setup.py upgrade-cluster.yml
cluster.retry code-of-conduct.md CONTRIBUTING.md docs inventory LICENSE mitogen.yaml OWNERS_ALIASES RELEASE.md requirements.txt roles scripts setup.cfg tests Vagrantfile
root@jenkins:/kubespray# sudo ansible-playbook -i inventory/prod/hosts.ini -become -become-user=root cluster.yml -vvvv
  • Connect to single master and look cluster-config in ~/.kube/config
Configure kubectl client with this config and run ./services.sh
Check traefik-ingress
  • Open kubernetes dashboard and look the all resources.
kubernetes-dashboard with heapster-controller and metrics-servers.

ENJOY — Kubernetes HA-CLUSTER in Baremetal !!!