Deploying Scalable ELK Cluster on Openstack

As you know Elasticsearch is based on Lucene, very fast and scalable for searching operations. Elasticsearch is good for data analysis, logging and error monitoring and alerting so can be used to search all kinds of documents.

OpenStack Heat is an orchestration engine to launch multiple cloud applications based on templates in yaml format. You can orchestrate cloud applications using a declarative yaml template through an OpenStack REST API.

In this documentation, we will see how to deploy and scale Elastic Search cluster with Kibana and Logstash. Before start, we need to have

We need to have OpenStack Platform (tested on Newton) up and running with these main components (nova, keystone, glance, neutron with LBaaS support and Heat)

Also, NFS server accessible from OpenStack platform is required. All installation files (jdk, elasticsearch.tar.gz, logstash.tar.gz, kibana.tar.gz, logstash.conf, license.json file for x-pack plugin) will be placed in NFS share for use.

My GitHub repo provides Heat Template yaml files and it is explained briefly what templates are used for.

To sum up, It can be provisioned ELK cluster with 1 Logstash + 3 ES Master + 3 ES Data + 2 ES Ingest + 1 Kibana server at first, and ES Data and Ingest Layer can be scaled any time. ES Data and ES Ingest nodes are behind Openstack LBaaS for redundancy. Only ES Masters, Logstash, and Kibana has floating IP so, there is no need for floating IP in order to scaling ES Data or Ingest nodes.

For any questions, help or advice, please drop me a mail,