Deploy Elasticsearch and Kibana Cluster on Kubernetes with Elasticsearch Operator

Happy devSecOps

(λx.x)eranga

Published in

Effectz.AI

5 min readJul 31, 2022

Background

The Elasticsearch Operator which also known as Elastic Cloud on Kubernetes(ECK) is a Kubernetes Operator to orchestrate Elastic applications (Elasticsearch, Kibana, APM Server, Enterprise Search, Beats, Elastic Agent, and Elastic Maps Server) on Kubernetes. It relies on a set of Custom Resource Definitions (CRD) to declaratively define the way each application is deployed. ECK simplifies deploying the whole Elastic stack on Kubernetes, giving us tools to automate and streamline critical operations. It focuses on streamlining all those critical operations such as, Managing and monitoring multiple clusters, Upgrading to new stack versions with ease, Scaling cluster capacity up and down, Changing cluster configuration, Dynamically scaling local storage (includes Elastic Local Volume, a local storage driver), Scheduling backups etc. In this post I’m gonna discuss about deploying scalable Elasticsearch cluster on Kubernetes using ECK. All the deployments which related to this post available in gitlab. Please clone the repo and continue the post.

Cluster Architecture

Elasticseach cluster contains three types of nodes, Master nodes(handle cluster-wide management and configuration), Data nodes(stores data and executes data-related operations search) and Client nodes(forwards cluster requests to the master node and data-related requests to data nodes). These nodes are deployed as pods in Kubernetes cluster. The best practice is to use 7 pods in the Elasticsearch cluster, 3 Master node pods, 2 Data node pods and 2 Client node pods. Following figure shows the Cluster architecture with these pods. Data node pods are deployed as a Stateful Set with a headless service to provide stable network identities. Master node pods are deployed as a Replica Set with a headless service which will help in auto-discovery. Client node pods are deployed as a Replica Set with a internal service which will allow access to the Data nodes for R/W requests.

Install ECK Operator

To deploy Elasticsearch on Kubernetes, first I need to install ECK operator in Kubernetes cluster. There are two main ways to install the ECK in a Kubernetes cluster, 1) Install ECK using the YAML manifests, 2) Install ECK using the Helm chart. In this post I have installed the ECK with using YAML manifest. Once installing the ECK on Kubernets cluster following components will be installed and updated.

CustomResourceDefinition objects for all supported resource types (Elasticsearch, Kibana, APM Server, Enterprise Search, Beats, Elastic Agent, and Elastic Maps Server).
Namespace named elastic-system to hold all operator resources.
ServiceAccount, ClusterRole and ClusterRoleBinding to allow the operator to manage resources throughout the cluster.
ValidatingWebhookConfiguration to validate Elastic custom resources on admission.
StatefulSet, ConfigMap, Secret and Service in elastic-system namespace to run the operator application.

In my scenario, I have installed the ECK on Minikube-based Kubernets cluster on local machine. Following is the way to install ECK Operator.

Deploy Elasticsearch Cluster

Now that ECK is running in the Kubernets cluster, I have the access elasticsearch.k8s.elastic.co/v1 API(which provided the ECK operator). I can deploy Elasticsearch cluster with this API. Following is the Elasticsearch cluster deployment with different types of nodes. Please note that in the deployment I have only used 1 Master node pod, 1 Data node pod and 1 Client node pod for the demonstration purpose(in here only 3 pods will be deployed instead of 7). To increase the number of pods, you just need to increase the count in the YAML deployment(e.g count: 3 in Master, count: 2 in Data and count:2 in Client).

The Master node sets with node.master: true, data node sets with node.data: true, Client node sets with node.ingest: true. When applying the deployment it will deploy three pods for Elasticsearch nodes. Behind the scene it automatically creates three PersistentVolumeClaims and three PersistentVolumes for respective Elasticsearch nodes. When applying the deployment, it will creates ClusterIP service rahasak-elasticsearch-es-http for the cluster. We can port-forward that ClusterIP service and access Elasticsearch HTTP API.

Test Elasticsearch Cluster

When deploying the Elasticsearch, the ECK Operator deploy several Kubernetes Secret objects for the cluster. The Elasticsearch cluster password is stored in the rahasak-elasticsearch-es-elastic-user Secret object(by default EKC Operator enables basic/password authentication for the Elasticsearch cluster). A default user named elastic is automatically created with the password stored in a Kubernetes secret. We can get the password from the Secret object and access the Cluster.

As mentioned above, when applying the deployment, it will creates ClusterIP service rahasak-elasticsearch-es-http for the cluster. We can port-forward that ClusterIP service and access Elasticsearch HTTP API as below.

Deploy Kibana

Once setup the Elasticsearch, I can deploy Kibana and integrate with Elasticsearch. Following is the 1 node Kibana deployment.

When applying the deployment it will create 1 node Kibana. The Kibana service will expose with ClusterIP service rahasak-elasticsearch-kb-http for the cluster. We can port-forward this ClusterIP service and access Kibana API.

The same Elasticsearch user credentials(which we have obtained in previous step via Secret) can be used to access the Kibana, Following is the way access Kibana with port forwarding ClusterIP service rahasak-elasticsearch-kb-http.