A Guide to Deploy Elasticsearch Cluster on Google Kubernetes Engine
This is a simple guide that helps you to deploy Elasticsearch cluster on Google Kubernetes Engine. This guide should be relevant on any Kubernetes cluster with a simple tweak on the persistent volume part.
The overall step-by-step is like the following:
- Enable the persistent volume via Storage Classes.
- Enable the Elasticsearch node discovery via Headless Service.
- Deploy the Elasticsearch Cluster via Stateful Sets.
1. Persistent Volume
The first step is to create a Storage Class on your cluster. Create new file called storage.yaml
with the following content:
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: ssd
provisioner: kubernetes.io/gce-pd
parameters:
type: pd-ssd
zone: asia-southeast1-a
Note that the Storage Class provisioner
that we use is GCE ( kubernetes.io/gce-pd
) with the following parameters:
type: pd-ssd
to enable the SSD as persistent disk. You may update the parameters type withtype: pd-standard
to enable the standard disk as persistent diskzone: asia-southeast1-a
for the compute zone. You may update this to your compute zone. You can also usezones
parameter for multiple zones like the following:zones: asia-southeast1-a, asia-southeast1-b
.
Then enable the storage class using the following command:
kubectl apply -f storage.yaml
You should be able to see the Storage Class on the dashboard:
If you deploy the Elasticsearch Cluster on other than Google Kubernetes Engine, please update the provisioner and the parameters. You can see the available provisioners and parameters in here. The rest of this guide will applicable to any Kubernetes Cluster.
2. Elasticsearch Node Discovery
The second step is to enable elasticsearch node discovery via headless service. Create new file called service.yaml
with the following content:
apiVersion: v1
kind: Service
metadata:
name: es
labels:
service: elasticsearch
spec:
clusterIP: None
ports:
- port: 9200
name: serving
- port: 9300
name: node-to-node
selector:
service: elasticsearch
Then enable the headless service using the following command:
kubectl apply -f service.yaml
You should be able to see the service on the dashboard:
Now, every pod that have label service: elasticsearch
should be accessible via $PODNAME.es.default.cluster.local
inside a kubernetes cluster. This will helps our elasticsearch nodes to discover each other and form a cluster.
3. Elasticsearch Cluster
The last step is to deploy the elasticsearch cluster using the StatefulSet. Create new file called elasticsearch.yaml
with the following content:
to deploy the Elasticsearch cluster, run the following command:
kubectl apply -f elasticsearch.yaml
It may takes time before the cluster is ready, it depends on the size of the cluster. You can see if the cluster is ready or not by accessing the workloads dashboard.
If your cluster is ready, you can check if cluster is created or not by accessing one of the elasticsearch node via port-forward:
kubectl port-forward elasticsearch-0 9200:9200
this will forward all request to http://localhost:9200 to the elasticsearch-0 node. Then:
curl http://localhost:9200/_cluster/state?pretty
It should show you that the cluster is formed by 5 elasticsearch nodes.
That’s it! now you can enjoy my favorite gif: