Setting up Elasticsearch and Kibana on Google Kubernetes Engine with ECK

Elasticsearch has long been considered the de facto stack for log collection and aggregation.

With Application Performance Monitoring (APM) and Security Information and Event Management (SIEM) offerings, and lot of open source tools such as Jaeger supporting Elasticsearch for storage, Elasticsearch has become even more relevant for overall system health and security.

Elastic, the corporate force behind all the innovation in Elastic Stack has been pushing the envelope further every year.

They started a fully managed Elastic Stack (which they call as Elastic Enterprise Search), a managed version of Elastic Stack on GCP and open sourced container images and helm chart for Kubernetes deployments.

Now they have GA’ed their Elastic Stack on Kubernetes (ECK) product, which is based on Kubernetes Operator pattern and it promises to simplify the Day-2 operations (Updates, Scaling and Security to name a few) of managing an Elasticsearch cluster.

In this post I’m going to share how to setup a fully functional Elasticsearch cluster using ECK on Google Kubernetes Engine. My focus would primarily be around how ECK handles Persistent Storage, Version Upgrades and Scaling (both horizontal and vertical).

Lets gets started.

Source: https://www.elastic.co/elastic-cloud-kubernetes

Table of Content

1. Prepping the K8S Cluster
2. Starting the Hello World cluster
3. Making room for Scaling
4. Adding Custom Persistent Storage
5. Starting Kibana
6. Final Thoughts

1. Prepping the K8S Cluster

Start a GKE cluster with a node pool of size 3 and n1-standard-2 machine types.

Once started, lets install the ECK CRD’s onto it. Current ECK version is 1.0 .

kubectl apply -f https://download.elastic.co/downloads/eck/1.0.1/all-in-one.yaml

After issuing the command, check for elastic-operator logs to see if the controller started successfully.

You’ll should see logs like below screenshot and there shouldn’t be any errors.

kubectl -n elastic-system logs -f statefulset.apps/elastic-operator

elastic-operator logs

You can also verify by checking resources running in elastic-system namespace.

kubectl get all -n elastic-system

You should see elastic-operator pod running for the stateful set. Also a service is created for elastic-webhook-server .

In next step we’ll start a 1 node Elasticsearch cluster to verify that our ECK installation is indeed working.

2. Starting the Hello World cluster

The CustomResourceDefinitions that we installed as part of previous step now makes Elasticsearch a first class resource in our cluster. We can create a resource of kind: Elasticsearch and the controller will start the necessary pods and services according to the specification. I’m using official quick start example from Elastic website.

Let us apply this file.

kubectl apply -f elasticsearch-1-node-yml

Verify if our resource is in green health.

kubectl get elasticsearch

Note: It might take some time, and you might see the Elasticsearch resource in ApplyingChanges Phase.

Once in green you should see following output.

Let us explore all the resources created by the controller for our Elasticsearch resource.

kubectl get all -n default

Resources deployed for custom Elasticsearch resource

You should see 1 Pod running Elasticsearch, 2 services to access the cluster and a Statefulset.

Now, lets verify if our Elasticsearch is indeed working.

You can access the Elasticsearch from within the cluster using the service quickstart-es-http at port 9200.

To access it from localhost you can forward the port of the service.

kubectl port-forward service/quickstart-es-http 9200

This will be a long running proxy. Keep it running.

From another tab you can now access the cluster via CURL utility. This deployment setup a master user elastic with password. The password is stored in a Kubernetes Secret. Lets get the password first.

PASSWORD=$(kubectl get secret quickstart-es-elastic-user -o=jsonpath='{.data.elastic}' | base64 --decode)

Then access the cluster via curl.

curl -u "elastic:$PASSWORD" -k "https://localhost:9200"

You should see the standard Elasticsearch response.

{
  "name" : "quickstart-es-default-0",
  "cluster_name" : "quickstart",
  "cluster_uuid" : "cpI65V-ZTF2RRVDI5wHNxA",
  "version" : {
    "number" : "7.6.2",
    "build_flavor" : "default",
    "build_type" : "docker",
    "build_hash" : "ef48eb35cf30adf4db14086e8aabd07ef6fb113f",
    "build_date" : "2020-03-26T06:34:37.794943Z",
    "build_snapshot" : false,
    "lucene_version" : "8.4.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

With that out of the way, let us index some documents and check the resiliency of our cluster.

Execute the script to index some records. Verify the data.

curl -u "elastic:$PASSWORD" -k "https://localhost:9200/_cat/indices?v"

The index is in yellow health because the replica shard has not been assigned to the node as we only have 1 node in our cluster.

Next, to simulate a real world node loss, lets forcefully delete the pod running the Elasticsearch server.

kubectl delete pod quickstart-es-default-0

Since its running as part of a Statefulset, Kubernetes will detect this and start a new Pod.

Also, you’ll have to stop and restart the port forwarding proxy.

Pretty Cool. Another cool thing to notice is that the controller also deploys a PersistentVolume to store the index data and Stateful set ensure that the new pod mounts the correct PersistentVolumeClaim . Hence even after loosing the Pod and a restart, we did not loose our index data.

You can verity it by accessing_cat/indices again.

However, by default the persistent volume is of size 1Gb and uses the default StorageClass of the cluster. In the subsequent section, we’ll see how the Elasticsearch resource let you specify your own PVC and PV configuration.

For next section, lets make our pokemon index green by adding one more node to our Elasticsearch deployment.

3. Making room for Scaling

Adding or removing nodes in a standard Elasticsearch cluster is a painfully tedious task. In this section you’ll start to see the benefits of the ECK operator. Adding a node to the cluster is as simple as changing nodeSets.count , re-apply and you are done. Lets make the change.

kubectl apply -f elasticsearch-2-nodes.yml

Once applied, verify that the quickstart has scaled to 2 nodes.

kubectl get elasticsearch

You’ll again need to restart the proxy. Verify that the pokemon index is in green health.

So horizontal scaling was quite easy. The custom resource also supports specifying compute resources using the familiar request and limit attributes. You can read more about it from official doc here.

Next let us see how we can use custom PersistentVolume and PersistentVolumeClaims .

4. Adding Custom Persistent Storage

Note: Updating PVC of a cluster can’t be done without deleting the entire cluster and recreating it, hence you should plan it before the initial deployment otherwise you’ll loose all of your data.

If you try to apply the new file you’ll be greeted by an error.

Volume claim templates cannot be modified

So you’ll need to delete the existing Elasticsearch resource and re-deploy it.

kubectl delete elasticsearch quickstart

Verify that now the size of persistent volume is 2Gi instead of the default 1Gi.

kubectl describe pvc

PVC with size 2Gi

So we were successful in attaching a custom PVC but again note that plan it before the deployment rather than how we did it in this post.

So far so good. The ECK not just adds custom resource for Elasticsearch but also for Kibana and ApmServer as well.

In the next section we’ll deploy Kibana and with LoadBalancer service to access Kibana UI.

5. Starting Kibana

Two things to note is that spec.version should match the version of the Elasticsearch resource and spec.elasticsearchRef should point to correct and existing Elasticsearch resource name.

Apply the resource.

kubectl apply -f kibana.yml

Verify if its successfully deployed.

kubectl get kibana

Also check the services deployed by Kibana.

By default this service is of type ClusterIP hence can only be used to access Kibana within the cluster. Let us update kibana.yml manifest to create this service of type LoadBalancer .

Apply the manifest and check the service type now.

Keep checking the EXTERNAL-IP should get populated in a while.

Once external IP is populated access the Kibana UI.

https://<EXTERNAL-IP>:5601

Do note to use https as our Kibana deployment has SSL enabled with a self signed certificate. You might get YOUR Connection is not Encrypted error in your browser. Ignore the warning.

Use the username elastic and password that we fetched from the secret.

PASSWORD=$(kubectl get secret quickstart-es-elastic-user -o=jsonpath='{.data.elastic}' | base64 --decode)
echo $PASSWORD

Re-index the pokemon data (if you deleted and re-created the cluster because of PVC changes).

You should be able to see your index data on Kibana.

So we’ve reached the end of the post. You can tear down the Kubernetes cluster if you don’t need it anymore to incur any extra cost. In the next section I’ll share my parting thoughts about ECK and the ease of operations that it brings.

6. Final Thoughts

ECK is a really amazing way of managing Elastic Stack resources on Kubernetes. Following are the highlighting feature of ECK for me:

1. How easy it is to create custom resources just by using declarative manifests. The CRDs really makes the Elastic Stack components as first class citizen in Kubernetes.
2. Adding and removing node is a one line change and is the real game changer for operations team.
3. The ECK also support seamless version upgrades if your old and new version are compatible for upgrade. You can read more about it here.
4. ECK also allows to configure nearly every aspect of your cluster right from node count to setting up custom elaticsearch.yml parameters in a declarative way that makes deployment repeatable and largely error free.
5. One feature that I wanted to explore but didn’t for sake of brevity, is that ECK also plays nicely with Service Meshes (Istio and Linkerd). Read more about it here.

Complete Documentation : https://www.elastic.co/guide/en/cloud-on-k8s/1.0/index.html

And there you have it. We successfully deployed Elasticsearch and Kibana on Kubernetes using ECK.

If you find any discrepancies in any part of the post or have any question in general, feel free to drop a comment.

Till then Happy Coding :)