Kubernetes Tutorial: Successful Deployment of Elasticsearch

Kubernetes is an open-source system developed for automating deployment, scaling, and management of containerized applications. It can handle outages and demand peaks by allowing its users to run multiple replicas of a single application while providing built-in scaling, health checks, and auto-healing mechanisms.

Briefly, Kubernetes works by having a description file that specifies how a container must work, the memory and CPU constraint of each container, how many replicas it should keep online, and what policy it should take to increase or decrease the number of instances. K8s also provide a mechanism (AKA deployment controller) to deploy new versions of the application while maintaining availability.

Beyond the complexity of this process, the steps described above do not work with stateful applications, out-of-the-box. Rather, stateful applications require special care for data management and scaling that wouldn’t be helped by simply running more instances.

Stateful Applications

Services can be classified as either stateless or stateful. Stateless services are those that do not lose any information if they are rebooted or crashed. In other words, you can easily increase or decrease the number of instances running within stateless services compared to instances running with a load balancer in front without having any service interruption or data loss. Examples of stateless services include proxy servers, web applications, or any application that consumes a database, for example.

Unlike stateless services, stateful services hold data that must not be lost. In addition to databases, which are a well-known type of stateful service, other types include user directories as well as messaging and authentication services. These services, however, can’t simply spin up new instances and work in a balanced way using a load balancer. First, each instance needs persistent storage that won’t be lost if the service shuts down. And second, each service has a policy on how to work in a distributed manner.

Kubernetes’ paradigm for handling persistent data is divided into three structures:

The Persistent Volume is set up by the cluster administrator to make it available for users just as nodes are available for computing. These volumes are managed by plugins like NFS, iSCSI, or cloud-provider solutions (such as Google, AWS, or Azure). A Persistent Volume claim is a user asking for persistent storage, much like asking for CPU resources for a container image. Claims don’t need to understand the underlying storage structure and are as generic as they can be.

Storage Classes are used for dynamic Persistent Volumes. If there is no Persistent Volume attending a claim, K8s will try to provision the storage claim based on Storage Classes-structures defined and named by the administrator that specify the constraints and resources used by each storage unit.

The stateful application itself has constraints that overtake the resources available on Kubernetes when it comes to distributed computing: They need to be engineered to handle multiple instances and fallbacks.

Master/slave architecture is the most common and also the easiest architecture to deploy in a distributed environment. The architecture dictates that only one instance receive read and write requests, while the slaves only keep the data updated. In some specific cases, slave instances are also allowed to serve read requests. If there is an outage with the master instance, one of the slaves will automatically assume the role of master.

In addition to the master/slave architecture, there are many others that provide more than one writeable node. However, the development of a high-performance stateful cluster must be carefully handled by an expert due to the increased complexity and the risk of having poorer performance than a single instance if badly configured.

Even with one-instance services, caution should be exercised. While it may be easy to upgrade/downgrade a service, one must take special care with stateful applications. Usually, once these applications are upgraded, the data stored on a Persistent Volume is changed. That change renders the original data unusable with an older version. This means if you need to roll back a version upgrade, the application may not be running as expected. The solution for this problem does not rely on K8s; it relies on how the application is designed to handle rollbacks.

Another well-known architecture is sharding, which will be discussed in greater detail in the next section.

Elasticsearch Deployment Overview

Elasticsearch (ES) is a special database focused on search and analytics. The database specializes in multi-indexing different data types, such as textual, geospatial, numerical, and unstructured data. The service also comes with some peripheral tools such as Kibana-developed to visualize analytics information-and Logstash — for parsing large batches of logs.

Distributed by design, Elasticsearch provides different ways to store data through replication while offering reliability and scalability. Keep in mind that the company behind Elasticsearch, Elastic, also provides a SaaS solution for Elastic Stack (Elasticsearch + Kibana + Logstash).

Elasticsearch has different moving parts that must be deployed to work reliably:

The topology in above image, features three main components which can be combined. But due to the ease of deployment with Kubernetes components, it is recommended to separate each into different computing units.

Client nodes are deployed as single deployments with Horizontal Autoscaler (if available). These nodes act as a reverse proxy to the entire cluster and may need to increase or decrease the number of nodes dynamic by request. Client nodes query master nodes to discover how data is distributed between each . Queries go to multiple data nodes and are consolidated in the requested client.

Master nodes are also published as deployments but can have a fixed number of instances: ideally three for high availability. Kubernetes will make sure there are always three instances running on different machines. A headless service will be deployed for autodiscovery. Master nodes hold meta-information about data nodes, such as location, sharding, and data replication.

Data nodes are where things get interesting. Deployed as StatefulSets, each instance has a Persistent Volume claim. If an instance goes down, Kubernetes ensures the recovered instance will be attached to the correct claim. Data nodes split the indices into shards and also replicate information between nodes, so that in cases of node data loss or node outage this doesn’t provoke a cluster data loss.

Elasticsearch Deployment

This section details how to easily deploy Elasticsearch with Kubernetes and check everything you’ve learned so far.

Before Beginning

The examples should work on any updated Kubernetes instance. The initial setup started with three nodes with, at least, 4 GB of RAM. Please follow your cloud provider’s instructions on how to enable Kubectl to your cluster ( Google Cloud , Microsoft Azure, , Digital Ocean ,).

Run the command to install the dashboard:

kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0-beta8/aio/deploy/recommended.yaml

Now, proxy the connection:

kubectl proxy

You should get a screen similar to the picture below:

Deploying Elasticsearch with Helm

Install Helm on your local machine and in Kubernetes cluster: https://helm.sh/docs/intro/install/

After installed, add Elastic Helm repository:

helm repo add elastic https://helm.elastic.co

Now, install Elastic by using the command:

helm install ${es_cluster_name} elastic/elasticsearch
helm install esarticle elastic/elasticsearch

You can follow the deployment steps by checking pod deployment:

kubectl get pods --namespace=default -l app=elasticsearch-master -w

This should return something similar to the image below:

NAME READY STATUS RESTARTS AGE elasticsearch-master-0 1/1 Running 0 2d elasticsearch-master-1 1/1 Running 0 2d elasticsearch-master-2 1/1 Running 0 2d

By default, the ES Helm Package creates three master nodes that act as client, master, and data nodes. Each replica will thus have its persistent volume claim:

kubectl get pv
NAME CAPACITY ACCESS MODES RECLAM POLICY STATUS CLAIM STORAGECLASS REASON AGE pvc-2b7f3e47-c2a0-4654-baed-94ad32b507c2 30g RWD Delete Bound default/elasticsearch-master-elasticsearch-master-0 gp2 2d pvc-9078604-elfe-4097-897b-ba8bf84af863 30g RWD Delete Bound default/elasticsearch-master-elasticsearch-master-2 gp2 2d pvc-b857fdie-b387-4b54-b144-00ed66045047 30g RWD Delete Bound default/elasticsearch-master-elasticsearch-master-1 gp2 2d

Helm also created a service with a load balancer, so it is possible to connect to the ElasticSearch service using the load balancer:

kubectl get service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE elasticsearch-master ClusterIP 10.245.80.223 <none> 9200/TCP,9300/TCP 2d elasticsearch-master-headless ClusterIP None <none> 9200/TCP,9300/TCP 2d

Deploying Kibana with Helm

is a graphical user interface for Elasticsearch designed to query data and create (near real-time) dashboards. There is also a Helm chart, so you can easily install it. Since you’ve already added the Elastic Helm repository into your local machine, type the following command:

helm install kibanaesarticle elastic/kibana

Note that you will probably need a new node on your Kubernetes cluster. Each ES master instance has received most of the previous Kubernetes node resources, and K8s cannot deploy Kibana. You can check if that’s the case by using the Kubernetes dashboard.

Once everything is deployed, you can check the service by creating a reverse proxy:

kubectl port-forward svc/kibanaesarticle-kibana 5601:5601

Bottom Line — Stateful Applications Can Work Well With Kubnernetes

Most posts about Kubernetes and container orchestrators detail how to deploy stateless and network bound services into the cluster. These problems, although complex, are very well documented on Kubernetes nowadays.

Stateful applications are applications that require a reliable unit of persistent storage: Data losses are not an option. Besides that, these applications are much more difficult to distribute and scale. New instances must be carefully synchronized with existing ones. The shutting down of instances also needs to be carefully orchestrated with other instances of the same service to avoid data loss.

Issues also arise when upgrading or downgrading stateful applications. Version upgrades often lead to data structure changes, so a rollback plan should be tested and in place prior to production.

Kubernetes is the most popular container orchestrator available. It provides effective solutions for increasing application reliability by creating fallbacks and continuously monitoring running applications, among other things. Frequently, stateful applications have not been taken into consideration for orchestrators. Generally, these products have an entire team dedicated to operating and maintaining its inner parts. With Persistent Volumes, however, Kubernetes shows that it is possible to achieve the level of customization stateful applications require without losing the many advantages an orchestration system offers.

Originally published at https://spot.io on March 18, 2020.