Configuring Elasticsearch Cross Cluster Search(CCS)
Hello everyone, in this article we will cover how to configure cross clusters in Elasticsearch. Before we dive deep into how to set up Cross Cluster in Elasticsearch, we’ll take a look at data Elasticsearch and its concepts.
What is Elasticsearch?
Elasticsearch is a highly scalable open-source full-text search and analytics engine built on Apache Lucene and developed in Java.
It allows you to store, search, and analyze big volumes of data quickly and in near real-time. It is generally used as the underlying engine/technology that powers applications that have complex search features and requirements.
It uses a documentation-based structure rather than tables and schemas. It has extensive REST APIs for storing or searching data. It is also the main component of the ELK(Elasticsearch +Logstash + Kibana) stack.
Its backend components include Cluster, Node, Shards and Replicas. ELK cluster is a group of one or more Elasticsearch nodes instances that are connected together.
What is Cross Cluster Search?
Cross-cluster search (CCS) allows you to configure multiple remote clusters across different regions and providers and enable unified search queries across all configured clusters. This remote connection between Elasticsearch clusters functions as a one-way traffic flow on port 9300.
Why do we need CCS?
By using CCS, you can easily visualize and manage your multi-region Elasticsearch clusters on a Single Central Kibana.
You can use a cross-cluster search to filter and analyze log data stored on clusters in different data centers.
Configuring CCS
To set up a Cross-Cluster search, you must fulfill the following prerequisites.
1. The local cluster must be defined.
The local cluster is the set used to initiate the cross-cluster search. In our example design, Cluster-US and Central Kibana are located in the same region. For this reason, master nodes of Cluster-US are defined in the Kibana Config file as follows.
server.port: 5601server.host: “kibana-ip-address”elasticsearch.hosts:[“http://master1_ip:9200","http://master_2_ip:9200","http://master_3_ip:9200"]
2. To perform a cross-cluster search, You must have at least one remote cluster.
To register a remote cluster, it is necessary to connect the local cluster to the nodes in the remote cluster using sniffing mode (default) or proxy mode.
2.1. Sniff mode, a cluster is created using 9300 port addresses of a name and node list. This mode requires the gateway node to have its broadcast addresses accessible to the local cluster.
2.2. Proxy mode, a cluster is created using a name and a single proxy address. Proxy mode does not require remote cluster nodes to have accessible broadcast addresses.
Open Central Kibana dashboard, then go to Management > Stack Management > Remote Clusters.
To set up a new remote, click Add a remote cluster. Give the cluster a unique name and define the seed nodes for cluster discovery.
Cluster-US defined a local cluster in Central Kibana is defined in sniff mode and click save button.
APAC and EMEA clusters are also defined by proxy mode and click save button.
You can see that all the regions are connected to the Central Kibana.
3. To create an index pattern to search across clusters, use the same syntax that you’d use in a raw cross-cluster search request in Elasticsearch: <cluster-names>:<pattern>.
Open Central Kibana dashboard then go to Management > Stack Management > Kibana > Index patterns > Create index pattern
You can view the index pattern you created in the discover section.
Today we completed the CCS configuration at Elasticsearch.
Thanks for reading! See you in my next post.
