Elasticsearch Cluster Migration Across Data centers
Downtime is no longer acceptable!
Public Cloud vs. Data Center is an all time debate leading to companies migrate between one another. In such migrations, moving things with Zero downtime is a key responsibility. Here I will discuss various strategies of migrating Elasticsearch cluster across Data centers.
Data Migration Strategies-
Every cluster migration can be done either with Downtime or Zero Downtime approach. Some require application changes, some require additional consumers and some require just property file changes. I will share detailed steps of Zero Downtime approach with no application changes along with overview of other approaches as well.
Let’s say we need to migrate cluster from AWS to data center, we will call them DC1 and DC2 resp. And C1,C2 for cluster running in DC1,DC2 resp, where C1 is LIVE cluster with all the data and C2 is new blank cluster.
Strategy with Downtime:
Approach1 — Snapshot and Restore (Full+Incremental)
This is the most common and well documented approach by Elastic
https://www.elastic.co/guide/en/elasticsearch/reference/7.6/snapshot-restore.html
Step-1: Register a snapshot repository
Step-2: Take a snapshot of indices
Step-3: Restore indices from a snapshot
Step-4: Monitor snapshot and restore progress
Step-5: Stop Application flow of Elasticsearch
Step-6: Repeat step 2, this will automatically take incremental snapshot from last full snapshot.
Step-7: Repeat step 3 & 4
Step-8: Once restoration is done, enable traffic on application.
Pros:
1. Easiest approach of migrating data.
2. Activity duration time is less.
Cons:
1. Application downtime required.
Strategy without Downtime:
Approach 1 — Extended Cluster with Rolling Node Termination
This approach consists of action items on Elasticsearch cluster only with just IP change in application’s property and no other code changes.
Step-1: Stop Shard rebalancing in C1, this is to ensure no shard movement is happening due to auto rebalancing nature of ES until activity is completed.
##C1 clustersudo curl -X PUT $(hostname):9200/_cluster/settings -H 'Content-Type: application/json' -d'
{
"transient": {
"cluster.routing.rebalance.enable": "none"
}
}'
Step-2: Ensure cluster.name is same in both C1 and C2
Step-3: If C2 cluster has already been setup, stop Elasticsearch process on all nodes of C2.
Step-4: Add C1N1 IP in C2N1 discovery.zen.ping.unicast.hosts and start Elasticsearch service. It will became part of C1 and now cluster size is 7 nodes. No change in C1 cluster required.
I have taken C1N1 here considering it as master_node, I would suggest you to get master_node of your C1 cluster and put its IP. This is not mandatory but my recommendation because this added node would be terminated at the end so would avoid multiple master election.
##C2 cluster
#cat /etc/elasticsearch/elasticsearch.ymlcluster.name: mediumnode.name: C2_Nodename1node.attr.rack: rack1path.data: /data/elasticsearch
path.logs: /var/log/elasticsearch
path.repo: /data/elasticsearch/backupbootstrap.memory_lock: true
network.host: C2N1
http.port: 9200discovery.zen.ping.unicast.hosts: ['C2N1', 'C2N2', 'C2N3', 'C2N4', 'C2N5', 'C2N6', 'C1N1']discovery.zen.minimum_master_nodes: 4
gateway.recover_after_nodes: 2action.destructive_requires_name: true
cluster.routing.allocation.awareness.attributes: rack#systemctl start elasticsearch
We just need to add single IP of C1 in single node of C2, rest of the C2 nodes have IPs of each other, hence they will be able to communicate with all and become part of combined cluster. To deeply understand how this happens, go through -
##C1 cluster
#sudo curl $(hostname):9200/_cat/nodesC1N1 65 100 2 2.50 2.08 1.80 mdi * C1_Nodename1
C1N2 56 95 2 1.25 1.53 1.64 mdi - C1_Nodename2
C1N3 70 100 2 1.72 1.79 1.77 mdi - C1_Nodename3
C1N4 62 99 2 1.44 1.62 1.57 mdi - C1_Nodename4
C1N5 67 97 2 1.26 1.48 1.53 mdi - C1_Nodename5
C1N6 72 100 2 1.35 1.87 2.10 mdi - C1_Nodename6
C2N1 23 70 1 0.30 1.01 1.14 mdi - C2_Nodename1
Step-5: Simply start Elasticsearch service on rest of the C2 nodes one by one. They will all become part of one cluster and we will end up having extended cluster or C1C2 cluster having 12 nodes.
Step-6: Change default values of concurrent shard recoveries and limit of inbound/outbound node recovery traffic according to your cluster size and utilization.
##C1 clustersudo curl -X PUT $(hostname):9200/_cluster/settings -H 'Content-Type: application/json' -d'
{
"transient": {
"cluster.routing.allocation.node_concurrent_recoveries": "20",
"indices.recovery.max_bytes_per_sec": "200mb"
}
}'
Step-7: Next step is to begin removal of C1 nodes from extended C1C2 cluster. Before you commence the process of removing a node, exclude that IP using cluster level shard allocation filtering, it will automatically move shards data present on it to rest of the nodes without impacting state of the cluster (even if you have replication 0).
Keep monitoring relocating_shards in cluster health and once all the shards have been reallocated you can shutdown the node.
curl -X PUT "$(hostname):9200/_cluster/settings?pretty" -H 'Content-Type: application/json' -d'
{
"transient": {
"cluster.routing.allocation.exclude._ip": "C1N2"
}
}'Why didn’t we start with C1N1 exclusion?
Considering C1N1 is master_node, we don’t want unnecessary master election multiple times in whole activity. If we would stop C1N1, then new master would be elected from rest of the 11 nodes (5old+6new). If it happens to be one of 5 old nodes then reelection will again happen on its termination.
Step-8: Continue step-6 for rest of the C1 nodes one by one except C1N1.
Step-9: Change application pointing to all new C2 nodes only.
Step-10: Continue Step-6 for remaining C1N1 node.
Step-11: Now complete data movement has been done to new cluster C2, all C1 nodes have been shutdown and cluster size has become to original number 6 again. But there is still one C1N1 IP present in C2N1 node (added in step-4).
Reset shard allocation IP filtering and disable cluster routing allocation to avoid shard movement while restarting of C2N1
sudo curl -X PUT $(hostname):9200/_cluster/settings -H 'Content-Type: application/json' -d'
{
"transient": {
"cluster.routing.allocation.exclude._ip": null,
"cluster.routing.allocation.enable": "none"
}
}'Step-12: Remove C1N1 from C2N1 added in step-4 and restart Elasticsearch service.
#cat /etc/elasticsearch/elasticsearch.ymlcluster.name: mediumnode.name: C2_Nodename1node.attr.rack: rack1path.data: /data/elasticsearch
path.logs: /var/log/elasticsearch
path.repo: /data/elasticsearch/backupbootstrap.memory_lock: true
network.host: C2N1
http.port: 9200discovery.zen.ping.unicast.hosts: ['C2N1', 'C2N2', 'C2N3', 'C2N4', 'C2N5', 'C2N6']discovery.zen.minimum_master_nodes: 4
gateway.recover_after_nodes: 2action.destructive_requires_name: true
cluster.routing.allocation.awareness.attributes: rack#systemctl restart elasticsearch
Step-13: Enable cluster routing allocation and shard rebalancing and change node_concurrent_recoveries to default one.
sudo curl -X PUT $(hostname):9200/_cluster/settings -H 'Content-Type: application/json' -d'
{
"transient": {
"cluster.routing.*": null,
"indices.recovery.max_bytes_per_sec": null
}
}'Pros:
1. No code change required.
2. No additional cost of infra or effort.
Cons:
1. Time consuming.
2. High latency. (It is recommended to perform the activity in off business hours to reduce the impact.)
3. Unreliable connectivity.
Approach 2 — Extended Cluster with Shard Rerouting
This approach is similar to the previous one, except movement of shards. In the previous approach, movement was automatic and randomly distributed to the remaining nodes, whereas in this approach, it will be a controlled movement.
Step-1: Disable cluster routing allocation to avoid auto shard movement during the activity.
sudo curl -X PUT $(hostname):9200/_cluster/settings -H 'Content-Type: application/json' -d'
{
"transient": {
"cluster.routing.allocation.enable": "none"
}
}'Next 6 steps are similar to first 6 steps of previous approach, ie step-1 to step-6
Step-8: Use cluster reroute API to manually move shards to the new nodes. Shards selection is completely your choice, if you want to maintain same state of cluster then you can do node by node otherwise can be picked up randomly.
You can also write small shell script to move multiple shards in one go.
curl -X POST $(hostname):92000/_cluster/reroute?pretty -H 'Content-Type: application/json' -d'
{
"commands" : [
{
"move" : {
"index" : "medium", "shard" : 0,
"from_node" : "C1N1", "to_node" : "C2N1"
}
}
]
}'Step-9: Change application pointing to all new C2 nodes only.
Step-10: Shutdown all C1 nodes one by one.
Step-11: Remove C1N1 from C2N1 discovery.zen.ping.unicast.hosts and restart Elasticsearch service.
Step-12: Enable cluster routing allocation and shard rebalancing.
Pros:
1. No code change required.
2. No additional cost of infra or effort.
Cons:
1. Time consuming.
2. High latency. (It is recommended to perform the activity in off business hours to reduce the impact.)
3. Unreliable connectivity.
Approach 3 — Data Queuing and Consumers
If you are queuing data before pushing to the ES cluster, then this approach can be an another option.
Step-1: Write an another consumer to read from queue and write in C2.
Step-2: Take Full Snapshot of indices in C1 and Restore it in C2 using Approach1 — Snapshot and Restore
Step-3: Once Restoration is completed, stop consumer which was writing data in C1.
Cons:
1. Additional effort of consumer for DC2
2. Additional resource requirement for consumers.
Approach 4 — Application writing in both ES Clusters
This is also a Zero downtime approach but requires change in application code.
Step–1: Application to start writing data in both C1 and C2.
Step-2: Take Full Snapshot of indices in C1 and Restore it in C2 using Approach1 — Snapshot and Restore
Step-3: Once Restoration is completed, change in application to stop writing in C1 and just point to C2.
Cons:
1. Change in application required to write data in both ES clusters.
Feel free to reach out to me at sanchitbansal26@gmail.com for any queries.
Also share your views if you have any better approach.
