TLDR Migrating a 130TB Cluster from Elasticsearch 2 to 5 in 20 Hours with 0 Downtime by Fred de Villamil

Replicate it using built-in ES capabilities, then reindex delta from your main data source. Also — it’s so cool to have everything saved in ~kafka~ a log.

77 nodes cluster, with 200TB storage, 4.8TB RAM, 2.4TB being allocated to Java, and 924 CPU core. It is made of 3 master nodes, 6 ingest nodes, and 68 data nodes. The cluster holds 1137 indices, with 13613 primary shards, and 1 replica in a second data center.

1 option — Cluster Restart: close every index, upgrade the software version, start the nodes, then open the indexes again. Downside — maintenance downtime =(

Better option — expand cluster 2x times, replicate everything 2 more times, split cluster into two, then catch-up with changes:

• added 90 new servers with the same ES 2.3.
• “number_of_replicas : 1” becomes “number_of_replicas : 3" and ES takes care of copying every index and shard onto new servers.

Transferring 130TB of data at up to 4Gb/s puts lots of pressure on the hardware. The load on most machines was up to 40, with 99% of the CPU in use. Iowait went from 0 to 60% on most of our servers.

• mitigate problems of serving clients from the busy hardware by using ES “zones” to split the data into cold and hot parts and dedicated some resources to serve the hot part to clients without reduced QoS.
• split the cluster: shutdown new servers, disconnect them from the cluster in terms of auto-discovery, start them separately:

• close all the indexes on the new cluster and now upgrade the ES to 5.0
• reopen indexes and catchup with changes by reindexing the delta from kafka source.
• switch ES clients to use new cluster.

Profit!

Read more about ElasticSearch troubles and monitoring on okmeter.io blog.