Managing Elasticsearch at scale at PhonePe — Part 2
In the previous article, I talked briefly about the architecture of our Elasticsearch and various strategies we have used to scale out cluster to handle millions of writes — https://medium.com/@nitishgoyal13/managing-elasticsearch-at-scale-at-phonepe-part-1-d1b58bc1c76c
In this article, I will talk about our cluster configuration and cluster settings along with performance results of various load tests
Most of the PhonePe infrastructure is hosted on self managed data centre. We run our Elasticsearch data nodes on Intel Servers
Typical BareMetal configuration
Cluster Configuration
DataNode Configuration
Over the years, we have changed our data node configuration as below
Elasticsearch on BareMetal → 2 VMs per BareMetal → 4 VMs per BareMetal
Data nodes are provisioned as 4 VMs on a dedicated BareMetal. So, these 800+ data nodes are provisioned atop 200+ BareMetal
In the next section of the article, I will talk about various performance comparisons and issues we have encountered over the years
1. Authorisation performance issue with larger number of shards in the cluster
We encountered a major bug in Elasticsearch 7.10.2 version related to Authorisation which impacted the write throughput of the cluster. The impact was more severe when we had more indices in the cluster. For clusters having indices in single digits, impact was minimal.
You can see in below chart, the throughput drop was too steep in Elasticsearch 7.10.2
More details about the bug can be found here — https://github.com/elastic/elasticsearch/issues/67987
2. Performance comparison across various Elasticsearch versions
As per our load tests, we are seeing higher throughput in Elasticsearch 8.9.0 as compared to Elasticsearch 7.x versions. Will publish a detailed article at later point in time with full RCA of this performance difference
Below is the CPU utilisation graph of our 2 clusters with exactly the same workload but different Elasticsearch versions
3. Cluster configurations and their impact on throughput
4. Experiment configurations which didn’t yield positive results
i) Running Elasticsearch on 4 VMs per BareMetal vs Running Elasticsearch as 4 different processes without VM on the BareMetal.
As per out load tests, VM based elasticsearch performance was better than process based elasticsearch. Unfortunately, we weren’t able to spend a lot of time to debug the reasoning for the same.
In the next part of this article, we will talk more about our algorithm which helps us decide on which fields we should enable indexing and doc_values — https://medium.com/@vashusharma180707/managing-elasticsearch-at-scale-at-phonepe-part-3-3bff56c9fb58
