In depth guide to running Elasticsearch in production

The basics: Clusters, Nodes, Indices and Shards

A more in-depth look at Elasticsearch

Quorum

node.master: true

How nodes join the cluster

Segments and segment merging

Message routing

So how do I run Elasticsearch in production?

Sizing

RAM

CPU

Disk

  • Scheduler: cfq and deadline outperform noop. Kyber might be good if you have nvme but I have not tested it
  • QueueDepth: as high as possible
  • Readahead: yes, please
  • Raid chunk size: no impact
  • FS block size: no impact
  • FS type: XFS > ext4

Index layout

Shards

  • for write heavy workloads, primary shards = number of nodes
  • for read heavy workloads, primary shards * replication = number of nodes
  • more replicas = higher search performance
node_throughput*number_of_primary_shards
node_throughput*(number_of_primary_shards + number_of_replicas)

Size

number_of_indices = (140 * number_of_nodes) / (number_of_primary_shards * replication_factor)
index_size = (number_of_nodes * disk_size) / number_of_indices

Segment merging

Cluster layout

Monitoring

  • number of segments
  • heap usage
  • heap GC time
  • avg. search, index, merge time
  • IOPS
  • disk utilization

Conclusion

Resources

--

--

--

Devops / AWS Freelancer

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Announcement About Cat Star Level Upgrade & Arena Rewards

JSHeroes 2019 Transparency Report — Part 2 (follow-up)

Bad Memory? Let Stax Help!

How To Add A Double Jump To Your Character Controller

What will come next?

QuarkWorks: Meet the Team Series — Chao Shi

350% performance improvement in fetching data from the server, by using RxJava

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Mattis Haase

Mattis Haase

Devops / AWS Freelancer

More from Medium

WSO2 API Manager 4.0.0.0 installation in OpenShift Console

What Is DevOps And Its Key Principles

4 phases plan for DevOps Implementation for Advanced Projects

Java web app containerization and migration to Azure Kubernetes Service