Deploying the ELK stack on Amazon ECS, Part 3: ElasticSearch
Igor Kantor
16610

Igor, Thank you very much for putting together all the instructions. I was able to setup the ES cluster on AWS. Only one thing that I had to do differently is to overide the new Launch Configuration Name directly on the Cloud Formation template as its not taking effect on ASG. If I do it on ASG, the Cloud Formation is overriding it. Anyway, that is the only difference.

In your proposed setup, every ES node, by default, acting as data, master and ingest. I m trying to configure differently by separating them as individual nodes — ex : 3 data ONLY, 2 master ONLY and one ingest ONLY node. To set that up, I created 3 different Task Defintions and corresponding ECS Service for each TD. I used the environment variables to control the node behaviour using these threee variables IS_MASTER,IS_DATA and IS_INGEST. I started the Service with Data Nodes and cluster came up with 3 nodes perfectly fine. Then I started the master Service but thats not able to join the cluster. For Data Node Service, I provided explict port mappings 9200 →9200 and 9300 → 9300 and for master I configured dynamic port mapping to avoid the port conflicts 0 → 9300 as both master, data and ingest run on same machine. Not sure if that is the culprit. Anyway, Have u ever come across this setup. can you shed some light what might be happening?

Error : “failed to send join request to master reason [RemoteTransportException[[jVb1EHy][172.17.0.2:9300][internal:discovery/zen/join]]; nested:“

elasticsearch.yml

cluster.name: "stage-elasticsearch"

bootstrap.memory_lock: false

network.host: 0.0.0.0

network.publish_host: _ec2:privateIp_

transport.publish_host: _ec2:privateIp_

discovery.zen.hosts_provider: ec2

discovery.ec2.tag.ElasticSearch: stage-esnode

discovery.ec2.endpoint: ec2.${REGION}.amazonaws.com

s3.client.default.endpoint: s3.${REGION}.amazonaws.com

cloud.node.auto_attributes: true

cluster.routing.allocation.awareness.attributes: aws_availability_zone

node.master: ${IS_MASTER:true}

node.data: ${IS_DATA:true}

node.ingest: ${IS_INGEST:true}