Does Logstash uses X-PACK?

Hi there. Here i am, again. First, i need to tell you: containers and Elasticsearch works great. We are using Elastic’s Logstash and Elasticsearch containers since February and never had any kind of trouble, but, this Tuesday night(2017–05–02) wasn’t any given day.

I was working on call and was sleeping when i got an alert from Sensu at 5am saying that Redis had more than 1mi messages stored in a single queue.

I’ll tell you more about our infrastructure: we index more than 1bi messages per day (to be more precise we indexed 1025796647 documents (381GB) on 2017–05–03, in a cluster with 16 slaves machines). We are migrating two clusters from Elasticsearch 1.7.3 to 5.3.0. One is already migrated, the other one has:

  • 18 machines (2 masters and 14 slaves) running Elasticsearch 1.7.3
  • 10 machines (2 masters and 8 slaves) running Elasticsearch 5.3.0

Our Logstash’s config has two outputs: one send the message to the old cluster and the other one sends the message to the new cluster, so, Logstash must connect to both clusters at the same time.

We are taking our time to migrate because of Kibana. Our clients (other teams) are used to Kibana 3 and Kibana 4 and 5 are so disruptive that we need to teach them how to use it.

Ok, now that things are explained, let’s go to the main event. Tuesday night i received The queue alert. When i logged in, i saw that the messages were not being indexed. I mean, not even one message was being indexed, like if Logstash refused to word at all.

Taking a look at Logstash’s logs, this part caught my attention:

09:32:17.395 [Ruby-0-Thread-9: /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-6.2.6-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:222] WARN  logstash.outputs.elasticsearch - Attempted to resurrect connection to dead ES instance, but got an error. {:url=>#<URI::HTTP:0x2ecff50 URL:http://logstash_system:xxxxxx@localhost:9200/_xpack/monitoring/?system_id=logstash&system_api_version=2&interval=1s>, :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://logstash_system:xxxxxx@localhost:9200/][Manticore::SocketException] Connection refused (Connection refused)"}
09:32:17.839 [Ruby-0-Thread-17: /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-6.2.6-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:222] INFO logstash.outputs.elasticsearch - Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX:9200/, :path=>"/"}
09:32:17.843 [Ruby-0-Thread-17: /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-6.2.6-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:222] WARN logstash.outputs.elasticsearch - Attempted to resurrect connection to dead ES instance, but got an error. {:url=>#<URI::HTTP:0x2ddf77d8 URL:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX:9200/>, :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX:9200/][Manticore::ClientProtocolException] XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX:9200 failed to respond"}

This particular part:

URL:http://logstash_system:xxxxxx@localhost:9200/_xpack/monitoring/?system_id=logstash&system_api_version=2&interval=1s

So, now, Logstash uses X-PACK? How? When? What?!

So, i’ve been working with Logstash and the ELK Stack for the past 3 to 4 years and, i confess, i haven’t read Logstash’s documentation, not even once. This time, i had to. When reading the documentation, i’ve learned that some configurations in my Elasticsearch cluster is set wrongly and i must correct them. I’ll tell you about it in another post.

Don’t panic! I’ve already said that, if you can, you must use X-PACK. 
The company that i work for can’t afford it, but we wish we could. That being the case, we must disable it. According to this issue on Github, this is how to do it: put the following option in logstash.yml:

xpack.monitoring.enabled: false

But, what if this don’t solve your problem? In my case and i believe it should be yours, one of my clusters was red. After i assigned the shards that were unassigned, everything went smooth.

Lesson learned: always read the Docs.