Ansible playbook — Creating an elasticsearch cluster on ubuntu nodes using the GCE discovery plugin

Setting up an elasticsearch engine cluster is easy. I wanted to make it even easier and that is why I made this quick Ansible playbook.

Before I start explaining about my playbook, let’s understand how Elastic and Google are helping us with the “Discovery Plugin”, well…

What is the GCE discovery plugin?

In order to create a cluster between a few nodes, the nodes need to “know” what other nodes are looking for a party, and we can manage it in a few ways. Today we are here to talk about one of the many ways, GCE Discovery Plugin.

Elastic has published a plugin for Google cloud servers. First, the plugin needs to be installed with the following command:

sudo bin/elasticsearch-plugin install discovery-gce

After that, we need to add the following lines to the elasticsearch.yml file (you can find it in /etc/elasticsearch if you did not make any changes during installation):

cloud:
  gce:
      project_id: <your-google-project-id>
      zone: <your-zone>
discovery:
      zen.hosts_provider: gce

• note that you can create a cluster with nodes even if they are in different zones, just use the following syntax:

zone: ["us-east1-b", "us-east1-c", "us-west2-a"]

Now that we understand how GCE discovery plugin works, let’s see how I used it in my Ansible-playbook for creating a cluster within one simple command.

Ansible is the simplest way to automate apps and IT infrastructure

This playbook will perform three steps:

1. Install elastic GPG-Key
2. Install Java
3. Install and config elasticsearch

In this particular post, I chose to talk about step number three and how I used Ansible variables and of course the GCE discovery plugin settings for this playbook. You can take a look at my playbook in Github. There are a few things we need to notice when configuring a cluster:

cluster.name:

I used the Ansible group name from Ansible’s inventory as a variable.

node.name:

I used Ansible’s hostname as a variable.

node.master:

I used “true” (not as a variable), but this setting may be different depending on your cluster.

node.data:

I used “true” (not as a variable), but this setting may be different depending on your cluster.

network.host:

[_local_, _site_] (not as a variable)

discovery.zen.minimum_master_nodes:

“Elastic” recommends to use this equation:

(master_eligible_nodes / 2) + 1

After dividing the master_eligible_nodes by two, you need to floor the result, for example, 3/2=1.5 (floor the number to 1). I have three nodes in my cluster and they are all set to:

node.master: true

In this playbook, I am using the equation on all of the nodes as per “Elastic”’s recommendation.

Now it is the part where the GCE discovery plugin needs to be set:

project_id:

I am using Google’s API in order to get the nodes project id.

zone:

I am using Google’s API in order to get the node’s zone.

Please notice that you need to have the following Google’s permission to be able to query Google’s API for “project_id” and “zone”:

--scopes=compute-rw

*Note that granting these permissions will allow your server to make changes in your GCE.

That’s it!

We covered the important parts in my playbook which you can clone from the git repository and play it over your ubuntu nodes.

Don’t hesitate to ask questions if you have any, I will be glad to assist!