Automation with Ansible: AWS Elasticsearch Service
No-fuss AWS-managed Elastic clusters
In this article we look at using Ansible for automating the configuration of AWS-managed Elasticsearch clusters in Amazon’s Elasticsearch Service. If you’re not sure what you’re doing here, maybe peek at the introduction, and take note that the automation here builds in part on what was built in the previous article about building a VPC. The scope of the automation will handily build the following:configure
- IAM Role for AWS ES: a service-linked IAM role that AWS Elasticsearch Service requires to operate
- Elasticsearch cluster: configure the cluster itself, inside a VPC, with settings provided by Ansible configuration
- Route 53 DNS entry for the cluster: configure a more friendly CNAME record that will point to our new cluster endpoint
Take note this article is not aiming to explain how these AWS resources work, but specifically how to automate their configuration with Ansible. Let’s get started!
No module? No problem
Ansible has loads of modules for handling the AWS cloud, but unfortunately the AWS Elasticsearch Service isn’t one of them! In the absence of a module to do the heavy lifting for us, we’re going to leverage using the Ansible command module to call out to the AWS CLI and get our work done. When doing this, it is important to take the time to preserve that core principle of Ansible: idempotence.
Idempotency
An operation is idempotent if the result of performing it once is exactly the same as the result of performing it repeatedly without any intervening actions.
To give a simple example: the Ansible we write that creates an Elasticsearch cluster using the AWS CLI, when we run it a second (or third, or fourth) time, it should have the smarts to work out it has already been created and avoid trying to create it again and crashing. We’ll achieve this by using the AWS CLI to first describe our cloud resources, then using Ansible’s powerful conditionals to optionally execute our resource creation tasks.
Service-linked IAM Role
AWS Elasticsearch Service leverages service-linked roles to operate, so our starting point is to quickly get that set up. But again, no Ansible module here: just call the AWS CLI to list our roles, have a look if our role for Elasticsearch exists using set_fact, and create it with the AWS CLI if we need to. Easy stuff, very little configuration.
Elasticsearch domain
Now the meat of it: creating our Elasticsearch domain. There is a little bit in here, this is what it looks like at a high level:
- check if our Elasticsearch domain exists already (idempotence!)
- create the Elasticsearch cluster, if required
- poll the AWS CLI and wait for our Elasticsearch cluster’s endpoint to appear
- update the Route 53 entry for our Elasticsearch cluster
Waiting here for our Elasticsearch domain to become available gives us a great example of how we can use Ansible’s until to poll a command and wait for a certain condition before we continue. In this case, we poll hitting the AWS CLI and are looking for the Endpoints object to show up on the output for our domain. retries and delay gives us control on how often and for how long we perform this check before crashing out.
We parameterise the creation of our Elasticsearch domain using a JSON template, and do the same again for our access policy inside that configuration template:
Some basic templating here allows us to feed the bulk of the Elasticsearch configuration from our Ansible host configuration, which, as we’ll see later, will give us a flexible base for creating a variety of Elasticsearch domains.
Note that I will admit that the access policy defined here is very basic — any entity within the current AWS account can perform any action on the cluster. But the value here is illustrating feeding that access policy as a template; from here you can take that and run with it to create whatever access scheme you need!
Hosts and Groups
Additions to our hosts and groups are fairly straightforward:
- define an
elasticsearchgroup into which we will put logical hosts (hosts that logically represent a piece of infrastructure, as opposed to ones that we connect to and run commands) to represent our Elasticsearch domains - in our new
elasticsearchgroup, cover some of the variable definitions in our Elasticsearch tasks with some handy defaults - to our new
elasticsearchgroup, add two example hosts:big.elasticsearch.ansibleandsmall.elasticsearch.ansible - define host vars files for our two new hosts to define the rest of their configuration not covered by the
elasticsearchgroup
In creating our elasticsearch group, we also make sure we add it to the :children of our high-level project.ansibled group in order to inherit the high-level definitions there.
In defining our two example Elasticsearch hosts we try and illustrate some of the breadth available in the configuration:
big.elasticsearch.ansibled: 4r4.xlargedata nodes configured with 80gb storage, and 2m4.largededicated master nodessmall.elasticsearch.ansibled: 2r4.large(default size) data nodes with 20gb storage, and no dedicated masters
Our elasticsearch group fills in any gaps with defaults: Elasticsearch version, number of nodes, node type, and dedicated master nodes turned off.
Playbook
Finally, gluing it all together, a playbook:
If you have read my previous article about configuring a VPC with Ansible, you might recall we wrote some helpful “facts” tasks for our VPC that discovered and defined some useful facts for us, such as mappings from subnet and security group names to AWS identifiers. If you were extra sharp before, you might have noticed that in our Elasticsearch configuration we leveraged these mappings, specifically vpc_security_group_ids and vpc_subnet_ids. So as part of our playbook we include those VPC facts tasks, before we configure our elasticsearch hosts.
And with the playbook in place, that basically covers it! Thanks for reading!
I love automation. This series of articles, Automation with Ansible, is the documenting of some of the Ansible bits ‘n bobs that make my life easier when managing software infrastructure.
Check it all out on GitHub, or the other entries in the series:
- Automation with Ansible: Introduction
- Automation with Ansible: Building a VPC
- Automation with Ansible: AWS Elasticsearch Service (this article)
- Automation with Ansible: Aurora RDS Clusters
- Automation with Ansible: Logentries
