Automating with Ansible: Aurora RDS Clusters

Managing Amazon’s fully-managed relational database service

In this article we will use Ansible to automate the configuration of Amazon Aurora managed databases. If you’re not sure what you’re doing here, maybe peek at the introduction, and take note that the automation here builds in part on what was built in a previous article about building a VPC. The scope of the automation will handily build the following

• Subnet and cluster parameter group: define the VPC subnets that our databases will live in, and a default MySQL 5.7 parameter group for our cluster to use
• Aurora cluster: configure an Aurora cluster, with settings provided by Ansible configuration
• Route 53 DNS entry for the cluster: configure a more friendly CNAME record that will point to our new cluster endpoint
• Aurora database instances: populate our Aurora cluster with database instances, with settings provided by Ansible configuration

Take note this article is not aiming to explain how these AWS resources work, but specifically how to automate their configuration with Ansible. Let’s get started!

No module? No problem

Ansible has loads of modules for handling the AWS cloud, including one for managing RDS databases, but unfortunately this module lacks support for Aurora database clusters. In the absence of a module to do the heavy lifting for us, we’re going to leverage using the Ansible commandmodule to call out to the AWS CLI and get our work done. When doing this, it is important to take the time to preserve that core principle of Ansible: idempotence.

Idempotency
An operation is idempotent if the result of performing it once is exactly the same as the result of performing it repeatedly without any intervening actions.

To give a simple example: the Ansible we write that creates our Aurora cluster using the AWS CLI, when we run it a second (or third, or fourth) time, it should have the smarts to work out it has already been created and avoid trying to create it again and crashing. We’ll achieve this by using the AWS CLI to first describe our cloud resources, then using Ansible’s powerful conditionals to optionally execute our resource creation tasks.

Subnet group and cluster parameter group

Before we can create an Aurora cluster we need two things:

1. Subnet group: defines the VPC subnets that our Aurora cluster instances can exist in
2. Cluster parameter group: defines the database (in our case, MySQL) settings that will be shared by all member instances of our cluster

When you create an Aurora cluster using the AWS Console, these things are automatically created for you behind the scenes. When we’re configuring these through the API, we need to take care of creating them ourselves. Luckily, they’re really easy, and don’t require much configuration. Some task files for creating them:

For the subnet group we can easily leverage the rds_subnet_group module. For the cluster parameter group Ansible is lacking module support (note: there is a module for parameter groups, just not cluster parameter groups), so we fall back to our tried-and-true method (query CLI, parse CLI response, conditionally create using CLI) to create a default Aurora MySQL 5.7 parameter group. Easy.

Aurora cluster

In creating our Aurora cluster we’re also lacking Ansible module support, and so we need to leverage the AWS CLI here too. Similarly to the subnet group module, the rds Ansible module does exist for administering RDS database instances, but lacks support for Aurora clusters.

The task list here looks very similar to the task list we wrote for creating an Elasticsearch domain, following these steps:

• check for our existing cluster using the AWS CLI
• parse the output of our query for details of the existing cluster
• if the cluster doesn’t exist, create it, capturing the output
• parse the output of our creation for details of the new cluster
• update our Route 53 entry using the entrypoint specified in our cluster details (which might be existing or new, doesn’t matter)

Configuration for creating the cluster is provided by evaluating a template file.

Aurora cluster: hosts, groups, and playbook

In our Ansible inventory we are going to track an Aurora cluster as another logical host (a host that logically represents a piece of our stack but isn’t an actual host that we connect to and run commands against), that is a member of an aurora.cluster group.

We define a single example cluster host in our new aurora.cluster group, and ensure it is added as a sub-group of the high-level project.ansibled group.

The aurora.cluster group defines the default connection port for the cluster, and specifies the default security groups, availability zones, and subnets for our clusters:

Our example cluster.aurora.ansibled host fills out the other, more specific variables that are required: cluster name, domain name, and master user credentials (sensitive information encrypted with Ansible Vault):

The playbook is what we’ve seen before: tie together the different task lists we’ve created, and provide credentials for the AWS CLI in the environment. Note that in our aurora.cluster group definitions we’re again using our “vpc facts”, vpc_security_group_ids and vpc_subnet_ids, so we ensure the task list for collecting those facts is included in our playbook before we go to work:

Next, we will extend our Ansible here to populate our Aurora cluster with database instances. Stay with me!

Aurora database instances

Task list to create an Aurora database instance is just like so many other things at this point: look for it using the AWS CLI, and create it if necessary, passing configuration from a template.

Aurora database instances: hosts, groups, and playbook

Aurora database instances are some more logical hosts, defined by anaurora.db group. Two database instances are defined for our cluster, in zones a and b, linked to our cluster by the aurora_cluster_name variable. We define that we want our database in zone a to be the primary one by defining a lower aurora_promotion_tier to it. Notably, our aurora.db group defines no group vars.

To integrate creating our Aurora databases into our playbook, we simply add a second play, that runs for all aurora.db hosts, that executes our task list:

Easy! Thanks again for reading!

I love automation. This series of articles, Automation with Ansible, is the documenting of some of the Ansible bits ‘n bobs that make my life easier when managing software infrastructure.

Check it all out on GitHub, or the other entries in the series:

• Automation with Ansible: Introduction
• Automation with Ansible: Building a VPC
• Automation with Ansible: AWS Elasticsearch
• Automation with Ansible: Aurora Clusters (this article)
• Automation with Ansible: Logentries

tomwwright/ansibled