Ansible and the AWS CLI: No module, no problem

Tips on integrating the AWS CLI when Ansible modules are letting you down

If you have picked up Ansible as a tool for managing your AWS cloud environments, then I know how it’s going. Things are going great. Ansible’s rich library of modules for AWS (159 at last count) is enabling you to bash out playbooks for bits of your stack at an alarming rate: EC2, DynamoDB, S3, Route 53, you’ve got it all. You are swimming in idempotent automation that makes your job a breeze. Life is good.

A quick example Ansible playbook to illustrate the idempotency of the S3 module

Lots of Ansible modules are idempotent, able to handle running from different states

The cracks start to show

That is, until you need to build something you don’t have a module for. For example: your team is building a new service that leverages Aurora clusters. Time for some more automation. You pull up your trusty list of Ansible modules… Hmm. Nothing.

(disclaimer: the rds module does support creating instances on the Aurora engine, but it is a bit spotty and there isn’t a module for the clusters themselves yet is the point here)

This doesn’t have to be a showstopper. One of the great strengths of AWS is being able to control things through the AWS CLI, and one of the great things about Ansible is how flexible you can be with things like command to execute shell commands. So we’ll just invoke the AWS CLI from Ansible and create an Aurora cluster that way:

A quick example Ansible playbook for creating an Aurora Cluster with the AWS CLI

It’s nice — it means we can still leverage Ansible’s configuration management and templating to keep things neat. But, it doesn’t end up quite as nice as usual. I mean, it does crash if the cluster already exists, because it isn’t idempotent. 🙅

Yeah, nice — AWS CLI throws us an error if the cluster already exists so the whole thing blows up

We can, however, write a little more Ansible code around the edges of our usage of the AWS CLI to make it more robust and more idempotent.

Filling in the gaps

Here’s what we can to do to soup it up a little:

• first, list all our Aurora clusters with the AWS CLI, storing the result using register
• take advantage of the fact AWS CLI communicates in JSON to use Ansible’s from_json and json_query filters to check out the result and work out if the Aurora cluster already exists
• use Ansible’s when to conditionally run the AWS CLI using command to create the cluster if it doesn’t exist

Improving our Ansible playbook with a lookup to make it idempotent

Happy and idempotent — successive runs of our Ansible’d AWS CLI calls don’t crash now

Nice, things are looking much better and our Aurora cluster creation is idempotent like the rest of our Ansible.

A note on filtering

This list-parse-create logic works great, but it is important to understand the differences in filtering between some AWS APIs, as it affects whether your Ansible can just ask AWS specifically about the object you want, or if you’ll need to list them all and filter yourself. This solution is not one size fits all! Some APIs have no filtering implemented, some APIs will happily return an empty set when filtering, some will throw an error when filtered if nothing can be found!

I think it is easiest to just illustrate with some examples, see below :)

Happy automating!
Tom

I love automation. This series of articles, Automation with Ansible, is the documenting of some of the Ansible bits ‘n bobs that make my professional life easier when managing software infrastructure.

Check it all out on GitHub, or the other entries in the series:

• Automation with Ansible: Introduction
• Automation with Ansible: Building a VPC
• Automation with Ansible: AWS Elasticsearch
• Automation with Ansible: Aurora RDS Clusters
• Automation with Ansible: Logentries

tomwwright/ansibled