Ansible and the AWS CLI: No module, no problem

Tips on integrating the AWS CLI when Ansible modules are letting you down

Tom Wright
Oct 1, 2018 · 3 min read

If you have picked up Ansible as a tool for managing your AWS cloud environments, then I know how it’s going. Things are going great. Ansible’s rich library of modules for AWS (159 at last count) is enabling you to bash out playbooks for bits of your stack at an alarming rate: EC2, DynamoDB, S3, Route 53, you’ve got it all. You are swimming in idempotent automation that makes your job a breeze. Life is good.

A quick example Ansible playbook to illustrate the idempotency of the S3 module
Lots of Ansible modules are idempotent, able to handle running from different states

That is, until you need to build something you don’t have a module for. For example: your team is building a new service that leverages Aurora clusters. Time for some more automation. You pull up your trusty list of Ansible modules… Hmm. Nothing.

(disclaimer: the rds module does support creating instances on the Aurora engine, but it is a bit spotty and there isn’t a module for the clusters themselves yet is the point here)

This doesn’t have to be a showstopper. One of the great strengths of AWS is being able to control things through the AWS CLI, and one of the great things about Ansible is how flexible you can be with things like command to execute shell commands. So we’ll just invoke the AWS CLI from Ansible and create an Aurora cluster that way:

A quick example Ansible playbook for creating an Aurora Cluster with the AWS CLI

It’s nice — it means we can still leverage Ansible’s configuration management and templating to keep things neat. But, it doesn’t end up quite as nice as usual. I mean, it does crash if the cluster already exists, because it isn’t idempotent. 🙅

Yeah, nice — AWS CLI throws us an error if the cluster already exists so the whole thing blows up

We can, however, write a little more Ansible code around the edges of our usage of the AWS CLI to make it more robust and more idempotent.

Here’s what we can to do to soup it up a little:

  • first, list all our Aurora clusters with the AWS CLI, storing the result using register
  • take advantage of the fact AWS CLI communicates in JSON to use Ansible’s from_json and json_query filters to check out the result and work out if the Aurora cluster already exists
  • use Ansible’s when to conditionally run the AWS CLI using command to create the cluster if it doesn’t exist
Improving our Ansible playbook with a lookup to make it idempotent
Happy and idempotent — successive runs of our Ansible’d AWS CLI calls don’t crash now

Nice, things are looking much better and our Aurora cluster creation is idempotent like the rest of our Ansible.

This list-parse-create logic works great, but it is important to understand the differences in filtering between some AWS APIs, as it affects whether your Ansible can just ask AWS specifically about the object you want, or if you’ll need to list them all and filter yourself. This solution is not one size fits all! Some APIs have no filtering implemented, some APIs will happily return an empty set when filtering, some will throw an error when filtered if nothing can be found!

I think it is easiest to just illustrate with some examples, see below :)

Happy automating!
Tom

I love automation. This series of articles, Automation with Ansible, is the documenting of some of the Ansible bits ‘n bobs that make my professional life easier when managing software infrastructure.

Check it all out on GitHub, or the other entries in the series:

Energetiq

Find out more about what our tech heads get up to at…

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store