Automation with Ansible
A real life introduction
Everyone loves automation. It saves time and allows us to be focused on more important stuff. Sometimes it’s plain lifesaving, Like when we need to patch all servers ASAP from a zero day vulnerability.
Ansible is self proclaimed “Simple IT Automation”. It can be used for various automation tasks, configuration management (like puppet of chef), app deployment or continuous delivery. It’s agentless (works over ssh), needs almost no software installed on managed hosts (basically a quite old version of python, almost all distros have it preinstalled) and DEAD SIMPLE. Why don’t we give it a try? Ansible “hello world” is a quite useless “ping” example, so I’ll try to show a useful real life Ansible usage examples. Let’s dive right in:
Monitoring is good, everyone loves monitoring. And if you are on the AWS cloud then Amazon provides some simple reporting scripts allowing the server to report to AWS CloudWatch with some server stats (disk, cpu, memory). But installing and setting up AWS cloudwatch reporting scripts can be a little tedious, Amazon supplies no simple “install and forget” package you can just “apt-get install’ and be over with. Looks like a good candidate to learn some Ansible with.
*Note: In order to follow this mini tutorial you should setup (or use an existing) remote cloud instance, you have ssh access to. You can follow the code here in this github gist
**Note this is an Ubuntu specific example but can be easily modified to other linux distros.
Install and setup (straight from the docs):
sudo apt-add-repository ppa:ansible/ansible
sudo apt-get update
sudo apt-get install ansible
mkdir automation && cd automation
touch ansible.cfg && hosts
mkdir -pv roles/aws_monitoring/tasks/main.ymlAnd setup a basic configuration (Change the apps data in the host file to fit your setup) in the project root folder with an ansible.cfg and hosts file
#The ansible.cfg
[defaults]
hostfile = hosts#The hosts file
[app_servers]
app1.example.com ansible_ssh_user=ubuntu
app2.example.com ansible_ssh_user=ubuntu
And using the following folder structure:
— roles
— aws_monitoring
— tasks
— main.yml
ansible.cfg
hosts
setup_aws_reporting.ymlThe magic is mostly done in the main.yml file inside the role. The playbook just calls various roles or tasks and for this example I’m using a playbook with one play and a play with one role so let’s take a look inside the role.
- name: Updating apt-cachesudo: yesapt: update_cache=yes cache_valid_time=3600- name: Installing perl pre requirements and unzipsudo: yesapt: pkg={{item}} state=installedwith_items:- libwww-perl- libdatetime-perl- libswitch-perl- unzip
- name: Checking if cloudwatch already Downloaded to this machinestat: path=~/aws-scripts-monregister: cloudwatch_folder- name: Downloading aws cloudwatch reporting scriptget_url: url=http://ec2-downloads.s3.amazonaws.com/cloudwatch-samples/CloudWatchMonitoringScripts-v1.1.0.zip dest=~/CloudWatchMonitoringScripts-v1.1.0.zipwhen: not cloudwatch_folder.stat.exists- name: Unzipping cloudwatchunarchive: src=/home/ubuntu/CloudWatchMonitoringScripts-v1.1.0.zip dest=/home/ubuntu/ copy=nowhen: not cloudwatch_folder.stat.exists- name: removing zipfile: path=/home/ubuntu/CloudWatchMonitoringScripts-v1.1.0 state=absent- name: setup crontab reporting to aws cloudcron: name="report aws" minute="1" job="~/aws-scripts-mon/mon-put-instance-data.pl --mem-util --mem-used --mem-avail --disk-path=/ --disk-path=/home --disk-space-avail --disk-space-used --disk-space-util >/dev/null 2>&1"
We can see that this file is actually a list of tasks, that would be run in that order when calling the role in a playbook. I chose to use a quite diverse selection of modules here, for example sake and I’ll try to break down in some detail what every task does:
- name: Updating apt-cachesudo: yesapt: update_cache=yes cache_valid_time=3600
The first task uses the Apt module and runs an `apt-get update` if it didn’t run in the given period of time. Notice we add the `sudo: yes ``` directive here, since apt-get must run as sudo.
- name: Installing perl pre requirements and unzipsudo: yesapt: pkg={{item}} state=installedwith_items:- libwww-perl- libdatetime-perl- libswitch-perl- unzip
The second task uses the apt module to install (or verify installation) of certain packages we need for the aws scripts (and some tasks in this role) to run. Notice the `with_items` directive that supplies a list of items that the apt directive would run against;. basically replacing the {{ item }} template in the item from ```with_items```. This is a very useful feature of ansible, allowing us DRY task files, instead of endless lists of the same apt-get install commands. Apt (or yum, depending on distro) would probably be one of your most used modules.
- name: Checking if cloudwatch already Downloaded to this machinestat: path=~/aws-scripts-monregister: cloudwatch_folder
Now we get to the stat module, the stat module doesn’t do anything but retrieves information on files and folders, information we can use in our tasks later. In this specific playbook I’ll use them in the `when clause` Notice we need to register this information into a variable, which we’ll use later.
- name: Downloading aws cloudwatch reporting scriptget_url: url=http://ec2-downloads.s3.amazonaws.com/cloudwatch-samples/CloudWatchMonitoringScripts-v1.1.0.zip dest=~/CloudWatchMonitoringScripts-v1.1.0.zipwhen: not cloudwatch_folder.stat.exists
In order to use the amazon supplied package we need to download it. For this we’ll use the get_url module that can handle downloads from ftp, http and https.
unarchive: src=/home/ubuntu/CloudWatchMonitoringScripts-v1.1.0.zip dest=/home/ubuntu/ copy=nowhen: not cloudwatch_folder.stat.exists
Now we need to unarchive the downloaded zip file. We’ll do that with the unarchive module (Notice we need unzip to be installed before for this to work..)
intermezzo: conditionals
Notice that the previous two tasks had a `when` directive. as the name suggests it expects a conditional (if you are familiar with python `if` statement you already know how to use it) and triggers (or not) the task according to the result. Here we use the variable with the stat results we registered earlier, using it to make the task idempotent: don’t run it if this specific state already exists.
And now back to the tasks in our role:
- name: removing zipfile: path=/home/ubuntu/CloudWatchMonitoringScripts-v1.1.0 state=absent
We don’t want to leave a mess in the home folder so we should remove the zip file. The file module can handle this with ```state=absent```, but also can do much more: handling everything from symlinks, group membership, creating files and directories and similar tasks.
- name: setup crontab reporting to aws cloudcron: name="report aws" minute="1" job="~/aws-scripts-mon/mon-put-instance-data.pl --mem-util --mem-used --mem-avail --disk-path=/ --disk-path=/home --disk-space-avail --disk-space-used --disk-space-util >/dev/null 2>&1"
Having the Aws scripts exists in some folder, without having a regular cron job to run them doesn’t help us much. So we use the cron module to handle this. The module can add a cron job to the crontab file. A small gotcha here: this module method of checking the task already exists is based on the name argument passed to the cron module and recorded as a comment in the crontab file, it doesn’t know how to detect it if the requested cron job already exists if set without the ansible cron module, so unless you check it in some other way and you have the job set up already by some other means then you’ll have this job twice.
Now It’s time to run this!
ansible-playbook setup_aws_reporting.yml -vNotice the -v flag I use to get a little more verbose output, it can get very verbose with -vvvv
The result would look something like this:
PLAY [app_servers] *********************************************************************GATHERING FACTS ***************************************************************ok: [app1.example.com]ok: [app2.example.com]TASK: [aws_monitoring | Updating apt-cache] ***********************************ok: [app1.example.com] => {“changed”: false}ok: [app2.example.com] => {“changed”: false}TASK: [aws_monitoring | Installing perl prerequisites and unzip] ************ok: [app1.example.com] => (item=libwww-perl,libdatetime-perl,libswitch-perl,unzip) => {“changed”: false, “item”: “libwww-perl,libdatetime-perl,libswitch-perl,unzip”}ok: [app2.example.com] => (item=libwww-perl,libdatetime-perl,libswitch-perl,unzip) => {“changed”: true, “item”: “libwww-perl,libdatetime-perl,libswitch-perl,unzip”}TASK: [aws_monitoring | Checking if cloudwatch already Downloaded to this machine] ***ok: [app1.example.com] => {“changed”: false, “stat”: {“atime”: 1435952439.932752, “ctime”: 1436168022.9624867, “dev”: 51713, “exists”: true, “gid”: 1000, “gr_name”: “ubuntu”, “inode”: 786480, “isblk”: false, “ischr”: false, “isdir”: true, “isfifo”: false, “isgid”: false, “islnk”: false, “isreg”: false, “issock”: false, “isuid”: false, “mode”: “0775”, “mtime”: 1436168022.9624867, “nlink”: 2, “path”: “/home/ubuntu/aws-scripts-mon”, “pw_name”: “ubuntu”, “rgrp”: true, “roth”: true, “rusr”: true, “size”: 4096, “uid”: 1000, “wgrp”: true, “woth”: false, “wusr”: true, “xgrp”: true, “xoth”: true, “xusr”: true}}ok: [app2.example.com] => {“changed”: false, “stat”: {“atime”: 1435952439.932752, “ctime”: 1436168022.9624867, “dev”: 51713, “exists”: false, “gid”: 1000, “gr_name”: “ubuntu”, “inode”: 786480, “isblk”: false, “ischr”: false, “isdir”: true, “isfifo”: false, “isgid”: false, “islnk”: false, “isreg”: false, “issock”: false, “isuid”: false, “mode”: “0775”, “mtime”: 1436168022.9624867, “nlink”: 2, “path”: “/home/ubuntu/aws-scripts-mon”, “pw_name”: “ubuntu”, “rgrp”: true, “roth”: true, “rusr”: true, “size”: 4096, “uid”: 1000, “wgrp”: true, “woth”: false, “wusr”: true, “xgrp”: true, “xoth”: true, “xusr”: true}}TASK: [aws_monitoring | Downloading aws cloudwatch reporting script] **********skipping: [app1.example.com]changed: [app2.example.com] => {“changed”: true, “checksum”: “a357935d5df0aeb283c3a9571f81afa2cb3ec78f”, “dest”: “/home/ubuntu/CloudWatchMonitoringScripts-v1.1.0.zip”, “gid”: 1000, “group”: “ubuntu”, “md5sum”: “f6c726b48e49fbba5c1e7b3f7486ac21”, “mode”: “0664”, “msg”: “OK (17627 bytes)”, “owner”: “ubuntu”, “sha256sum”: “”, “size”: 17627, “src”: “/tmp/tmpRFD2GY”, “state”: “file”, “uid”: 1000, “url”: “http://ec2-downloads.s3.amazonaws.com/cloudwatch-samples/CloudWatchMonitoringScripts-v1.1.0.zip"}TASK: [aws_monitoring | Unzipping cloudwatch] *********************************skipping: [app1.example.com]changed: [app2.example.com] => {“changed”: true, “check_results”: {“unarchived”: false}, “dest”: “/home/ubuntu/”, “extract_results”: {“cmd”: “/usr/bin/unzip -o \”/home/ubuntu/CloudWatchMonitoringScripts-v1.1.0.zip\” -d \”/home/ubuntu/\””, “err”: “”, “out”: “Archive: /home/ubuntu/CloudWatchMonitoringScripts-v1.1.0.zip\n extracting: /home/ubuntu/aws-scripts-mon/awscreds.template \n inflating: /home/ubuntu/aws-scripts-mon/CloudWatchClient.pm \n inflating: /home/ubuntu/aws-scripts-mon/LICENSE.txt \n inflating: /home/ubuntu/aws-scripts-mon/mon-get-instance-stats.pl \n inflating: /home/ubuntu/aws-scripts-mon/mon-put-instance-data.pl \n inflating: /home/ubuntu/aws-scripts-mon/NOTICE.txt \n”, “rc”: 0}, “gid”: 1000, “group”: “ubuntu”, “handler”: “ZipArchive”, “mode”: “0755”, “owner”: “ubuntu”, “size”: 4096, “src”: “/home/ubuntu/CloudWatchMonitoringScripts-v1.1.0.zip”, “state”: “directory”, “uid”: 1000}TASK: [aws_monitoring | removing zip] *****************************************ok: [app1.example.com] => {“changed”: false, “path”: “/home/ubuntu/CloudWatchMonitoringScripts-v1.1.0”, “state”: “absent”}ok: [app2.example.com] => {“changed”: true, “path”: “/home/ubuntu/CloudWatchMonitoringScripts-v1.1.0”, “state”: “absent”}TASK: [aws_monitoring | setup crontab reporting to aws cloud] *****************ok: [app1.example.com] => {“changed”: true, “jobs”: [“report aws”]}ok: [app2.example.com] => {“changed”: false, “jobs”: [“report aws”]}PLAY RECAP ********************************************************************app1.example.com : ok=7 changed=0 unreachable=0 failed=0app2.example.com : ok=7 changed=3 unreachable=0 failed=0
app2.example.com : ok=7 changed=3 unreachable=0 failed=0
Horrey! we did it, we’ve setup AWS reporting on out two app servers, (one of them already had it) in a fast (parallel) automated and reproducible way. Mission accomplished!
Beware the shell — A preemptive intervention
My first try with ansible was actually just using it as a remote shell scripts executer, almost all tasks were using the “shell” module to execute a shell command, which is quite logical since I started by migrating existing bash scripts to ansible. This proved to be an ansible anti pattern, and everytime I run a playbook ansible politely commented on that. Now, when I find myself using the ansible shell module, I consider it guilty until proved necessary. But why? There are two problems with using the shell module this way:
- Someone already done this: lot’s of shell stuff is already packed as ansible modules, cp, files, stat, unarchive, archive, etc.. So you can get the same results with less verbose code on your side and with a (probably) better implementation than your own hacky shell script.
- Idempotency: Most ansible modules are idempotent, they don’t do something that has already been done before, either by ansible or as given state. Apt won’t install an existing package (unless forced to), route53 won’t set up an existing dns record, etc. If you use shell scripts as tasks you’ll have to handle idempotency, checking existing state or risk having your task fail the second time you run them (The file has already been deleted!) or just take much longer then than actually needed.
- Security: The shell module loads the user shell, including environment etc. this is considered less secure, if you must run a linux cli tool, you can use the command module to run it without engaging the remote user shell (Notice you won’t have shell redirection tools, pipes | etc)
So resort to the shell script as a last resort, and even then try to use the command module if possible.
Where to go from here
Check out the official documentation, It’s quite good. don’t forget to take a look at the modules list, You don’t need to be too detailed, just enough to be aware that those exist. sometimes in the future when you try to script an complex ansible playbook you’ll be grateful for the existing core module, doing JUST the thing you were looking for.
If you need a more specific entry level tutorial you can check out one of Digital Ocean (Shout out to DO you are a great source for this kind of stuff!) excellent tutorials such as this one.
I hope this post would help you on your ongoing automation path and one last thing, this post is just a gentle introduction to Ansible. It’s a huge framework with lots of advanced features and it probably have you covered no matter what you are trying to do. So.. when you try to do something new that proves to be not trivial with your existing Ansible knowledge take a look at the docs, there just might be a better and documented way to solve the problem at hand.
I hope this post would help you on your ongoing automation path and one last thing, this post is just a gentle introduction to Ansible. It’s a huge framework with lots of advanced features, when you try to do something new that proves to be not trivial with your existing Ansible knowledge take a look at the docs, there just might be a better and documented way to do this.
