Attaching a persistent EBS volume to a self-healing instance with Ansible

Sometimes, you don’t want to start from scratch when an instance is terminated and comes back up.

Take, for example, setting up a private package registry — once your packages have been built and are stored there, you don’t want to lose them, so you store everything on an attached drive. EFS is easier to manage, but costs more. EBS volumes can present a problem, however, if your instance is also in an auto-scaling group (set to one desired/one minimum/one maximum) for the sake of self-healing. There’s no way, using Cloudformation alone, to create a volume, have it not delete on instance termination, and have that same volume attach to the instance that takes its place.

Luckily, that’s where Ansible comes in.

Cloudformation

This time, I’m setting up a different registry, Nexus Sonatype. I have a Cloudformation template that sets up all the infrastructure — the load balancer, security group, autoscaling group, and the instance permissions. I have a few parameters, but most importantly, VolumeSize.

In the user data, I install the basics to get set up — git, ansible and boto3. The Ansible playbook and tasks/files/handlers are in their own git repository, which gets checked out straight on the instance, and is run like this:

- Fn::Sub:
- '/usr/local/bin/ansible-playbook --extra-vars "volumesize=${VolumeSize}" -i "localhost," -c local /home/ec2-user/temp/nexus/playbook/nexus-playbook.yaml'
- { VolumeSize: !Ref VolumeSize }

This passes the volume size onto Ansible, which takes care of the the rest.

Ansible

The playbook is relatively straightforward, with the structure like so:

- nexus-playbook.yaml
- files/
-- nexus-nginx.conf
-- nexus.rc
-- nexus.vmoptions
-- nginx-main.conf
- tasks/
-- existing_volume.yaml
-- first_time_installation.yaml
-- main.yaml
- handlers/
-- main.yaml

First, it runs tasks/main.yaml, which has the initial instance setup (installing more packages, creating the nexus user/group, etc). Then, it captures the instance ID and searches for EBS volumes tagged app:nexus, registering these values to aws_instance_idand existing_volume respectively. Finally, there are two conditional include_tasks statements — if existing_volumes.volumes does not have a value, it runs first_time_installation.yaml, which will create an EBS volume, tag it app:nexus, name it nexus_volume, set it not to delete on termination, and proceed with the nexus and nginx (for proxying) installation/first-time setup. If it does find a volume with the tag, it runs existing_volume.yaml, which simply attaches the volume and starts up the app.

Here’s how it all comes together:

The playbook

nexus-stack.yaml:

---
- hosts: 127.0.0.1
remote_user: ec2-user
become: true
tasks:
- import_tasks: tasks/main.yaml
- name: get instance id
uri:
url: http://169.254.169.254/latest/meta-data/instance-id
return_content: yes
register: aws_instance_id
- name: search for existing volume
ec2_vol_facts:
filters:
"tag:app": "nexus"
region: us-west-2
register: existing_volume
- include_tasks: tasks/first_time_installation.yaml
when: not existing_volume.volumes
- include_tasks: tasks/existing_volume.yaml
when: existing_volume.volumes
handlers:
- import_tasks: handlers/main.yaml

Creating the volume (from first_time_installation.yaml):

- name: create ebs volume
ec2_vol:
region: us-west-2
instance: "{{ aws_instance_id.content }}"
name: nexus_volume
volume_size: "{{ volumesize }}"
device_name: /dev/sdb
delete_on_termination: no
tags:
app: nexus

The ec2_vol module requires the instance ID to create and attach the volume, which is why it needed to be gotten and saved in the main playbook before.

Attaching an existing volume (from existing_volume.yaml):

- name: attach existing ebs volume
ec2_vol:
region: us-west-2
instance: "{{ aws_instance_id.content }}"
name: nexus_volume
device_name: /dev/sdb

Interestingly, when attaching an existing volume to an instance with the ec2_volmodule, you can’t feed it the volume ID — it goes by name. This is why the volume is given a name when it’s created — otherwise, it would have created a brand new volume and called it nexus_volume.It would not, however, have tagged the volume, so it wouldn’t be found the next time the playbook was run, and would have run the first_time_installationagain.

Limitations

Unfortunately, this particular method will only work if your autoscaling group will remain at one desired/one minimum/one maximum instance (though going down to zero and coming back up to one would work too, if you ever needed to, I guess). Once you start scaling to two or more instances, there’s an extremely good chance you’ll run into problems with new instances trying to attach unavailable volumes (unless that can be filtered for), or end up with a bunch of extra EBS volumes that never get terminated if you scale up once and then never again.


There you have it! An instance in an autoscaling group with a persistent EBS volume. Hopefully that makes sense — let me know if there’s anything I can clarify, or if you’ve any neat tricks of your own.

Like what you read? Give Eva Gonciarczyk a round of applause.

From a quick cheer to a standing ovation, clap to show how much you enjoyed this story.