Low-risk Package Updates with Ansible and vSphere

Keeping systems up-to-date is hard, but important. You want to ensure critical vulnerabilities are patches as soon as possible and that you keep up to date with bugfixes and the latest features, but don’t want to run the risk of breaking changes.

Conveniently, as we use Ansible for pretty much everything internally, it’s a doddle to upgrade all the installed packages on a system (admittedly, naively)

$ ansible all -m package -a 'name=* state=latest'

…or in playbook form:

- name: upgrade all packages
  package:
    name: *
    state: latest

This is fairly naive — it doesn’t do any verification that things are running smoothly afterwards, and there’s no reliable way to roll back a botched upgrade.

vSphere has a feature called snapshotting which allows you to take a point-in-time snapshot of a VM’s state and later restore it. We could leverage this in a workflow like so:

1. Take a snapshot of the VM
2. Gracefully drain all services from the VM
3. Update the system
4. Restart services on the VM and run a set of smoke tests to verify functionality
5. If the smoke tests fail, roll back to the snapshot

In playbook form, that would look something like this:

An example playbook that safely upgrades packages — note that some parameters to the vmware_* modules are omitted for brevity

This playbook uses the rescue feature of Ansible playbooks to recover from errors and restore the snapshot.

Your smoketest.yml playbook would run a variety of tests against the host to verify its functionality, such as checking an HTTP server

- name: check http server
  wait_for:
    port: 80
  when: inventory_hostname in groups['webservers']

If your tests are strong enough you can be reasonably confident that things have gone to plan and you can proceed with the upgrades.

Now we can upgrade our entire fleet in a safe, rolling fashion with a single command.

$ ansible-playbook upgrade.yml

Voila!

If you found this interesting, check out how we use Ansible to automate the installation of our VMs with Kickstart.