Restarting slow daemons

There is a patter for many services and applications: They become alive (start to process incoming connections) some time after they started. If you connect to them at that moment you may be able to connect, but many operations would fail.

One of those examples is InfluxDB. If I install influxdb and instantly start to create databases, there is a some chance for this error to happen:

Traceback (most recent call last):
  File "<stdin>", line 113, in <module>
  File "<stdin>", line 105, in _ansiballz_main
  File "<stdin>", line 48, in invoke_module
  File "/tmp/ansible_influxdb_database_payload_AvI6jE/__main__.py", line 143, in <module>
  File "/tmp/ansible_influxdb_database_payload_AvI6jE/__main__.py", line 133, in main
  File "/tmp/ansible_influxdb_database_payload_AvI6jE/__main__.py", line 94, in create_database
  File "/usr/lib/python2.7/dist-packages/influxdb/client.py", line 533, in create_database
    self.query("CREATE DATABASE {0}".format(quote_ident(dbname)))
  File "/usr/lib/python2.7/dist-packages/influxdb/client.py", line 392, in query
    in data.get('results', [])
  File "/usr/lib/python2.7/dist-packages/influxdb/resultset.py", line 23, in __init__
    raise InfluxDBClientError(self.error)
influxdb.exceptions.InfluxDBClientError: open /var/lib/influxdb/meta/meta.dbtmp: no such file or directory

This is clearly influx bug, but it shows the type of pattern I’m talking about.

The solution is simple: sleep after restart for second or two. How to do this in Ansible, together with handlers?

Sleeping handlers

After the explanation earlier, it’s easy to understand what this handler do:

---
- name: start influxdb
  become: yes
  systemd: name=influxdb state=started enabled=true
  notify: sleep
- name: sleep
  pause: seconds=2

The key advantage of ‘sleep’ handler is that it’s doing sleep only if ‘start influxdb’ handler was triggered (and i twas finished in ‘changed’ state).