I run ansible to run ansible to make ansible runnable on remote host
Higher order generic inventories in an awkward language
This is my working blog entry. Things I do sometimes in Ansible is kinda on the verge of sanity, and I know the more crazy things I wrote, the shittier is a code.
But sometimes a carrot ahead is too tempting.
The carrot is reduction in total CI/CD pipeline time. Without my despicable lunatic tricks it’s about 1 hour 35 minutes 44 seconds.
With my mad brain-frying hacks, it’s… 17 minutes.
So, my poisonous gifts are hard to decline.
Here is one of the tricks I do. I run testinfra tests locally. Instead of painfully slow host.run … or even slower host.ansible.uri I just copy tests to a remote machine and run them there. It’s about 30 times acceleration on tests.
Side note for beginners: There is a so-called infrastructure testing, the thing to keep project alive. Use it. testinfra is one of those. You can use ansible modules from testinfra, and you can use ansible inventory to specify groups where to run tests.
Second thing I have is so-called ‘secret inventory’. I don’t use vault (as it’s impossible to maintain in a large team), instead we use sops inventory plugin to encrypt a special inventory called ‘secret inventory’. It’s joined with the main inventory (which doesn’t contain any secrets).
Side note for beginners: You can use something else than ansible-vault. There are ‘inventory plugins’ to use, and not all of them are dynamic inventories for cloud providers. You can have any inventories joined together: just list them as -i inventory1.yaml -i inventory2/ -i inventory3 in a command line, or via environment variable ANSIBLE_INVENTORY=inventory1,inventory2.
So, to run tests locally I need to copy tests and inventory onto remote server. I need inventory because tests often peek into inventory variables and group membership for tests is defined via host membership in inventory.
I run tests locally by redefining ansible_connection: local.
Side note for beginners: You can run a play (or a task) on local host instead of remote server. Normally it’s done via ‘delegate_to’ keyword, but you can set ansible_connection: local and have the same result. ansible_host is ignored and code is run locally.
You can’t just copy inventory onto a remote server. First, we need to update it with ansible_connection. We need to update it because we can’t use -e or -c options for Ansible, because this inventory is used indirectly by testinfra. Moreover, we can’t use whole inventory because inventory contains all servers, and we need to have ‘reduced’ inventory for each server, containing only itself.
And, we don’t need to copy secret inventories. So we can’t just grab all inventories and copy them onto remote.
So, what we do?
Oh, it’s juicy.
We query host variables and put them into inv variable.
Then we save them remotely as host vars (and it does not matter what kind of variables it was before, we put them all as host vars):
Why so hard? Why not use hostvars[inventory_hostname]|jo_nice_yaml ? Because of the jinja. If you have any jinja it’s going to be evaluated and may be misevaluated if it has some jinja expressions. Also, !!unsafe stops been unsafe.
Note for beginners: If you have some crazy go templates in the file, use {% raw %} in multiline jinja or use !!unsafe “string” in yaml.
Instead we saving inv.stdout directly into file, without Jinja evaluation. Yes, my gifts are poisonous. If you try to maintain this code without deepest possible understanding of those subtle details, everything will collapse.
Heh, it was just the old stuff. It was run in separate job without secrets. Now I got to the point when I need to move my code into job with ANSIBLE_INVENTORY containing secret inventory, and I don’t those secrets on remote server.
I need to fetch inventory without secrets. Behold!
Do you need help to understand this?
I use ANSIBLE_INVENTORY variable (at the controller), split it by comma, remove all elements with word ‘secret’ in them, join back by comma and set ANSIBLE_INVENTORY to this value for call to ansible-inventory on the same controller host to retrieve host vars compiled from the reduced inventory.
I use ansible to run ansible to mangle inventory for ansible.
Poisonous, I know. But it reduce life of your CI by about 5 minutes… Hard to reject, I know. So, bear with me. It worth it.
Note for beginners: Don’t do things like this. I know where to stop.
Write simple code. No includes, no dynamic variables, no joined dictionaries. Just simple playbooks with simple roles doing only one simple thing at a time in linear fashion.
