Handling nasty expressions in Ansible with grace

There are inevitable moments when you need to do loop calculations in Ansible. You have a group, and you need to extract some variable from all hosts in that group (except those who don’t have this variable), and use it for iteration, or for other purposes.

It’s nasty in Ansible with Jinja. In better programming languages you can use Option type with iterators, monads, generators and all other beauty you don’t have in Jinja. There are many ways around, and here I present the one I believe has the most readability. With emphasis on readability, not on shrewdness, cleverness, or number of characters. That means the next person (you, in the next three years) can read and understand it without efforts, and modify it without issue. It’s also has a very good debug-ability.

As a testcase we want to gather all used values of ansible_python_interpreter for all groups in a variable foobar. I want a list with all values, and each value should be in the list only once. Obviously, some hosts may not have ansible_python_interpreter value, so we should skip those.

I’ll start from the solution and explain later. The solution I propose is ‘block Jinja for yaml’ + ‘from_yaml’ filter.

- name: List of python interpreters for foobar
  ansible.builtin.set_fact:
     foobar_pythons: '{{ py_yaml | from_yaml | default ([], True) | unique }}'
  vars:
    py_yaml: |
      ---
      {% for grp in foobar %}
      {% for h in groups[grp] %}
      {% if (hostvars[h]).anisble_python_interpreter | default(False) %}
       - {{ (hostvars[h]).anisble_python_interpreter }}
      {% endif %}
      {% endfor %}
      {% endfor %}

Let’s decode this, and then discuss why I believe it’s the most readable way to solve the puzzle.

1. py_yaml is task-local variable, which gets a text value. We use yaml feature for ‘pre’ text, which is ‘|’ after the variable name. This allows us to use multiline Jinja without fear of formatting issues.
2. It contains a normal YAML templatization. It’s not pretty or concise, but it’s very readable. Two nested loops, with one if. We write list element (starting with ‘-’ if there is a value).
3. Then we use this variable in set_fact value, from_yaml , which should yield us a proper list. If this is an empty value (falsy) we replace it with empty list. Then we apply the unique filter.
4. You may have noticed an odd default(False) expression in the block jinja. It’s not a mistake. We could rewrite that expression into ‘is defined’, but there is no way back, if the variable is defined, you can’t undefine it. But you can override it to be False. See my very old article about this.

That was ‘solution’ part. Now let’s discuss why I consider this better than any fancy map/selectattr trickery for which you may have a finger itch.

Readability

The key property of ‘readability’ is that a person can ‘read’ it without building complicated mental model. If you use filters in Jinja for selecting and iterating, you use a poor tool (compared to iterators in Rust!), and you need to reason about invariants for all cases for each filter. Because most ‘list filters’ in Jinja do not give you an invariant, you get a few variants. Then every variant is processed by next filter producing a few new variants for every input variant. You have a variant cardinality blast in your head, and, more importantly, in reader’s head, which tries to understand what it will produce for all interesting cases (including an empty list in foobar variable, empty groups, hosts without a defined variable, hosts been in few groups all listed in foobar , etc, etc). That’s breaking people’s souls.

Now look at the loops. They are visually readable (each stanza on a new line), there is a single variant, which either gives you a line with a value, or an empty line. Both loops are invariant, and you have combination of a ‘start of yaml’, may be elements of the list, and may be empty lines. YAML magic says that empty lines do not count, and what we have is a simple list of items. Easy to understand, easy to update. Also, YAML does not need ending case (like ‘no comma in last element’ in old JSON), so it’s easy to write. Also, we have --- at the start, which assure us that Ansible won’t try to be smart and accept that this is a ‘string’, not some odd jinja to interpret.

One nasty thing I have to use here, is | default([], True) . Default is fine, but that ‘True’ at the end is ugly. It says ‘use this default if the value is ‘falsy’”. It solves the problem but is ugly, and I don’t know how to step over this. Inline ‘if’ is even more horrible, so this is the best of the worst we have in Jinja.

Finally, if you look at the ‘set_fact’ expression, it keeps an invariant, making it very easy to reason about:

py_yaml | from_yaml | default ([], True) | unique

• from_yaml is either parse success or error. Success may be any value, but we produce a list or empty yaml, so it should be [], “” or null.
• default will check if the value is false, an empty line or an empty list and replaces it with a list, therefore, we have only lists at that stage.
• unique is applied to a list and produce a new list without duplicates

With exception of a small ‘jitter’ around types produced by from_yaml, this code is linear and has invariant, therefore it’s easy to read.

Last small notice: I called the variable my_yaml to fit the expression in a single line the post. My original variable name was foobar_pythons_yaml .

Conclusion

When I try to improve readability I keep things linear (invariant), use multiline expression for complicated moments, and avoid variant cardinality multiplications.