Some notes on using SaltStack at home

I recently relocated, and took the time to re-do a lot of my home network post-move, and decided my fleet had grown such that I was overdue for a revamp of how things are managed. I elected to use Saltstack to manage my machines on the network, and thought rather than writing a standard beginner’s guide, I’d demonstrate a use case for my local development, deployment, and run pipelines from the perspective of a user not-unfamiliar with configuration management, but new to the Saltstack toolset.

That represents a significant commitment of time to manage manually, so I turned a few years ago to configuration management to handle it all, but in this case, my preference for agent-less management left a lot to be desired (in this specific case), and I decided to try out Salt after years of my only experience with an agent-based configuration management system, in production, was Chef, and ever since being an adherent of Ansible and Terraform for my conf management and provisioning needs.

My current set-up looks like this:

  1. A hypervisor (32 GB RAM, 240 GB SSD OS disk, 7 TB storage array)
  2. A jumphost (8 GB RAM, 240 GB SSD), which also provides internal DNS, and acts as the Salt master.
  3. Client machines (VMs providing various services — a Docker registry, Git server, Object Storage APIs, CI/CD, among others — , laptops used by my household, etc.)

All of the above are managed in Salt. So, with that in mind, let’s take a look at my Salt repo’s topfile (the file mapping states to machines that match a custom grain — role — to identify their purpose in my network):

jmarhee@tonyhawkproskater2 /srv/salt $ cat top.sls                                                                                                                                                                         
base:
'*':
- ssh_git
'role:docker_registry':
- match: grain
- docker_registry
'role:laptop':
- match: grain
- laptop
'role:windows':
- match: grain
- windows_bootstrap
'role:ci':
- match: grain
- drone

So, you’ll see that the base role is applied to all hosts (signified by the wildcard, *), and that various roles have additional states appended to them when a host matching that grain is found.

For example, when my topfile recognizes a host its about to highstate (applying this top-level configuration to the hosts specified when one runs a command like salt “*" state.highstate, rather than applying a single state to a single host, like salt “*" state.sls ssh_git) a docker_registry and applies a secondary state to it, docker_registry .

The directory structure for my Salt repo ( /srv/salt) is pretty straight forward. In the root is my top.sls , and any other relevant state files referenced therein (this can, obviously, be more complex, if you have multiple environments, etc. but for this use case, my structure is simple), and a files directory where I store configs, scripts, keys, etc. More advanced use cases can use things like Hiera to manage secrets, etc. and other Salt modules to leverage more advanced management.

Let’s take a look at this base state (in the base directory, ssh_git.sls) that applies to all hosts:

jmarhee@tonyhawkproskater2 /srv/salt $ cat ssh_git.sls                                                                                                                                                                     
{% if grains['kernel'] == "Linux" %}
install_git:
pkg.installed:
- name: git-core
- name: zsh
{% if grains['role'] == "ci" %}
- name: golang
- name: make
- name: build-essential
{% endif %}
sync base_cron:
file.managed:
- name: /root/base_cron
- source: salt://files/base_cron
- user: root
- group: root
- mode: 700
base_cron:
cmd.script:
- require:
- file: /root/base_cron
- source: salt://files/base_bootstrap.sh
- user: root
- group: root
- shell: /bin/bash
base_sudo:
file.managed:
- name: /etc/sudoers.d/myOverrides
- source: salt://files/sudo_overrides
- user: root
- group: root
- mode: 700
{% if grains['role'] == 'jumphost' or grains['role'] == 'hypervisor:kvm' or grains['role'] == 'ci' %}
sync keypair_pub:
file.managed:
{% if grains['id'] == 'git.boulder.gourmet.yoga' %}
- name: /home/git/.ssh/id_rsa.pub
{% else %}
- name: /home/jmarhee/.ssh/id_rsa.pub
{% endif %}
- source: salt://files/id_rsa.pub
{% if grains['id'] == 'git.boulder.gourmet.yoga' %}
- user: git
- group: git
{% else %}
- user: jmarhee
- group: jmarhee
{% endif %}
- mode: 600
sync keypair_pvt:
file.managed:
{% if grains['id'] == 'git.boulder.gourmet.yoga' %}
- name: /home/git/.ssh/id_rsa
{% else %}
- name: /home/jmarhee/.ssh/id_rsa
{% endif %}
- source: salt://files/id_rsa
{% if grains['id'] == 'git.boulder.gourmet.yoga' %}
- user: git
- group: git
{% else %}
- user: jmarhee
- group: jmarhee
{% endif %}
- mode: 400
{% endif %}
sync authorized_keys:
file.managed:
{% if grains['id'] == 'git.boulder.gourmet.yoga' %}
- name: /home/git/.ssh/authorized_keys
{% else %}
- name: /home/jmarhee/.ssh/authorized_keys
{% endif %}
- source: salt://files/authorized_keys
{% if grains['id'] == 'git.boulder.gourmet.yoga' %}
- user: git
- group: git
{% else %}
- user: jmarhee
- group: jmarhee
{% endif %}
- mode: 600
sync ssh_config:
file.managed:
{% if grains['id'] == 'git.boulder.gourmet.yoga' %}
- name: /home/git/.ssh/config
{% else %}
- name: /home/jmarhee/.ssh/config
{% endif %}
{% if grains['id'] == 'iampizza.local' %}
- source: salt://files/ssh_config_laptop
{% else %}
- source: salt://files/ssh_config
{% endif %}
{% if grains['id'] == 'git.boulder.gourmet.yoga' %}
- user: git
- group: git
{% else %}
- user: jmarhee
- group: jmarhee
{% endif %}
{% endif %}

So, that’s a lot of conditionals. The reason, however, that it gets applied to all hosts is that it manages files I need on all of my hosts, just implemented differently.

For example, it starts by checking if the kernel grain is Linux, thus excluding a couple of Windows machines in my home (which have different requirements). Then, unless the id is such that it indicates I may use a name other than jmarhee, in this case, a git server, it changes the upload path.

So, in this case, let’s take a machine like registry.boulder.gourmet.yoga which is a local Docker registry, it applies this above base state, but also docker_registry, which looks like this:

sync registry_cron:
file.managed:
- name: /home/jmarhee/docker_cron
- source: salt://files/docker_cron
- user: jmarhee
- group: jmarhee
- mode: 600
check_and_add_cron:
cmd.script:
- require:
- file: /home/jmarhee/docker_cron
- source: salt://files/load_cron.sh
- user: jmarhee
- group: jmarhee
- shell: /bin/bash

which basically adds a crontab file, and then a script to load variables into that file before setting the crontab to use those commands. You’ll recall that in top.sls I mapped this file to machines matching that docker_registry role.

So, looking back at the above, say I want to know what will happen to my laptop ( iampizza.local) if I run salt "iampizza*" state.highstate or salt — grain “role:Laptop" state.highstatebut don’t want to test the output (you can do this by adding test=True to your Salt commands for a dry-run), I can check top.sls and see :

'role:laptop':
- match: grain
- laptop

so in addition to the above in ssh_git.sls I’ll also be doing:

sync laptop_cron:
file.managed:
- name: /home/jmarhee/laptop_cron
- source: salt://files/laptop_cron
- user: jmarhee
- group: jmarhee
- mode: 700
laptop_cron:
cmd.script:
- require:
- file: /home/jmarhee/laptop_cron
- source: salt://files/laptop_bootstrap.sh
- user: root
- group: root
- shell: /bin/bash

So, pretty similar to what I did on docker_registry , in fact, so similar that it makes sense to unify these states, right? Easy enough to do:

sync cron:
file.managed:
{% if grain['role'] == "Laptop %}
- name: /home/jmarhee/laptop_cron
- source: salt://files/laptop_cron
{% elif grain['role'] == "docker_registry" %}
- name: /home/jmarhee/docker_cron
- source: salt://files/docker_cron
{% endif %}
- user: jmarhee
- group: jmarhee
- mode: 700
load_cron:
cmd.script:
- require:
{% if grain['role'] == "Laptop %}
- file: /home/jmarhee/laptop_cron
- source: salt://files/laptop_bootstrap.sh
{% elif grain['role'] == "docker_registry" %}
- file: /home/jmarhee/docker_cron
- source: salt://files/load_cron.sh
{& endif %}
- user: root
- group: root
- shell: /bin/bash

So let’s call this new state load_cron.sls , and modify our topfile to map this new state to docker_registry and laptop roles:

'role:docker_registry':
- match: grain
- load_cron
'role:laptop':
- match: grain
- load_cron

Possibilities for refactoring this state file are pretty endless as it stands, but, you reduced how many states you mean to manage for identical, grain-filtering behavior by 50%! That’s a start.

So, once this is all saved, I can proceed to apply this state. For the sake of this example, let’s assume these two roles are the only hosts affected by this change, and I want to highstate (apply all states mapped to a given grain, in this case, their role) only those. I’ll, in this case, I know the hostnames, and want to use some regex to target those IDs:

sudo salt -E "(iampizza*|registry*)" state.highstate

this command basically does this, relative to what we detailed above:

First, it applies the base ( ssh_git) state:

salt "iampizza.local" state.sls ssh_git.sls

then, applies the load_cron state we just wrote (which we mapped to my laptop in the above topfile change; this would apply top any laptop with the role of Laptop). The process, then, repeats for the second half of our regex string ( iampizza or registry* , which refers to the hostname, or Salt ID of the node in question, rather than the grain based approach I demonstrated above, but either can be used here), using the same logic, specific to that matched host’s rule.

I recommend, if you’re entirely new to Saltstack, that you take a look at this book:

It has an excellent introduction to states, Jinja templating, the reactor functionality, etc. in SaltStack as well as the introduction to high-level Salt concepts.

One clap, two clap, three clap, forty?

By clapping more or less, you can signal to us which stories really stand out.