Some notes on using SaltStack at home

Joseph D. Marhee

I recently relocated, and took the time to re-do a lot of my home network post-move, and decided my fleet had grown such that I was overdue for a revamp of how things are managed. I elected to use Saltstack to manage my machines on the network, and thought rather than writing a standard beginner’s guide, I’d demonstrate a use case for my local development, deployment, and run pipelines from the perspective of a user not-unfamiliar with configuration management, but new to the Saltstack toolset.

That represents a significant commitment of time to manage manually, so I turned a few years ago to configuration management to handle it all, but in this case, my preference for agent-less management left a lot to be desired (in this specific case), and I decided to try out Salt after years of my only experience with an agent-based configuration management system, in production, was Chef, and ever since being an adherent of Ansible and Terraform for my conf management and provisioning needs.

My current set-up looks like this:

  1. A hypervisor (32 GB RAM, 240 GB SSD OS disk, 7 TB storage array)
  2. A jumphost (8 GB RAM, 240 GB SSD), which also provides internal DNS, and acts as the Salt master.
  3. Client machines (VMs providing various services — a Docker registry, Git server, Object Storage APIs, CI/CD, among others — , laptops used by my household, etc.)

All of the above are managed in Salt. So, with that in mind, let’s take a look at my Salt repo’s topfile (the file mapping states to machines that match a custom grain — role — to identify their purpose in my network):

jmarhee@tonyhawkproskater2 /srv/salt $ cat top.sls                                                                                                                                                                         
base:
'*':
- ssh_git
'role:docker_registry':
- match: grain
- docker_registry
'role:laptop':
- match: grain
- laptop
'role:windows':
- match: grain
- windows_bootstrap
'role:ci':
- match: grain
- drone

So, you’ll see that the base role is applied to all hosts (signified by the wildcard, *), and that various roles have additional states appended to them when a host matching that grain is found.

For example, when my topfile recognizes a host its about to highstate (applying this top-level configuration to the hosts specified when one runs a command like salt “*" state.highstate, rather than applying a single state to a single host, like salt “*" state.sls ssh_git) a docker_registry and applies a secondary state to it, docker_registry .

The directory structure for my Salt repo ( /srv/salt) is pretty straight forward. In the root is my top.sls , and any other relevant state files referenced therein (this can, obviously, be more complex, if you have multiple environments, etc. but for this use case, my structure is simple), and a files directory where I store configs, scripts, keys, etc. More advanced use cases can use things like Hiera to manage secrets, etc. and other Salt modules to leverage more advanced management.

Let’s take a look at this base state (in the base directory, ssh_git.sls) that applies to all hosts:

jmarhee@tonyhawkproskater2 /srv/salt $ cat ssh_git.sls                                                                                                                                                                     
{% if grains['kernel'] == "Linux" %}
install_git:
pkg.installed:
- name: git-core
- name: zsh
{% if grains['role'] == "ci" %}
- name: golang
- name: make
- name: build-essential
{% endif %}
sync base_cron:
file.managed:
- name: /root/base_cron
- source: salt://files/base_cron
- user: root
- group: root
- mode: 700
base_cron:
cmd.script:
- require:
- file: /root/base_cron
- source: salt://files/base_bootstrap.sh
- user: root
- group: root
- shell: /bin/bash
base_sudo:
file.managed:
- name: /etc/sudoers.d/myOverrides
- source: salt://files/sudo_overrides
- user: root
- group: root
- mode: 700
{% if grains['role'] == 'jumphost' or grains['role'] == 'hypervisor:kvm' or grains['role'] == 'ci' %}
sync keypair_pub:
file.managed:
{% if grains['id'] == 'git.boulder.gourmet.yoga' %}
- name: /home/git/.ssh/id_rsa.pub
{% else %}
- name: /home/jmarhee/.ssh/id_rsa.pub
{% endif %}
- source: salt://files/id_rsa.pub
{% if grains['id'] == 'git.boulder.gourmet.yoga' %}
- user: git
- group: git
{% else %}
- user: jmarhee
- group: jmarhee
{% endif %}
- mode: 600
sync keypair_pvt:
file.managed:
{% if grains['id'] == 'git.boulder.gourmet.yoga' %}
- name: /home/git/.ssh/id_rsa
{% else %}
- name: /home/jmarhee/.ssh/id_rsa
{% endif %}
- source: salt://files/id_rsa
{% if grains['id'] == 'git.boulder.gourmet.yoga' %}
- user: git
- group: git
{% else %}
- user: jmarhee
- group: jmarhee
{% endif %}
- mode: 400
{% endif %}
sync authorized_keys:
file.managed:
{% if grains['id'] == 'git.boulder.gourmet.yoga' %}
- name: /home/git/.ssh/authorized_keys
{% else %}
- name: /home/jmarhee/.ssh/authorized_keys
{% endif %}
- source: salt://files/authorized_keys
{% if grains['id'] == 'git.boulder.gourmet.yoga' %}
- user: git
- group: git
{% else %}
- user: jmarhee
- group: jmarhee
{% endif %}
- mode: 600
sync ssh_config:
file.managed:
{% if grains['id'] == 'git.boulder.gourmet.yoga' %}
- name: /home/git/.ssh/config
{% else %}
- name: /home/jmarhee/.ssh/config
{% endif %}
{% if grains['id'] == 'iampizza.local' %}
- source: salt://files/ssh_config_laptop
{% else %}
- source: salt://files/ssh_config
{% endif %}
{% if grains['id'] == 'git.boulder.gourmet.yoga' %}
- user: git
- group: git
{% else %}
- user: jmarhee
- group: jmarhee
{% endif %}
{% endif %}

So, that’s a lot of conditionals. The reason, however, that it gets applied to all hosts is that it manages files I need on all of my hosts, just implemented differently.

For example, it starts by checking if the kernel grain is Linux, thus excluding a couple of Windows machines in my home (which have different requirements). Then, unless the id is such that it indicates I may use a name other than jmarhee, in this case, a git server, it changes the upload path.

So, in this case, let’s take a machine like registry.boulder.gourmet.yoga which is a local Docker registry, it applies this above base state, but also docker_registry, which looks like this:

sync registry_cron:
file.managed:
- name: /home/jmarhee/docker_cron
- source: salt://files/docker_cron
- user: jmarhee
- group: jmarhee
- mode: 600
check_and_add_cron:
cmd.script:
- require:
- file: /home/jmarhee/docker_cron
- source: salt://files/load_cron.sh
- user: jmarhee
- group: jmarhee
- shell: /bin/bash

which basically adds a crontab file, and then a script to load variables into that file before setting the crontab to use those commands. You’ll recall that in top.sls I mapped this file to machines matching that docker_registry role.

So, looking back at the above, say I want to know what will happen to my laptop ( iampizza.local) if I run salt "iampizza*" state.highstate or salt — grain “role:Laptop" state.highstatebut don’t want to test the output (you can do this by adding test=True to your Salt commands for a dry-run), I can check top.sls and see :

'role:laptop':
- match: grain
- laptop

so in addition to the above in ssh_git.sls I’ll also be doing:

sync laptop_cron:
file.managed:
- name: /home/jmarhee/laptop_cron
- source: salt://files/laptop_cron
- user: jmarhee
- group: jmarhee
- mode: 700
laptop_cron:
cmd.script:
- require:
- file: /home/jmarhee/laptop_cron
- source: salt://files/laptop_bootstrap.sh
- user: root
- group: root
- shell: /bin/bash

So, pretty similar to what I did on docker_registry , in fact, so similar that it makes sense to unify these states, right? Easy enough to do:

sync cron:
file.managed:
{% if grain['role'] == "Laptop %}
- name: /home/jmarhee/laptop_cron
- source: salt://files/laptop_cron
{% elif grain['role'] == "docker_registry" %}
- name: /home/jmarhee/docker_cron
- source: salt://files/docker_cron
{% endif %}
- user: jmarhee
- group: jmarhee
- mode: 700
load_cron:
cmd.script:
- require:
{% if grain['role'] == "Laptop %}
- file: /home/jmarhee/laptop_cron
- source: salt://files/laptop_bootstrap.sh
{% elif grain['role'] == "docker_registry" %}
- file: /home/jmarhee/docker_cron
- source: salt://files/load_cron.sh
{& endif %}
- user: root
- group: root
- shell: /bin/bash

So let’s call this new state load_cron.sls , and modify our topfile to map this new state to docker_registry and laptop roles:

'role:docker_registry':
- match: grain
- load_cron
'role:laptop':
- match: grain
- load_cron

Possibilities for refactoring this state file are pretty endless as it stands, but, you reduced how many states you mean to manage for identical, grain-filtering behavior by 50%! That’s a start.

So, once this is all saved, I can proceed to apply this state. For the sake of this example, let’s assume these two roles are the only hosts affected by this change, and I want to highstate (apply all states mapped to a given grain, in this case, their role) only those. I’ll, in this case, I know the hostnames, and want to use some regex to target those IDs:

sudo salt -E "(iampizza*|registry*)" state.highstate

this command basically does this, relative to what we detailed above:

First, it applies the base ( ssh_git) state:

salt "iampizza.local" state.sls ssh_git.sls

then, applies the load_cron state we just wrote (which we mapped to my laptop in the above topfile change; this would apply top any laptop with the role of Laptop). The process, then, repeats for the second half of our regex string ( iampizza or registry* , which refers to the hostname, or Salt ID of the node in question, rather than the grain based approach I demonstrated above, but either can be used here), using the same logic, specific to that matched host’s rule.

I recommend, if you’re entirely new to Saltstack, that you take a look at this book:

It has an excellent introduction to states, Jinja templating, the reactor functionality, etc. in SaltStack as well as the introduction to high-level Salt concepts.

Joseph D. Marhee

Written by

Systems Engineer

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade