Salt Essentials

Giandomenico Avelluto
Quantyca
Published in
16 min readJun 19, 2019

Hello everyone!

I’m enthusiastic about writing my first article on Medium! I will begin with a brief introduction of myself: My name is Giandomenico Avelluto and I work as Lead Site Reliability Engineer @Quantyca.

We are an Information Technology (IT) consulting company that works with data and system integration, Big Data architectures, reporting, and analysis.

1. Introduction

In this article, I want to introduce the software application that we use to manage software deployment, automation, and remediation flows on both Cloud and on Premises infrastructures for our projects: Salt.

We will deepen the following topics:

  • What Salt is, why we chose it and how it works
  • How to install Salt
  • Components walkthrough
  • Conclusions and opinions

2. Requirements

Requirements for step by step procedures:

in addition only for Windows users:

  • Git Bash (windows users must use this as terminal for this tutorial)

3. What Salt is

Salt is a software to automate the management and configuration of any infrastructure or application at scale.

There are two versions of Salt:

Salt Open manages systems entirely via a command-line interface. Thanks to the Salt REST API you can integrate it with external softwares, for example: Jenkins or Rundeck.

  • SaltStack Enterprise (that needs a subscription)

SaltStack Enterprise is built on top of Salt open version and moreover provides role-based access control (RBAC) and an Enterprise Web Console.

3.1 Benefits of SaltStack

  • SaltStack minimizes the servers overall configuration time from hours to minutes, managing server provisioning in parallel
  • SaltStack provides support for cloud infrastructure as well, and integrated compatibility with leading cloud service providers like AWS, Google Cloud or Azure
  • SaltStack is very fast and lightweight and uses ZeroMQ for communication that provides the foundation for the remote execution engine

4. Why Salt?

Salt employes a distributed messaging system to send out remote commands. This means that unlike other configuration tools such as Ansible, Salt is very scalable since the server is not opening up thousands of SSH sockets with the agents but it exposes a couple of convenient queues where the agents can subscribe to.

The result is:

  • a massively scalable automation system that supports thousands of nodes
  • a predictive orchestration system through event-driven automation that can react to infrastructure events

Following I will discuss how to install Salt and which are the main components.

5. Architecture

Salt utilizes a server-agent model, where the server is called the salt-master and the agents the salt-minion.

Architecture

Operating systems that are supported by SaltStack are:

6. Communication Model

Salt communicates with managed systems (minions) using a publish-subscribe pattern. Connections are initiated by the Salt minions, which means that you do not need to open any incoming ports on those systems. The Salt master uses two ports:

  • 4505 → All Salt minions establish a persistent connection to the publisher port where they listen for messages. Commands are sent asynchronously to all connections over this port, which enables commands to be executed over large numbers of systems simultaneously.
  • 4506 → Salt minions connect to the request server as needed to send results to the Salt master and to securely request files and minion-specific data values. Connections to this port are 1:1 between the Salt master and Salt minion (not asynchronous).

6.1. Salt Minion Public Key Authentication

Salt Minion Public Key Authentication

When the minion starts for the first time, it searches the network for a system named salt (though this can be easily changed to an IP or different hostname). When found, the minion initiates a handshake and then sends its public key to the Salt master. After this initial connection, the Salt minion’s public key is stored on the server, and it must be accepted on the Salt master.

7. How to install Salt

Salt can be installed from a package manager, pip, directly from source, or using a bootstrap script. SaltStack also provides dedicated tools to create virtual machines and install Salt on public and private clouds (salt-cloud and salt-virt). However, my preferred method and the simplest, is to install Salt through the bootstrap script.

For example, using Vagrant you can follow these steps to install Salt:

7.1 Create two vagrant instances

  • salt-master
  • salt-minion

Open the terminal on your host which we will call “master terminal”, and create the needed directories:

mkdir -p ~/vagrant/salt-{master,minion}

Now run these commands on master terminal to configure Salt Master vagrant instance:

cd ~/vagrant/salt-master
cat <<EOF > Vagrantfile
Vagrant.configure("2") do |config|
config.vm.box = "debian/bullseye64"
config.vm.network "private_network", ip: "192.168.56.2"
config.vm.hostname = "saltmaster"
config.vbguest.auto_update = false
config.vm.provider "virtualbox" do |vb|
vb.name = "saltmaster"
vb.memory = 1024
vb.cpus = 1
end
config.vm.provision "shell",
inline: "sudo apt-get update && sudo apt-get install -y curl"
end
EOF

and then start the new salt-master instance:

vagrant up

Do the same to configure the minion instance:

cd ~/vagrant/salt-minion
cat <<EOF > Vagrantfile
Vagrant.configure("2") do |config|
config.vm.box = "debian/bullseye64"
config.vm.network "private_network", ip: "192.168.56.3"
config.vm.hostname = "saltminion"
config.vbguest.auto_update = false
config.vm.provider "virtualbox" do |vb|
vb.name = "saltminion"
vb.memory = 1024
vb.cpus = 1
end
config.vm.provision "shell",
inline: "sudo apt-get update && sudo apt-get install -y curl"
end
EOF

Then start also the new salt-minion instance:

vagrant up

7.2 Install Salt

Login on Salt Master instance as root user and run the bootstrap script to install the salt-master component. So from master terminal run:

cd ~/vagrant/salt-master && vagrant ssh -c "sudo su -"

then:

curl -L https://bootstrap.saltstack.com -o install_salt
sh install_salt -P -M -N -x python3

The last command will only install the Salt Master component.

Let’s take a look at the command options:

  • -P → allow pip based installations
  • -M → install salt-master
  • -N → do not install salt-minion
  • -x → Changes the Python version used to install Salt

Now, open a new terminal on your host which we will call “minion terminal” and login to the Salt Minion instance to install the minion component:

cd ~/vagrant/salt-minion && vagrant ssh -c "sudo su -"

then:

curl -L https://bootstrap.saltstack.com -o install_salt
sh install_salt -A 192.168.56.2 -i webserv -P -x python3

The last command will install only the Salt Minion component:

  • -A → Pass the salt-master DNS name or IP. In our case, it’s the IP that we’ve assigned previously, during the creation of the Salt Master vagrant instance.
  • -i → set the minion id

Note: Without passing any specific option, the Salt Master component is not installed.

Now we have to add that minion to the master, so switch to the master terminal and run these commands:

salt-key

Output:

Accepted Keys:Denied Keys:Unaccepted Keys:webservRejected Keys:

Salt Master has received the minion request, so we can add the minion by running:

salt-key -a webserv -y

That’s it! Now we can run some test commands from master terminal to verify that it works:

salt webserv test.ping

Note: the test.ping function will verify that the communication with the target is correctly configured

Output:

webserv:True

Here we go! The connection is working.

8. Salt core components

Here are the main Salt concepts which make it unique:

Core components

8.1 Execution modules

A Salt execution module is a Python module that runs on a Salt minion. It performs tasks and returns data to the Salt master.

Execution modules syntax

For instance, to get details about disks usage percentage you can run from master terminal:

salt webserv disk.percent

Output:

webserv:
----------
/:
8%
/dev:
0%
/dev/shm:
1%
/run:
2%
/run/user/1000:
0%
/sys/fs/cgroup:
0%

Lesson learned: You can use the execution modules to push/pull data from you systems. The level of remote execution functionality available out of the box is impressive. Check out the official documentation to see the full list.

8.2 States

The core of the Salt State system is the SLS, or SaLt State file, which is built on top of modules. The SLS file is a representation of the state in which a system should be in and is set up to contain this data in a simple format. By default, Salt represents the SLS data in YAML (YAML Ain’t Markup Language), a human-readable data serialization language.

By default, Salt State files are stored in the filesystem on path /srv/salt. But you can use other backends (e.g: git). Check out the official documentation to get the full list of supported backends.

For instance, we can create a state file named install_network_packages.sls in the /srv/salt . So from master terminal run:

mkdir /srv/salt
cat <<EOF > /srv/salt/install_network_packages.sls
install_network_packages:
pkg.installed:
- pkgs:
- rsync
- lftp
- curl
EOF

That state makes sure that listed packages are installed on systems. Now, to apply that state we need to run the state.apply module passing the state file name as argument without the extension:

salt webserv state.apply install_network_packages

Learn everything you need to know about Salt State system checking out the official documentation.

8.3 Grains

Salt comes with an interface to derive information about the underlying system. This is called the Grains interface. Grains are collected for the operating system, domain name, IP address, kernel, OS type, memory, and many other system properties.

These grains are also expandable to include other bits of static information that you’d like to have assigned to a minion such as a custom role. You can use these grains to retrieve information from minions dynamically.

For instance, to retrieve all grains from systems we can run the following command from master terminal:

salt webserv grains.items

You can also use grains in a Salt State file. You can access it via grains[‘key’]:

For example, let’s create a state file with the above content and save it in /srv/salt/install_apache.sls:

cat <<EOF > /srv/salt/install_apache.sls
Install apache:
pkg.installed:
{% if grains['os_family'] == 'RedHat' %}
- name: httpd
{% elif grains['os_family'] == 'Debian' %}
- name: apache2
{% endif %}
EOF

and apply it running:

salt webserv state.apply install_apache

In the state above, grains are used to choose which apache package should be installed based on the target Operating System.

As you can notice in the above Salt state file example, sometimes a state may require programming logic or inline execution. This is accomplished with module templating. The default module templating system used is Jinja2. Check out the official documentation for more details.

8.4 Pillar

The Pillar system lets you define secure data (variables) that is ‘assigned’ to one or more minions, using targets. You can store values such as ports, file paths, configuration parameters, and passwords.

Salt pillar uses a Top file to match Salt pillar data to Salt minions.

For example, we can assign the pillar defined in a file to our webserv minion. So from the master terminal, create the /srv/pillar directory, and then create a new file called top.sls in the new pillar directory:

mkdir /srv/pillar
cat <<EOF > /srv/pillar/top.sls
base:
'webserv':
- default_users
EOF

Next, create a file named default_users.sls in the same pillar directory:

cat <<EOF > /srv/pillar/default_users.sls
users:
john: /bin/bash
peter: /bin/shell
jacob: /bin/shell
andrew: /bin/bash
EOF

This file contains a simple data structure, which is a list of users with each one assigned value.

Now, in order to retrieve the pillar data we need to refresh pillars by running:

salt '*' saltutil.refresh_pillar

and then we can get that data by calling the execution module:

salt webserv pillar.get users

Output:

webserv:
----------
andrew:
/bin/bash
jacob:
/bin/shell
john:
/bin/bash
peter:
/bin/shell

The more powerful aspect of this component is that Salt pillar keys are available in a dictionary, in Salt states.

For example, create the /srv/salt/users directory and then a file called init.sls in that directory:

mkdir /srv/salt/users
cat <<EOF > /srv/salt/users/init.sls
{% for user, shell in pillar.get('users', {}).items() %}
Create the user {{user}}:
user.present:
- name: {{user}}
- shell: {{shell}}
{% endfor %}
EOF

So, we can execute that state by running:

salt webserv state.apply users

Note: As you know, /srv/salt/users is a directory. In this case, Salt will check if a init.sls file exists in that directory and if found runs the content of that init.sls state.

Output:

webserv:
----------
ID: jacob
Function: user.present
Result: True
Comment: New user jacob created
Started: 19:17:15.176896
Duration: 57.154 ms
Changes:
----------
fullname:
gid:
1001
groups:
- jacob
home:
/home/jacob
homephone:
name:
jacob
other:
passwd:
x
roomnumber:
shell:
/bin/shell
uid:
1001
workphone:
----------
ID: john
Function: user.present
Result: True
Comment: New user john created
Started: 19:17:15.234546
Duration: 30.497 ms
Changes:
----------
fullname:
gid:
1002
groups:
- john
home:
/home/john
homephone:
name:
john
other:
passwd:
x
roomnumber:
shell:
/bin/bash
uid:
1002
workphone:
----------
ID: peter
Function: user.present
Result: True
Comment: New user peter created
Started: 19:17:15.265282
Duration: 28.9 ms
Changes:
----------
fullname:
gid:
1003
groups:
- peter
home:
/home/peter
homephone:
name:
peter
other:
passwd:
x
roomnumber:
shell:
/bin/shell
uid:
1003
workphone:
----------
ID: andrew
Function: user.present
Result: True
Comment: New user andrew created
Started: 19:17:15.294427
Duration: 29.11 ms
Changes:
----------
fullname:
gid:
1004
groups:
- andrew
home:
/home/andrew
homephone:
name:
andrew
other:
passwd:
x
roomnumber:
shell:
/bin/bash
uid:
1004
workphone:
Summary for webserv
------------
Succeeded: 4 (changed=4)
Failed: 0
------------
Total states run: 4
Total run time: 145.661 ms

The above state defines a “for cycle” that iterate on “users” pillar and create a user with correspondent shell associated for each iteration.

8.5 Beacons

Beacons let you use the event system to monitor processes continually. So when monitored activity occurs in a system process, an event is sent on the event bus.

Let’s enable your first beacon by creating the configuration file /etc/salt/minion.d/beacons.conf from the minion terminal:

cat <<EOF > /etc/salt/minion.d/beacons.conf
beacons:
service:
- services:
apache2:
onchangeonly: True
delay: 5
EOF

then restart the minion service:

systemctl restart salt-minion

The above example will fire an event 5 seconds after the state of apache2 service changes.

In order to check the configured monitoring events in real-time, Salt provides a command that displays them as they are received on the Salt Event Bus.

So check them by running, from the master terminal:

salt-run state.event pretty=True

Next, switch on the minion terminal and run:

systemctl restart apache2

After 5 seconds you should see on the master terminal, a similar event fired on the Salt Event Bus:

salt/beacon/webserv/service/apache2 {
"_stamp": "2019-05-02T15:07:52.569437",
"apache2": {
"running": true
},
"id": "webserv",
"service_name": "apache2"
}

Then press Ctrl+c on the master terminal to exit the routine.

So, we have a way to monitor our services but now we want to do something more than that. For example, you may wish to send alerts or run a remediation flow. To accomplish this task we can use the Salt Reactor System.

Refer to the official documentation for a complete explanation of how to use the Salt Beacon system and create a powerful monitoring system.

8.6 Reactor

Reactor system gives Salt the ability to trigger actions in response to an event. It’s an interface that watches Salt’s event bus for event tags that match a given pattern, then runs one or more actions in response.

Let’s begin by configuring a simple reactor state, based on the previously configured beacon.

On the master terminal create the /srv/salt/reactor directory:

mkdir /srv/salt/reactor

Next, create a file in that directory named restart_apache2.sls:

cat <<EOF > /srv/salt/reactor/restart_apache2.sls
{%- if data[data['service_name']]['running'] == False %}
start {{data['service_name']}}:
local.service.start:
- tgt: {{data['id']}}
- args:
- name: {{data['service_name']}}
{%- endif %}
EOF

Then, we need to tag it with your beacon in the master config file /etc/salt/master.d/reactor.conf:

cat <<EOF > /etc/salt/master.d/reactor.conf
reactor:
- 'salt/beacon/*/service/*':
- salt://reactor/restart_apache2.sls
EOF

Now restart the salt-master service to reload the configurations.

systemctl restart salt-master

and watch again the Event Bus:

salt-run state.event pretty=True

Ok, on minion terminal let’s try to stop the apache2 service again and wait for 5 seconds:

systemctl stop apache2

You should see on master terminal a series of events that represent our newly configured remediation flow.

Next, try to check the status of apache2 service:

systemctl status apache2

The service is up again!

Lesson Learned: Using Salr Reactor system you can create complex remediation flows to assure that your applications are always up & running.

8.7 Highstate

In order to manage groups of machines, an administrator needs to be able to create roles for those groups. In Salt, the file which contains a mapping between groups of machines on a network and their configuration roles is called top file.

Let’s check the Salt root directory tree:

/srv/salt
├── install_apache.sls
├── install_network_packages.sls
├── reactor
│ └── restart_apache2.sls
└── users
└── init.sls

Alright, from master terminal, let’s create the top.sls file in the /srv/salt directory and map some state files to some minion targets:

cat <<EOF > /srv/salt/top.sls
base:
# All minions get the following two state files applied
'*':
- install_network_packages
- install_apache
# Minions that have a grain set indicating that they are
# running the Debian operating system will have the state file
# called 'users' applied.
'os_family:RedHat':
- match: grain
- users
EOF

and run:

salt webserv state.apply

Output:

webserv:
----------
ID: install_network_packages
Function: pkg.installed
Result: True
Comment: All specified packages are already installed
Started: 21:20:10.553978
Duration: 1753.861 ms
Changes:
----------
ID: apache
Function: pkg.installed
Name: apache2
Result: True
Comment: All specified packages are already installed
Started: 21:20:12.308331
Duration: 31.571 ms
Changes:
Summary for webserv
------------
Succeeded: 2
Failed: 0
------------
Total states run: 2
Total run time: 1.785 s

This action is referred to as a “highstate”. When this function is executed, a minion will download the top file and attempt to match the expressions within it. When the minion does match an expression the modules listed for it will be downloaded, compiled, and executed.

As our Minion virtual instance has Debian 11 operating system, only the first mapping in top file is executed.

8.8 Orchestrate

While the execution of states or highstate is perfect when you want to ensure that the minion is configured in the way you want, sometimes you want to configure a set of minions all at once.

For example, if you want to set up a load balancer in front of a cluster of web servers you can ensure the web servers are set up first, and then the configuration is applied consistently to the load balancer.

So, let’s start to create two new vagrant instances. Open a new terminal on your host and run the following commands:

mkdir ~/vagrant/salt-minion2
cd ~/vagrant/salt-minion2
cat <<EOF > Vagrantfile
Vagrant.configure("2") do |config|
config.vm.box = "debian/bullseye64"
config.vm.network "private_network", ip: "192.168.56.4"
config.vm.hostname = "saltminion2"
config.vbguest.auto_update = false
config.vm.provider "virtualbox" do |vb|
vb.name = "saltminion2"
vb.memory = 1024
vb.cpus = 1
end
config.vm.provision "shell",
inline: "sudo apt-get update && sudo apt-get install -y curl"
end
EOF

and then start the new virtual instance:

vagrant up

Next, let’s create the second virtual instance:

mkdir ~/vagrant/salt-minion3
cd ~/vagrant/salt-minion3
cat <<EOF > Vagrantfile
Vagrant.configure("2") do |config|
config.vm.box = "debian/bullseye64"
config.vm.network "private_network", ip: "192.168.56.5"
config.vm.hostname = "saltminion3"
config.vbguest.auto_update = false
config.vm.provider "virtualbox" do |vb|
vb.name = "saltminion3"
vb.memory = 1024
vb.cpus = 1
end
config.vm.provision "shell",
inline: "sudo apt-get update && sudo apt-get install -y curl"
end
EOF

and then start the new virtual instance:

vagrant up

Now, install the Salt Minion on both new virtual instances:

cd ~/vagrant/salt-minion2 && vagrant ssh -c "sudo su -"

then:

curl -L https://bootstrap.saltstack.com -o install_salt
sh install_salt -A 192.168.56.2 -i webserv2 -P -x python3
exit
exit

the same for the last VM:

cd ~/vagrant/salt-minion3 && vagrant ssh -c "sudo su -"

then:

curl -L https://bootstrap.saltstack.com -o install_salt
sh install_salt -A 192.168.56.2 -i loadbalancer -P -x python3
exit
exit

Next, let’s add the new minions to the Salt Master from master terminal, by running:

salt-key -A -y

Now, let’s verify that all is correctly working by running:

salt '*' test.ping

Output:

webserv:
True
webserv2:
True
loadbalancer:
True

Let’s modify the /srv/salt/install_apache.sls state adding some additional configurations:

cat <<EOF > /srv/salt/install_apache.sls
Install apache:
pkg.installed:
{% if grains['os_family'] == 'RedHat' %}
- name: httpd
{% elif grains['os_family'] == 'Debian' %}
- name: apache2
{% endif %}
Deploy a simple web page:
file.managed:
- name: /usr/share/apache2/default-site/index.html
- contents: |
<!doctype html>
<html lang="en">
<head>
<style>
body {
width: 100%; height: 100%; top: 0; left: 0;
background: url(https://media.licdn.com/dms/image/C561BAQHk032ec5S2uQ/company-background_10000/0?e=2159024400&v=beta&t=2y7J65LrJZAPKuCEiRVc2XmP4Bh6IyWvAK409Y-hDm0) no-repeat center top; position: fixed; z-index: -1;
-webkit-background-size: cover;
-moz-background-size: cover;
-o-background-size: cover;
background-size: cover;
}
div {
height: 200px;
width: 400px;
position: fixed;
top: 20%;
left: 50%;
margin-top: -100px;
margin-left: -200px;
}
</style>
<meta charset="utf-8">
<title>Congratulation</title>
<link rel="icon" href="https://media.glassdoor.com/sqll/1166445/saltstack-squarelogo-1548910587917.png">
<meta name="description" content="Demo">
<meta name="author" content="Giandomenico Avelluto">
</head>
<body>
<div align="center" style="display: block; text-align: center; color: crimson;">
<h1>Hello! My Minion id is: {{ grains['id'] }} and actually my disk occupation is: {{ salt.disk.percent()['/'] }}</h1>
</div>
</body>
</html>
Restart service if configuration changes:
service.running:
- name: apache2
- watch:
- file: Deploy a simple web page
EOF

Let’s check the two added functions in the state above:

  • file.managed: this function downloads files from the salt master and places them on the target system. In this case we pass to the minion the html content to expose
  • service.running: this function ensure that the specified service is running, in this case we ensure also that the apache2 service will be restarted if the html content changes. That is accomplished by the watch keyword

Now, let’s set the HAProxy load balancer:

mkdir /srv/salt/haproxy
cat <<EOF > /srv/salt/haproxy/init.sls
Install haproxy package:
pkg.installed:
- name: haproxy
Configure haproxy:
file.managed:
- name: /etc/haproxy/haproxy.cfg
- contents: |
global
log 127.0.0.1 local0 notice
maxconn 2000
user haproxy
group haproxy
defaults
log global
mode http
option httplog
option dontlognull
retries 3
option redispatch
timeout connect 5000
timeout client 10000
timeout server 10000
listen stats
bind *:81
stats enable
stats uri /haproxy?stats
stats realm Strictly\ Private
stats auth admin:admin
frontend localnodes
bind *:80
mode http
default_backend nodes
backend nodes
mode http
balance roundrobin
option httpclose
option forwardfor

{%- for server, addr in salt['mine.get']('webserv*', 'network.ip_addrs').items() %}
server {{ server }} {{ addr[0] }}:80 check
{%- endfor %}
Start haproxy service:
service.running:
- name: haproxy
- watch:
- pkg: Install haproxy package
- file: Configure haproxy
EOF

Finally, we can configure our orchestration by creating the file /srv/salt/apache_orchestration.sls :

cat <<EOF > /srv/salt/apache_orchestration.sls
Install apache on webservers minions:
salt.state:
- tgt: 'webserv*'
- sls:
- install_apache
Send mine function to loadbalancer:
salt.function:
- tgt: '*'
- name: mine.send
- arg:
- network.ip_addrs
- kwarg:
interface: eth1
Configure the Load Balancer only if the webserv minions are correctly configured:
salt.state:
- tgt: 'loadbalancer'
- sls:
- haproxy
- require:
- salt: Install apache on webservers minions
EOF

Now are you ready to run the orchestration:

salt-run state.orch apache_orchestration

That’s all! We can check out that all is working asking Salt which is the load balance IP:

salt loadbalancer network.ipaddrs eth1

and then copy and past it in a browser. You should see something like this:

Lesson learned: Orchestration system allows you to define the infrastructure as code. By running a single command you can orchestrate and deploy complex applications.

I also recommend you to check out the following story to understand how to take advantage of Salt Orchestration component:

9. Conclusion

The operations that can be accomplished using SaltStack are literally endless. I hope that this introduction will be useful for you to create a powerful configuration management system and complex remediation flows.

I also recommend to check out the new Saltstack SecOps Platform that harnesses event-driven automation technology to deliver full-service, closed-loop automation for IT system compliance and vulnerability remediation.

If you see room for improvement, let me know and follow us on our Linkedin profile!

--

--