Vagrant Provisioning with SaltStack

Provisioning Virtual System using Masterless Salt Stack

This tutorial covers provisioning systems with a Vagrant provisioner for Salt Stack.

About Salt Stack

Salt Stack is a distributed remote execution platform with change configuration capability. The Salt Stack scripts called states, are essentially a DSL package in a YAML format. Salt Stack also has states grouped into a component structure called a formula. A formula can contain several states, local variables, files, and templates. In addition to states and formulas, Salt Stack uses pillars to store variables and arbitrary data and secrets.

Activities within the Salt Stack system are coordinated with Salt Master server(s) and a message queuing control bus, with minions installed onto the clients. This organizational structure allows for not only high performance scenarios, but for managing cloud based infrastructure as well as services that exist as clusters, like Elastic Search, Apache Kafka, Apache Spark, Apache Storm, or Kubernetes.

Prerequisites

Before getting started you need to have Vagrant and Virtualbox installed. These instructions use Curl and Bash shell, so if this is not your shell, you will need to convert the shell commands to something that makes sense in your environment.

I have published some previous guides on installing Vagrant and Virtualbox on macOS, Windows, and Linux operating systems.

Windows 8.1

Using Chocolatey to install the requirements:

macOS High Sierra 10.3

Using Homebrew to install the requirements:

Fedora 28

Using native package manager Dandified YUM, or in short DNF, to install requirements:

Part I: Salt Formula with Intelligent Defaults

We’ll start this tutorial with a Salt Stack Formula called hello_web that will install an Apache web server and copy over HTML content on Ubuntu system. Vagrant will call Salt Stack to run this formula using masterless mode.

Staging Area Structure

We need to lay down some roots… Using Bash shell, we can create our overall staging area and structure for this tutorial.

mkdir -p ~/vagrant-salt/roots/{pillar,salt/hello_web/files}
cd ~/vagrant-salt
touch Vagrantfile roots/salt/top.sls \
roots/pillar/{top.sls,hello_web.sls} \
roots/salt/hello_web/{defaults.yaml,init.sls,map.jinja}
cat <<-'HTML' > roots/salt/hello_web/files/index.html
<html>
<body>
<h1>Hello World!</h1>
</body>
</html>
HTML

The resulting structure will look like this, with our formula under the salt directory.

~/vagrant-salt
├── Vagrantfile
└── roots
├── pillar
│ ├── hello_web.sls
│ └── top.sls
└── salt
├── hello_web
│ ├── defaults.yaml
│ ├── files
│ │ └── index.html
│ ├── init.sls
│ └── map.jinja
└── top.sls

Vagrant Configuraiton

Update the Vagrantfile to have this content, using the Ubuntu box from the Bento project:

Vagrant.configure('2') do |config|
config.vm.box = 'bento/ubuntu-16.04'
config.vm.network 'forwarded_port', guest: 80, host: 8085
  ####### File Share #######
config.vm.synced_folder './roots/salt/', '/srv/salt'
config.vm.synced_folder './roots/pillar', '/srv/pillar'
  ####### Provision #######
config.vm.provision :salt do |salt|
salt.masterless = true
salt.run_highstate = true
salt.verbose = true
end
end

We can see provisioner doesn’t do much of anything, except use the default behavior, which are looking for the top.sls in /srv/salt and /srv/pillar. These are mounted from out local roots directory.

Create Top Salt State

In the top file, we can organize and group several systems, and then specify what to do on them, such as formulas to run. For our one machine scenario, we’re going to run the formula hello_web on all systems.

Update the state/top.sls to mach the following:

base:
'*':
- hello_web

Create Top Pillar

Similar to the top salt state, we want to configure variables and data for our systems, which is all systems again, all systems being our one guest Ubuntu system.

Update the pillar/top.sls to match the following:

base:
'*':
- hello_web

Formula Defaults

Now we can dive into developing our formula. First let’s create intelligent default variables that make sense for an Ubuntu system.

In the hello_web formula, update the defaults.yaml with the following:

hello_web:
docroot: /var/www/html
package: apache2
service: apache2

Formula Map

Now we can create a map that glues in all of our variables that we would like to use for this formula. The map is essentially a Jinja template.

In this map, we want to we will import our defaults, optionally specify our own local variables, and then merge in variables from pillar.

In the hello_web formula, update the map.jinja with the following:

{% import_yaml 'hello_web/defaults.yaml' as default_settings %}
{% set hello_web = salt['pillar.get']('hello_web', default=default_settings.get('hello_web'), merge=True) %}

Formula Salt State

The main course has arrived with our Salt State file. In the hello_web formula, update init.sls with the following:

{% from "hello_web/map.jinja" import hello_web with context %}
hello_web:
pkg.installed:
- name: {{ hello_web.package }}
service.running:
- name: {{ hello_web.service }}
- enable: True
- reload: True
file.managed:
- name: {{ hello_web.docroot }}/index.html
- source: salt://hello_web/files/index.html

This needs some explanation…

The first thing here to to make a bold declaration!!! Er, um, rather, I mean an identity declaration that I like to use to group all of the states together under hello_web, but serves a functional purpose to declare a high state component within Salt Stack.

Under this ID declaration, each of the following members will have will be a state module (pkg, service, file) followed by a corresponding state function (installed, running, managed). Under these state module and state function keys, we supply a list of parameters that will be passed to the function, such as a name parameter that makes sense to the state, like a package name, a service name, and a file name.

Salt States can use Jinja, reference variables, whether they are local default variables of the formula, or variables set in a pillar.

Testing the Results

Now let’s bring up our virtual guest, provision the system, and then use lovable curl command on the host:

vagrant up
curl -i 127.0.0.1:8085

This should result in the following:

HTTP/1.1 200 OK
Date: Sat, 11 Aug 2018 10:48:33 GMT
Server: Apache/2.4.18 (Ubuntu)
Last-Modified: Sat, 11 Aug 2018 10:47:41 GMT
ETag: "3c-5732696eef3bc"
Accept-Ranges: bytes
Content-Length: 60
Content-Type: text/html
<html>
<body>
<h1>Hello World!</h1>
</body>
</html>

Part II: Salt Formula and a New Pillar

Let’s shift gears and use the delicious Linux flavor of CentOS from the Bento folks. As the default variables in the hello_web formula will not work well for with CentOS, we’ll need to specify different variables using pillar. The hello_world formula will scoop these up and use them instead of the default variables.

Update the Vagrantfile with the following:

Vagrant.configure("2") do |config|
config.vm.box = "bento/centos-7.5"
config.vm.network "forwarded_port", guest: 80, host: 8083
  ####### File Share #######
config.vm.synced_folder "./roots/salt/", "/srv/salt"
config.vm.synced_folder "./roots/pillar", "/srv/pillar"
  ####### Provision #######
config.vm.provision :salt do |salt|
salt.masterless = true
salt.run_highstate = true
salt.verbose = true
salt.pillar "hello_web" => {
"package" => "httpd",
"service" => "httpd",
"docroot" => "/var/www/html"
}
end
end

We can add new pillar values with the salt provisioner by calling the pillar method and passing in a single ruby hash. Vagrant converts this ruby hash to a Python dict format and passes it on the command line to salt-call state.highstate in our case. For more information, see Vagrant docs on Pillar Data.

Side note: For those unfamiliar with ruby, this is passing in salt.pillar({ "hello_web" => { … } }), but with some syntax sugar to lose the outer curly braces (as ruby knows this is a single hash value), so salt.pillar("hello_web" => { … }). Also, in ruby, you do not need parenthesis for the parameter list, if there is only one space after the method name, so now we have salt.pillar "hello_web" => { … }.

Trying Out New Results

First destroy our existing virtual guest running the previous distro, and bring this new one up with CentOS, and then test again with curl:

vagrant destroy --force
vagrant up
curl -i 127.0.0.1:8085

This will give us the following:

HTTP/1.1 200 OK
Date: Sat, 11 Aug 2018 11:01:25 GMT
Server: Apache/2.4.6 (CentOS)
Last-Modified: Sat, 11 Aug 2018 11:01:08 GMT
ETag: "3c-57326c70410e0"
Accept-Ranges: bytes
Content-Length: 60
Content-Type: text/html; charset=UTF-8
<html>
<body>
<h1>Hello World!</h1>
</body>
</html>

Final Thoughts

Well, not stating the obvious, but this with Vagrant you can achieve a new pillar of success or high state of achievement by sprinkling a little salt across your systems.

You can quickly develop, test, and learn Salt Stack environment in a controlled and repeatable way across a variety of distributions. I hope this tutorial has helped begin that journey.