Just-in-Time Nomad: Templating HashiCorp Nomad Jobs with Nomad Packs

Published in

Dzero Labs

10 min readFeb 26, 2022

Mural at the back of Osler Fish Market, on the West Toronto Rail Path. Photo by Adri Villela.

If you’re familiar with Kubernetes, chances are, you’ve heard of Helm, a templating and packaging tool for your Kubernetes deployments. Unfortunately, there was no such tool for Nomad…that is, until recently. On November 16th, 2021, HashiCorp introduced Nomad Pack in Tech Preview, with the announcement of Nomad 1.2.

I love me a good comparison between Kubernetes and Nomad, so I’ve been dying to try out Nomad Pack since it was announced. But alas, other things have gotten in the way of my explorations. Welp…I finally sat down this week to get a little taste of Nomad Pack, and today, I’ll be sharing my learnings with you!

Are you ready?! LET’S DO THIS!! 💪

Objective

Today, we will:

Learn Nomad Pack basics
Learn how to install Nomad Pack
Create our own Nomad Pack for an existing Nomad Jobspec

Assumptions

Before we move on, I am assuming that you have a basic understanding of HashiCorp Nomad. If not, mozy on over to my Nomad intro post.

Pre-Requisites

Note: If you’ve followed this series, you’ll know that I’m a huge fan of using HashiQube for my Nomad explorations on my local machine. It is a virtualized environment that runs a bunch of the Hashi tools together, including Vault, Consul, Nomad. I’ll be running the example using HashiQube; however, if you have a different setup, feel free to skip the “Provision a Local Hashi Environment with HashiQube” section below.

In order to run HashiQube, you’ll need the following:

Oracle VirtualBox (version 6.1.30 at the time of this writing)
Vagrant (version 2.2.19 at the time of this writing)

Part 1: Creating a Nomad Pack + An Overview

As I mentioned before, Nomad Pack is a tool for templating your Nomad jobspecs. You might be wondering, “Why in Space would I need such a tool?”The main use case is the ability to use the same Nomad jobspec for different environments. That is, you inject environment-specific configs into a single jobspec. One jobspec to rule them all! MUAHAHAHA! 🦹‍♀️

Now that we’ve got that out of the way…

It’s worth noting that HashiCorp already has a collection of canned Nomad Packs in their Nomad Pack Community Registry, including one that’s near and dear to my heart: an OpenTelemetry Collector pack submitted by fellow Observability geek, Jason Harley.

That’s all well and good, but chances are, that you will need to build your own. Lucky for you, I’m here to guide you!

Nomad Pack Example

First things first, when you create your own Nomad pack, you need to follow a specific directory structure. It looks like this:

.
└── README.md
└── packs/
    └── <PACK-NAME-A>/
        └── ...pack contents...
    └── <PACK-NAME-B>/
        └── ...pack contents...
    └── ...packs...

Where <PACK-NAME-A> and <PACK-NAME-B> are, as you would imagine, the name of your Nomad Packs. Create one pack, or a zillion, but they must all be housed in the packs directory.

For our example, we’ll be using…you guessed it…my trusty ‘ole 2048-game!

Note: You don’t have to create any of the files in this section, since the tutorial repo in Part 2 already has them; however, it might still be good to do, just to get a feel for creating them yourself.

1- Open up a terminal window, and create a folder for our Nomad Packs project

mkdir nomad-packs-examples

2- Navigate over to our newly-created directory above

cd nomad-packs-examples

3- Create the Nomad Pack directory structure and bare-bones readme for our example

mkdir -p packs/2048_game/templates
echo "#2048-game Pack" >> packs/2048_game/README.mdcd packs/2048_game

Our Nomad Pack, called 2048_game, resides in a directory inside the packs directory, which we called 2048_game, to keep things consistent. I’ll tell you about the templates directory shortly. But first, we need to create some supporting files.

4- Create metadata.hcl

This file contains information about the pack. Here’s what ours looks like:

Let’s create ours from the above gist:

git clone https://gist.github.com/000d887f27cbc7d68d2caf1e17cfdfb5.git# Cleanup - move metadata.hcl to our 2048_game folder
mv 000d887f27cbc7d68d2caf1e17cfdfb5/metadata.hcl .
rm -rf 000d887f27cbc7d68d2caf1e17cfdfb5

So…a few things about metadata.hcl:

app.url: The URL of your application. When we deploy the 2048-game app later on, you’ll have it running on http://2048-game.localhost, so that’s what I’m using.
app.author: The app author’s name. Credit where credit is due. The author of the 2048-game I’ve been using is GitHub user alexwhen.
pack.name: The name of your pack. It’s good to keep it consistent with your directory name.
pack.description: Short description of your pack.
pack.url: Location of your pack. I’ve added the location of the pack in the GitHub repo I’m using for this tutorial (not the Gist).
pack.version: Version of the pack. Probably a good idea to keep that up-to-date, for the sake of your users. 😃

Note: If you’re wondering why we didn’t name our pack 2048_game instead of game_2048, it’s because you can’t start a pack name with a number. You also can’t use hypens (-) in your pack name.

5- Create variables.hcl

This file contains the variables that you’ll be injecting into your template file. Here’s what ours looks like:

If you’re familiar with Terraform or Waypoint, the variable definition will look familiar to you. The basic structure looks like this, with a few variations based on data type:

variable "<variable_name>" {
  description = "<some_description>"
  type        = <some_datatype>
  default     = <some_default_value>
}

In our variables.hcl above, you’ll notice that we use different variable types, including object (map), list (array), number, and string. For more on variable types, I recommend checking out the Terraform docs, since the concepts translate to Nomad Packs.

Let’s create our variables.hcl:

git clone https://gist.github.com/cf98baa82365953e4fb989889a7169d3.git# Cleanup - move variables.hcl to our 2048_game folder
mv cf98baa82365953e4fb989889a7169d3/variables.hcl .
rm -rf cf98baa82365953e4fb989889a7169d3

6- Create outputs.tpl

This file prints a nice little output message after your pack is created. Here’s what ours looks like:

You may have noticed some of the funny-looking notation, such as this:

[[ .game_2048.app_count ]]

That’s the templating notation. It says, “Look at the variables.hcl file, and in the pack called game_2048, find me a variable called app_count.” The value game_2048 comes from our metadata.hcl file that we created earlier.

Pro Tip: If the name of the pack in ouputs.tpl doesn’t match up the value of pack.name in metadata.hcl, Nomad Pack will scream at you.

Or, with a more complex example in which we’re referencing a field within a variable of type object, we’d reference it like this:

[[ .game_2048.docker_artifact.image ]]

And finally, we also have this:

[[ .nomad_pack.pack.name ]]

Where .nomad_pack refers to Nomad Pack system variables.

Let’s create outputs.tpl now:

git clone https://gist.github.com/cc8d11d49190f5313ca8483dd2ff1244.git# Cleanup - move outputs.tpl to our 2048_game folder
mv cc8d11d49190f5313ca8483dd2ff1244/outputs.tpl .
rm -rf cc8d11d49190f5313ca8483dd2ff1244

7- Create the jobspec template, 2048-game.nomad.tpl

The jobspec template mostly looks like a regular ’ole jobspec, except that it has the template notation, like we saw in outputs.tpl.

Pro Tip: Your template file MUST reside in the templates folder, otherwise Nomad Pack will get mad at you.

Here’s what our template looks like:

As you can see, we’ve parameterized a number of values in 2048-game.nomad.tpl. You can parameterize as little or as much as you want, depending on your situation.

I also want to highlight line 4 of 2048-game.nomad.tpl, since it’s something we didn’t see in outputs.tpl. If you recall our datacenters variable (lines 7–11) in variables.tpl, you’ll notice that it’s a list.

variable "datacenters" {
  description = "A list of datacenters in the region which are eligible for task placement."
  type        = list(string)
  default     = ["dc1"]
}

To display values from a list, your template looks like this (line 4 of 2048-game.nomad.tpl):

datacenters = [ [[ range $idx, $dc := .game_2048.datacenters ]][[if $idx]],[[end]][[ $dc | quote ]][[ end ]] ]

Basically, it’s just a loop.

If you want to go super-fancy, you can go wild and create helper templates. (Helm has a similar concept.) These files are aptly named _helpers.tpl and reside in the same directory as your jobspec template (i.e. in the templates folder). Helper templates let you re-use template snippets and do all sorts of fancy things, like conditionals and loops, much like you can do with jinja templates in the world of Python.

Note: If you’re looking for a really good advanced example of using helper templates, I highly recommend that you check out the _helpers.tpl in the OpenTelemetry Collector Nomad Pack.

Let’s create our 2048-game.nomad.tpl:

git clone https://gist.github.com/84fd986b73256e1740545de7cd53b349.git# Cleanup - move 2048_game.nomad.tpl to our 2048_game/templates folder
mv 84fd986b73256e1740545de7cd53b349/2048-game.nomad.tpl templates/.
rm -rf 84fd986b73256e1740545de7cd53b349

Okay…now that we know the basics, we’ll learn how to run our Nomad Pack in the next section.

Part 2: Applying a Nomad Pack

I will be using a modified version of the HashiQube Repo (a fork of servian/hashiqube) for today’s tutorial. It includes all of the source files that you created in Part 1.

If you’re curious, you can see what modifications I’ve made here.

1- Provision a Local Hashi Environment with HashiQube

Start HashiQube by following the detailed instructions here.

Note: Be sure to check out the “Gotchas” section, if you get stuck.

Once everything is up and running (this will take several minutes, by the way), you’ll see this in the tail-end of the startup sequence, to indicate that you are good to go:

Final output of the Vagrant VM startup sequence

You can now access the services below:

Vault: http://localhost:8200
Nomad: http://localhost:4646
Consul: http://localhost:8500
Traefik: http://traefik.localhost
Waypoint: https://192.168.56.192:9702

2- Install the Nomad Pack Binary

To install the Nomad Pack binary open up a terminal window on your host machine (i.e. your own computer, not the Vagrant VM), and run the following:

git clone https://github.com/hashicorp/nomad-packcd ./nomad-pack \
  && make devexport NOMAD_PACK_DEST="/usr/local/bin/nomad-pack"
cp bin/nomad-pack $NOMAD_PACK_DEST

Let’s see if it worked! Run the following in your terminal:

nomad-pack registry list

The firs time the above command is run, nomad-pack pulls all Packs from the Nomad Pack community registry repo, and saves them to a local cache. This cache is located at:

$HOME/Library/Caches/nomad/packs on Mac
$HOME/.nomad/packs on Linux

Your output will look something like this:

Okay! We’re in business! Are you getting excited??

3- Run the Nomad Pack

Note: We’ll be running our commands assuming that you’re starting from the repo root in your terminal on the host machine (i.e. the hashiqube directory). If you prefer, you can cd over directly to the pack directory, hashicorp/nomad/packs/2048_game. Once you’ve done that, you would replace any occurrences of hashicorp/nomad/packs/2048_game in the commands below with ..

Before we run the Nomad Pack, let’s do a couple of checks, to make sure that everything is looking good.

First, let’s see if we can get the info about our pack. Run the command below:

nomad-pack info hashicorp/nomad/packs/2048_game

Sample output:

As you can see, we can see some info about our game_2048 pack. The info command pulls data from our metadata.hcl and our variables.hcl files, which we learned about in the previous section, “Part 1”.

Before applying our game_2048 Pack to Nomad, it’s good idea for us to render our template. This helps us verify whether the Jobspec that we’re posting to Nomad looks the way we expect it to. It also runs some validatons, checking for any boo-boos in our template. We do this by running this command:

nomad-pack render hashicorp/nomad/packs/2048_game

Note: If you’re familiar with Helm, the above command is similar to the helm template command.

Sample output:

As you can see per the output above, all occurrences of [[ .game_2048.<var_name> ]] have been replaced with the appropriate values from our variables.hcl file. Pretty neat, huh?

Now that we’ve checked to make sure that things look good, let’s deploy our pack to Nomad by running this command:

nomad-pack run hashicorp/nomad/packs/2048_game

The above command will:

Render the jobspec (same as the nomad-pack render command)
Deploy the jobspec to Nomad (same as the nomad job run command)

Upon successful completion of the job in Nomad, it prints a nice message…the same message that we configured in our outpus.tpl file. Whaaat? 🤯

We can see that our job has been deployed to Nomad by checking the UI:

Well, hello my prettyyyyy!

Or by running the following command from the CLI:

nomad job status 2048-game

Sample output:

Thing of beauty!

If you want to nukify your job, you can run this command:

nomad-pack destroy hashicorp/nomad/packs/2048_game

Sample output:

To check that our job has gone bye-bye:

nomad job status 2048-game

It’s gone!!! 😳

So there you have it. You have created and deployed your first Nomad Pack! Woo hoo!! 🎉

Conclusion

Congrats! You’ve just created your very first Nomad Pack! You should be super proud of yourself!! Quick recap of what we’ve done:

We’ve learned that Nomad Pack is a templating tool for Nomad jobspecs — like Helm for Kubernetes
We learned the anatomy of a Nomad pack by creating our own for the 2048-game
We used Nomad Pack to deploy a jobspec to Nomad

Now, please enjoy this photo of a white yak on a brown field.