No Workload Left Behind: How to orchestrate with HashiCorp Nomad, Part 2

Published in

Globant

13 min readMar 15, 2023

This is the second part of my introductory article to HashiCorp Nomad, a generic workload orchestrator that can orchestrate anything besides containers. In that article, we reviewed Nomad's features, architecture, and how it compares with other orchestrators like Kubernetes.

In this article, we will review Nomad's installation methods and set up a local development environment to run some Jobs to test some of the core features we covered in the first article. And in the following article, we will review some of the features that make Nomad unique. But before that, we will delve into some essential concepts to understand how Nomad works.

Nomad concepts

You need to understand Nomad's core concepts to work with it.

Job: Declares the desired state of a workload for Nomad to run.
Driver: the interface or context under which the task will run. For example: docker, java or win_iis.
Task: the smallest unit of work and specifies the configuration, constraints, and resources required to run.
Task Group: Related tasks that must be executed on the same client node and cannot be separated. A task group is the equivalent of a pod in Kubernetes.
Client: basically the equivalent of a worker node in Kubernetes. Runs and manages tasks using its resources and must report to a server.
Server: the equivalent of the control plane in Kubernetes. They are responsible for managing jobs, resource allocation, and running evaluations. Servers talk to each other and use a leader/follower load balancing method for HA.
Allocation: a task group in a job executed in a client node. The Nomad servers create allocations as part of scheduling decisions made during an evaluation.
Evaluation: a mechanism to validate if everything is in its desired state; if not, it allows the servers to perform corrective actions and perform changes in allocations if necessary.
Data Center: an abstract grouping of clients within a region.
Region: Contains one or more data centers. A region comprises servers, which can connect across regions to make a global network.
Raft consensus protocol: Allows the cluster to elect a leader and set the followers.
Gossip protocol: Securely shares information between servers and clients about its configuration and topology.

How to run Nomad?

As mentioned in the previous article, Nomad is available as a pre-compiled binary or as a package for Linux, Windows, or macOS. For a local installation, it is as simple as downloading a single binary and running a server and at least one client. You can also test Nomad using Vagrant, but the recommended way is for a production cluster using Terraform and Packer to do all the provisioning.

Local Install

Remember that the Nomad binary is used for both server and client, and its role is set according to how it is configured in the configuration file that is passed to the binary at runtime. The binary can be executed like this:

$ sudo nomad agent -config /etc/nomad.d/nomad.hcl

The /etc/nomad.d/nomad.hcl file contains the server or client configuration. This is an example of a basic server configuration:

Nomad basic server configuration

At the top of the file, there are some global configuration options, like:

data_dir, the directory where Nomad's configuration and state files are stored
bind_addr, the IP address Nomad will bind to
datacenter, the name of the region this Nomad server belongs to

Then there is the server stanza, which is the place where server configuration parameters are configured. For example, bootstrap_expect sets the number of expected servers on the cluster (1, 3, or 5 according to the Raft consensus protocol) to be able to start and reach a consensus. This option is mandatory for server configuration.

The consul stanza is used to point the Nomad cluster to a remote Consul cluster. But if Consul is already running on the same server where a Nomad server or client is running, Nomad will detect it and register itself on it without having to use the consul stanza. The vault stanza points Nomad to a Vault server for secrets management.

The acl stanza is used to configure access control to the Nomad server through the web GUI or the CLI, and to control Job deployment by defining ACL policies. It is worth noting that this feature comes disabled by default and needs to be enabled and properly configured for a production setup. I guess that the reason for this is to facilitate testing and Job development by developers and to allow for complete customization according to each project's needs; ACL configuration is a complex topic that would need a whole article just for that.

This is an example of a client configuration:

Nomad basic client configuration

The main difference with the previous server configuration file is the client stanza. This configuration parameter defines Nomad's configuration as a client instead of a server. Here are defined things like the servers this client should join, or to enable the drivers to run different kinds of workloads, according to what the node where the client is running supports (i.e., you cannot enable the Java driver in a node that does not have a JRE installed). Other options are common to the server configuration, like the consul or vault stanzas.

These binaries could be configured as system services, so they are launched at instance boot. In the case of Linux, you could define a Systemd file like the following:

Example Nomad Systemd file

There are also instructions on how to set up a Windows service in the official documentation. But to play around with nomad in your dev laptop, you can run the Nomad binary using the -dev parameter, which will launch Nomad as both server and client:

$ sudo nomad agent -dev -bind 0.0.0.0 -log-level INFO

Vagrant

You can also use Vagrant to set up a development environment for Nomad. Vagrant is another HashiCorp tool for building and managing local virtual machine environments, which is very useful for development or doing tests. After having installed Vagrant, to run Nomad in it, you need a Vagrantfile. HashiCorp provides a Vagrantfile that starts a small Nomad cluster. First, create a new directory for your Vagrant environment and cd into it:

$ mkdir nomad-demo
$ cd nomad-demo

Then download the Vagrantfile:

$ curl -O https://raw.githubusercontent.com/hashicorp/nomad/master/demo/vagrant/Vagrantfile

Now, start the VM:

$ vagrant up

An Ubuntu VM image will be downloaded and launched and then provisioned with Docker and Nomad:

Bringing machine 'default' up with 'virtualbox' provider…
==> default: Importing base box 'bento/ubuntu-18.04'…
…
==> default: Running provisioner: docker…

Wait a few minutes, and the Vagrant box will be ready to use. Once the Vagrant box is running, use the ssh command to start a shell session.

$ vagrant ssh

To test that the nomad command is available, run this command:

$ nomad version
Nomad v1.3.5 (1359c2580fed080295840fb888e28f0855e42d50)

Terraform & Packer

The previous installation methods are oriented towards a development or learning setup. Still, when deploying Nomad in a production environment, you will need a more automated deployment strategy that can scale over time as your infrastructure grows. For this task, then Terraform and Packer could be adequate options. Nomad could be packaged as part of a VM image using Packer, and Terraform can be used to deploy (and upgrade) a set of nodes in the cloud that use that image. But this topic deserves its own article; for now, you could check this example project that deploys a Nomad cluster in AWS using both Terraform and Packer as a starting point for your own cluster.

How to run a workload?

To run anything in Nomad, we need to define a Job. These jobs are files written in the HCL language, the same language used by other HashiCorp tools like Terraform. If you know the HCL language to write Terraform code, you won't have trouble writing Nomad client/server configuration files or Jobs.

A Job defines the desired state of the application; it defines the job configuration, things like the resources it needs, network configuration, and the Nomad region it should run. The Nomad servers are responsible for ensuring the actual state matches the user desired state, which is done by finding an optimal placement for each task such that resource utilization is maximized while satisfying all constraints specified by the job. These constraints could be technical requirements such as hardware architecture, the availability of GPUs, or a specific operating system and kernel version, etc.

Tasks are scheduled on the nodes in the cluster running Nomad, configured as a client. When a job is launched, a server creates an evaluation to find the most suitable client for the task groups defined for that job in particular. According to the evaluation result, it creates a Job allocation in the selected client nodes. An evaluation is created when the external state changes, either desired or emergent. This means that every time a Job is submitted and an evaluation determines that the desired state has changed, a current job will be updated (or deregistered).

*Nomad job scheduling (Source:* *HashiCorp*)

The following is an example of a Nomad job:

Example Nomad job file

The general hierarchy for a job is as follows:

job
  \_ group
      \_ task

Each job file has only a single job; however, a job may have multiple groups, and each group may have multiple tasks. The purpose of groups is co-locating tasks on the same machine. Each task is a unique action for Nomad to execute. Tasks define the task driver to use, its configuration, and the resources required by the task.

Nomad Demo

Now that we have covered Nomad's core concepts, we are ready to continue and set up a local environment to try it out. We are going to do a local Nomad install and launch three different jobs to test the following features:

Run an example job and test the basic features anyone can expect in an orchestrator, like job deployment, job escalation, and rollbacks.
Run other kinds of workloads different from containers, like a static binary and a Java application.

Nomad installation

Let's start by installing Nomad locally and executing it in dev mode. The following instructions are for Debian/Ubuntu Linux; if you are using a different OS, please refer to the installation instructions and look up the commands for your OS. Run the following commands in a terminal to add the HashiCorp deb repository as a source and install Nomad:

$ wget -O- https://apt.releases.hashicorp.com/gpg | gpg - dearmor | sudo tee /usr/share/keyrings/hashicorp-archive-keyring.gpg
$ echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
$ sudo apt update && sudo apt install nomad

After finishing the installation, in a terminal launch nomad in dev mode:

$ sudo nomad agent -dev -bind 0.0.0.0 -log-level INFO

You will see the output of Nomad starting up. When you see these messages:

2022–09–19T22:53:30.902–0500 [INFO] nomad: cluster leadership acquired
2022–09–19T22:53:30.903–0500 [INFO] nomad.core: established cluster id: cluster_id=d99996a7–8ab2-eba4–91d4–52c8d759366c create_time=1663646010903589813
2022–09–19T22:53:30.904–0500 [INFO] nomad: eval broker status modified: paused=false
2022–09–19T22:53:30.904–0500 [INFO] nomad: blocked evals status modified: paused=false
2022–09–19T22:53:30.926–0500 [INFO] client: node registration complete
2022–09–19T22:53:31.929–0500 [INFO] client: node registration complete

Based on the info log, you can be sure that the Nomad server and agent started successfully, and we are ready to deploy a new job. Now, open a second terminal, from here we are going to launch a job (you can use something like tmux or screen commands if you are connecting remotely). First, let's check Nomad's available nodes. Run the command nomad node status to check that a node is up and running and ready to work:

$ nomad node status
ID DC Name Class Drain Eligibility Status
c0d2fa96 dc1 nomad-demo <none> false eligible ready

Now that we verified that one client node is running, we can run some jobs.

Running an example job

We are going to start with a simple job. To create an example job file, we can use the nomad job init command. This command will output a basic example of a job.

$ nomad job init

An example.nomad file will be created. The example job file is very well documented; it explains each configuration parameter in the job file. If you want a cleaner file without any comments, we can pass the -short parameter to the nomad job init command (this is the example job we presented before when explaining how to run Nomad). This example job runs a Redis database and defines the data center where the job should be deployed. If you remember from the nomad status command output, "dc1" is the name of the data center. It also defines the name of the Docker image to use and the resources to assign to it. Now let's deploy the job. For this, we use the nomad job run command.

$ nomad job run example.nomad

The command outputs information about the scheduling process:

==> Monitoring evaluation "6792e7a6"
    Evaluation triggered by job "example"
    Evaluation within deployment: "99297d1d"
    Allocation "635a981a" created: node "3825e530", group "cache"
    Evaluation status changed: "pending" -> "complete"
==> Evaluation "6792e7a6" finished with status "complete"

Here, the output indicates that an evaluation was performed to find a suitable node to deploy this job. The result of that was the creation of an allocation that is now running on the local node. Now, let's check the status of the job with nomad job status command:

$ nomad job status example        
ID            = example
Name          = example
Submit Date   = 2022-09-23T09:56:08-05:00
Type          = service
Priority      = 50
Datacenters   = dc1
Namespace     = default
Status        = running
Periodic      = false
Parameterized = false

Summary
Task Group  Queued  Starting  Running  Failed  Complete  Lost  Unknown
cache       0       0         3        0       0         0     0

Latest Deployment
ID          = f351d71d
Status      = successful
Description = Deployment completed successfully

Deployed
Task Group  Desired  Placed  Healthy  Unhealthy  Progress Deadline
cache       3        3       3        0          2022-09-23T10:06:19-05:00

Allocations
ID        Node ID   Task Group  Version  Desired  Status   Created    Modified
a52e34e0  956f8047  cache       1        run      running  1h6m ago   1h5m ago
e2773027  956f8047  cache       1        run      running  1h6m ago   1h5m ago
bb177e47  956f8047  cache       1        run      running  1h27m ago  1h5m ago

When Nomad runs a job, it creates allocations based on the task groups within the job. These allocations are equivalent to pods in Kubernetes. To inspect an allocation, use the nomad alloc status command.

$ nomad alloc status a52e34e0
ID                  = a52e34e0-4d30-ddd6-14ca-0b3a23201c2d
Eval ID             = 2cd0caed
Name                = example.cache[1]
Node ID             = 956f8047
Node Name           = nomad-demo
Job ID              = example
Job Version         = 1
Client Status       = running
Client Description  = Tasks are running
Desired Status      = run
Desired Description = <none>
Created             = 1h5m ago
Modified            = 1h5m ago
Deployment ID       = f351d71d
Deployment Health   = healthy

Allocation Addresses
Label  Dynamic  Address
*db    yes      127.0.0.1:24974 -> 6379

Task "redis" is "running"
Task Resources
CPU        Memory           Disk     Addresses
4/500 MHz  2.4 MiB/256 MiB  300 MiB   

Task Events:
Started At     = 2022-09-23T14:56:09Z
Finished At    = N/A
Total Restarts = 0
Last Restart   = N/A

Recent Events:
Time                       Type        Description
2022-09-23T09:56:09-05:00  Started     Task started by client
2022-09-23T09:56:08-05:00  Task Setup  Building Task Directory
2022-09-23T09:56:08-05:00  Received    Task received by client

We can see details of this particular allocation, like the hardware resources in use. Now let's see the logs of the Redis task:

$ nomad alloc logs a52e34e0 redis
1:C 23 Sep 2022 14:56:09.775 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 23 Sep 2022 14:56:09.793 # Redis version=7.0.4, bits=64, commit=00000000, modified=0, pid=1, just started
1:C 23 Sep 2022 14:56:09.793 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
1:M 23 Sep 2022 14:56:09.793 * Increased maximum number of open files to 10032 (it was originally set to 1024).
1:M 23 Sep 2022 14:56:09.793 * monotonic clock: POSIX clock_gettime
1:M 23 Sep 2022 14:56:09.794 * Running mode=standalone, port=6379.
1:M 23 Sep 2022 14:56:09.794 # Server initialized
1:M 23 Sep 2022 14:56:09.794 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl v
m.overcommit_memory=1' for this to take effect.
1:M 23 Sep 2022 14:56:09.794 * Ready to accept connections

Until now, we have been working on the command line, but Nomad also has a web UI. To access it, open the http://IP_ADDRESS:4646/ui/ address in your browser, replacing IP_ADDRESS with the address where you are running the Nomad binary. If you are doing it in your laptop, then you can use localhost or 127.0.0.1, but if you are running it remotely or in a VM, you will need to find out the corresponding IP address, as in http://192.168.0.22:4646/ui/.

As we are running Nomad in a development mode, there isn't any kind of authentication, but ACLs should be enabled and configured for production deployment. After we enter the web UI, we land on the jobs page, where we can see the example job we launched on the command line:

We can see the list of jobs being executed in the Nomad cluster. We can see other information in the left panel, like the client and server nodes. Going back to the Jobs page, if we select a Job, we can see some more detailed information, like its execution status, the task groups, and allocations:

When clicking on the Task Group, we can see the allocations of this group:

When clicking on an allocation, its details are shown, like resource utilization, the files part of the tasks part of the allocation, and the details of the tasks:

Here we can see the details of this particular task, the resources used, the list of the task's events, and log output and files:

To see the logs, click on the Logs tab at the top:

Conclusions

In this article, we tested some of Nomad's core features described in the first article. We reviewed some important concepts to better understand Nomad's inner workings to prepare for the hands-on lab and learned different installation methods. About Nomad's job definition, I think we could say that using the HCL language to do the Job definition helps in making them simple, self-documented, and clear to understand without the need to use multiple files to run a single application. The client CLI is clear and straightforward to use, and it is nice to have an accompanying web GUI included as part of the server.

Finally, we ran an example job to test some core features expected in any modern orchestrator, but still, we have yet to test some of the features that make Nomad unique.

In the next part of this article series, we will test some of these features that differentiate Nomad from other orchestrators, such as being able to run workloads different from containers. Stay tuned.