Nomad on Windows Server 2019 — scheduling a simple ASP.NET Docker Windows container

Welcome to the first part in a series discussing running Nomad in a Windows environment. We’ll start by setting up a basic Nomad lab on Windows Server 2019 and schedule a simple ASP.NET Docker Windows container that you can access from a web browser.

Audience

Before we jump in, I want to clarify a few things about the target audience and assumptions for this series:

• If you’re not familiar with Nomad or what problems a scheduler solves, I recommend checking out the Nomad Intro page on the HashiCorp Developer platform and watching the introduction video by HashiCorp’s Co-Founder & CTO Armon Dadgar.
• You should have basic knowledge of Docker
• This blog is primarily aimed at those who manage a fleet of Windows servers and are interested in modernizing both new and legacy applications.

Now that we’ve covered the groundwork, let’s dive right into the world of Nomad and explore how it can revolutionize your application scheduling process. Get ready for a hands-on journey that will empower you with the knowledge to leverage Nomad’s capabilities effectively.

Why Nomad on Windows? A quick elevator pitch

I wanted to give a quick elevator pitch on why would you even choose Nomad on Windows.

*Puts on salesman hat*

In today’s business landscape, cloud computing has become an important tool, but traditional datacenters still play a vital role and are here to stay. Many companies are adopting multi-cloud strategies to reduce risk and costs, leveraging existing investments in datacenters and colocations. However, traditional data centers often house legacy applications that cannot be modernized due to a variety of reasons.

Nomad offers a distinctive solution that caters to legacy workloads built on and designed for Windows. Its cloud and platform-agnostic nature provides great flexibility, allowing organizations to crawl, walk, and run through the application modernization journey without being tied to a specific cloud provider. To elaborate, this provides a smooth transition to different application frameworks or underlying deploying methods. For example, moving from a .NET 4.8 app that is executing directly on the OS to a Docker container can easily be done by changing a couple of lines within the Nomad job manifest file.

Simple

Nomad’s brilliance is in its simplicity compared to other solutions. You’ll see below that getting Nomad up and running is extremely simple. It’s a single binary with a config file and that’s it. The config file is written in HCL, so it’s extremely easy to read and understand. Nomad documentation explains what each option does to easily give you the greatest flexibility. I’ve also outlined what each option does in the lab below.

But wait, there’s more!

When we think of a scheduler, we usually think of Kubernetes, which can schedule only CRI workloads. However, Nomad can schedule just about anything.

• Batch jobs
• Docker
• LXC
• Podman
• Raw executables (The raw application itself)
• Java
• IIS (community support)

With those benefits in mind, lets get into the lab.

Lab: Installing Nomad server/client

If you are familiar with HashiCorp Vault or Consul, then you’ll feel right at home with Nomad’s architecture.

Single Region

Multi-Region

• Nomad uses a server-client architecture.
• Server’s are the control plane and API endpoint. The brains of the cluster.
• Clients are the worker (compute) nodes. A single binary agent runs on the node and executes the actual workloads that the server nodes schedule on them.
• Raft is used as the consensus protocol for the Nomad servers in a 3 or 5 node configuration to maintain quorum. With Nomad Enterprise, you can leverage read-only replicas and multi-region federation for scale.
• RPC is used for communication between client agents and servers.
• Serf is used for WAN gossip.

For some bedtime reading, you can check out more in-depth information about Nomad’s architecture on our documentation page.

Goal

For this lab we are going to create 3 Windows Server 2019 VM’s, install Nomad, cluster them together, and schedule an ASP.NET Windows Docker container.

Create 3 VMs

I am running the Nomad nodes in both client and server mode, which is not ideal for production, but can be perfectly fine for small test/dev deployments and labs. Pick your platform of choice (I am using my homelab which is running Proxmox on an HP Proliant 1U server) and perform the following:

• Download Windows 2019 Server and install
• Install Windows updates and any drivers needed. For example: virtio if running on kvm.
• For production, it’s recommended this be a standardized image using something like HashiCorp’s Packer.
• For optimal performance, we suggest a minimum of 4 cores and 8 GB memory per VM. Keep in mind that Windows consumes a significant amount of resources, so it’s necessary to have sufficient overhead to run a decent number of applications. To reduce overhead, you can consider using Server Core if you’re familiar with Powershell.

Install Docker Engine and Dockercli

Microsoft provides an easy Powershell script to enable all the features required for Windows containers and install Docker. You can go to Microsoft’s Docker Quickstart Guide to download and run it.

If the service doesn’t get created, you can run:

& "C:\Program Files\Docker\Docker\resources\dockerd" --register-service

Run:

& "C:\Program Files\Docker\Docker\DockerCli.exe" -SwitchWindowsEngine

This will switch the daemon to Windows containers. By default, Docker uses Linux containers and only one can be running at a time.

Note

At the time of writing this, the only way to download dockercli.exe is to download Docker desktop for Windows. If you are running this in a lab, this isn’t an issue. However, if you are running this is in production there are licensing constraints with Docker desktop, so you may need to use something like HashiCorp Packer to build the image with dockercli baked in without Docker desktop.

Install Nomad

1. Download binary: https://developer.hashicorp.com/nomad/downloads
2. Create directory: C:\Nomad\data
3. Unzip binary to newly created directory
4. Configure Nomad as a Windows service

New-Service -Name "Nomad" -BinaryPathName "C:\Nomad\nomad.exe agent -config=C:\Nomad\nomad.hcl" -StartupType Automatic

Firewall rules

Ensure the following ports are open on the Windows firewall:

HTTP API — TCP — 4646
RPC — TCP — 4647
Serf WAN — TCP/UDP — 4648

Configure Nomad

1. Create a file in C:\Nomad called nomad.hcl

2. Paste the config below. Each option is explained below.

Notice the double backslash \\ for directories. This is necessary because HCL by default uses \ as an escape character, which means that to define directory paths correctly, you need to use a double backslash to enable HCL to interpret the directory accurately.

#Local directory used to store agent state. Client nodes use this directory by
#default to store temporary allocation data as well as cluster information. 
#Server nodes use this directory to store cluster state, including the 
#replicated log and snapshot data.

data_dir = "C:\\Nomad" 

#Specifies which address the Nomad agent should bind to for network services, 
#including the HTTP interface as well as the internal gossip protocol and 
#RPC mechanism.

bind_addr = "0.0.0.0"

#Specifies the path for logging.

log_file = "C:\\Nomad\\logs.txt"

#By default, logs will grow in size forever until the disk is filled to 100%, 
#so if this is a long lived lab environment, you may want to consider adding 
#in the log rotation options below. Max space logs will take up is 5mb.

log_rotate_bytes = "100000"
log_rotate_max_files = "5"

#The `server` block configures the Nomad agent to operate in server mode to 
#participate in scheduling decisions, register with service discovery, 
#handle join failures, and more.

server {
  #Default is false so you must explicitely tell Nomad to run in server mode.
  
  enabled = true

  #Specifies the number of server nodes to wait for before bootstrapping. 
  #For production you want 3 or 5, but for a lab, 1 is acceptable especially 
  #if you are lacking resources to run 3 VMs.
  
  bootstrap_expect = 1

  server_join = {
  
 #Replace these with the IPs of this VM and the other 2 Nomad VMs you deployed.
    retry_join = [ "1.1.1.1", "2.2.2.2", "3.3.3.3" ]
    retry_max = 3
    retry_interval = "15s"
  }
}

#The `client` block configures the Nomad agent to accept jobs as assigned by 
#the Nomad server, join the cluster, and specify driver-specific configuration.

client {
#Specifies if client mode is enabled. Default is false so you must explicitely 
#tell Nomad to run in client mode.
  enabled = true

#Specifies a list of server addresses to join and will continue to attempt 
#to join even after a failure at a specified occurence.
  server_join {
  
 #Replace these with the IPs of this VM and the other 2 Nomad VMs you deployed.
    retry_join = [ "1.1.1.1", "2.2.2.2", "3.3.3.3" ]
    retry_max = 3
    retry_interval = "15s"
  }
}

#For learning, definitely enable the UI. You can enable this on one node or 
#all of them. It's up to you.

ui { enabled = true }

#Tells Nomad to utilize Docker to handle downloading docker containers, 
#mapping ports, and starting, watching, and cleaning up after containers.

plugin "docker" {
  config {
  enabled = true
#Allows tasks to bind host paths inside their container. By default, a 
#container can only access directories and files within its allocation 
#directory due to security reasons. However, in some cases, you may have an 
#SMB mount on the host that you want your containers to access. 
#In such scenarios, this feature can be quite useful as it allows a container 
#to access a directory on the host that lies outside of its allocation 
#directory. We will talk more indepth about storage in a later blog post.

  volumes { enabled = true }

#This is required due to a bug in Nomad and/or Docker. 
#For lab purposes "all" is acceptable, however with production, 
#you will want to explicitely state what is allowed.

   allow_caps = ["all"]
  }
}

#Use to exec commands on the Windows host without isolation. 
#Useful for installing and running application right on the host.
#We will be discussing this in a later blog post.
plugin "raw_exec" {
  config { enabled = true }
}

1. If not started, start Nomad service:
2. Start-Service Nomad

Interact with Nomad remotely

To interact with your Nomad cluster from a remote location (e.g. From your local laptop), perform the following:

Windows

1. Download the binary
2. Create a directory called “Nomad” at C:\
3. Unzip to C:\Nomad
4. Run the following in Powershell: $Env:PATH += "C:\Nomad\"
5. Run the following in your shell session:
   export NOMAD_ADDR=<ip of nomad server node>:4646

Mac & Linux

1. Download the binary
2. Extract to /usr/bin
3. Run the following in your shell session:
   export NOMAD_ADDR=<ip of nomad server node>:4646

Your first job

Now the exciting part! Let’s schedule our first job using Docker.

For more bedtime reading, you can read how Nomad scheduling works under the hood.

IPv6

After deploying a job without modifying any network settings outside of a static IP, Nomad and, in turn, Docker will advertise the IPv6 address as the endpoint for jobs. For learning purposes, I recommend disabling IPv6 entirely.

1. Go to Network interface settings > Properties
2. Uncheck IPv6
3. Apply

Job file example

job "windows-legacy" {

group "windows-legacy" {
  count = 1
  network {
    port "http" {
      to = 80
      static = 80
    }
  }

task "windows-helloworld" {
  driver = "docker"
  config {
    image = "mcr.microsoft.com/dotnet/samples:aspnetapp"
    image_pull_timeout = "20m"
    ports = ["http"]
  }
 }
}
}

After performing a nomad job run, you should be able to navigate to http://<ip of node> and receive a simple ASP.NET web page.

Conclusion

I hope your introduction to Nomad was enjoyable and I inspired you to continue learning more about Nomad and how you can leverage the capabilities of an advanced scheduler like Kubernetes without all the complexity. I will be adding onto this series and discussing more advanced topics, such as:

• Deep dive into the different Nomad task drivers and the recommended patterns
• Networking patterns
• Storage patterns
• Consul, Vault, and other integrations
• and more

If I’ve successfully sparked your interest in Nomad and you’re eager to dive deeper, I encourage you to visit HashiCorp’s Developer site for Nomad. There, you’ll find a plethora of tutorials covering a wide range of topics that will help you get started and satisfy your Nomad itch!