Tutorial on running Nomad and Consul as a systemd daemon service, part 2

Introduction

Consul is a service discovery application which can make each individual service in the cluster visible to each other.

This is a powerful feature in a distributed system scenario where nodes are running applications independently yet still depend on other services running on the same node or even different nodes.

For instance by configuring Consul as a DNS server, we can get the ip address where a specific service is running.

ping elasticsearch.service.consul
PING elasticsearch.service.consul (192.168.1.29) 56(84) bytes of data.
64 bytes from 192.168.1.29: icmp_seq=1 ttl=64 time=0.086 ms
64 bytes from 192.168.1.29: icmp_seq=2 ttl=64 time=0.159 ms

Besides service discovery, consul supports health checking feature. Each consul client can provide health check info of services running on the same host as well as the status of local node.

Basic Architecture

Consul agent runs on each node. Each agent is responsible for monitoring the health of local registered services as well as the local node itself. And consul agent can talks to one or more consul servers.

Consul servers are where data is stored and replicated. If any service need to discover other services or nodes, they can make a query to any of the Consul servers or any of the Consul agents. The agents forward queries to the servers automatically.

Setting Consul agent

On Chip1 and Chp2:

Add the consul program binary in:

/usr/bin/consul

It’s a little bit tricky about making consul service as a systemd daemon. To enable the consul service we need to start the agent and then tell the agent which one is the consul server. So we put these steps in a script file. And the systemd service we call this script.

Add the following shell file in /etc/consul/start-consul-client.sh:

mkdir /etc/consul
vim /etc/consul/start-consul-client.sh

In start-consul-client.sh:

#! /bin/bash
mkdir /run/consul
mkdir /etc/consul.d
/usr/bin/consul agent -data-dir=/tmp/consul -node=$(/bin/hostname)-$(/sbin/ifconfig wlan0 | /bin/grep 'inet addr:' | cut -d: -f2 | /usr/bin/awk '{ print $1}') -bind=$(/sbin/ifconfig wlan0 | /bin/grep 'inet addr:' | cut -d: -f2 | /usr/bin/awk '{ print $1}') -config-dir=/etc/consul.d  -pid-file=/run/consul/consul.pid &
n=0
until [ $n -ge 5 ]
do
 /usr/bin/consul join $(ping -c 1 gateway.local | head -1 | awk -F "(" '{print $2}' | awk -F ")" '{print $1}') && break  # substitute your command here
  n=$[$n+1]
  sleep 2
done

The systemd service file /etc/systemd/system/consul-client.service:

[Unit]
Description=IoT iaas consul agent 
Wants=network-online.target
After=network-online.target 
[Service]
Type=forking
PIDFile=/run/consul/consul.pid
Restart=on-failure
ExecStartPre=-/bin/rm -f /run/consul/consul.pidExecStart=/etc/consul/start-consul-client.sh  
[Install]
WantedBy=multi-user.target

The last, enable the service:

sudo systemctl enable consul-client.service
sudo systemctl start consul-client.service

On server (Chip3):

Add the consul program binary in:

/usr/bin/consul

The systemd service file /etc/systemd/system/consul-server.service:

[Unit]
Description=IoT iaas consul server 
Wants=network-online.target
After=network-online.target 
[Service]
ExecStart= /bin/sh -c "/usr/bin/consul agent -server -bootstrap-expect=1 -data-dir=/tmp/consul -node=$(echo `hostname`) -bind=$(ifconfig enp3s0 | grep 'inet addr:' | cut -d: -f2 | awk '{ print $1}') -config-dir=/etc/consul.d"
Restart=alwaysRestartSec=10 
[Install]
WantedBy=multi-user.targetIntroduction
Consul is a service discovery application which can make each individual service in the cluster visible to each other.
This is a powerful feature in a distributed system scenario where nodes are running applications independently yet still depend on other services running on the same node or even different nodes.
For instance by configuring Consul as a DNS server, we can get the ip address where a specific service is running.
ping elasticsearch.service.consul
PING elasticsearch.service.consul (192.168.1.29) 56(84) bytes of data.
64 bytes from 192.168.1.29: icmp_seq=1 ttl=64 time=0.086 ms
64 bytes from 192.168.1.29: icmp_seq=2 ttl=64 time=0.159 ms
Besides service discovery, consul supports health checking feature. Each consul client can provide health check info of services running on the same host as well as the status of local node. 
Basic Architecture
Consul agent runs on each node. Each agent is responsible for monitoring the health of local registered services as well as the local node itself. And consul agent can talks to one or more consul servers.
Consul servers are where data is stored and replicated. If any service need to discover other services or nodes, they can make a query to any of the Consul servers or any of the Consul agents. The agents forward queries to the servers automatically.
Setting Consul agent
On Chip1 and Chp2:
Add the consul program binary in:
/usr/bin/consul
It’s a little bit tricky about making consul service as a systemd daemon. To enable the consul service we need to start the agent and then tell the agent which one is the consul server. So we put these steps in a script file. And the systemd service we call this script.
Add the following shell file in /etc/consul/start-consul-client.sh:
mkdir /etc/consul
vim /etc/consul/start-consul-client.sh
In start-consul-client.sh:
#! /bin/bash
mkdir /run/consul
mkdir /etc/consul.d
/usr/bin/consul agent -data-dir=/tmp/consul -node=$(/bin/hostname)-$(/sbin/ifconfig wlan0 | /bin/grep 'inet addr:' | cut -d: -f2 | /usr/bin/awk '{ print $1}') -bind=$(/sbin/ifconfig wlan0 | /bin/grep 'inet addr:' | cut -d: -f2 | /usr/bin/awk '{ print $1}') -config-dir=/etc/consul.d  -pid-file=/run/consul/consul.pid &
n=0
until [ $n -ge 5 ]
do
 /usr/bin/consul join $(ping -c 1 gateway.local | head -1 | awk -F "(" '{print $2}' | awk -F ")" '{print $1}') && break  # substitute your command here
  n=$[$n+1]
  sleep 2
done
The systemd service file /etc/systemd/system/consul-client.service:
[Unit]
Description=IoT iaas consul agent 
Wants=network-online.target
After=network-online.target 
[Service]
Type=forking
PIDFile=/run/consul/consul.pid
Restart=on-failure
ExecStartPre=-/bin/rm -f /run/consul/consul.pidExecStart=/etc/consul/start-consul-client.sh  
[Install]
WantedBy=multi-user.target
The last, enable the service:
sudo systemctl enable consul-client.service
sudo systemctl start consul-client.service
On server (Chip3):
Add the consul program binary in:
/usr/bin/consul
The systemd service file /etc/systemd/system/consul-server.service:
[Unit]
Description=IoT iaas consul server 
Wants=network-online.target
After=network-online.target 
[Service]
ExecStart= /bin/sh -c "/usr/bin/consul agent -server -bootstrap-expect=1 -data-dir=/tmp/consul -node=$(echo `hostname`) -bind=$(ifconfig enp3s0 | grep 'inet addr:' | cut -d: -f2 | awk '{ print $1}') -config-dir=/etc/consul.d"
Restart=alwaysRestartSec=10 
[Install]
WantedBy=multi-user.target
And enable the service:
sudo systemctl enable consul-server.service
sudo systemctl start consul-server.service
Finally, we can test if the magic happens!
consul members
Node     Address            Status  Type    Build  Protocol  DC
chip     192.168.1.33:8301  alive   client  0.7.5  2         dc1
chip     192.168.1.34:8301  alive   client  0.7.5  2         dc1
chip     192.168.1.29:8301  alive   server  0.7.3  2         dc1
Now we see that, we can find two chips running as a client mode and one chip as a server. And we get their ip address!

And enable the service:

sudo systemctl enable consul-server.service
sudo systemctl start consul-server.service

Finally, we can test if the magic happens!

consul members
Node     Address            Status  Type    Build  Protocol  DC
chip     192.168.1.33:8301  alive   client  0.7.5  2         dc1
chip     192.168.1.34:8301  alive   client  0.7.5  2         dc1
chip     192.168.1.29:8301  alive   server  0.7.3  2         dc1

Now we see that, we can find two chips running as a client mode and one chip as a server. And we get their ip address!