Open in app

Sign in

Write

Sign in

WAX Technical How To #24

Ross Dold
EOSphere Blog

--

Having clear visibility and being notified when things aren’t working correctly is crucial for ensuring network health when providing WAX Guild public services.

Setting up your Guild’s infrastructure monitoring and alerts can certainly be accomplished using an array of software platforms, bespoke applications and scripts. However this can take a significant amount of time and effort to get right.

The EOSphere Guild have created an out of the box solution called HeadsUp that accomplishes key infrastructure monitoring and alerts within a quick 30mins of setup.

Checkout Our Public HeadsUp Showcase Dashboard

This guide will cover how to quickly configure and deploy HeadsUp so your Guild can start getting some solid WAX blockchain visibility.

Configure and Deploy HeadsUp Monitoring and Alerts

HeadsUp is designed for WAX Guilds to run their own private instance of the platform on their internal network to monitor and alert on:

  • WAX Software Nodeos
  • WAX Hyperion Full History
  • WAX Atomic Assets API

HeadsUp uses docker compose to run three docker containers configurable in a single docker-compose.yaml file. The HeadsUp platform hardware and software requirements are quite minimal.

Website Container
The Website is the central hub of the HeadsUp platform. The website receives metrics and alerts from the Ingestor (over a websocket) and forwards them onto whoever needs to receive them. This can be the website frontend, email or slack recipients.

Ingestor Container
The Ingestor is the engine of the HeadsUp platform. The Ingestor is responsible for querying all monitored nodes and providing structured metric data to the Website.

Postgres Container
The database which holds the current configuration information and a list of recent alerts.

Hardware and Software Requirements

  • 4 Core CPU
  • 64GB Disk
  • 2GB RAM
  • Ubuntu 22.04 (Recommended)
  • docker compose v2.27.0 (Recommended)

This guide follows installing HeadsUp on a server running Ubuntu 22.04 however other docker compatible distributions will work with minor adjustments. The steps to a running production HeadsUp instance are the following:

  1. Install docker compose and associated applications
  2. Configure the docker-compose.yaml
  3. Initial startup
  4. Configure Chains
  5. Configure Nodes
  6. Configure Metrics
  7. Set Alerts

Install docker compose and associated applications

Add Docker’s official GPG key:

sudo apt update

sudo apt install ca-certificates curl

sudo install -m 0755 -d /etc/apt/keyrings

sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc

sudo chmod a+r /etc/apt/keyrings/docker.asc

Add the repository to apt sources:

echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
  $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

Install:

sudo apt update

sudo apt install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

Check docker compose version:

docker compose version

Docker Compose version v2.27.0

NB: Docker requires the user that will be running docker is in the docker user group, assuming you don’t want to run HeadsUp as root add your user to the docker group as below: Add user to the docker group:

sudo usermod -aG docker ${USER}

Configure the docker-compose.yaml

Download the latest HeadsUp example docker-compose.yaml and configure the necessary fields with your own details using the EOSphere example below as a guide.

The important fields at this stage are:

  • headsup_server_masterpassword (used for website admin unlock)
  • headsup_server_jwtsecret (random string of characters used to generate a private key to sign authentication tokens)
  • headsup_server_postgrespassword (website & postgres sections must match)
  • headsup_server_rooturl (used for calculating the full URLs for silencing alerts)
wget https://raw.githubusercontent.com/eosphere/HeadsUp-Monitoring-Alerts/main/example-docker-compose.yaml

cp example-docker-compose.yaml docker-compose.yaml

nano docker-compose.yaml

website:
  environment:
    - headsup_server_masterpassword=headsup123pass
    - headsup_server_jwtsecret=NqPFk9icjklswHgKZZ5J
    - headsup_server_postgrespassword=postgres123
    - headsup_server_rooturl=https://headsup.eosphere.io
    - headsup_server_logduration=7
    - headsup_server_failedpingsthreshold=3
    - headsup_server_reminderfrequency=15

#Alerts Email, Telegram and Slack Webhook - EOSphere Example#
    - headsup_server_emailalerts=true
    - headsup_server_emailsender=alerts@eosphere.io
    - headsup_server_emailto=ross@eosphere.io,shelley@eosphere.io,john@eosphere.io
    - headsup_server_emailsmtpserver=smtp.gmail.com
    - headsup_server_emailsmtpssl=true
    - headsup_server_emailsmtptls=false
    - headsup_server_emailsmtpport=465
    - headsup_server_emailsmtpuser=alerts@eosphere.io
    - headsup_server_emailsmtppassword=E6oNZPfZ6r5g
    - headsup_server_telegramalerts=true                                                                                                                                                                                            
    - headsup_server_telegrambot=7867365320:ABHkXyeDb7eB-ZiplTx-5BCcChQqE9FiJMg                                                                                                                                                     
    - headsup_server_telegramchat=-4379740104  
    - headsup_server_slackalerts=true
    - headsup_server_slackhook=https://hooks.slack.com/services/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o

postgres:
  environment:
    - POSTGRES_PASSWORD=postgres123

Alerts are always shown in the dashboard, please configure your specific email, telegram and slack alert credentials as required. The example-docker-compose.yaml has these fields hashed out, they can also be set to true/false.

Initial Startup

HeadsUp requires a once-off manual sequenced start of it’s containers for initial startup.

1. Start the postgres Database container

docker compose up postgres

This will download the postgres container and then run it which initialises the database. We do this because of the time it takes to initialise this container. When the website container starts it tries to run the database migrations immediately, if the database is not ready this causes an issue.

Once the database container is initialised press Ctrl+C to stop it.

2. Start the Website container to create and add an Access Key

docker compose up website

Once it has started Access the HeadsUp Website through a web browser http://<SERVER_IP_ADDRESS>:8000

Login with the master password by clicking the lock on the top right.

Admin Access Login

Click the Setup on the left hand side

Click the Plus/+ sign next to Ingestors

Give your Ingestor a Name (Typically Ingestor 1) and set the Sort Order to 0

Click the Save button

Once it has saved click the Ingestor Key and Plus/+ icon to “generate new key”

Click Generate Key

Generate New Access Key Window

Click Copy to copy the Access Key and then click Done

In your terminal press Ctrl+C to stop the Website container.

Edit your docker-config.yaml file to configure the new copied access_key in the ingestor: section

nano docker-compose.yaml

services:
  ingestor:
  environment:
    - headsup_name=Ingestor 1    
    - headsup_accesskey=c16ea810ab45f24bf605697cb72b1b5dfb0a5f15305df565135d13122be886f4

The manual initialisation process is complete.

3. Start HeadsUp with docker compose

docker compose up -d

This will orchestrate the launch of the HeadsUp platform and will run docker compose in the background.

You can check running docker processes with:

docker ps

or

docker stats --all

In this configuration docker compose will automatically restart the HeadsUp platform on reboot.

You can now access the fully functional dashboard and move onto configuring chains, metrics and alerts.

Configure Chains

HeadsUp has the ability to monitor and alert on multiple Antelope blockchains at the same time, each new chain is essentially a discrete blockchain network Mainnet/Testnet etc.

Connect to the website and Login as the master user.

Click Home on the left column

You should see a blank screen as you have not set up any chains.

Click + Add Chain

Configure the following fields:

Chain Name A human readable name for the chain. This name is the name used to identify the chain in the HeadsUp’s user interface.

Chain Type You can choose the chain type, this will prefillout some of the values (chain id and Vote Decay Type) with that chain’s known settings and use the correct chain icon. Unknown chains can use the Private Chain type.

Icon This is the icon to use for this chain, you cannot customise this setting.

BP Name For metrics which monitor your Block Producer this is the name of your on-chain Block Producer Account.

Vote Decay Type The original EOS mainnet halved the values of votes after 52 weeks, WAX uses 13 weeks. You can only use one of the preset options.

Chain Id The WAX Chain Id. Each Mainnet and Testnet (or Private Network) will have a unique one of these.

Sort Order The order in which to display this chain in the on-screen list.

Click Create > and the new chain will be added to the left hand panel.

Add Chain Window

Configure Nodes

Nodes are configured within each chain, they monitor specific Nodeos, Atomic or Hyperion API’s. As HeadsUp is designed to be run privately, for best operation a node should be configured to directly query the API rather than a loadbalancer or any other intermediary.

Connect to the website and Login as the master user.

Click Home on the left column

Click on the relevant Chain, WAX MAINNET for Example

Click + Add Node

Configure the following fields:

Node Name A human readable name for the node. This name is the name used to identify the node in the HeadsUp user interface.

Node Type
Block Producer — The primary Nodeos Block Producer Node

Nodeos Follower — The secondary Nodeos instances that measure their block sync status against the Block Producer Headblock

Hyperion — The Hyperion API

Atomic Assets — The Atomic Assets API

Website — Monitor and alert on your website, chains.json and bp.json files

The difference between “Block Producer” and “Nodeos Follower” is the headblock number metric. For the purposes of alerting the headblock number of the follower node in compared to the headblock number of the Block Producer Node.

Node Types

IP Address Address of the node, can also be a DNS name

Port TCP Port to connect to the node on

Sort Order On screen display order of the nodes in Headsup

Use SSL for connections Connect via HTTPS otherwise unencrypted HTTP

Ingestor It is possible to have multiple ingestors although at this point we only use Ingestor 1

Click Create >

A Default set of Metrics and Alerts will be configured upon creation of each node

Add Node Window

Configure Metrics

Metrics define what is displayed on the HeadsUp dashboard as well as make various alerts available.

Connect to the website and Login as the master user.

Click Home

Click on the relevant Chain

Click on the relevant Node

Click the Bell Icon in the top right hand corner

On the left side Metrics are divided into Metric Modules of related values. Metrics are different depending on whether a Nodeos, Hyperion or Atomic Node is configured.

Node Metric Modules

Various metrics are selectable within each Metric Module and are enabled by selecting Monitor / Monitoring

Node Metrics

Once a specific metric is selected and Saved, it will be viewable in the dashboard node page and may open up options in the alert menu.

Dashboard Node Page

Set Alerts

Metrics only advise on a value in the dashboard, while alerts will send a message when a certain condition is met. Depending on your configuration these alerts notify via the website dashboard, email, telegram and/or slack.

Alerts are set on each node and are available when a corresponding metric has been selected.

Connect to the website and Login as the master user.

Click Home

Click on the relevant Chain

Click on the relevant Node

Click the Bell Icon in the top right hand corner

Alerts are displayed on the right hand side and are grouped into Alert Modules of related values and typically would already have some default alerts applied. Alerts are different depending on whether a Nodeos, Hyperion or Atomic Node is configured.

Alert Modules

Each alert has a condition field which is what HeadsUp will test against, there is an explanation of the condition field next to each alert.

Alert Condition Fields

Configure your desired alert threshold, the defaults are a good place to start.

When an alert is generated they will be shown on the dashboard as well as sent to your configured email, telegram or slack.

Alert shown on Node Page
Alert shown in Alerts History Page

You should now be up and running with a clear view of your infrastructure with all the important alerting when things misbehave.

We would appreciate any feedback or feature requests in our GitHub or please join our Telegram Group for assistance and further discussions on HeadsUp.

EOSphere Guild is a Block Producer on the WAX Protocol Network as well as many other Antelope based Blockchains.

If you find our work helpful, please vote us on the WAX Mainnet: eosphereiobp

If you prefer to proxy your vote, our proxy account is : blklotusprxy

Connect with EOSphere via these channels:

TELEGRAM | MEDIUM |YOUTUBE | FACEBOOK | TWITTER | INSTAGRAM

--

--

Ross Dold
EOSphere Blog

Written by Ross Dold

81 Followers
Editor for

Co-Founder of EOSphere Australia