Open in app

Sign in

Write

Sign in

Telos Technical How To #5

Ross Dold
EOSphere Blog

--

Our previous Telos Technical How To articles have focused primarily on building and running EOSIO Software, but that is only part of the picture when providing reliable services to the Telos Ecosystem.

This 5th Telos Technical How To Edition will walk you through the process of building and running a Load Balancer service to reliably scale to the Telos Network security and performance demands.

How to Set Up a Reliable Telos Load Balancer

Scaling your Telos Network API services to millions of requests per day can be a challenge if you hadn’t planned ahead for the load and availability requirements of providing Quality Block Producer level service delivery.

Implementing Load Balancers at the public facing side of your infrastructure are a fantastic way to reliably scale your API service offering.

A Load Balancer is primarily implemented for the following benefits:

  • To balance user requests across multiple backend servers
  • Offloads SSL handshake and encryption/decryption
  • Decreases the burden of backend servers having to manage application and network user sessions
  • Creates a security demarcation point for your Telos service offering to users out on the internet

EOSphere have had tremendous success utilising HAProxy software to serve our Load Balancer requirements.

HAProxy is a free, very fast and reliable solution offering high availability, load balancing, and proxying for TCP and HTTP-based applications. It is particularly suited for very high traffic web sites and powers quite a number of the world’s most visited ones. Over the years it has become the de-facto standard opensource load balancer, is now shipped with most mainstream Linux distributions, and is often deployed by default in cloud platforms

This article will provide an example of how to build and configure HAProxy for use on the Telos Network based on our own experience and recommendations.

Requirements

Hardware (Running is a virtual environment works great)
Note: Depending on what traffic you receive you may need to scale up as required

  • 2 Core+ CPU
  • 32GB+ Disk
  • 1GB+ RAM
  • Public Network Interface
  • Private Network Interface

Operating System

Internet

  • Modern Broadband / Fibre Connection (100Mb/s synchronous and above)
  • Static Public IP Address (Either terminating on this node or forwarded with NAT)
  • Domain Name and DNS Provider
  • SSL Certificate

Install the Software

In this example we will be installing HAProxy from a community edition recommended Personal Package Archive (PPA) maintained by Vincent Berant and using the latest as of August 2022 HAProxy LTS branch v2.6.0 (Supported until Q2 2027)

This build has native support for Prometheus allowing you to view and export metrics and enables cool observability as well as logging and tracing.

Installation Process

Use the below process to install HAProxy v2.6.0:

> sudo apt install --no-install-recommends software-properties-common> sudo add-apt-repository ppa:vbernat/haproxy-2.6> sudo apt update> apt install haproxy=2.6.\*

You can check the installed version as below, make sure you can see Prometheus support:

> haproxy -vvBuilt with the Prometheus exporter as a service

Configuration

Now that you have a fresh HAProxy install let’s configure the instance for production. In this example the goal is to:

  • Load Balance incoming requests to two Telos API nodeos servers
  • Offload SSL encryption/decryption of the HTTPS sessions
  • Observe HAProxy metrics utilising the Prometheus service

This example has a public internet interface which is client facing as well as a separate private LAN interface where the servers are hosted.

All HAProxy configuration is found in haproxy.cfg The configuration is divided into 4 sections.

global    
# global settings go here defaults    
# defaults go here frontend    
# the frontend accepts requests from clients backend    
# servers to fulfill client request go here

There is a default haproxy.cfg that you can make a copy of for future reference:

> sudo mv /etc/haproxy/haproxy.cfg /etc/haproxy/old_haproxy.cfg

Follow the below to configure each section in the new haproxy.cfg :

> sudo nano /etc/haproxy/haproxy.cfg

global

global
        log /dev/log    local0
        log /dev/log    local1 notice
        maxconn 100000
        chroot /var/lib/haproxy
        stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
        stats timeout 30s
        tune.ssl.default-dh-param 2048
        user haproxy
        group haproxy
        daemon# Default SSL material locations
        ca-base /etc/ssl/certs
        crt-base /etc/ssl/private# Default ciphers to use on SSL-enabled listening sockets
        ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS
        ssl-default-bind-options no-sslv3 no-tlsv10 no-tlsv11

defaults

defaults
        log     global
        mode    http
        http-reuse always
        option  httplog
        option  dontlognull
        timeout connect 10000
        timeout client  25000
        timeout server  25000
        errorfile 400 /etc/haproxy/errors/400.http
        errorfile 403 /etc/haproxy/errors/403.http
        errorfile 408 /etc/haproxy/errors/408.http
        errorfile 500 /etc/haproxy/errors/500.http
        errorfile 502 /etc/haproxy/errors/502.http
        errorfile 503 /etc/haproxy/errors/503.http
        errorfile 504 /etc/haproxy/errors/504.http

frontend

In this section you will be enabling your public internet interface to accept incoming client requests, you will also be binding your SSL Certificate to be able to accept HTTPS and provide SSL Offloading.

frontend http-in
    bind <PUBLIC IP ADDRESS>:80
    bind <PUBLIC IP ADDRESS>:443 ssl crt /etc/ssl/certs/eosphere-local-cert-key-2021.pem alpn h2,http/1.1
    http-response set-header Access-Control-Allow-Origin "*"
    http-response set-header Access-Control-Allow-Headers "Origin, X-Requested-With, Content-Type, Accept, Authorization, JSNLog-RequestId, activityId, applicationId, applicationUserId, channelId, senderId, sessionId"

EOSphere use Comodo provided wildcard SSL certificates, your results may vary but the construct of the eosphere-local-cert-key-2021.pem in this example is the following and in the order below:

your_company.keyyour_company.crtSectigoRSADomainValidationSecureServerCA.crtUSERTrustRSAAAACA.crt

Copy and past the text in the above files into the new single file and save this created file to /etc/ssl/certs/<yourcompany_key>.pem

In addition to a typical frontend configuration this example will use an access-list to recognise traffic based on the requested destination domain. This is handy if you were supporting other API services with similar backend requests but with different domain names. For example Telos Mainnet (telos.eosphere.io) and Telos Testnet (telos-testnet.eosphere.io).

Add the following to the frontend http-in section to cover the access-lists:

acl telos_acl hdr(host) -i telos.eosphere.iouse_backend telos_api_servers if telos_acl { path_beg /v1/chain }
use_backend telos_api_servers if telos_acl { path_beg /v1/node/get_supported_apis }

This example is matching for requests to telos.eosphere.io

As our backend servers don’t support history we haven’t forwarded /v1/history .. you are now starting to see the granular control that HAProxy can provide.

backend

This is where the two Telos nodeos servers will be specified and the load balancing algorithm will be configured roundrobin in this case:

backend telos_api_servers
    balance roundrobin
    default-server check maxconn 3000
    server telos-pn-1 <PRIVATE LAN IP>:8888 cookie server1
    server telos-pn-2 <PRIVATE LAN IP>:8888 cookie server2

Prometheus

Enable the Prometheus service with the following additional front end section:

frontend stats
        mode http
        bind <PRIVATE IP>:8404        
        http-request use-service prometheus-exporter if { path /metrics }
        stats enable
        stats uri /stats
        stats refresh 10s

Save and Start

Now that that the haproxy.cfg file is successfully configured save and exit.

Check the configuration for errors:

> haproxy -c -V -f /etc/haproxy/haproxy.cfg
Configuration file is valid

Restart the HAProxy Service

> sudo service haproxy restart> sudo service haproxy status
Active: active (running)

The HAProxy server will now accept requests to the statistics dashboard powered by Prometheus where you can observe the operation of your newly built Telos HAProxy Load Balancer.

HAProxy Statistics Dashboard

http://<PRIVATE IP>:8404/stats
Prometheus Dashboard — Telos API Backend Detail

And of course test an external request to your Telos nodeos load balanced servers after you have pointed your DNS A record to the configured HAProxy public IP address.

NB: As we have an access-list configured to match interesting requests, the test request will have to be made using the DNS name and not an IP address.

https://<Your Company DNS_Name>/v1/chain/get_info

The Next Edition of Telos Technical How To .. will look into further configuring your Load Balancer deployment to stop threats and API abuse.

Be sure to ask any questions in the EOSphere Telegram

EOSphere Guild is a Block Producer on the Telos Mainnet and Testnet Blockchains as well as many other EOSIO based Blockchains.

If you find our work helpful, please vote us on the Telos Mainnet: eosphereiobp

Connect with EOSphere via these channels:

TELEGRAM | MEDIUM |YOUTUBE | FACEBOOK | TWITTER | INSTAGRAM

--

--

Ross Dold
EOSphere Blog

Written by Ross Dold

81 Followers
Editor for

Co-Founder of EOSphere Australia