Setup Cosmos Validator Relay Network

7 min readJan 2, 2019

This is Part III of a series of posts dedicated to exploring the topic of staking and validation as Cosmos Validator. With launch of Cosmos Hub coming, we will witness the biggest adversarial BFT distributed network. Though Tendermint gives us safety, we still need to fight for liveness of network. In this article, I will talk about how to defend the liveness with relay nodes.

Part I. Anti-DDoS solution for validators

Part II. Key Management System for validators

Current Status of Cosmos Testnet

The first adversarial testnet of Cosmos, aka Game of Stakes has just started a few days ago. It was the largest BFT network ever existed with 200+ validators securing the consensus of network. As a permissionless public blockchain, we see validators come from all over the world.

According to some network traffic analysis within validator community, we see an imbalance data outbounded from different cloud server providers. A majority of data come from USA , then Europe and Asia counts for less than 10%. My guess is price difference leads to this location clustering. Since Cosmos-SDK is based on Tendermint and we need to collect 2/3+ signatures from all the validators, this kind of geo distribution gives some validators inadvantage.

To become a well connected network world wide, more work needs to be done. And today I’d like to promote the idea of relay network.

Why do we need a Relay Node Network?

There are already some discussions about what a validator operator should do to build a reliable infrastructure. The sentry node solution is to make validator node resilient to DDoS attack, which is the most common attack factor in Comsos network. The discussion about sentry node leads to the idea of relay-node. According to Zaki,

A Relay Node is a full node that only make persistent connections to white list IP of sentry nodes of other validators. It runs with pex disabled and . The firewall on the Relay node blocks all connections from ip address other than what is on a white list.

The sentry node architecture will make each validator node strong, while relay node network will establish private connections between several validators’s sentry nodes. And relay nodes will consume less resources than public sentries. The logical layout is illustrated below:

In this relay node network, there are two validator nodes: A and B. The trusted links are established between relay node of A and B. In this way, even when public sentries of B are totally not working, B’s prevote & precommit message could still be gossiped to the whole network though B’s public sentry could be attacked or connected with unreliable peers.

In the future, we should be able to see several relay node clusters: Asian cluster, European cluster and American cluster. Every validator won't have to have a public sentry in every cluster. He only need to make sure his relay node is well connected to join a joint VPN network. Traffic will be reduced due to fixed connections and less resourced required. P2P messages from Tendermint will still be gossiped around the world and reach consensus.

How to setup a Relay Network?

P2P VPN solutions

A virtual private network establishes a secure private network across a public network. The key to build a relay network is to have a VPN connection. Most traditional VPN solutions follow the client-server principle, which means that all participating nodes connect to a central server. This creates a star topology, which leads to single point failure.

A virtual network built by P2P VPN uses a full mesh topology. All nodes talk directly to each other, there is no need for a central server. If one node goes down, the rest of the network is unaffected.

Next, I will talk about how to setup relay node network with the help of 2 popular P2P VPN solution.

Tinc

Tinc is a Virtual Private Network daemon that uses tunneling and optional encryption to create a secure private network between hosts on the Internet. Tinc is free open source software. Since this VPN works at the IP level network (layer 3) as a normal network device, there is no need to adapt existing software. This allows VPN sites to share and send information with each other over the public Internet on a secure way without exposing the information to others.

Features of Tinc are:

Minimum adaption for existing software
Encryption, authentication and compression
Automatic mesh routing and fully decentralized management.
Easily expandable VPN
Runs on many operating systems and supports IPv6.

The key advantage of using tinc is that it’ll automatically configure all your nodes as soon as you connect a new one, effectively creating a mesh network of VPN endpoints. This also means that if one of the nodes fails, your remaining nodes in the mesh should continue to function without any issues.

How to Install

To set up a two-node VPN with Tinc, you need to install tinc on each server. Both of these machines are connected to the internet and are able to reach eachother.

Install Tinc

sudo apt-get install tinc

2. Create config files /etc/tinc/relaynet/tinc.conf for Tinc on relay1:

Name = relay1
Device = /dev/net/tun
AddressFamily = ipv4

Name - This is a daemon-specific name within the VPN. It should be unique.
Device - Determines the virtual network to use. Tinc will automatically detect what kind of device you are using.
AddressFamily - Tell tinc which type of address to use.

Now you need to generate the relay2’s configuration file /etc/tinc/relaynet/tinc.conf in similar way:

Name = relay2 
Device = /dev/net/tun 
AddressFamily = ipv4 
ConnectTo = relay1

ConnectTo - This value points to the tinc daemon you want to connect.

3. Create the hosts configuration file. Because tinc is built using a peer-to-peer model each node needs to communicate with the others. The basic host file for the application server is: /etc/tinc/relaynet/hosts/relay1

Address = public-ip-node-1
Subnet = 10.0.8.1

Similarly, create a host file for the second relay node:/etc/tinc/linodeVPN/hosts/relay2

Address = public-ip-node-2 
Subnet = 10.0.8.2

4. Add the public key of each node.

tinc can create the key pair using the following command:

sudo tincd -n linodeVPN -K 4096

This creates the private key (/etc/tinc/relaynet/rsa_key.priv) and appends the public key to the ams hosts configuration file that we recently created (/etc/tinc/relaynet/hosts/relay1 ).

5. Create VPN Control Scripts

Control scripts are responsible for setting up virtual interfaces on each server. They’re needed on both servers. From the application server, create the following file to enable the tinc interface: /etc/tinc/relaynet/tinc-up

#!/bin/sh 
ip link set $INTERFACE up 
ip addr add 192.168.100.209 dev $INTERFACE 
ip route add 192.168.100.0/24 dev $INTERFACE

Create a script to disable the interface:/etc/tinc/linodeVPN/tinc-down

#!/bin/sh 
ip route del 192.168.100.0/24 dev $INTERFACE 
ip addr del 192.168.100.209 dev $INTERFACE 
ip link set $INTERFACE down

Create a similar set of scripts on second relay node.

After creating the control scripts, you will need to change the permissions on both servers:

6. Test tinc

Start the tincd daemon on the first relay node:

sudo tincd -n relaynet -D -d3

Do the same on second relay node:

If everything was configured successfully, you should see an output similar to this:

Then you could a quick test with ping between two relay nodes using its VPN address. Great! Now you could use VPN addresses for connecting relay nodes.

PeerVPN

PeerVPN is a software that builds virtual ethernet networks between multiple computers. Such a virtual network can be useful to facilitate direct communication that applications like file sharing or gaming may need. Often, such direct communication is made impossible or very difficult by firewalls or NAT devices.

Features of PeerVPN are:

Ethernet tunneling support using TAP devices.
IPv6 support.
Full mesh network topology.
No NAT reconfiguration necessary, only one node with a public address is required.
Shared key encryption and authentication support.

How to Install

Download the binary files to your servers from here
Create peervpn.conf for each node.

port 7000
networkname RelayNet
psk yoursecretpassword
enabletunneling yes
interface peervpn0
ifconfig4 10.0.0.1/24

This will open UDP port 7000 and create a virtual ethernet interface with the name peervpn0 and the IP address 10.0.0.1.

3. Test

Similar to tinc to test if everything works, start PeerVPN on both nodes. Each node should now have created its own peervpn0 interface. It may take some time until the VPN tunnel is built. Try to ping 10.0.0.2 from one node or ping 10.0.0.1 from another. If you get a response, the VPN has been set up successfully!

In Close

In this article, I explained two ways to build a P2P VPN network for Cosmos validators. Relay nodes will play a key part in establishing a world-wide validator network. I wish to see more implementation of P2P VPN network in testnet.

Reference: