EOS Block Producer Configuration for High-Availability, Scalability and more Control on Network Security

In order to have high-availability, scalability and more control on network security, an EOS Block Producer has to consider having front-end cluster nodes (that handle user and peer-to-peer requests) and back-end cluster nodes (that handle data operations), in addition to isolating database files. This article discusses those considerations and provides a suggestion of the internal network topology of a one EOS Block Producer.

Consider that this article was written at the early time of Dawn 4.0, where there were very few resources on EOS, and the official documentations was changing very frequently. So, we advise to double check the official wiki to be sure that you are up to date.

Discussion

Is it possible to configure two or more front-end cluster nodes where:

• One (or more) of the nodes handles the peer-to-peer operations that is related to block producing synchronization.
• The other node(s) can receive http(s) user requests (blochchain transactions).

And what about back-end? Can we have multiple nodes, which perform the heavy operations, arranged in a cluster to avoid single point of failure?

Finally, how to set data files to be in an isolated storage?

Suggested Topology

The front-end can be configured as a set of non-producing nodes to serve as a cluster. Those nodes should be the ones exposed to handle user requests. They also could be set behind a load-balancer that redirects the p2p requests to a specific node(s), while redirecting the http(s) end-user requests to a different node(s). An HTTP DDOS protection could be applied on the http(s) port. While a low-level, and possibly a customized, DDOS protection could be applied on the p2p port.

However, the back-end could be another nodes cluster that is connected to the non-producing nodes and configured to do block-producing. Whereas, those back-end nodes should not be exposed.

Additionally, database files are better to be isolated on some storage server.

Taking all those into consideration, the following diagram is provided as a suggestion:

Suggested Topology for one EOS Block Producer

Note that the diagram is just a suggestion and you may came with a different one. However, in both ways, it should give you an insight.

Configurations

Front-End Cluster

To achieve the suggested topology for the front-end nodes, follow the steps on this Wiki page https://github.com/EOSIO/eos/wiki/Testnet-Single-Host-Multinode that contains detailed instructions regarding bringing up a multi-node network. But to have non-producing nodes, don’t add them to the setprods JSON data, or create producer accounts for them.

However, the Peer-to-Peer relay node also needs a peer-private-key to authenticate itself with the producing node, but should retain allowed-connection = any.

Note: The p2p relay node will sign handshakes sent to nodes outside of the cluster as well, but if those nodes don’t have its peer-key, they will simply ignore the signature.

Actaully, there can be as many http-servicing nodes as you wish (or need dictates), and the protocol supposes to work as expected when they’re set up behind a typical http load-balancer. (reference)

Back-End Cluster

The mentioned production cluster may be tightly coupled with authenticated connections using the allowed-connection =, peer-key =, and peer-private-key = options provided for the purpose and could be used as follow on both the block producing (and http-servicing) nodes (reference):

• Set allowed-connection = specified to each of them.
• Add a peer-private-key to each of them.
• And provide each of them with the public peer-key of the other.

File Server

To have the data on a separate storage location(s), you may consider the following:

nodeos Runtime Data

nodeos stores runtime data (e.g., shared memory and log content) in a custom data folder. The location of this folder is determined by your system.

• Mac OS: ~/Library/Application Support/eosio/nodeos/data
• Linux: ~/.local/share/eosio/nodeos/data

A data folder can be specified using the — data-dir command line argument to nodeos. (EOS Wiki)

Location of the Produced Blocks

The location of the blocks directory (absolute path or relative to application data dir) is set in the config.ini as follow:

blocks-dir = "blocks"

MongoDB

Finally, to have data on a separate storage, you can set MongoDB database directory bath to the desired location as mentioned at: https://docs.mongodb.com/manual/tutorial/manage-mongodb-processes/#specify-a-data-directory

mongod —dbpath /srv/mongodb/

However, bear in mind that the mongo DB plugin may not survive long term.

To have more discussions, this article has been added in a question-and-answer at eosio.stackexchange.com; Feel free to provide your suggestions.