Enter the Dōjō: Building Celo Validator Infrastructure

Keyko’s how-to guide for setting up and deploying the infrastructure needed to run a Celo validator.

At Keyko, we’re attracted to projects that make a difference, and Celo is the perfect example of a project we love. Celo is a proof-of-stake (PoS) blockchain network with smart contracts aimed at providing an open financial platform for all. It’s inclusive and aimed at tackling some the world’s most pernicious issues. Celo also offers the added benefit of potential return on investment for anyone who joins as either as as a validator or in support of another validator.

Keyko joined the journey to the Celo project by actively contributing tools (explorer, celostats, validator-explorer), and also running our own validator. In this post, we will explore how you can set up your own validator.

Source: Wikimedia Commons

Choosing the environment

Choosing the right environment for your validator is handy. You can choose to use your own infrastructure, or you can think about running in some Cloud environment. The availability, stability, and connectivity are key aspects that will impact the quality of your validator, as well as security.

At Keyko, we are convinced about using approaches that improve the availability and agility without compromising in any sense the security of the setup. For our first validator, we chose to use Google Cloud, as we had extensive experience in this cloud and also could also gain experience contributing to the Celo Network in tandem.

The setup

We use the recommended proxy-validator setup, limiting access to the validator via port 30503 by the Proxy Node. For deployment tool we use Terraform and Vault for secret backing. For simplicity purposes, we will omit this part on this example and will explain how to set up both nodes. Additionally, we will show you how to set up basic monitoring, though a more convenient pipeline is also suggested. Most of the steps are based on the official documentation page.

Accounts

We need to generate two Ethereum key pairs, one used for the validator node etherbase account (and must be authorized as validator signer), and another used for the proxy nodekey. If you do not have these accounts previously, you can generate using `celocli` in your workstation:

$ celocli account:new # For the proxy
This is not being stored anywhere. Save the mnemonic somewhere to use this account at a later point.
mnemonic: ketchup such lens inflict fashion pole material vehicle awful alone age unveil eye win road catalog lyrics warm zero stock alcohol sound brief cancel
privateKey: da7a0e953403261c160958ff9354a3156e1ea03de73a99f7b1fb58798ce0af7e
publicKey: 04bf119ab562c743dc30f13a8b2a40732dd7bb7c00a1702228c75171e86bc51232af0182fbb5254ef17735201315152eccd3c99e39286cd03161644fb2655e17b1
accountAddress: 0x0Df0A22cF680Fcd172a56786485dF19CAcA59309
$ celocli account:new # For the validator
This is not being stored anywhere. Save the mnemonic somewhere to use this account at a later point.
mnemonic: picture verb school dawn type maze ghost decline hollow impact over crumble clinic trick blush hammer bracket hire eight enjoy follow habit nephew silly
privateKey: 709ba35677332783f45435d992a08f0d9809d41bc75222754679ffb28a912215
publicKey: 041d19c20e7af5159d00cc2608167e31305a96b51b5e97bf15181fec893d4b4c794aaaf9decaf248e89425514d16be7d54267d15820bc0a2d634f48d23bf1e3995
accountAddress: 0xBF3BDFb790e6F4A18C4087DD177A5456bf77f962

These values must be stored securely.

Base system and dependencies

Celo artifacts are distributed as Docker containers, so the portability and compatibility among most any modern Linux distribution is assured. In this example, we are using Debian 10 Buster, but you can easily adapt them to your favorite distribution.

The first step would be installing Docker:

$ apt update
$ apt install -y apt-transport-https ca-certificates curl software-properties-common gnupg2
$ curl -fsSL https://download.docker.com/linux/debian/gpg | apt-key add -
add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/debian $(lsb_release -cs) stable"
$ apt update
$ apt install -y docker-ce

We will configure the Docker log options to keep the local storage consumption low (for the Keyko validator, the logs are forwarded to Stackdriver for better monitoring):

$ cat <<'EOF' > '/etc/docker/daemon.json'
{
  "log-driver": "json-file",
  "log-opts": {
  "max-size": "10m",
  "max-file": "3"
  }
}
EOF
$ systemctl restart docker

Proxy

Proxy should allow connections to ports 30303/tcp and 30303/udp, so it must be publicly exposed to improve the connectivity and be able to establish connections to other peers and validators. You can configure a more advanced network setup, but the simplest solution is assigning a public IP to this instance and configure the firewall rules or security groups accordingly.

$ GETH_NODE_DOCKER_IMAGE=us.gcr.io/celo-testnet/celo-node:rc1
$ docker pull $GETH_NODE_DOCKER_IMAGE
$ BOOTNODE_ENODE=enode://5c9a3afb564b48cc2fa2e06b76d0c5d8f6910e1930ea7d0930213a0cbc20450434cd442f6483688eff436ad14dc29cb90c9592cc5c1d27ca62f28d4d8475d932@34.82.79.155:30301
$ DATA_DIR=/root/.celo
$ GETH_PASSWORD=”<your_secret_password>”
$ echo “$GETH_PASSWORD” > $DATA_DIR/account/accountSecret
$ PROXY_ADDRESS=”0x0Df0A22cF680Fcd172a56786485dF19CAcA59309”
$ VALIDATOR_ADDRESS=”0xBF3BDFb790e6F4A18C4087DD177A5456bf77f962”
$ NODEKEY=”709ba35677332783f45435d992a08f0d9809d41bc75222754679ffb28a912215”
$ echo “$NODEKEY” > $DATA_DIR/pkey
$ PUBLIC_IP=<node_public_ip>
$ docker run \
  -v $DATA_DIR:$DATA_DIR \
  --name geth-init \
  --rm \
  --entrypoint /bin/sh \
  -d \
  $GETH_NODE_DOCKER_IMAGE -c "geth init /celo/genesis.json”
$ cat <<EOF >/etc/systemd/system/geth.service
[Unit]
Description=Docker Container %N
Requires=docker.service
After=docker.service
[Service]
Restart=always
ExecStart=/usr/bin/docker run \\
  --name geth \\
  --restart=always \\
  --net=host \\
  -v $DATA_DIR:$DATA_DIR \\
  --entrypoint /bin/sh \\
  $GETH_NODE_DOCKER_IMAGE -c "\\
  geth \\
    --bootnodes $BOOTNODE_ENODE \\
    --password=$DATA_DIR/account/accountSecret \\
    --unlock=$PROXY_ADDRESS \\
    --etherbase=$PROXY_ADDRESS \\
    --nousb \\
    --rpc \\
    --rpcaddr 127.0.0.1 \\
    --rpcapi=eth \\
    --nodekey=$DATA_DIR/pkey \\
    --networkid=42220 \\
    --syncmode=full \\
    --consoleformat=json \\
    --consoleoutput=stdout \\
    --verbosity=2 \\
    --maxpeers=100 \\
    --nat=extip:$PUBLIC_IP \\
    --metrics \\
    --proxy.proxy \\
    --proxy.proxiedvalidatoraddress $VALIDATOR_ADDRESS \\
    --proxy.internalendpoint :30503 \\
  "
ExecStop=/usr/bin/docker stop %N && /usr/bin/docker rm %N
[Install]
WantedBy=default.target
EOF
$ systemctl daemon-reload
$ systemctl enable geth.service
$ systemctl restart geth.service

Validator

Validator setup follows the same pattern. However, as said before, it should not be exposed directly to the Internet. Notice that for the `PROXY_PUBLIC_KEY` we trim the first `04` bytes from the output returned with `celocli`:

$ GETH_NODE_DOCKER_IMAGE=us.gcr.io/celo-testnet/celo-node:rc1
$ docker pull $GETH_NODE_DOCKER_IMAGE
$ BOOTNODE_ENODE=enode://5c9a3afb564b48cc2fa2e06b76d0c5d8f6910e1930ea7d0930213a0cbc20450434cd442f6483688eff436ad14dc29cb90c9592cc5c1d27ca62f28d4d8475d932@34.82.79.155:30301
$ DATA_DIR=/root/.celo
$ GETH_PASSWORD=”<your_secret_password>”
$ echo “$GETH_PASSWORD” > $DATA_DIR/account/accountSecret
$ PROXY_ADDRESS=”0x0Df0A22cF680Fcd172a56786485dF19CAcA59309”
$ VALIDATOR_ADDRESS=”0xBF3BDFb790e6F4A18C4087DD177A5456bf77f962”
$ PKEY=”709ba35677332783f45435d992a08f0d9809d41bc75222754679ffb28a912215”
$ echo “$PKEY” > $DATA_DIR/pkey
$ PROXY_PUBLIC_KEY=”bf119ab562c743dc30f13a8b2a40732dd7bb7c00a1702228c75171e86bc51232af0182fbb5254ef17735201315152eccd3c99e39286cd03161644fb2655e17b1”
$ PROXY_INTERNAL_ENODE_URL="enode://$PROXY_PUBLIC_KEY@<PROXY_ENODE>:30503"
$ PROXY_EXTERNAL_ENODE_URL="enode://$PROXY_PUBLIC_KEY@<PROXY_ENODE>:30303"
$ PROXY_URL=”$PROXY_INTERNAL_ENODE_URL:$PROXY_EXTERNAL_ENODE_URL”
$ docker run \
  -v $DATA_DIR:$DATA_DIR \
  --name geth-init \
  --rm \
  --entrypoint /bin/sh \
  -d \
  $GETH_NODE_DOCKER_IMAGE -c "geth init /celo/genesis.json && geth account import --password $DATA_DIR/account/accountSecret $DATA_DIR/pkey”
$ rm -f $DATA_DIR/pkey
$ cat <<EOF >/etc/systemd/system/geth.service
[Unit]
Description=Docker Container %N
Requires=docker.service
After=docker.service
[Service]
Restart=always
ExecStart=/usr/bin/docker run \\
  --name geth \\
  --restart=always \\
  --net=host \\
  -v $DATA_DIR:$DATA_DIR \\
  --entrypoint /bin/sh \\
  $GETH_NODE_DOCKER_IMAGE -c "\\
  geth \\
    --password=$DATA_DIR/account/accountSecret \\
    --unlock=$VALIDATOR_ADDRESS \\
    --etherbase=$VALIDATOR_ADDRESS \\
    --mine \\
    --nousb \\
    --rpc \\
    --rpcaddr 127.0.0.1 \\
    --rpcapi=eth \\
    --networkid=42220 \\
    --syncmode=full \\
    --consoleformat=json \\
    --consoleoutput=stdout \\
    --verbosity=2 \\
    --maxpeers=1 \\
    --metrics \\
    --nodiscover \\
    --proxy.proxied \\
    --proxy.proxyenodeurlpair=\\"$PROXY_URL\\" \\
"
ExecStop=/bin/sh -c '/usr/bin/docker stop %N && /usr/bin/docker rm %N'
[Install]
WantedBy=default.target
EOF
$ systemctl daemon-reload
$ systemctl enable geth.service
$ systemctl restart geth.service

Source: “Subway Tunnel” by Nappa

Basic Monitoring

In the last step, we will set up a basic monitoring scheme based on https://github.com/gruntwork-io/health-checker to check that the node is syncing the latest blocks. These steps can be reproduced on both nodes.

Firstly we will install `health-checker` and `jq` as a dependency:

$ curl -L "https://github.com/gruntwork-io/health-checker/releases/download/v0.0.5/health-checker_linux_amd64" -o /usr/local/bin/health-checker
$ chmod +x /usr/local/bin/health-checker
$ apt install -y jq

Now we will set up one script that will check that our node is syncing blocks. This script can be adapted to check additional things:

$ cat <<’EOF’ >/root/check-sync.sh
#!/bin/bash
delta_max=${1:-30}
block=$(curl -s -X POST --data '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}' localhost:8545 -H 'Content-Type: application/json' | jq -r .result)
timestamp=$(curl -s -X POST --data "{\"jsonrpc\":\"2.0\",\"method\":\"eth_getBlockByNumber\",\"params\":[\"$block\", true],\"id\":1}" localhost:8545 -H 'Content-Type: application/json' | jq -r .result.timestamp)
timestamp=$((${timestamp}))
now=$(date "+%s")
delta=$((now-timestamp))
if (( delta >= delta_max )); then
exit 1
else
exit 0
fi
EOF
$ chmod +x /root/check-sync.sh

Finally, we setup the systemd unit for the health-checker and verify it works:

$ cat <<’EOF’ >/etc/systemd/system/health-checker.service
[Unit]
Description=Health-checker
[Service]
User=root
Type=simple
ExecStart=/usr/local/bin/health-checker --listener "0.0.0.0:6000" --port 8545 --script /root/check-sync.sh --script-timeout 2
[Install]
WantedBy=multi-user.target
EOF
$ systemctl daemon-reload
$ systemctl enable health-checker.service
$ systemctl start health-checker.service
$ curl -I localhost:6000
HTTP/1.1 200 OK
Date: Thu, 28 May 2020 08:48:49 GMT
Content-Length: 2
Content-Type: text/plain; charset=utf-8

Now you can configure your favorite alerting tool to check `/` on port 6000.

Source: Wikipedia

I would like to thank all of the Keyko team for helping and supporting this process. Also from Keyko, we would like to thank Celo.org for allowing us to take part in their Journey to make financial tools accessible to all.

Here are some links with tools we help contributing to Celo Platform:

Keyko - Celo validator

Celo Stats

Celo Explorer

Vote for Us!

Keyko is looking for more votes on our Validator Group to run another Validator and stay elected, and we’d love to welcome you into our dōjō! Here’s a little bit of background to entice you through our doors:

• We assisted in implementing the Celo Blockchain as a launch partner https://keyko.io/#projects-top.
• We are looking for at least 1 million more votes to run another validator.
• We have one validator that’s already has accumulated uptime, so voters can start earning right now. Currently it has a score of: 90+%!
• The Keyko validator has 100% uptime in every epoch that it was elected in.
• Details about the Validator Group can be found here: https://celo.keyko.io.

www.keyko.io