Regen Network Testnet — El Choco Upgrade Guide
What to do when the upgrade time comes?
This weekend, on August 3rd, the governance vote for the El Choco upgrade passed. As stated in the proposal, “This upgrade changes the PowerReduction from 1e6 to 1e2”, to allow validators to connect with less than 1 million trees (an unfortunate default configuration from cosmos-sdk). Rather than resetting the chain, this will be the first test of Regen’s upgrade module on a public testnet. The upgrade will take place at 15:00 UTC tomorrow, August 6th. All validators on the current testnet should be ready for a quick upgrade procedure at that time.
Overview
When the first block with a timestamp greater than 2019–08–06T15:00:00Z is minted and sent to the xrnd process to be processed, the xrnd daemon will print “CONSENSUS FAILURE… upgrade needed” in the logs and stop processing blocks. The process will not die and the server will continue to gossip over p2p.
The general process is to kill the xrnd daemon, switch out the binaries, and restart it (probably happens automatically if you run it as a service). Since the upgrade module calls a config/do-upgrade script if present, you can include auto-update code there.
As soon as 2/3 of the voting power has finished the upgrade, consensus will proceed and new block will be minted. All blocks until this point will be processed with v0.4.0 of regen-ledger. All blocks after this point will be processed with v0.4.1 of regen-ledger.
I highly recommend to manually place the el-choco (v0.4.1) binaries in a safe location on your server in advance of the upgrade rather than download them at that moment. So you just need to copy them and can upgrade in a matter of seconds.
Reference Setup
Below, I will explain an upgrade procedure I have manually tested on a small development network. However, since all validators have different setups, I will share my setup here, so you can see where the paths come from and adjust as needed. The main design is that the xrnd binary is running on a Linux system, as a systemd service. If you use docker or another supervisor process, you will have to make major adjustments. Otherwise, you will probably just need to change the paths.
Here is the file I installed in /lib/systemd/system/xrnd.service:
[Unit]
Description=Regen Xrnd
After=network-online.target[Service]
User=root
ExecStart=/usr/local/bin/xrnd start --home /root/data
StandardOutput=file:/var/log/xrnd/xrnd.log
StandardError=file:/var/log/xrnd/xrnd_error.log
Restart=always
RestartSec=3
LimitNOFILE=4096[Install]
WantedBy=multi-user.target
Detail of manual upgrade
This is the first way I tried out, and it is the simplest approach that worked, I could not automate it. When trying to copy a new xrnd binary over a running process, I get the following error:
$ cp /root/el-choco/xrnd /usr/local/bin/xrnd
cp: cannot create regular file ‘/usr/local/bin/xrnd’: Text file busyThe solution is simple: you have to stop the process before copying the binaries.
$ systemctl stop xrnd
$ cp /root/el-choco/xrnd /usr/local/bin/xrnd
$ systemctl start xrndxrnd will restart with the v0.4.1 binary and be able to proceed beyond the upgrade module block.
Detail of auto-upgrade
As mentioned above, you can automate this entirely with a config/do-upgrade script. This script is executed as a subprocess of the xrnd process when the block is hit. Ideally, this will copy over the new binary and kill the parent process, systemd restarts it automatically. However, you cannot overwrite the binary file while running. To work around that, we can use symlinks to change versions of the binary. This is a common approach used in Linux and the standard approach for eg. Go binaries:
$ ls -l /usr/bin/go
lrwxrwxrwx 1 root root 21 feb 28 14:48 /usr/bin/go -> ../lib/go-1.12/bin/go
$ ls -ld /usr/lib/go*
lrwxrwxrwx 1 root root 7 feb 28 14:48 /usr/lib/go -> go-1.12
drwxr-xr-x 4 root root 4096 ene 8 2019 /usr/lib/go-1.10
drwxr-xr-x 4 root root 4096 feb 18 12:41 /usr/lib/go-1.11
drwxr-xr-x 4 root root 4096 jul 29 12:59 /usr/lib/go-1.12What we want to set up (before the upgrade date) is the following structure:
$ ls -l xrnd*
lrwxrwxrwx 1 root root 11 ago 5 12:07 xrnd -> xrnd-v0.4.0
-rwxrwxr-x 1 root root 37182093 ago 5 12:09 xrnd-v0.4.0
-rwxrwxr-x 1 root root 37190526 ago 5 12:10 xrnd-v0.4.1We can set the following script in config/do-upgrade:
#!/bin/bashset -e -o pipefail
set -xXRND_LINK=/usr/local/bin/xrnd
XRND_TARGET=/usr/local/bin/xrnd-v0.4.1
ln -sf $XRND_TARGET $XRND_LINK# and kill parent process
kill -9 $PPID
Make sure that the xrnd process has access to write the XRND_LINK file. Either it is running as root (probably only in dev setups), or you place the xrnd binaries in a special directory where the process can write. This script will be run with the same permissions as the xrnd process.
Also note the last line (kill -9 $PPID). If you try this out on a terminal, it will kill your terminal window. However, in the setup above, it will kill the (hung) xrnd process, after the binaries have been replaced, which will cause systemd supervisor to restart the process, using the new binary linked to from /usr/local/bin/xrnd. If you checked all binaries, paths, and permissions before the upgrade block comes, this should all happen automatically. But please watch it the first time (or try out in a dev setup).
Gotchas:
- The config/do-upgrade script must be executable (chmod +x do-upgrade), or it will not run when the time comes. And make sure it is in the same directory as the genesis.json file.
- Ensure the binaries are correctly set up (xrnd-v0.4.1 version — long)
- Ensure the xrnd user can write to both the symlink, as well as the directory the symlink is in. (If xrnd is run as root, this is no issue. If it is run as a user process, the symlink and binaries should be in a user folder, not /usr/local/bin)
Log Entries (Reference)
For those who like to read log files, here are some log entries from my tests of this feature set, so you have an idea of what to expect. These can be used as references when validating your upgrade worked:
$ systemctl status xrnd
$ journalctl -f -u xrnd(coming soon)
