Telos Upgrade Guide to Leap 3.1

Anyone running a node on Telos Testnet or Mainnet must upgrade their node software in order to continue syncing with the network. This guide has been created to assist you on your upgrade path.

• Preparing for the Consensus Upgrades
• Planning for the Upgrade
• Preparing your node for the upgrade
• Updating Binaries
• System Contracts and Applications
• Additional Considerations
• Conclusion

Preparing for the Consensus Upgrades

Telos has two networks that need to be upgraded, Telos Testnet and Telos Mainnet. This is done through a coordinated consensus upgrade proposed in a multisig to Leap 3.1 contracts.

Operators running a node on the Telos Testnet must upgrade their node software to Leap v3.1.0 before September 28th 2022 to continue syncing with the network.

and

Operators running a node on the Telos Mainnet must upgrade their node software to Leap v3.1.0 before October 26th 2022 to continue syncing with the network.

The coordinated consensus upgrade to the Leap 3.1 release includes priorities guided by network consensus. With this release, the Antelope Coalition officially replaces the previously inattentive EOSIO developers with the EOS Network Foundation (ENF) core development team. This emancipation enables cross-chain collaboration and pushes the network into the future.

The Leap upgrade represents the network assuming ownership of its own codebase.

As guardians of the network, block producers on Telos assume responsibility for stable network upgrades. However, with consensus upgrades, all nodes in the network need to be updated.

For months now, many people have been testing Leap 3.1, and the ENF engineering team has pushed out release candidates regularly. This testing continues and the EOS Mainnet is preparing for their consensus upgrade on the 21 September 2022, with the Telos network upgrading a month after.

Like the previous EOSIO 1.8 release, Leap 3.1 requires coordination between block producers and other network participants to ensure a smooth network consensus upgrade. Now that the software is released it’s time to upgrade on Telos Testnet and Mainnet , node operators must validate prerequisites, remove obsolete plugins, and change configuration settings before updating binaries.

This document provides an upgrade checklist to facilitate the node upgrade process and avoid disrupting operations. Some specific steps may vary based on existing node configuration, but these general steps work for most nodes.

Planning for the Upgrade:

• Make sure a test server is available to try out these steps before performing them on production nodes.
• Use one of the supported operating systems (Ubuntu 18.04, 20.04, 22.04). You can install Leap 3.1 on a fresh server or upgrade an existing server. The instructions assume you are upgrading.
• Decide if you are using the pre-built binaries or compiling binaries yourself. (details).
• Ensure nodes are NOT configured to use any plugins that are no longer available in Leap 3.1 (as noted under deprecations in the GitHub Release pages for Leap 3.1 and the releases codenamed Mandel).

Nodes that use these plugins must adapt their solutions prior to activation, ideally immediately:

• History v1 “history_plugin” + “history_api_plugin”
• MongoDB “mongo_db_plugin”

Operators can migrate their solutions to services like Hyperion or Chronicle. The trace history plugin is a native migration option.

At this point, if you’re upgrading your 2.0 (NOT 2.1) node with a snapshot, you can move on to preparing for the upgrade.

• EOSIO 2.1 used state history plugin (SHIP) files and blocks.log files that are incompatible with Leap. SHIP files from 2.1 are always incompatible with 3.1. However, blocks.log files may be compatible if the node upgraded from an earlier version of EOSIO.
• Leap 3.1 is based on EOSIO 2.0, so upgrading nodes from EOSIO 2.0 is straightforward.

If you are using EOSIO 2.1, you may have additional work:

• 2.1 Nodes that don’t keep a blocks.log file or state history are the easiest to upgrade. Just start the node from a snapshot.
• For 2.1 nodes that keep a blocks.log file but do not keep state history, the version of the blocks.log file is important. If your blocks.log was created before EOSIO 2.1, the blocks.log file is compatible with 3.1 and you can start from a snapshot without removing existing block logs.
  - If you are using EOSIO 2.0 your block log is compatible.
  - If you are using EOSIO 2.1, your block log may or may not be compatible. An incompatible blocks.log file needs to be removed. To determine if your blocks.log file is compatible, use the following command with your node:
  apt-get install bsdmainutils
hexdump <nodeos data directory>/blocks/blocks.log | head
(The <nodeos data directory> will be the “data” directory where nodeos stores blockchain data, for example, “.local/share/eosio/nodeos/data”)
  The second number in the output (top row, second column from the left) corresponds with the blocks.log version. 0001, 0002, or 0003 is ok. 0004 is from EOSIO 2.1, and is not compatible.
  0000000 0003 8000 6a7f 017e e473 5a38 0827 d7e6
0000010 8804 fb34 07c1 2f9f b1ab 3c7b 5b12 6a14
The output continues. Other than the bolded section, the content will be different.
• If your node is running EOSIO 2.0 and needs to keep blocks and state history, you can use a snapshot for your upgrade.
• If your node is running EOSIO 2.1 and needs to keep state history, you need to either replay your block log or resync from the network. Replay is faster, as it saves network overhead.
  - You can replay your blocks log if you have a compatible version of blocks.log. A replay can take weeks.
  - If your blocks log is not compatible, you need to resync from another node on the network. You can synchronize from either 2.0 or 2.1, and a resync creates a compatible blocks.log file. Resync can take weeks. When resyncing, blocks.log and state history are created together.
  - Ask in the Telos Mainnet or Testnet Telegram Groups if you are after a 2.0 blocks.log file for download.

Figure 1, Table of steps the nodes running 2.1 must take depending on their configuration
(cells highlighted in yellow and red represent steps that can take weeks to complete)

Figure 1

If you have any questions about the upgrade process, please reach out in the Telos Leap v3.1.0 Launch Group.

Preparing your node for the upgrade

• Download Pre-built Binaries to install Leap. Operators who wish to build Leap themselves can find instructions in the Readme.
• Before installing any binaries, make sure you have a compatible snapshot, if needed.
  - If you are running EOSIO 2.0 (but not EOSIO 2.1), you can generate a snapshot from within a local node by using the command:
  curl -X POST http://127.0.0.1:8888/v1/producer/create_snapshot
To perform a snapshot, nodes must be started with the configuration option plugin = eosio::producer_api_plugin
Make sure your node is not exposed to the public when enabling the producer_api_plugin as it is only for administrators.
  - You can also download snapshots from a trusted block producer. Be aware that there is no way for a node to verify the integrity of these snapshots.

https://snapshots.eosnation.io/ (unzip zst, use latest)

https://snapshots.greymass.network/telos/ (unzip tar.gz, use latest)

https://snapshots.eosamsterdam.net/public/telos/ (unzip zst, use latest)

http://seed01.eosusa.news/snaps/telos/ (unzip tar.gz, use last modified)

https://snapshots.eosphere.io/ (unzip zst, use latest)

backup.cryptolions.io (unzip zst, use latest)

https://teloscentral.com/snapshots/view (unzip tar.gz, use latest)

https://snapshots.telosunlimited.io/ (unzip tar.gz, use latest)

Replaying blockchain history from the start is an alternative to using a snapshot, but it can take weeks to complete.

• Several configuration options have changed. Nodeos will fail to start if invalid configuration options are present.
  - The Reversible Block Database configuration has been removed, and operators should remove these parameters from config.ini and other scripts:
  reversible-blocks-db-size-mb
reversible-blocks-db-guard-size-mb
fix-reversible-blocks
import-reversible-blocks
export-reversible-blocks
  - The Block log splitting feature from nodeos 2.1 has been removed and changed to log block rotation. Remove the following parameters:
  blocks-log-stride
max-retained-block-files
blocks-archive-dir
  - And replace them with:
  block-log-retain-blocks
• Any nodes currently running EOSIO 2.1 should remove the state history.
  - From the data directory, delete files from the /state-history directory using
  rm <nodeos data directory>/state-history/*
• Any nodes that have block log version 0004 from 2.1 should remove Block logs:
  - From the data directory, delete files from the /blocks directory using
  rm <nodeos data directory>/blocks/*
rm <nodeos data directory>/blocks/reversible/*

The following installation uses snapshots. Node operators who need to replay transactions for SHIP or block logs should know that a replay can take weeks to complete.

Notes for improving resync speed for blocks.log files (For 2.1 nodes):

• By default, nodeos syncs 100 blocks at a time. If your network is stable, you can potentially speed up resync using the –sync-fetch-span option, which changes the number of blocks synced at a time. This option is particularly effective with blocks that are mostly empty. Operators have reported setting the number as high as 1000–5000. Once the resync is complete, you should set this number back to 100.
• Not all peers will have blocks.log files that include the first block. Nodes that start from just a snapshot do not include pre-snapshot blocks in the blocks.log file. You can improve resync speed with a change to the p2p list. If you include only p2p peer nodes that have a genesis block, this will speed up the sync process. See the list of peer nodes that include a full blocks.log file all the way to genesis.
• Keep your p2p-peer-address list short. Ideally, limit it to the 3 or 4 closest nodes geographically. You should have at least 2 or 3 nodes that extend all the way to the genesis block.
  When resyncing, nodeos requests the same batches of blocks from all peers. This slows down the resync process, and is often unnecessary for trusted peers. A sync with a limited number of peers rather than all peers, speeds up sync by avoiding duplicate data. If you feel confident, you can even pick the single closest peer node from which to sync.
• It is fastest to sync from a single peer located in the same datacentre. To do this, run an existing node with a blocks.log that contains all blocks back to genesis. List this node as a p2p peer when you re-sync from genesis block with Leap. Even if your seed node is running nodeos 2.1 or 2.2, it can still be used to provide past blocks.
  You can further accelerate sync if you run Leap v3.1 on the same server but on a different p2p port from the currently running node. With this, you would list your own node at 127.0.0.1:<p2p_port> as a peer when you begin the re-sync. Be sure you have a fresh set of /blocks and /state directories and enough NVMe or disk space for the replay.

If you use one of these practices to increase sync speed, make sure to revert options to their default settings afterward to avoid unexpected behaviour.

Updating Binaries

Do these steps on a test machine before repeating the steps with production machines.

1. Start with a fresh Ubuntu node or update an existing node by stopping nodeos and removing the old binaries. To remove old binaries, you can generally use:
   dpkg --remove <old-pkg-name>
   or
   sudo apt remove eosio
2. Remove the state file. To delete the files in this directory, you can use:
   rm <nodeos data directory>/state/*
3. Update binaries
   - Download the most recent Ubuntu binaries from the GitHub releases page
   - Open a Terminal window and run dpkg -i filename.deb or sudo apt install ./filename.deb , replacing filename.deb with the file name of the downloaded Ubuntu package.
4. Start nodeos with the snapshot.

After these steps, a node is ready for the consensus upgrade.

System Contracts and Applications

After Leap installation, node operators are ready for block producers to activate new features. At some point after activation, block producers can update the system contracts.

Leap’s new features bring powerful UX improvements. However, users won’t be able to use these tools immediately because a time lag may occur while developers update their apps to take advantage of the new features.

Applications will take time to incorporate new Leap features into their workflows. While the release of a software development kit (SDK) will accelerate the availability of Leap features, adoption will still take time.

Additional Considerations

Block producers activate all of the new features in Leap 3.1 simultaneously. Although issues are unlikely, these changes may impact smart contracts and dApps. Developers can use the Telos Testnet to ensure their dApps continue to work as expected on Mainnet. Developers can also use the Testnet to test new Leap features.

The Contract Development Toolkit (CDT) has been upgraded, and anyone using this tool should be aware of some minor naming changes:

• CDT 3.0 changes the names of its binaries to replace “eosio” prefixes with “cdt” instead, as in cdt-cpp instead of eosio-cpp.
• For CMake projects, use find_package(cdt) instead of find_package(eosio.cdt).

The next release will likely include more naming changes.

Conclusion

The Antelope Coalition directly developed and financed the Leap 3.1 upgrade, with arms wide open to suggestions from developers and operators. This release represents a new era of development for chains built on the EOSIO codebase and a step forward onto solid ground after years mired in the unpredictable bog of a negligent steward.

The Road to Antelope and Leap

The community fork of EOSIO is now known as “Antelope”

The Antelope protocol has branched off of an earlier version of the EOSIO protocol implemented in version 2.0 of the nodeos software. The Telos network runs on EOSIO software and will now be following the Antelope branch along with three other blockchain networks in the Antelope Coalition.

Mandel 3.1 is now “Leap 3.1”

The main C++ implementation of the prior EOSIO protocol was never given a distinct name which led to confusion regarding whether one was discussing the protocol itself or a particular implementation of the protocol. This ambiguity is resolved by the first community-run implementation of the Antelope protocol: Leap v3.1.

For a summary of events leading up to these changes, please see this article:

Telos Forms Part of Community-Led Antelope Protocol

List of peer nodes with blocks.log files extending to genesis:

Telos Mainnet:

• p2p.telosunlimited.io:9876
• p2p.dailytelos.net:9876
• seed.telosmadrid.io:9876
• telos.p2p.boid.animus.is:5252
• telos.p2p.boid.animus.is:5151
• kandaweather-mainnet.ddns.net:9876
• peer1-telos.eosphere.io:9876
• peer2-telos.eosphere.io:9876
• telos.seed.eosnation.io:9876
• p2p.telosgermany.genereos.io:9876
• telos.eosusa.io:9876

Telos Testnet:

• telostest.seed.eosnation.io:9876
• seed-testnet.telosmadrid.io:9886
• p2p.testnet.telosunlimited.io:9876
• telos.testnet.boid.animus.is:3535
• telos.testnet.boid.animus.is:5050
• kandaweather-testnet.ddns.net:8765
• peer1-telos-testnet.eosphere.io:9876
• test.telos.p2p.eosusa.io:19876