Ankr Contributing to BSC and Implementing Erigon Performance Upgrade

Published in

Ankr

8 min readDec 10, 2021

Binance Smart Chain began life as a fork of go-ethereum. The aim was to bring interoperability and programmability alongside compatibility with existing Ethereum Smart Contracts. The increasing success of Binance Smart Chain has led to significant scalability issues that trace back to its origins as a fork of Geth.

One of the key issues for running a BSC Archive Node has been demands on local storage. Archive Nodes can store approximately 17TB of data.

This cumbersome storage size requires significantly more time to index or compact the data, and reduces the performance of the Geth client when requesting data. Geth (Go Ethereum) is a command-line interface used to operate in conjunction with the EVM — the Ethereum Virtual Machine.

EVM compatibility is one of the major benefits of the Binance Smart Chain as it supports the extensive existing Ethereum tooling, block time speeds, and low costs. We need to create a more efficient storage mechanism to keep the BSC operating at peak performance levels while ensuring Geth is running smoothly.

Introducing the Erigon Solution

Erigon originally started as turbo-geth and an improvement of Geth. However, it soon evolved into a very different product to Geth hence the new name.
Ankr is implementing an Erigon BSC version based on https://github.com/ledgerwatch/erigon to solve the existing problems in the BSC. The new version will be open-source and available to all. Implementation of the Erigon client aims to improve efficiency in data storage and increase blockchain performance in several key ways:

Erigon features a modular design enabling parallelized development of the client.
There are fewer Read/Write operations with the database when interacting with state.
Preprocessing of data outside the storage engine improves database efficiency significantly.
Synchronization via a series of steps allows for much faster sync.

Why Erigon?

The improvements in the Erigon client have significant benefits over Geth and other existing ethereum clients. Specifically:

Drastically reduced disk storage — 1.2TB for Archive Node, 430GB for Pruned Node
Faster sync speed > 10 blocks per second
Fully bootstrapped archive node in < 3 days.
Crash resilience — Erigon’s database can withstand power failures.
Modularity — P2P and web3 RPC services can be run as components.

This solution is mutually beneficial for BSC and Ankr. Creating Erigon as open-source means that BSC node operators can freely use it and be able to contribute their computing power to the Ankr Protocol and multi-chain RPC network. With an incredibly fast and reliable network of BSC nodes, Ankr will be better able to serve the blockchain and dApp developers that depend on the protocol to communicate with the BSC.

Challenges and Improvements With Ankr’s Implementation

Erigon is an amazing solution for the above reasons. However, it is not fully compatible with BSC since the BSC has a unique way of handling consensus and block processing. For that reason, Ankr has been working with the BSC team to merge Erigon capabilities to the Binance Smart Chain and resolve some key migration challenges. A key priority has been to ensure compatibility with all nodes throughout the migration from BSC Geth to Erigon.

Features that were migrated from BSC node:

Genesis blocks and chain configurations — Erigon stores all genesis information inside sources itself instead of JSON files and doesn’t require pre-initializing the blockchain state before running the node. This is useful but requires additional support for all possible geneses in source files. It also helps to run nodes by providing a chain flag with BSC value.
Parlia consensus engine — BSC uses its own consensus engine called Parlia whilst Erigon’s consensus interfaces are not fully compatible with Geth’s. We therefore decided to modify some interfaces to make them compatible with both systems without sacrificing compatibility with block processing. This makes it much easier to merge new Erigon changes.
BSC’s compatible system calls — Erigon already supports system calls but they’re not designed to be compatible with BSC’s system calls due to differences in callee and gas limits. The key difference of BSC is that every system transaction requires an intermediary block finalization that is not supported by Erigon. Additionally, this can lead to some gas calculation issues if multiple system calls modify the same state objects.
Forks — We support all BSC forks, including Ramanujan, Niels, MirrorSync, and Bruno.
Block processing — BSC uses a somewhat tricky block processing method that appends system transactions and receipts onto every block execution. Due to block modification, it also changes transactions, receipts root, and state root. That is why the block processing part of Erigon was also changed because the terms of the Erigon consensus engine cannot modify the state; it can only verify it.
System contracts — Of course, all BSC system contracts are supported with a transport layer that facilitates BNB cross-communication functionality.

Key Milestones

In the first release of Erigon for BSC, we decided not to enable Validator Mode and instead let it work just as a full blockchain node. Currently, Erigon doesn’t support accounting to avoid creating disruptions to the Erigon node. We will support this function with Erigon in the future.

The first milestone will be to ensure that the Erigon solution works without errors in implementation with BSC full nodes (both Mainnet and Testnet).

Once full node capabilities are complete, we will verify that Erigon is compatible with the BSC validator nodes.

Erigon’s Improvements on BSC

Increased Speed & Performance — Erigon implements a much faster MDBX storage library (instead of LevelDB) that can reduce database size and improve BSC performance. MDBX is one of the fastest embedded databases and the most lightweight. Of course, MDBX has some limitations. For example, it supports only 2³¹ filesystem pages to be addressed, and it limits database size to 8 TB, but it can be fixed by increasing the page size on FS. Currently, it takes around 2.5TB to store a full archive BSC node in a database using Erigon and an MDBX driver. MDBX drivers bring performance up by 30% compared to other databases like RocksDB or LevelDB.
New RPC Daemon Capabilities — One of the key features of Erigon is the RPC daemon. By removing all RPC functionality from the node, we can create an independent transport function between the node and RPC daemons to let developers scale their RPC using remote database functionality. This is a very powerful mechanism that makes it possible to build clusters from the node by only setting one Erigon full sync mode and increasing its scalability.
Faster Synchronization Process — Syncing nodes can be a long and arduous process, but Erigon speeds this along with stages that batch data together more efficiently and minimize database overwrites. The key idea of staged sync is to split the synchronization process into 15 parts, like downloading headers or bodies, block execution, or state verification. Such division allows the utilization of all resources of the machine by using batch processing and shared caches. All intermediary caches are dropped after each phase which saves disk space and IO operations. Most of the stages allow specific optimizations, which are hard to implement without a staged sync architecture. The heaviest stage is still block execution. This is hard to parallelize due to the strict sequential nature of transactions themselves. However, the impact of all other parts on the sync process is minimized.

Benchmark Testing and Results

Upon benchmark testing, the Erigon BSC version performed remarkably well. Here is a brief overview of our process.

Hardware Specs

We deliberately used a below-average server from Digital Ocean to show the excellent performance of Erigon for BSC. The server used included:

4 vCPU
8 GB memory
160 GB disk for OS
1T GB disk for BSC testnet archive data

Synchronization Process

The testnet synchronization process started from [INFO] [12–06|21:34:41.691] and lasted to the latest height at [INFO] [12–07|22:54:31.442] [6/15 Execution] Completed on block=14778509.

The entire process took 25 hours and 10 mins. However, Erigon for BSC is using only one process of the CPU, which leaves a lot of room for future improvements. Having only 1 CPU makes it very challenging to speed up because blocks & transactions executions are inherently consecutive, not parallel. At Ankr, our preliminary estimate of this code refractory could be approximately 4–5 times faster than it is now although it is very challenging to implement. But again, this is much better than the current Go-Ethereum based clients already.

Benchmark

The Ankr team used ethspam and versus to test the performance of the Erigon client. Conducting tests with 100, 200, 300 concurrent requests, respectively, we encountered unquestionably outperforming results with 0% error. We compared this to an identical server using the Geth Goreli testnet with 100 requests, and the test encountered 6.75% connection errors.

Erigon with 100 concurrent requests

Erigon with 200 concurrent requests

Erigon with 300 concurrent requests

GETH Goreli testnet with 100 concurrent requests

We choose a Digital Ocean server with the exact same specifications for similar benchmark testing. The results show much worse performance (in terms of tp95 tp99) with 6.75% connection errors.

Future Work

There are even more opportunities for improvement that Ankr can follow now. And we will be in conversations with the BSC team to coordinate and plan our efforts. However, here are some potential priorities:

Keep improving the performance of Erigon for BSC by utilizing multi-core of CPU
Enable validator mode

Follow our GitHub to learn more about Erigon and follow future test results.

Benefits for the Binance Smart Chain Ecosystem

Ankr has a close relationship with the BSC ecosystem, and we are proud to announce this new project that will benefit the BSC community and applications.

Ankr offers a further boost to BSC developer capabilities with our bscrpc.com public RPC. This open access, free RPC allows anyone to make request calls to the chain and receive responses exactly as if they were running a BSC full node themselves.

“The Binance Smart Chain is seeing an all-time high in transactions as emerging markets heavily adopt it for DeFi, NFT’s, and gaming. The low gas fees and EVM compatibility make it a favorite. However, this comes at a cost: nodes are struggling to keep up. With Ankr open-sourcing our custom implementation of Erigon, we can help node providers keep at block height, ensuring the BSC community keeps thriving.”

- Josh Neuroth, Ankr’s Head of Product

“We are proud that BSC is reaching lofty goals as an open protocol and community-driven ecosystem.”

- Samy, BSC’s Ecosystem Coordinator

BSC projects and users will benefit hugely from upgrades to the network. As we evolve the BSC to become even faster and more efficient, it will be able to handle enormous amounts of traffic and transactions. That means more adoption from all corners of crypto as new projects look to build on the highly evolved BSC and new users take advantage of fast BSC dApps.

Follow Ankr here