Emerald ParaTime Updates — February 2022

Published in

Oasis Foundation

5 min readFeb 7, 2022

Emerald, Oasis’s official EVM compatible ParaTime, launched in November 2021. It has brought a wealth of capabilities to the Oasis Network, including easy integration with EVM-based DApps and the launch of the first DeFi projects on the Oasis Network in January 2022. These launches generated huge interest and sparked rapid usage growth on Emerald. To keep pace with this rapid growth, the Oasis team released a series of upgrades to ensure continued good user experience for Emerald users.

As of today, Emerald version 6.2.0 has been running smoothly on Mainnet for over a week since its release on January 28. Fifty-five Oasis Network validators are running Emerald nodes, and Emerald is processing nearly 200k transactions per day. In this blog, we provide more details on the rapid growth that Emerald achieved in January and how the Oasis team solved the issues that arose as a result of that growth.

Significant Usage Growth

Transaction volume on Emerald increased rapidly during January after its initial launch. It began with just a few dozen transactions per day through January 5, climbing to over 18k per day on January 10. The first significant increase in transaction volume occurred on January 11, when the number of transactions reached over 150k per day. This coincided with the launches of the Wormhole Bridge integration and YuzuSwap, the first DEX built on the Oasis Network, which surpassed $100M TVL within 12 hours of its launch.

Rapid growth continued, eventually peaking at a maximum of 950k transactions per day on January 19, at which point we set a non-zero minimum gas price on Oasis-hosted web3 gateways. After that, it stabilized at around 200k transactions per day through the end of the month.

Ensuring the Health and Scalability of the Network

The Oasis team is constantly monitoring the status of Emerald to ensure that whenever issues arise, solutions are quickly developed and deployed. In response to the significant growth in transaction volume, Oasis and Network node operators took immediate steps to meet the demand and ensure the health of the ParaTime.

When the first big surge in usage occurred on January 11, the team immediately increased the hardware capacity of the existing web3 gateway servers and provisioned new instances. This swift action from both the Network and its community partners allowed the endpoints to scale to meet the increased demand, which is currently averaging around 23M requests per day.

Supporting High Demand with Emerald 6.1.0

Zero-fee Transaction Spam

To allow users to get started on Emerald without having to pay transaction fees, we launched Emerald on Mainnet without setting an explicit minimum transaction gas price. When Emerald usage started to take off, some users began to take advantage of this policy and set up bots that spammed the network with zero-fee transactions. One possible goal of such activity is to attempt to profit by placing speculative trade orders on DEX platforms.

As this activity increased, we worked to prevent it from affecting the experience of legitimate non-bot users of the network. First, we increased the minimum gas price on Oasis-hosted web3 gateway endpoints and later rolled out an upgrade that allowed operators of Emerald ParaTime nodes to set a non-zero minimum gas price on their node. This meant that all nodes participating in the Emerald ParaTime worker committee that upgraded to the new version would be able to enforce a minimum gas price and reject zero-fee transactions.

Improper Transaction Scheduling Beyond the Block Gas Limit

As increased demand led to higher transaction volume, we discovered an issue with transaction gas limits not being properly considered when scheduling batches of transactions. As a result, the Emerald transaction scheduler could schedule batches that used more gas than the block gas limit. Such batches could be scheduled but could not be executed successfully since both transaction and block gas limits are verified correctly during execution.

This resulted in some transactions incorrectly failing with “batch out of gas” errors, for example:

batch out of gas (limit: 10000000 wanted: 10000111)

Users were still charged gas fees even though their transactions were not processed due to the batch exceeding the maximum allowed block gas limit. As with all Emerald gas fees, those fees were distributed among nodes that were scheduled to the Emerald committee during that time.

The Emerald ParaTime 6.1.0 release addressed both zero-fee transaction spam and the block gas limit bug issues by:

Enabling Emerald nodes to set a non-zero minimum gas price, and
Properly considering transaction gas limits when scheduling batches.

Testing the upgrade began on Testnet on January 19, and it was rolled out on Mainnet on January 26.

Improved Transaction Scheduling in Emerald 6.2.0

After addressing zero-fee transaction spam and the block gas limit issue with the release Emerald 6.1.0, the team focused on another issue with Emerald transaction scheduler that arose due to the way Ethereum handles unused gas fees. In Ethereum (and Emerald), a user can set a transaction’s gas limit to a huge number, e.g., 5M gas units, and if the transaction only uses 100k gas units, the user gets back the unspent gas fee. However, because the ParaTime transaction scheduler used by Emerald considered only the gas limit requested by a transaction, a denial of service attack was possible in which an attacker could consume large amounts of available capacity in Emerald blocks by declaring very large transaction gas limits while actually using, and paying for, much smaller amounts.

The solution to this issue was to augment the ParaTime SDK with a new transaction scheduler that is aware of how much gas transactions actually consume. By considering how much gas transactions actually use when constructing batches, the new transaction scheduler ensures that ParaTimes that adopt Ethereum’s gas semantics can pack blocks efficiently.

Emerald 6.2.0, which switches to the new transaction scheduler, was deployed on Testnet on January 27, and the Oasis team proposed it for Mainnet just one day later, on January 28.

The Emerald 6.1.0 and 6.2.0 releases were both backwards compatible, enabling the ParaTime to remain in operation while participating nodes performed the upgrades. This allowed both upgrades to be completed without causing any downtime.

Quick Response by Emerald Node Operators

Despite the announcement of Emerald 6.2.0 happening late Friday night in Asia and Europe time zones, node operators responded quickly to update their Emerald ParaTime and Oasis node versions. In less than four hours after the official announcement of 6.2.0, more than half of the 65 total Emerald nodes were upgraded, and 56 out of 65 were upgraded in less than 24 hours.

We are very grateful for our amazing community of dedicated and passionate node operators running on the Oasis Network!