Mango Markets Simulation on Solana cluster

Blockchain technology has brought about revolutionary changes in the world of finance and technology. One of the latest and most promising entrants in this space is Solana. It is a highly performant, fast, and secure blockchain that is designed for high-throughput and decentralized applications. To ensure that the Solana network functions seamlessly and efficiently, rigorous testing is necessary. Testing is an essential part of the software development process, and the Solana community has recognized this fact. Recent outages of the Solana network have further highlighted the need for a comprehensive test suite that closely mimics real-world usage patterns.

Mango Markets is a decentralized exchange (DEX) built on the Solana blockchain. It allows users to trade a wide range of cryptocurrency assets, including Bitcoin, Ethereum, and Solana, as well as synthetic assets such as Perpetual Futures contracts. Mango Markets aims to create a decentralized financial ecosystem that is fast, secure, and accessible to everyone. Perpetual Futures Contracts is a type of derivative that allows users to trade the price of an asset without owning the underlying asset itself. Perp markets on Mango Markets allow users to trade long or short positions on a variety of assets.

In this article I will explain why we have created a new repository to simulate Mango Markets on Solana, and how to use this repository to test any changes in banking stages, especially transaction scheduling. We will also explore how these tests could be used to mimic mainnet traffic patterns on testnet or other clusters and could be used as a smoke test before rolling out new release.

Testing Solana

Solana’s blockchain is a highly complex and sophisticated technology. It is designed to handle a massive number of transactions per second (TPS) and support a large number of decentralized applications (dApps). However, to ensure that the network functions smoothly, testing is necessary.

Testing on Solana Blockchain serves several purposes, including:

1. Identifying Potential Issues: By testing the Solana network, developers can identify any potential issues, bugs, or vulnerabilities in the codebase. This helps ensure that the network remains secure and resilient, and that any issues are addressed before they become a problem.
2. Enhancing Performance: Through rigorous testing, developers can optimize the network’s performance and ensure that it is capable of handling a large number of transactions per second.
3. Improving User Experience: Testing helps ensure that the user experience is seamless, fast, and efficient. This is essential to attract and retain users to the network.
4. Testing New Developments: Testing new versions of the Solana with existing test and benchmark suites we can find out potential features or bugs the new version will introduce in the cluster.

Current tests on Solana Blockchain:

Testing on Solana Blockchain currently involves several steps, including:

1. Unit Testing: This is the first stage of testing, where developers test individual components of the codebase. It helps ensure that each component functions as intended and that there are no bugs or errors.
2. Load Testing: Load testing involves stressing the network to identify how it handles a large number of transactions. This helps ensure that the network can handle a high volume of traffic without slowing down or crashing.
3. User Acceptance Testing : Carried out by developers and other members of the community before releasing on mainnet.

Motivation

Solana Bench TPS tests are a popular way to measure the network’s transaction processing speed. These tests involve sending many transactions to the Solana network and measuring the TPS the network achieves. However, it is essential to note that Solana Bench TPS tests do not read the current state of the cluster to create new transactions; there is no dependency between transactions created. It accesses different accounts with minimum write lock contention between the transactions. This means that the transactions in these tests are not competing for write access to the same account and are therefore processed with the maximum degree of parallelism. In a real-world scenario, transactions usually depend on the state of accounts and multiple transactions may need to write to the same account simultaneously, which can cause write lock contention. This can slow down the transaction processing speed and reduce the overall TPS achieved by the network since accounts must be read to create transactions, and these transactions must be executed in sequence.

To accurately measure the network’s transaction processing speed and identify any potential performance issues, it is important to conduct tests that simulate real-world scenarios. These tests should involve transactions that compete for the write access to the same account, as well as transactions that perform operations such as transferring assets, executing smart contracts, and more. By conducting realistic tests, developers can gain a better understanding of the Solana network’s performance capabilities and identify any potential bottlenecks or issues that need to be addressed. This can help ensure that the network remains fast, secure, and efficient, even under high loads and intense competition for resources.

We have implemented a new set of tests to benchmark the mango v3 smart contract.I want to highlight that the goal of the tests is not to increase TPS on the solana network but to ensure that for the given load most of the transactions are successfully executed without any errors. In this test we try to mimic the behaviour of a typical high frequency trading market maker.

One of the key features of Mango v3 is its reliance on keeper instructions. Keeper instructions are instructions that will update the data in Mango accounts to keep the price and other required data up to data. They are executed at regular intervals by bots. If the keeper instructions are not executed, then user transactions on Mango v3 will start failing. This is because data such as prices, interest, and funding rates will not be properly updated, triggering a failsafe in the contract to prevent inaccurate pricing and health calculations. In order to prevent this from happening, it is important for keepers to execute their transactions in a timely and efficient manner.

The main goal of this new test suite is to have maximum successful user transactions executed on the cluster with various prioritization fees while keeper instructions executed properly at every valid interval. This will create a robust environment to do smoke testing on the Solana cluster. We also want to make these tests as complete as possible to test all the features and components involved. For example, before each cluster upgrade, we can run these tests to run continuously for multiple hours with a heavy load while we update each node at a time and see the effects on the cluster.

Important Terms

If you are from the Solana ecosystem then I think you already know about Accounts, PDA’s, Transactions, Instructions, RPC, TPU, TVU, Prioritization fees, Tokens etc. If not I highly recommend going through https://docs.solana.com/ before continuing. Here I will just list some important terms related to mango markets.

1. Orderbook : A book in which orders are entered and buyers are matched with sellers. A well known concept in finance.
2. Oracle : They are providers for price data of different tokens in the Solana ecosystem.
3. Perp Market : An orderbook where a perpetual future is sold for an asset.
4. Market Maker : An entity which places orders such that it can extract profits from the spread in the orderbook.
5. Mango Group : A solana account which configures an instance of mango markets.
6. Mango Account : A solana account which represents a user account on mango markets.
7. Keeper Instruction : Instructions which should be called at regular intervals so that the mango group is up to date with latest prices from the oracle. Without running keeper instructions, mango transactions would start failing because all the data will go stale.
8. Root Bank : A mango v3 structure which stores some token related data like rate of interest on the token.
9. Node Bank : A mango v3 structure to enable processing of the same token in parallel by multiple users which contains details like tokens available, borrowed etc. A Root bank has multiple Node banks.

Inner workings of mango markets v3

At the heart of Mango markets is the Mango group, which is an instance of the configuration that contains information about the tokens, markets, and settings on the platform. Tokens are added to the Mango group with oracles, which provide price data that is used to facilitate trading. When a user creates a mango account it is derived from the mango group.

Once the Mango group is correctly configured, users can begin trading on the spot and perpetual (perp) markets. These markets are designed to enable users to buy and sell assets at the current market price, or to speculate on their future price movements through leveraged positions.

To ensure that prices remain accurate and up-to-date, Mango markets employ a `cachePrices` instruction that is used to update the price of tokens from their corresponding oracles. This consolidates the prices from different oracles into one so that users always have access to the most reliable and accurate price information possible. If the last update is more than valid interval (an input variable while creating mango group) then the cache goes stale and all the instructions depending on price will start to fail. The same is true for instructions `updateRootBank`, `cacheRootBank`, and `updateFunding`. Accounts required for these instructions do not change over time, so we can just build a transaction once and send it after every constant interval by just updating the recent blockhash.

Perp markets are very similar to openbook/serum spot markets. There are multiple accounts like asks, bids and event queue for each market. The asks and bids accounts are used to store information about the orders that have been placed on the platform. The asks account stores information about sell orders, while the bids account stores information about buy orders. These accounts are constantly updated as new orders are placed on the platform, and they are used to determine the current market price of the asset being traded. The event queue is used to track all of the events that occur on the platform, including new orders being placed, existing orders being cancelled, trades being executed and changes in account balances.

An important keeper instruction for the perp market is `consumeEvents` which will consume events and update users balances. For responsive settlement of funds after orders are matched, this instruction must be executed multiple times per second for the highest volume markets. If the event queue becomes full then new orders may not be placed. Unlike other keeper instructions the accounts required are not constant, but instead depend on the accounts at the head of the event queue. This means we have to read the current state of the event queue account to build a valid transaction. The state of this account may change between the time it’s read by the client, and when the `consumeEvents` instruction is processed by the cluster, which makes writing and optimizing the keeper for the `consumeEvents` quite challenging.

There are two important repositories that will help to run these tests. One repository is used to configure mango instances and create a few market makers for mango. There is also a script for all keeper instructions which will send `cachePrice`, `updateRootBanks`, `cacheRootBank` and `updateFunding` once per second and `consumeEvents` 10 times per second. The other repository Mango Simulation, is used to generate realistic traffic and output useful statistics. Read on for a summary of each project and how to use it. These projects are still being actively developed.

Configure Mango

Github : Configure Mango

This repository is primarily used to configure mango v3. All the required binaries are in the `bin` folder. The Pyth oracle is mocked and is just a program that writes a buffer into an account. To create a local cluster you can either use directly the `configure_local.sh` script or you can deploy these programs manually using the provided keypair files for each program onto an existing cluster. You can also use `start_test_validator.sh` to start a test validator.

You can deploy the Mango program with the following command:

solana program deploy bin/mango.so --program-id bin/mango.json

The mango program has also been modified to ignore the static values for mango token, serum token and luna token, so that it easily works on other clusters where mint addresses may differ. In the `src` folder we have ‘mango_utils.ts`, `serum_utils.ts`, `pyth_utils.ts` which can help you to update the configuration accordingly.

Important scripts

Index.ts : Creates a new instance of mango with 50 users. This script will create multiple mints, create oracles for those mints, and create spot markets. Using these mints we then create and configure the mango group and create perp markets. Next we proceed to create 50 keypairs, create mango accounts for them, mint them a huge number of tokens for each mint and then fund them 1 SOL each to pay transaction fees. The data related to creation of mango users is then stored in `accounts-20.json` and the overall mango group in `ids.json`. You can set few environment variables like `$CLUSTER` to denote if it is `localnet` or `testnet`, `$ENDPOINT_URL` to point to the RPC node of the cluster and `$NB_USERS` to set number of users you want to create (50 by default). It also requires an `authority.json` which has at least 100 SOL to distribute among users and pay to create accounts and transactions. If the authority.json file is not provided then the script will create it and airdrop 100 SOL to it.

Keeper.ts : This script should be run continuously during the testing to run the keeper instruction on the cluster. It takes the ids.json file created by `index.ts` to run keeper instructions on each of the markets.

refund_users.ts : Fund existing users with SOL tokens to pay transaction fees.

How to run

# Clone repo
git clone https://github.com/godmodegalactus/configure_mango.git

# installing all dependencies
yarn install

# to start a local cluster in the background
./configure_local.sh &

# Set environment variables
export ENDPOINT_URL=http://127.0.0.1:8899
export CLUSTER=localnet
export NB_USERS=50

# configure mango
yarn ts-node index.ts

# running keeper
yarn ts-node keeper.ts

Mango Simulation

Github : mango-simulation

To mimic the behavior of market makers on the platform, Mango Simulation uses the “CancelAllPerpOrders” instruction and “PlacePerpOrder” instruction on both buy and sell sides of the market to create a transaction and send it to the cluster multiple times per second. This strategy enables to simulate the behavior of market makers, who are responsible for providing liquidity to the platform by placing orders on both sides of the market. By rapidly cancelling and opening new orders. Mango Simulation can test the performance of the platform under conditions of high market activity.

These transactions are then sent to the cluster using the TPU client instead of routing it through the RPC node. The TPU client is aware of the Solana leader schedule and will send the transactions to leaders of the next 12 slots. This skips the RPC nodes for sending transactions to avoid HTTP rate limiting, leaving the RPC node free to handle other requests. The only catch is that with QUIC the connections are dropped or de-prioritized if we do not authenticate with the identity of a validator with active stake in the cluster. While running the tests on `mainnet` or `testnet` clusters it is recommended to pass the identity of high stake validator as an argument to the the program the to reduce the transaction drop rate. The transactions are confirmed by retrieving blocks from the RPC node using a websocket subscription, then iterating on all the transactions in the block to find ones that have been sent by Mango Simulation.

All of the keeper instructions are already added into the mango simulation project so that we avoid using an external keeper. consumeEventinstruction can get the account update notifications from either RPC subscription or from Geyser plugins. To use geyser plugin we will require RPC nodes to support Geyser plugins and GRPC so the runtime can receive account updates with minimal latency, enabling us to dispatch consumeEvents frequently enough to keep up with events generated by order cancel & replace.

Running Mango Simulation

# Install rust from https://www.rust-lang.org/tools/install

# clone repo
https://github.com/blockworks-foundation/mango-simulation

# Build repo
cargo build --release

# To run mango simulation
# Arguments are described below
cargo run --release -- --accounts accounts.json \
--duration 120 \
--identity identity.json \
--url http://localhost:8899 \
--keeper-authority authority.json \
--mango-cluster localnet \
--mango ids.json \
--markets-per-mm 5 \
--prioritization-fees 70 \
--quotes-per-second 2 \
--transaction-save-file tlog.csv \
--block-data-save-file blog.csv

Arguments

• accounts : A JSON file created by `index.sh` in configure mango repo which contains details of all the market makers on the mango instance. It is called accounts-20.json by default.
• mango : A json file created by index.sh` in configure mango repo which contains details for mango group instance and mango configuration. It is called ids.json by default.
• duration : Number of seconds you want to run this test on the cluster.
• url : URL of a rpc node in the cluster.
• identity : Identity of a staked validator node to authenticate the connection with TPU client.
• keeper-authority : A keypair which has enough SOL to pay for keeper transactions.
• mango-cluster : Which cluster to use from ids.json file. It is specified as an environment variable while running index.ts in configure mango. It is `localnet` by default.
• markets-per-mm : Number of markets a market maker will trade on (default 5). These markets are randomly chosen for each market maker from the list of perp markets in ids.json.
• prioritization-fees : Percentage of transactions that will include prioritization fees randomly set between 100–1000 micro lamports. A value of 70 means that 70% of transactions will include a prioritization fee.
• quotes-per-second : Number of transactions that are submitted by a market maker per second for each market.
• transaction-save-file : Output CSV file detailing the status of each transaction that was sent to the cluster.
• block-data-save-file : Output csv file which has information about the blocks retrieved while running the tests.

The total number of transaction sent to the cluster during a test run is

Total Number Of Transactions sent = Number of users * markets-per-mm * quotes-per-second * duration

Analyzing Results

After running the mango simulation we get a small summary on how many transactions were confirmed, errored and timed out. Confirmed transactions are equal to successful + errored and timeouts are transactions dropped by the leader in various stages. After running the benchmark we noticed most of the transactions are dropped in the ‘Sigverify’ stage.

# summary
[2023-03-07T14:47:21.576878173Z INFO  solana_bench_mango] confirmed 58076 signatures of 60000 rate 96%
[2023-03-07T14:47:21.577594835Z INFO  solana_bench_mango] errors counted 453 rate 0%
[2023-03-07T14:47:21.584819201Z INFO  solana_bench_mango] timeouts counted 1471 rate 3%

For detailed results we can check the transactions CSV created by mango simulation. We log all the details about the transactions that were sent by mango simulation. If most of the transactions are in error state check or to understand more about errors, please check the errors list in the Mango V3 source code. You have to convert the hexadecimal error code to decimal and find the correct error. In particular, errors 0x20, 0x21 and 0xd are very common and are associated with keeper instructions not executing properly.

Errors:

0x20 : Stale root bank cache
0x21 : Stale token price
0xd : Too many open orders in the event queue. Should use consumeEvents to free up the queue.

Transaction logs contain other useful information such as signature, sent slot, confirmed slot, market maker, market, prioritization fees, errors, successful, timeout, processed slot, and blockhash.

Another equally important result is the block data CSV. It contains how many CUs were consumed in the block and how many market maker transactions were processed. The CUs consumed should be around 40–45 Million per block. The fields logged in the block data CSV are `block hash`, `block slot`, `block leader`, `total transactions`, `number of market makers transactions`, `block_time`, and `cu_consumed`.

How to set TPS?

It is advisable to first find the expected TPS for the cluster which is 5000 TPS, and the maximum compute resource available to a block which is 45 million non voting CUs. Each market maker transaction is expected to consume around 200,000 CUs.

The Solana cluster adds a block in 400 ms, 2.5 blocks per second.

(Q) = (45M * 2.5) / (200K * number of market makers * number of markets)

for default values :

Q = 112.5 M / (200K * 50 * 5) = 2.25 TPS

So at Q = 1 most of the transactions are expected to pass, for Q = 2 there are expected to have some drops and errors while for Q = 3 the rate at which transactions are dropped should increase significantly. You can reduce the number of market makers and or markets to adjust the number of transactions you want to send.

Summary

We have two repositories to help us test, benchmark, and stress the Solana cluster. The Configure Mango repository is used primarily to configure a cluster to run the benchmarks. It can deploy programs required by Mango, configure a Mango group instance, and create Mango users. The Mango simulation repository can be used to send market maker transactions on various markets. We can launch Mango simulation using arguments to send Q transactions per second per market per market-maker for N seconds targeting a specific cluster.

Solana remains very difficult to test completely because it is technically very advanced, as it combines many algorithms and ideas. Effectively testing each stage and then testing interactions with other stages remains a fairly complex task. Using Mango simulation we can easily emulate real-time transaction load on a Solana cluster, making these tests a very good candidate for smoke testing before deploying a new validator version on mainnet. The keeper part of Mango simulation also tests the RPC response time making it more realistic rather than blindly sending transactions without awaiting confirmation.

We would like other Solana projects to join us to implement more simulation tests for their dApps. We will continue our work on Mango v4 simulation tests so that we can test more recent features such as`Address Lookup Tables`, including tests for other features that are not yet used or that lack testing.

I would like to thank Max from Mango, Pan from Mango, and Kirill from Solana Labs for their efforts and contribution. Please feel free to contribute and share your ideas.

For more information check: https://github.com/solana-labs/solana/issues/28925