Spamming Solana: a Trip Report

By GM and Haseeb Qureshi

In our previous blog post, we did some back-of-envelope math to compare performance across L1s. Among the big L1s, Solana stood out as having a drastically different model, so we proceeded to try to load test Solana empirically. It turned out to be much more challenging than expected and warranted a solo deep-dive.

Before we go into the details of our load test, we first need to go over some background knowledge on Solana if you’re coming from an EVM background.

Account model

The first big difference between the EVM chains and Solana is the account model design. On Ethereum, each smart contract is an account that has its own storage. On Solana, any account can store state; smart contract storage is only used to store executable bytecode which is immutable; smart contract state is stored in other non-executable accounts. Solana developers must pass in a list of all the accounts that a transaction touches; that allows the Solana scheduler to more easily identify dependencies and thereby parallelize transactions. (There have also been theoretical designs to speed up the EVM by similarly separating state loading from code execution.)

Let’s trace the lifecycle of a Solana transaction. Once you have an account with some SOL in it, you sign a transaction using your private key. Your signed transaction is then sent to “the validators,” but Solana transactions are not gossiped into a mempool like other networks. Instead, they must be sent directly as a UDP packet to the leader of the consensus protocol. A user of course doesn’t know who the leader is, so users will send their transactions to an RPC server that forwards transactions to the current and next validator according to the leader schedule, which is known in advance.

MEV (maximum extractable value) is hence very different on Solana since there is no public mempool to monitor and snipe at. More on this later.

Courtesy: Jito Labs

Currently, transaction fees on Solana are deterministic within the same block: users are charged 0.000005 SOL per signature in the transaction. You can’t opt to pay higher fees for a better chance of inclusion.

Say you’re a validator and it’s your turn to be the leader. That means you’ll be responsible for deciding what transactions go into the next 4 blocks. You have a time limit of a total of 1.6s to pack 4 sequential slots (400ms each) into the blockchain, and then the crown is immediately transferred to the next leader. If you packed as many transactions as you could into the blockchain but have leftovers, you’re supposed to forward the leftovers along to your successor.

Here’s the tradeoff: you want to include as many transactions as possible to maximize your profits in transaction fees, but if you overdo it, it could backfire. A giant block will get streamed too slowly, and it is likely to get skipped by the network if the majority of validators won’t be able to replay it within a slot’s time.

You can get into the gory details about how a Solana validator processes Transaction Processing Unit (TPU) processes transactions in this excellent blog post by Jito Labs.

Transaction inclusion and ordering

Now let’s talk about MEV.

In PoW Ethereum, miners have control over:

1. Transaction Inclusion: Miners could insert their own transactions
2. Transaction Exclusion: Miners could censor certain transactions and not include them, potentially to capture the opportunity themselves
3. Transaction Ordering: Miners have ultimate control over how they would like to order the transactions in a block

Prior to the emergence of Flashbots, the market for inclusion and ordering all took place in the mempool. This worked well until blockspace got more competitive and users wanted granular control over the precise position within a block (mostly so they could frontrun or backrun particular transactions).

Soon users initiated bidding wars in the mempool to try to attain certain transaction priorities — known as “priority gas auctions” — which sent gas prices skyrocketing and caused huge amounts of wasted blockspace in failed transactions. Flashbots circumvented this by creating a separate off-chain market for transaction ordering where advanced users could bid in the Flashbots Auction instead of competing in mempool bidding wars.

An Ethereum block that contains Flashbots bundles

It’s all about incentives

You might ask: “Ok cool, but what does all this have to do with load testing Solana?”

Imagine if you were an aspiring NFT minter who wants to be the first to mint a new drop. Wat do?

Is this you, anon?

On Ethereum, prior to Flashbots: you mint by bidding a high gas price in the mempool bidding war

• On Ethereum, with Flashbots: you bid in the Flashbots private auction instead and send your transaction in a Flashbots bundle

Now, what about on Solana?

On Solana there is no way to express your preferences on inclusion or ordering — there’s no way to pay higher fees, there’s no special flag of “hey this transaction is super important.” So there’s only one rational option: spam your transaction as many times as you can.

The FIFO-ish nature of the current transaction ordering mechanism means those who value transaction inclusion are rational to spam the network to maximize their probability of winning, which is exactly what happened during the 9/14 Solana outage due to the Grape IDO.

This gives you a sense of the design tradeoffs within Solana’s transaction model. Given spamming Solana is already widespread, we figured: should be a breeze for us, right?

The trip

When we first started taking a look at Solana, we wanted to follow a similar approach with EVM chains: to find out the equivalent of block gas limit for Solana.

First, Solana does have something like gas, called compute units (CU), which is defined here. The cap is currently set at 48M CUs.

Second, only a limited number of CUs are writable to a single account in a single block. This limit is to prevent too many transactions writing to the same account, therefore reducing a block’s parallelism. The per-account limit is 12m. If you follow this 12m account CU limit, a 380ms block time on devnet, and a cost of 74,408 CU per Orca swap, we arrive at a theoretical limit of 424.40 swaps/sec.

To confirm our understanding, we decided to put the Solana devnet directly to the test with spam. We spun up 2 Node.js processes that submit trades on the SOL-ORCA pair in opposite directions: one spamming transactions that trade SOL for ORCA, the other spamming ORCA for SOL. We wanted to do this at a roughly equal rate to make sure the pool stayed balanced-ish.

Upon the first run of the test, transactions started landing on the devnet chain, but at an extremely low rate: roughly 10 tx/sec. The obvious suspect would be the RPC server. And upon further investigation, we were right: we got rate limited by IP. Bummer.

Now there are several ways we could potentially bypass this:

• Get closer to the action by running our own validator
• Could try to follow the leader schedule and directly send transactions to the TPU port in encoded form from multiple different servers. We may want to utilize the TPU Client for this. Since we would be only forwarding traffic, a laptop should be good. Or just send them to random validators in the hopes they forward incoming traffic.
• Use IPv6. Usually, you get an entire /64 block of IPv6 addresses. Could write a small program that binds UDP sockets to different IPv6 source addresses so we can send from multiple IPs just on one server.

Luckily, Richard Patel and Vlad Bukovsky from the Blockdaemon team spoiled me with a dedicated node for this test! And upon changing the RPC URL to Blockdaemon’s node (with no IP throttling), it worked.

And yet even after lifting the IP throttling, we weren’t able to land enough transactions to fill a block. Took us a while to figure out why this was still failing: turns out, it was because Solana transactions don’t have nonces. Let me explain.

In the EVM, every account has a nonce (this is analogous to a TCP sequence number). This ensures that every transaction is unique — your first transaction has a nonce of 0, the second transaction must have a nonce of 1, and if you try to then execute another transaction with a nonce of 0, it will fail. This is how Ethereum ensures that transactions are de-duplicated and processed in the intended order.

Solana does not use nonces, as it would require too much state synchronization. Instead, Solana references a previous block hash as a “nonce” — so long as that block is within the last 150 blocks, it will be accepted. If it sees any duplicate transactions with the same block hash, then those transactions are subsequently dropped as duplicates.

Therein was our mistake: we were spamming transactions making the same sized trade in opposite directions: 1 ORCA for SOL, 1 SOL for ORCA, over and over again. Although these trades were obviously sensible, Solana would not allow them without them all having different blockhashes. So we used the easiest workaround: randomize the trading amounts. If each trade size is rand(), then each transaction is now different and none will get deduped.

And with that, we were in business. We played around with the number of swaps we submit in a single batch (via a TypeScript Promise) and the sweet spot we found was ~350, which landed 184 Orca swaps in Devnet block 106784857.

In Devnet block 106784857, we managed to land 184 Orca swaps

Here’s our code for this naive test. Feel free to play with it and share your results. Interestingly, when we conducted the test around Jan 10, we were sending duplicate transactions trying to spam the devnet naively, which is exactly what later slowed down the mainnet around Jan 22nd: liquidation bots spamming duplicate transactions.

One thing we are curious about for future tests is to see how many markets could be processed in parallel. Theoretically, you could be able to do 4 assuming Solana is truly embarrassingly parallel. The test we’ve done is only on the ORCA-SOL pair which means all transactions are touching the same state, hence have to be processed serially.

More alpha, ser

The test was a quick and simple attempt at getting a feel for the network. Here are some ways you might be able to hit higher numbers (we take no responsibility for what you might do with these suggestions).

1. Try spamming ahead of a few slots. The transition from just forwarding packets to forwarding and holding is 20 slots before the validator becomes the leader.
2. Disabling preflight checks as that’ll disable the simulation done in the RPC server before sending
3. Make sure transactions have unique signatures, you can achieve this by looking at multiple recent block hashes and randomizing the transaction amounts. Pre 1.8.12, duplicate transactions will be caught by the deduper after signature verification, which was the main issue that crippled the network during the slowdown. Liquidation bots spammed the network with duplicate messages to land a few liquidations. Checking duplication after signature verification (which is an expensive operation) is extremely inefficient, causing validators to waste time verifying the same signatures over and over. This has been fixed in 1.8.14.
4. Another fun strategy is to deliberately include unnecessary signatures in your transaction than what’s required to artificially pay a higher fee. You might do this in the hope that validators have modified the vanilla client to actually favor higher fees.

Zooming out

The price we pay to get the high performance on Solana is obvious: decentralization. Currently, the hardware required to set up a Solana validator is so high that only a few can afford to run a fully participating node that creates blocks or verifies the chain: The Blockdaemon Solana devnet node we were fortunate enough to get our hands on had 256GB of RAM. A rough look at the number of nodes in the network also gives us a glimpse of the story: Eth2 has an estimated ~5000 validator nodes, while Solana has ~1,400 validator nodes (note that companies are likely running multiple nodes; you should take the Beacon Chain deposits and SOL stake distributions into account). Admittedly, block production is always going to be centralized due to network effects, but at least we can use techniques to make sure that block validation is still trustless and decentralized.

The Solana devs are now working on new fee market designs, including a recent discussion on adding user-specified fees to prioritize txs propagation to the leader/block. Teams like Jito Labs are building to address the need for an MEV infrastructure layer between users/searchers and validators. Solana is also exploring introducing flow control with QUIC protocol. I’m hopeful that Solana will continue to evolve and become an even more robust foundation, given all the innovation being built on top of it.

Acknowledgments: buffalu, Richard Patel, Alex Kroeger, Xinyuan Sun, Tom Schmidt, Rahul Bishnoi for the feedback!