What I learned after a month of intensive MEV bot study
What are you up against when you are building a MEV bot? Everything you need to know to get started.
MEV is a tempting space. It’s only natural that news of alpha will draw people in. But do we really understand what we are up against when we enter this arena?
Today, I would like to share some insights gained after almost a month of researching MEV. Bear in mind that one month is rarely enough to make someone an expert. So this post is sort of like an introduction to what will come after — “how to build your own working MEV bot”. Please take this content with a grain of salt.
Introduction
How long does it take to build a profitable MEV bot? And how should I begin?
In the posts to come, starting with this one, we’ll be asking ourselves the 5 questions below. By asking them and searching for answers, we will get a feeling of whether this quest is worth taking.
- Power dynamics of MEV ecosystem: Understanding order flows (Part 1)
- What are the base hardware requirements? And how much capital should I invest in to start? (Part 1)
- What language should I use? Python, Javascript, Golang, Rust? (Part 2)
- What is necessary to become profitable? (Part 3)
- How can we search for alpha? (Part 3)
We begin by analyzing the risks associated with MEV trading and then gradually progress to understanding how we can mitigate these risks and turn the tides in our favor.
Let us begin then.
Power dynamics of MEV ecosystem: Understanding order flows
This, in my opinion, is the most important aspect of MEV everyone should be aware of. We should always understand the dynamics of the game we are playing.
Order flow, in simple terms, refers to the flow of trading orders generated by multiple market participants. This can encompass trade and order book data on CEXs, as well as swap transactions on DEXs.
While some people may draw comparisons between MEV trading and HFT in TradFi, from what I have observed so far, the MEV space is quite distinct. There are certainly some similarities, as both require proximity to where orders and data/information are generated.
In HFT, it is crucial to generate high trading volumes in order to attain VIP status on an exchange and reduce commission fees. Additionally, many exchanges provide 2x to 4x faster connections to their HTTPS and WSS connections, allowing for an edge over other retail traders (although most exchanges won’t openly disclose this service if asked). HFT teams also colocate their servers next to or within the exchange to expedite order transmission.
Similar principles apply to MEV. Understanding how your transactions are routed before finalizing in a new block is imperative. By observing how MEV traders execute their orders, one can discern the notable differences compared to retail traders.
- Transaction watching/fishing from the public mempool
Users send their transactions to the mempool, with the hope of securing a spot in the newly added block. These transactions are then routed to public nodes worldwide. However, it’s important to note that these mempools lack synchronization, resulting in the absence of a global mempool concept. For those interested in exploring the source code for mempools, referring to geth would provide valuable insights. In geth, mempools are referred to as Transaction Pools.
Searchers (we’ll call MEV traders ‘searchers’) will access these public data from the mempool and select transactions that could potentially profit them.
2. Running simulations
Searchers will proceed to run simulations on the chosen transactions using their privately owned nodes. The purpose of these simulations is to observe how the execution of these selected transactions will impact the state of the blockchain.
One approach to achieve this involves creating a locally hardforked mainnet, where each transaction can be executed within an isolated EVM environment. By meticulously analyzing the opcodes, searchers can gain insights into how the transactions will unfold. To facilitate this process, searchers can employ tools such as revm and arbiter.
EVM creates a virtual environment where a single transaction can be executed to observe state changes. But, more often than not, we hope to simulate multiple transactions as bundles at once, so we would tweak the codebase a little to meet our needs, or use services like:
3. Sending transaction bundles to Builders
If the simulated output shows promise, searchers will proceed to bundle their transactions together with the ones selected from the public mempool. The purpose of this bundling is to ensure that their transactions are executed either before or after the public transactions, enabling strategies such as front-running, back-running, or sandwiching. Validators will then execute the transactions within the bundle in the specific order, ensuring that the simulated results unfold exactly as expected.
To accomplish this, searchers utilize private relays where transactions bypass the mempool and instead go through builders like Flashbots.
Below is a table listing 38 known builders:
Presently, there exist four dominant builders that hold nearly 90% of the market share. Searchers typically send their bundled transactions to all four builders, and sometimes even more, ensuring that their bundles have a high likelihood of being included (a practice followed by 89% of searchers). Unless other searchers are also attempting to add the same bundle, it is probable that the bundled transactions will find a place in a newly created block. Otherwise, searchers may find themselves involved in tipping/bribing wars to secure a position. Currently, the majority of the profit, over 90%, goes to the proposers. This situation worsens for opportunities that are widely recognized by the public. However, there are also lesser-known opportunities where validators can be incentivized with 30–80% of the profits.
Further information regarding these builders can be found below:
4. Validator with mev-boost will propose new blocks to the network
Any validator node has the capability to run MEV-Boost, a tool that allows them to receive bundled transactions constructed by builders such as Flashbots. These bundles come with incentives in the form of tips or bribes.
They will receive bundled transactions and propose a new block to the network.
5. Blocks get added to the blockchain
The proposed block gets added to the blockchain. Searchers earn profits from successfully executing transactions, and validators get paid their tips/bribes.
That pretty much sums up the order flow of MEV searchers. And as you can see, the process is quite centralized just like TradFi. Furthermore, whether liked or not, a significant portion of MEV profits is captured by validators who add bundles to new blocks, in addition to the rewards they receive for block validation.
This situation has raised concerns among regular Ethereum users, prompting efforts to decentralize various components of the MEV ecosystem. The aim is to maximize returns for users. To delve deeper into these endeavors, you can explore the Orderflow Auction by following the link provided below:
And another attempt is with MEV-Share (which I’ll go over in another post):
Having gained insight into the process of MEV searching and the associated costs, you may feel overwhelmed by its scale or disheartened by the centralizing forces at play.
So does this mean that as an individual trader, that it’s impossible to compete with the experienced MEV traders in this landscape?
While there is some truth to that notion, it doesn’t mean that all alpha opportunities have vanished. Alpha, as seasoned traders would understand, can never truly disappear. The crypto scene is still in its infancy with numerous dynamic elements, guaranteeing the persistence of inefficiencies in certain areas. By learning some tips to identify these inefficiencies, you can still find profitable opportunities.
What are the base hardware requirements? And how much capital should I invest in to start?
Now that you have a grasp of the order flows involved in MEV searching and you’re interested in diving into the MEV game, you may wonder how to begin.
The answer varies depending on your commitment. However, the good news is that you can start with just your laptop, making the initial cost effectively zero. Moreover, if you plan to leverage flashloans extensively, your starting capital can also be reduced to nearly zero, with the only requirement being sufficient funds to cover gas fees. (If you’re unfamiliar with flashloans, don’t worry — I’ll delve into them shortly when discussing how alpha can be extracted atomically using flashloans.)
But remember, there’s no such thing as a free lunch. The lower the level of risk you’re willing to take, the more challenging it becomes to generate profits.
So we’ll look at 2 parts where we’ll need to input as startup investment in this post — hardware and software.
- Hardware (a sort of capital investment)
- Software (a sort of complexity investment)
Cost of speed: Hardware
Just as speed is crucial for executing high-quality HFT strategies, it holds equal importance in MEV trading. Network speed often presents the biggest bottleneck. To quickly benchmark network speed, I conducted tests using various setups:
- Free-tier node provider service (Alchemy),
- Paid node provider service (Alchemy, Growth plan with $49/month),
- Local private full-node (geth + lighthouse).
It is essential that you understand how much of an edge you have just by setting up your own local blockchain node. So people seriously considering MEV trading should read through the benchmark results I’ve prepared.
Below is the Python code I used to benchmark my results. Nothing too complicated:
import os
import timeit
from web3 import Web3
from dotenv import load_dotenv
load_dotenv(override=True)
# create a .env file
ALCHEMY_FREE_RPC_URL = os.getenv('ALCHEMY_FREE_RPC_URL')
ALCHEMY_PAID_RPC_URL = os.getenv('ALCHEMY_PAID_RPC_URL')
def benchmark_tx_requests(w3: Web3, block_number: int):
for i in range(block_number - 100, block_number):
b = w3.eth.get_block(i)
# print(f'{i}/{block_number}', b)
if __name__ == '__main__':
free_w3 = Web3(Web3.HTTPProvider(ALCHEMY_FREE_RPC_URL))
paid_w3 = Web3(Web3.HTTPProvider(ALCHEMY_PAID_RPC_URL))
local_w3 = Web3(Web3.HTTPProvider('http://127.0.0.1:8545')) # you need a local node for this
block_number = free_w3.eth.get_block('latest').number
t1 = timeit.timeit(lambda: benchmark_tx_requests(free_w3, block_number), number=1)
print(f'Free tier: {t1} seconds')
t2 = timeit.timeit(lambda: benchmark_tx_requests(paid_w3, block_number), number=1)
print(f'Paid tier: {t2} seconds')
t3 = timeit.timeit(lambda: benchmark_tx_requests(local_w3, block_number), number=1)
print(f'Local: {t3} seconds')
The results are very interesting. Using a local node, your requests are handled almost instantaneously, and using node provider services looks impossible now.
Then, what about making interactions with smart contracts or subscribing to logs/events using WSS connections? We’ll try that next.
- Smart contract function call benchmark:
Let’s add another benchmark code like below:
# same code above...
def benchmark_contract_call(w3: Web3):
# USDC/ETH 0.05% pool
uniswap_v3_pool_address = '0x88e6A0c2dDD26FEEb64F039a2c41296FcB3f5640' # checksum address
uniswap_v3_pool_abi = [
{
'inputs': [],
'name': 'factory',
'outputs': [{'internalType': 'address', 'name': '', 'type': 'address'}],
'stateMutability': 'view',
'type': 'function'
}
]
pool = w3.eth.contract(address=uniswap_v3_pool_address, abi=uniswap_v3_pool_abi)
factory_address = pool.functions.factory().call()
# print(factory_address)
if __name__ == '__main__':
# previous benchmark code goes here...
# Contract calls
t1 = timeit.timeit(lambda: benchmark_contract_call(free_w3), number=10)
print(f'Free tier: {t1 / 10.0} seconds')
t2 = timeit.timeit(lambda: benchmark_contract_call(paid_w3), number=10)
print(f'Paid tier: {t2 / 10.0} seconds')
t3 = timeit.timeit(lambda: benchmark_contract_call(local_w3), number=10)
print(f'Local: {t3 / 10.0} seconds')
The above code is a very simple script that uses the same three endpoints, and this time we test out the time it takes to make a function call to the Uniswap V3 Pool contract. We simply call the “factory” function in that contract, and we do this 10 times.
The results this time are also as expected:
Again, you can’t even compare.
- Log subscription benchmark:
This time, we test out the websocket’s data retrieval/update speed. The Python script for this setup is a bit longer:
import os
import csv
import json
import asyncio
import datetime
import pandas as pd
from web3 import Web3
from pathlib import Path
from websockets import connect
from dotenv import load_dotenv
from multiprocessing import Process
load_dotenv(override=True)
LOG_DIR = Path('./logs')
os.makedirs(LOG_DIR, exist_ok=True)
ALCHEMY_API_KEY = os.getenv('ALCHEMY_PAID_RPC_URL').split('/')[-1]
PAID_URL = f'wss://eth-mainnet.g.alchemy.com/v2/{ALCHEMY_API_KEY}'
LOCAL_URL = 'ws://192.168.200.182:8546'
web3 = Web3(Web3.WebsocketProvider(PAID_URL))
async def stream_events(url: str, log_filename: str):
async with connect(url) as ws:
usdt_address = '0xdAC17F958D2ee523a2206206994597C13D831ec7'
transfer_event = web3.keccak(text='Transfer(address,address,uint256)')
subscription = {
'jsonrpc': '2.0',
'id': 1,
'method': 'eth_subscribe',
'params': [
'logs',
{'address': usdt_address, 'topics': [transfer_event.hex()]}
]
}
await ws.send(json.dumps(subscription))
subscription_response = await ws.recv()
print(subscription_response)
f = open(LOG_DIR / log_filename, 'w', newline='')
log = csv.writer(f)
last_updated = datetime.datetime.now()
while True:
try:
message = await asyncio.wait_for(ws.recv(), timeout=15)
response = json.loads(message)
params = response['params']['result']
log.writerow([
params['blockNumber'],
params['transactionIndex'],
params['logIndex'],
datetime.datetime.now().timestamp()
])
# run stream for 60 * 60 seconds (1 hour)
now = datetime.datetime.now()
if (now - last_updated).total_seconds() >= 60 * 60:
f.close()
print(f'{log_filename} complete')
break
except Exception as e:
# error handling here
raise e
def stream(url: str, log_filename: str):
loop = asyncio.get_event_loop()
loop.run_until_complete(stream_events(url, log_filename))
if __name__ == '__main__':
p1 = Process(target=stream, args=(PAID_URL, 'paid.csv'))
p2 = Process(target=stream, args=(LOCAL_URL, 'local.csv'))
_ = [p.start() for p in [p1, p2]]
__ = [p.join() for p in [p1, p2]]
paid_log_df = pd.read_csv(LOG_DIR / 'paid.csv', header=None)
local_log_df = pd.read_csv(LOG_DIR / 'local.csv', header=None)
log = pd.merge(paid_log_df, local_log_df, on=[0, 1, 2])
log.columns = ['block', 'tx', 'log', 'paid', 'local']
log['latency'] = log['paid'] - log['local']
print(f'Average latency in seconds: {log["latency"].mean()}')
print(log['latency'])The code above is creating two processes each subscribing to websocket endpoints using two different RPC URLs — one to Alchemy, and the other to my local node. (It says “ws://192.168.200.182:8546” and not “localhost”, because I’m working on my laptop now…)
The two websocket connections receive Transfer events from USDT and writes to the log file right away. One connected to Alchemy will write to “paid.csv”, and the other connected to my local node will write to “local.csv”.
I ran the test for 3 minutes and the results were quite interesting. I’ve calculated latency to be:
Latency = Time of paid service logged — Time of local node logged
As you can see from below, for the 3 minutes of testing, paid service received its data approx. 0.3 seconds slower than my local node on average.
I’ve also plotted a bar chart showing us the latency on each block updates. There’s sort of a cycle… I think I should dig deeper and test this out for a bit longer.
Now, testing for an hour:
Though latency fluctuates depending on the stability of my full-node’s P2P connection (I’m assuming), using a node service instead of your own node connection will on average cost you 0.25~0.3 seconds of lag time — which is pretty critical if you are aiming to be fast.
With these benchmarks we can easily figure out that setting up a local full-node is crucial to reducing network latency.
Anyone interested in the benchmark code to poke around can refer to this Github repo:
And you may be wondering how much it would cost to setup a full-node locally, and how long it will take as well.
Below is the SSD I bought few weeks ago to install Geth/Lighthouse on my local machine and start running my full Ethereum node. It’s 4 TBs large and it cost me 450 dollars. My desktop is 5~6 years old and was something close to 1500~2000 dollars back then.
I’m currently running Ubuntu 22 on my desktop. And it took me a week to fully install Geth on my machine. Geth takes up 2TBs of storage right now.
What language should I use? Python, Javascript, Rust?
Cost of complexity: Software
We saw how much performance boost we can gain from a bit of hardware investment, what about software then? What programming language should I use to expect the highest performance?
To answer this question, I would like to conduct a simple experiment to see if language selection does in fact impact the program performance. I’d also like to show you that using lower level languages does come with a pretty high cost for those that are just getting started, and that some programs may even run slower than the average Python, Javascript code if not done right!
We’ll deal with these questions in the next blog post! ☺️
