Inside Balancer Contracts — BasePool part 2 — An unexpected journey

Inside Balancer Contracts part 2 — An unexpected journey banner image

I think everyone has experienced one of those moments. You went on wikipedia for some information and 2 hours later you’ve landed on an article about Ruth Belville and how she earned a living by letting people look at her watch, which I have to admit seems like a pretty cool business. But nonetheless you ask yourself how on earth did I end up here? Well that’s kinda what happened to me when I wanted to refresh my knowledge on the BasePool contract and continue with the BaseMinimalSwapInfoPool contract to talk about swaps. But, it’s been a few hours, and I have not even touched that bad boy. Instead I’m fiddling around with the Remix debugger and reading up on memory layout.

So how did I end up here? Well, when we talked about the BasePool, I skipped over 2 functions, namely queryExit and queryJoin. So what do they do and why did I spend a whole day with them?

Returns the amount of BPT that would be burned from `sender` if the `onExitPool` hook were called by the Vault with the same arguments, along with the number of tokens `recipient` would receive.
This function is not meant to be called directly, but rather from a helper contract that fetches current Vault
data, such as the protocol swap fee percentage and Pool balances.

Neat right? We have a dry run for a pool exit or join and get the amount of burned / minted BPT. The second statement basically means we should use the BalancerQueries helper contract instead of calling this thing directly. Probably another contract we should talk about at some point.

But let’s see some code. The two functions do basically the same, proxy to the _queryAction function passing a function reference for the hook to call ( _onExitPool or onJoinPool) and the appropriate scaling function. We only look at the queryExit but I added some comments to highlight the difference in the _queryAction call

Do you spot something weird here? The_queryAction seems to return nothing, and we return the bptIn and amountsOut with default initialized values, meaning zeros. Also the comment on the return statement hints that we are entering dark magic territory.

So let’s look at the underlying _queryAction function. I’ve left out most of it for now or your brain would probably explode (at least mine would). We just focus on the else branch for now and ignore all magic

Looks kinda familiar right? We basically do the same as the onJoinPool and onExitPool functions just without the irrelevant parts for a dry run like minting or burning BPT. First call the _beforeSwapJoinExit() hook, which was actually added since my first article about the BasePool contract. It per default checks if the pool is not paused but allows implementing contracts to overwrite it and extend the functionality. We then normalize the tokens for 18 decimals and call the passed _action function, which would be either the_onJoinPool or the_onExitPool hook. Then we use the provided downscale function which for joins rounds up and for exits rounds down. That’s it, we have the BPT amount and the token amounts, but how do those values now get back out? If you look closely, _queryAction has no return type, and thats where the fun begins.

I’m just gonna drop the bomb and show you the full function now. You can also follow along in the Remix debugger by using this simplified version.

I don’t know about you, but I was like holy shit, what is this madness and why is this necessary? I mean maybe it’s just me being an ignorant noob and you wizards write code like this before breakfast, but I was definitely intimidated. Even after reading the comments, which are pretty in depth, I was lost. At first I was like ok, I know what this function is doing in the end, let’s leave this wizardry for the big boys, but then I couldn’t let go. I knew the time was right, I could feel it in air, it was time for some assembly.

Alright, let’s roll up our sleeves and get our hands dirty. The first comment says we should refer to the queryBatchSwap function of the Vault contract which uses the same technique, so thats what we do. The first comment section gives us some background info.

In order to accurately 'simulate' swaps, this function actually does perform the swaps, including calling the Pool hooks and updating balances in storage. However, once it computes the final Vault Deltas, it reverts unconditionally, returning this array as the revert data.

By wrapping this reverting call, we can decode the deltas 'returned' and return them as a normal Solidity function would. The only caveat is the function becomes non-view, but off-chain clients can still call it via eth_call to get the expected result.
This technique was inspired by the work from the Gnosis team in the Gnosis Safe contract: https://github.com/gnosis/safe-contracts/blob/v1.2.0/contracts/GnosisSafe.sol#L265
Most of this function is implemented using inline assembly, as the actual work it needs to do is not significant, and Solidity is not particularly well-suited to generate this behavior, resulting in a large amount of generated bytecode.

So I guess this answers the question on why assembly, avoiding large amounts of bytecode. Fair enough, with big contracts like those there is a good chance that you exceed the bytecode limit. We also see that this cannot be a view function and therefore has to be explicitly called via eth_call since it behaves close to a regular swap, or in our case join/exit with most of the state updates. But it’s only a toned down version of a regular join/exit. We don’t do things like minting BPT and the like and therefore we have to prevent this function from being accidentally used instead of the real join/exit or this could end in a a disaster. The technique to prevent this is mentioned in the referenced Gnosis Safe contract function which is used to estimate transaction gas

This method is only meant for estimation purpose, therefore two different protection mechanism against execution in a transaction have been made:
1.) The method can only be called from the safe itself
2.) The response is returned with a revert

Sound legit. Revert in the end and only let the contract itself call it. Let’s see how this is achieved in our case.

Here we see the basic execution flow with the assembly magic still omitted. Initially, if the call is coming from outside, we call the same function recursively with the same arguments

(bool success, ) = address(this).call(msg.data);

which then takes the else branch, since now the msg.sender is the contract itself. There we simulate the join or exit and revert with the result. So it’s ensured that the simulation always ends with a revert. We then parse the data from the revert reason and call the return op code which stops execution and returns the result. So the execution flow won’t go back to the calling function.

Cool, I think we’ve got the basic execution flow! It feels like we might have a chance to understand this thing after all. But then we haven’t even touched the assembly stuff yet, so let’s not get ahead of ourself. Time to dive deeper!

I’d say we start backwards by looking at how the simulated data gets encoded into the revert reason and then figure out how it reconstructs it. But before we dive in, we need to cover a few fundamentals regarding memory storage layout. From the solidity docs we have the basic layout

Scratch space can be used between statements (i.e. within inline assembly). The zero slot is used as initial value for dynamic memory arrays and should never be written to (the free memory pointer points to 0x80 initially).

Solidity always places new objects at the free memory pointer and memory is never freed (this might change in the future).

Elements in memory arrays in Solidity always occupy multiples of 32 byte.

Let’s start with a short example. Consider the following function:

Running this function results in the following memory allocation

As you can see in this beautiful image which highlights my exceptional designer skills, the first two 32 Byte slots are the empty scratch slots and can be used for anything you want but never expect them to be zero (unused). Then we have the free memory pointer which initially always points to 0x80. Since we have an array in memory, it now points to 0xe0 which is followed by 3 slots for the numbers array. The docs on array layout in memory are a bit sparse but we see that the first slot contains the length of the array followed by 2 slots with the values 123 and 542 which are the values we assigned. Also good to know, an array reference, in our example the numbers variable points always to the first slot of the array containing the arrays length. With that out of the way, let’s see if we can make sense of this magic.

To recap, we are in the else branch where the contract called itself with the same calldata and we do the actual dry run of the join/exit and want to return the result in a revert since we don’t ever want this to actually execute.

We can now somewhat understand the first comment in the assembly section. We have a 32 Byte slot for the bptAmount and then one slot for the tokenAmounts array length and a slot for each entry. For some context, the revert spec from the docs

revert(p, s)- end execution, revert state changes, return data mem[p..(p+s))

So for the revert call, we need the start address and the total size in Bytes we want to copy from there. So we start by calculating the Bytes used by the tokenAmounts items

let size := mul(mload(tokenAmounts), 32)

Remember, the reference of tokenAmounts points to the first slot containing the array length, so we load the array length and multiply by 32 (since each slot consists of 32 Bytes).

Now that we have calculated the only dynamic size, we can move on to line up all the items in memory so they can be returned by the revert message.

This is where it gets interesting. Value types such as bptAmount are stored on the stack and not in memory, so to be able to return it in the revert call we need to copy it over in memory. Since it has to be a consecutive memory range, we have to either put it right before or after the tokenAmounts array. In this case, we place it before. We need a 32 Byte slot so we take the address of tokenAmounts and subtract 32 (0x20 in hex) and store the bptAmount there.

let start := sub(tokenAmounts, 0x20);
mstore(start, bptAmount)

If we consider our showcase example before, we would now overwrite the zero slot

But actually, there is another memory array in scope, the scalingFactors . So we are most likely overwriting the last value in the scalingFactors array. But thats fine since we revert after this and don’t use it anymore.

Gotta say, I’m slowly getting the hang of this, and I hope you are too! So let’s move on, we’ve almost made it to the end of the first section. We also need to send the error signature which is as hex code0x43adbafb. We again prepend it in the slot before with

Again we subtract 32 Bytes from the current start and store the padded value. Sweet, now we need to set the new start address, but we don’t want to send the full padded 32 Bytes, we only need the last 4 of them for the error signature. So thats what we do.

start := sub(start, 0x04)

Excellent! I think we are ready to revert.

revert(start, add(size, 68))

Boom! Remember size was only the size of the array elements, so we need to add another 4 Bytes for the error signature, 32 Bytes for the bptAmount and another 32 Bytes for the array length, so we end up with size + 68. Starting to feel a little bit like a wizard myself.

This was a wild ride, but we are still not quite done yet since we have to unroll this thing again. But I have a feeling that this is gonna be child’s play. First a few definitions to op codes being used

• returndatacopy(t, f, s) — copy s bytes from returndata at position f to mem at position t
• returndatasize — size of the last returndata
• and(x, y) — bitwise and of x and y
• eq(x, y) — 1 if x == y, 0 otherwise

With this we can now decipher the code. For reference, I’ve included the full if branch again. We’ll go through it step by step.

First, we handle the error signature.

We copy the 4 Bytes into the 0 slot which is a scratch slot. Since we cannot be sure that the scratch slot has not been used already, we have to do a bit-wise and to ignore the other bits in the slot to get the signature. Now we do a pretty weird looking if statement.

eq returns 1 if true, and 0 if false, so the nested eq returns 0 if the error does not match the expected signature. The outer eq checks against 0 so is true if the inner is false basically meaning if the error code does not match. If this is the case we just copy the full return data into memory at slot 0 and revert with it. But this should never really happen.

Now that validation is done, we have to extract the bptAmount and the tokenAmounts and turn them into the ABI encoded values. So we copy the bptAmount which is the next slot in the returndata after the 4 Bytes of error signature into slot 0, overwriting the in-memory error signature which we don’t need anymore.

returndatacopy(0, 0x04, 32)

With ABI-encoded values, we have to keep in mind that dynamic values like the tokenAmounts array are not directly passed but rather the offset at which position the values start. So instead of the tokenAmounts just following directly after the bptAmount, we have an additional 32 Bytes describing where the array starts. So where does it start then? Well we have the 32 Bytes of the bptAmount plus the 32 Bytes describing the offset, so it would be at start + 64 . This is also nicely illustrated in the comments

Ok so let’s store the offset in the second 32 Bytes slot at position 0x20

mstore(0x20, 64)

Then we copy the array data starting in the next slot at 0x40 , skipping the first 4 Bytes of the error signature and the next 32 Bytes of the bptAmount

returndatacopy(0x40, 0x24, sub(returndatasize(), 36))

And for the glorious end, we return starting from slot 0. But how much data? The easiest way is by deriving it from the original return data size by removing the error signature which is not included and adding the offset entry which describes the starting point of the tokenAmounts array. So we end up with returndatasize — 4 + 32

return(0, add(returndatasize(), 28))

And we are done! Holy moly what a marathon. I’m not sure if anyone is still with me, but if so I hope you’re also kinda stoked. Some animated visuals would have been nice but thats beyond my skills, for now!

So what have we learned? We’ve seen an interesting technique for dry runs, but in regard of balancer tech, not too much I have to admit. I could have described this function in a few sentences and moved on, but not this time, no! We followed through and learned some fundamentals, and that’s how you grow in my opinion. Keep building with the tools you have, but every once in a while, move back to the fundamentals and build them up over time. Prior to this, I didn’t feel like I had much of a need for assembly and didn’t know much about memory storage. But now it feels like a new tool has been added to my tool belt. We might be not very skilled at its usage, but we have it with us and if the time is right, we know where it is.