Is the $6B locked in L2 secure?

TL;DR — Monitoring the state of complex systems deployed on Ethereum L1 is necessary for the security of users’ funds. It should be done by vendor-neutral third parties and it is not trivial. Read on to learn why it is necessary and how it could be done from the technical point of view.

Today L2BEAT lists nineteen Layer 2 solutions live on Ethereum Mainnet. To fully grasp the security of all the $6B locked in those systems one would need to analyse each and every one of them individually. At the time of this write-up, Arbitrum Optimistic Rollup is listed as #1 on our site, with over $2.4B dollars worth of capital moved to it from Ethereum. In this article we will explore its current deployment in great detail with the hope that our findings will apply to the broader ecosystem.

Screenshot from https://l2beat.com

L1 tokens are primarily bridged to Arbitrum through three contracts:

• Bridge 0x011B6E24FfB0B5f5fCc564cf4183C5BBBc96D515 holds over $1,286M worth of ETH
• L1CustomGateway 0xcEe284F754E854890e311e3280b767F80797180d holds over $636M worth of USDC, USDT and few smaller tokens
• L1ERC20Gateway 0xa3A7B6F88361F48403514059F1F16C8E78d60EeC holds over $464M worth of various ERC20 tokens (DPX, WBTC, SPELL, etc…)

Whoever controls these gateways (bridges) has indirect access to these funds and the first review of them quickly reveals that all three of them are instances of TransparentUpgradableProxy, meaning that their admin can upgrade the implementation at any time granting themselves full access to funds escrowed in the bridge even if the current implementation does not grant such access. Who then is the admin of these contracts?

Is it enough though to monitor these contracts and does “the security of the system” indeed rely on the of the admin of these contracts? As it turns out, the reality is much more complicated — these contracts are controlled by both the current MultiSig that can upgrade their implementation, but more importantly, they are also controlled by the Rollup’s implementation, i.e. the rest of the Rollup machinery. This is what we normally mean when we say that Rollups — as opposed to sidechains — “derive their security from L1 Ethereum” (if we choose to ignore the MultiSig that can upgrade the implementation). So who controls the rest of this machinery and how this is done?

Rollups are complex control systems

Most complex scaling projects in the space, Arbitrum included, have officially stated that until the technology matures, that they will retain “manual” controls over the system which will act as a security precaution in case something goes horribly wrong. They have been very forthright about it in both announcements and with their UIs and it should not come as a surprise to anyone.

UI warning present on the official Arbitrum One Bridge

One can then argue that until these controls are removed, the whole construction is no more secure than a simple bridge to a sidechain that is controlled and operated by a MultiSig. We, at L2BEAT, believe that this is fundamentally a wrong take — there is a huge difference between a system that is designed to inherit its security from Ethereum L1 (as both Optimistic and ZK Rollups aim to do) that is still equipped with some “training wheels” but with clear path towards decentralisation, and a system that is simply equipped with just the training wheels but there is nothing else securing the funds, and with no clear path toward decentralisation. In the first case, the system is secured by Ethereum L1 (via fraud proofs or validity proofs implemented on L1 contracts) and training wheels are an extra (and temporary) feature, while the latter case has nothing but training wheels.

How would we then try to attempt to get a full understanding of the security properties of the deployed, complex system on an Ethereum L1? Here’s how you should not do it — ask the team on social media, or read the whitepaper / documentation. Too many times we have found that what is actually deployed on L1 does not match claims stated in the documentation (granted — this was NOT the case with Arbitrum. Their documentation is fantastic and the team has always been very open and helpful when asked for any clarifications). Fundamentally it all comes down to the “don’t trust, verify” philosophy that is the cornerstone of public blockchains. If you, as a user, cannot independently verify, then why do we even bother building complex financial systems on blockchains. It goes without saying that the first and obvious prerequisite to the independent verification is access to verified source code on Etherscan.

One way of trying to understand complex contracts is to see them as “control systems” equipped with various knobs and levers:

When we then look at the actual system deployed on L1, we can ask ourselves:

• What are the knobs and levers of a system?
• Who controls them?
• What was the history of the changes in this system? Who performed them and when?
• Who can upgrade the control centre, i.e. introduce new knobs and perhaps remove the old ones that cannot be used anymore? Was such an upgrade performed before?
• Is there a system in place that monitors current system parameters and if the particular knob or button is pressed, alerts interested users?

All these questions are highly relevant, as without proper understanding of the internal system architecture end users cannot be certain what effect a change of the particular system parameter would have on the security of their funds. If we take Arbitrum as an example, assuming that we have a decent understanding about how Optimistic Rollups, and in particular the Arbitrum implementation, work we might be interested in knowing:

• Who can submit fraud proofs? Anyone or is there a whitelist?
• If there is a whitelist, who adds and removes entries to this whitelist?
• What is the length of the fraud proof window? Can that be changed? By whom? Has it ever been changed? What are other important parameters that may affect how the Rollup actually works?
• If the answer to the above questions depends on the current implementation (and in most cases it does), can the implementation be upgraded? By whom? Has it been upgraded in the past, and if yes, what has changed in the new implementation?

Making sense of what is…

We, at L2BEAT, have run a script that “drills down” into the Arbitrum One deployment. Amongst other things, the script “understands” that most contracts are implemented using TransparentUpgradableProxy, some contracts are implemented using UpgradableBeacon and the main Rollup contract acts as an entrypoint that dispatches calls to two different logic contracts the UserFacet and AdminFacet. Conceptually the current deployment looks like this:

Our unofficial diagram presenting Arbitrum One L1 architecture

If it looks complex, that’s because it is. Our script wades through the hierarchy of contracts distilling whatever information we deemed to be important from the user’s point of view and produces information that will require further analysis:

Analysing the output we can deduce that contracts that are upgradable, are managed by two separate ProxyAdmin contracts that, in turn, are owned by the 4-out-of-6 Gnosis Safe MultiSig (which used to be 2-out-of-3 but it was upgraded). All the other controls with one exception of the L1GatewayRouter are also managed by the same 4-out-of-6 MultiSig.

In layman’s terms it means that with that one exception everything is managed by this MultiSig, and also that every single contract can be upgraded by that MultiSig. That one exception, the EOA that is currently the owner of the L1GatewayRouter, can manage the mapping of tokens to particular gateways (i.e. route user’s deposit/withdraw token requests to particular gateway implementations) but it cannot do anything else — the L1GatewayRouter itself can be only upgraded by the same MultiSig as everything else in the system.

…and making sense of what was

Who-can-do-what is important to know, but if you were interested in how these controls were used in the past, the script also needs to analyse the history of changes. Here’s the fragment of the script’s output for the Arbitrum One deployment:

Further analysis of this output reveals that:

1. On the 29th of May two Validators were set up (and no more, so right now only two whitelisted stakers can submit fraud proofs)
2. On the 28th of August some upgrades were performed to the main system by updating its facets and by updating three Beacon contracts that are used by NodeFactory, ChallengeFactory and old Outbox
3. New addresses have been added for the Sequencer role
4. Outbox has been replaced with a new implementation. Interestingly there were two updates, the first one (to EOA) was quickly removed afterwards
5. New gas limit of 120,000,000 has been set, presumably increasing the value that was set before
6. On 31st of August whitelists for accessing Inboxes and Gateways were removed entirely (set to 0x00000)
7. On 14th of September, a new owner for the whole Rollup was changed from the EOA to the current MultiSig)

How easy is it to extract this information from on-chain data? In theory it is all there, in practice this is the example of a transaction decoded using the EthTx Transaction Decoder that sets new gas limit for Arbitrum blocks:

https://ethtx.info/0x276c7dc9c1703664b312d1ea1075cc7b8d33e3e09d0904b79a3fc32a2d44145c/

As you can see there, the only thing you can deduct from Event itself is that something happened (OwnerFunctionCalled(id=11)) but specifics are not clear. You need to understand what each id in the Event means and you need to gather the exact data (like new settings that might have changed) from elsewhere. One can try to analyse the call trace (we will then see that from the Gnosis MultiSig’s execTransaction, setAvmGasSpeedLimitPerBlock was eventually called which sets this Rollup parameter). Analysing traces, as a technique to get blockchain history, is hard though and requires a lot of effort.

The second, even more important question is how can we be sure that we have fetched all relevant transactions that alter the state that we are interested in? Is it enough to track all OwnerFunctionCalled(id=11) events as in the above transaction?

It turns out that it is not enough. There is another transaction that alters this particular storage slot. It is much more complex as it deploys and initialises the Rollup and initially sets its many parameters. However, it isn’t obvious that this is the case just by looking at the trace.

State diffs to the rescue

So what is an alternative? It turns out that there is one team — at Token Flow Insights — that is currently building an easily queryable full state diff history for every single Ethereum transaction. You can think about it as a super-charged Ethereum archive node that will give you access to full state diff history. Token Flow Insights is also the team behind the EthTx Transaction Decoder.

With this dataset, accessing the history of changes in any storage slot does not need any additional knowledge about the contract logic. All we need to do is simply to find out what is the storage slot that holds the AvmGasSpeedLimitPerBlock and ask for its history without worrying about which method of which call in which implementation version has actually managed to change it (note that with implementation upgrades storage slots can change, so this needs to be tracked as well).

This is shown in the following extracts from Token Flow’s data warehouse:

This simple query gives you the full history of changes in the AvmGasSpeedLimitPerBlock parameter:

All that is needed then is the map of all the slots that we might be interested in monitoring but this can be also queried from the Token Flow Semantic Database:

Given that the first transaction that sets the initial state of the AvmGasSpeedLimitPerBlock does not emit the OwnerFunctionCalled event that we have used to track the parameters, we initially missed it. Additionally our assumption that the AvmGasSpeedLimitPerBlock was increased turned out to be wrong — it was actually decreased from 600,000,000 to 120,000,000.

Do we need blockchain vigilantes?

Monitoring state and extracting state change history is a complex endeavour. Who should do all that work and who should finance it? As stated at the beginning, with the “don’t trust, verify” philosophy, in our opinion, this is what a vendor-neutral third party should do. Moreover, it would be great if Rollup providers could help with financing such effort without actually leaving the burden to actual end users.

This is exactly what we aim to do at L2BEAT — we already report current TVL and Risk Profile (largely based on the claims from the documentation and very rudimentary analysis of the source code — if available), but this is just the beginning. What the community needs is live monitoring for all the major Rollups, alerting and — ultimately — tooling for users to opt out from whatever changes to the Rollup architecture and / or parameters are thrown at them by the Rollup administrators.

If you believe that this mission is worth pursuing, L2BEAT is currently participating in the GR12 Gitcoin Grant round.