(RESOLVED) Incident Report: Forwarding Events from Ethereum
This blog post will provide updates on the situation and expected timelines for resolution as soon as they are available. Updates will also be shared via the official Vega Twitter and Discord channels.
Date & Time
2023/11/21 12:30 GMT
Status
Resolved / Monitoring
MAJOR
For full details of how incidents are classified, please see this blog.
Description
The Vega project team is currently investigating an issue identified where no prices were sourced for the ETH/USD(USDT) perp market since around 10:30 GMT on 21/11/2023. This can be seen in the Vega block explorer when looking at the oracle for the market.
The proximate cause for this is that some validators are failing to forward all required events from the Ethereum network. The reason these events are not being forwarded is unknown and may relate either to unexpected or mishandled states coming from Ethereum RPC nodes run by validators or from infrastructure issues or rate limiting.
As the issue does not appear to be related to specific event types, it is likely that it may also impact deposits from Ethereum into Vega, however, this has not yet been confirmed.
No funds stored on the network are at risk, however, there is a risk of incorrect or delayed funding settlement on markets relying on Ethereum-sourced data.
This blog post will provide updates on the situation as more information becomes available and expected timelines for resolution when known. Updates will also be shared via our usual Twitter and Discord channels.
Updates
2023/11/21 15:30 (UTC)
The ETH/USD(USDT) perpetual market is now receiving oracle prices once more. The Vega project team has identified potential improvements to reduce the Ethereum RPC load and minimise the chance of future similar incidents. This issue will be monitored to ensure the network is functioning well.
2023/11/23 10:07 (UTC)
The Vega project team has released a new patch version of the core, which includes improvements to reduce the number of calls made to the Ethereum RPC. The validators have deployed v0.73.6 of the network. The issue will continue to be monitored.
