An Overview Of On-chain Event Logs Processing

While it is widely understood that smart contracts store data in their state, it is essential to track the journey towards the current state. One effective solution is to leverage an off-chain indexer. By utilizing an off-chain indexer, we can gain valuable insights into the progression leading up to the present state.

The overview of Polygonscan (Contract account) tabs

Before we delve into the topic, it’s essential to gain a brief understanding of Polygonscan. By familiarizing ourselves with Polygonscan, we can lay a solid foundation for our exploration.

1. Transactions: Native transactions. It means your address are directly involved in the transaction (i.e. your address is the from address or the to address). Each row represents one transaction ⇒ the txn Hash is unique.
2. Internal Txns: aka contract interactions. It shows some contract interactions within transactions so a few internal transactions may belong to same transaction.
3. ERC-20 Token Txns: Show the ERC20 tokens transfer records
4. ERC-721 Token Txns: Show the ERC721 NFT transfer records
5. ERC-1155 Token Txns: Show the ERC1155 NFT transfer records
6. Contract*: a place for interacting with the contract, e.g. verifying the contract code, read contract, write contract, etc.
7. Events*: All event logs in this contract (That is what we are going to talk about today!)

*: Only appears in Contract address

What are Event Logs?

Event logs are structured data that are generated and stored on the blockchain when specific events occur, such as transactions, smart contract executions, token transfers, and more. Each event log contains relevant details about the event, including the involved parties, timestamp, and additional data specific to the event type. These logs serve as an essential component of on-chain data processing in blockchain systems, capturing important information about occurrences or actions within smart contracts. They provide a decentralized and immutable source of truth, offering transparency and auditability. Event logs empower developers, applications, and users to monitor and react to specific events, facilitating the creation of decentralized applications (dApps), automated processes, and smart contract interactions. Leveraging event logs, developers can build robust applications that securely and reliably interact with on-chain events.

The Importance of Off-chain Indexer

Off-chain indexers play a crucial role in blockchain ecosystems by providing efficient and scalable access to on-chain data. While blockchain networks store vast amounts of valuable information, directly querying and processing this data on-chain can be resource-intensive and time-consuming. Off-chain indexers serve as intermediaries between applications and the blockchain, indexing and organizing on-chain data in a way that allows for faster and more targeted queries. By offloading the indexing process to dedicated off-chain infrastructure, applications can retrieve specific data efficiently, enabling real-time analytics, decentralized applications, and data-driven decision-making. Off-chain indexers also enhance scalability by reducing the burden on the underlying blockchain network, resulting in improved performance and responsiveness. They are essential for building robust and user-friendly applications that require access to on-chain data without sacrificing performance or scalability.

Data Providers:

Before we dive into the main content, let me introduce a few reputable data providers that I have personally tried.

1. Moralis

Moralis offers real-time blockchain data across popular blockchain networks like Ethereum, Binance Smart Chain, Polygon, Solana, etc. Moralis indexes all core aspects of blockchain data and provides access to this through a suite of data-focussed API endpoints, including Blocks, Transactions, NFTs, Tokens and Balance information.

2. Alchemy

Alchemy is a powerful blockchain infrastructure provider that offers a suite of developer tools and services. It simplifies the process of building and scaling blockchain applications by providing reliable and scalable infrastructure. Additionally, Alchemy offers a range of developer-friendly features, including raw transactions data, processed NFT data, analytics, monitoring, and debugging tools, to enhance the development and deployment process.

3. The Graph (Subgraph)

Different from Moralis and Alchemy, Subgraph does not provide raw transactions data directly. Subgraphs are open APIs on The Graph that organize and serve blockchain data to applications. Using subgraphs, developers and data consumers alike benefit from speedy access to indexed data.

Architectures:

Here is a brief overview of different approaches for event logs processing.

1. Subgraph / The graph + Lambda

1. a log handler of mapping the events with the schema
2. Publish the subgraph.
3. An EventBridge event triggers an ECS every minute.
4. Fetch latest data snapshot from subgraph.
5. Process the data we fetched and update database
6. Duplicate a same set of services for other contracts but with different logic and settings of step1.

2. Moralis API + Lambda

*Moralis can be replaced by alchemy or any other data provider.

1. An EventBridge event triggers EC2 instance every minute.
2. Fetch event logs from Moralis. (Get logs for contract / Get events for contract)
3. Process the logs and publish it to a corresponding topic
4. Each serverless application (SQS + Lambda) subscribes to one topics.
5. Formatting the data to fit our schemas.
6. Update the database.

3. Moralis Streaming + SNS + Lambda

*Moralis can be replaced by alchemy or any other data provider.

1. Subscibe to Moralis. (https://docs.moralis.io/streams-api/evm/monitor-multiple-addresses)
2. Send the data to correspondings topic
3. Each serverless application (SQS + Lambda) subscribes to one topics.
4. Formatting the data to fit our schemas.
5. Update the database.

Subgraph VS Moralis API VS Moralis Streaming — How to choose your approach?

Subgraph provides a convenient setup and eliminates the need to handle data sources manually. It automatically decodes event logs and delivers real-time data. With support for over 20 networks, such as Ethereum, Polygon, Binance Smart Chain, Goerli, and Mumbai, it offers extensive compatibility. However, it does not support webhooks or pub/sub mechanisms, which can limit its flexibility. Additionally, deploying a new subgraph is necessary to add support for additional contracts.

The API approach provides support for a wide range of networks, including Ethereum, Polygon, Binance Smart Chain, Goerli, and Mumbai. One notable advantage of this approach, particularly for junior developers, is its relative ease of use compared to the streaming solution, requiring less technical knowledge of handling streaming data. However, it is important to consider that the API approach may introduce latency due to EventBridge rules.

For real-time data in our own database, streaming solutions offer an optimal choice. Supporting over 10 networks, including Ethereum, Polygon, Binance Smart Chain, Goerli, and Mumbai, they facilitate simplified support for additional contracts and integration with data analytics services. However, implementing a streaming solution requires more effort and the need to handle each part independently.

So, which one is the best approach?

Consider these factors when evaluating and selecting the most suitable approach for accessing blockchain data, taking into careful consideration the specific requirements, objectives, and priorities of our project. By thoroughly assessing the advantages and drawbacks of each option, including Subgraph, API solutions like Moralis, and streaming solutions, we can make an informed decision that aligns with our project’s architecture, scalability needs, real-time data requirements, flexibility in data processing and storage, integration potential with data analytics services, and the level of effort and resources we are willing to invest. Ultimately, choosing the right approach will contribute to the success and efficiency of our project by ensuring seamless access to blockchain data while effectively addressing our unique business needs.