Polkascan Development Update #1
Progress towards the Polkascan Multi-chain Explorer
Hello from the team building Polkascan. We are humbled that our project has been included in Web3 Foundation’s first wave of grants. The blockchain community will also benefit from this as W3F Grants requires all work resulting from this grant to be open-sourced.
Our project contributes to the Polkadot ecosystem by providing an MVP for a generalized open-source block explorer for Substrate-based blockchains, such as the Polkadot relay chain (Alexander network). The block explorer harvests and decodes data from Substrate-nodes, decoded data is stored in a relational database and is disseminated by an API to be used in a block explorer user interface. This grant ensures that the Polkadot-ecosystem has access to an open source block explorer from day one!
With the Polkascan project we are working towards a multi-chain explorer. We aim to make multi-chain data accessible and understandable. In order to get to the Polkascan Multi-chain Explorer we are developing an explorer that works well with many individual chains. Explorers for these individual chains can in turn be aggregated into the Polkascan Multi-chain Explorer.
Substrate — as blockchain development framework — is ideal for this generalized approach. By making our explorers compatible with Substrate, we are also making our explorers compatible with all the blockchains that are being built on Substrate. An increasing number of teams are developing their blockchain projects with Substrate. All these teams and their respective communities will require explorers to some extent.
In this first update, we would like to give you a brief overview of our immediate development milestones as part of the ecosystem grant, and update you on our current progress.
Overview of the open source explorer
The generalized open source explorer for Substrate-based blockchains consists of a number of artifacts (applications, libraries and scripts).
- Python Harvester Application
The purpose of the Harvester Application is to transform a Substrate node’s raw data into relational data for various classes of objects, such as: blocks, runtime metadata entities, extrinsics, events and various runtime data entities, such as: timestamps, accounts and balances. - Python Substrate RPC Wrapper Library
One repetitive task of the Harvester Application is to query a Substrate node for required data. A generalized library will specialize in this task. This library should independently be useful for the wider Polkadot-ecosystem. - Python SCALE Codec Library
Most of the data that the RPCs output is encoded with the SCALE Codec. This codec is used by the nodes’ internal runtime. In order to get to meaningful data — which applies to most of the data of the block explorer — this data will need to be decoded. The SCALE Codec Library will specialize in this task. This library should independently be useful for the wider Polkadot-ecosystem. - Python Explorer API Application
The purpose of the Explorer API Application is to make the data which is produced by the Harvester Application accessible to developers of the respective blockchains. The Explorer API Application will provide endpoints to all harvested data. This artifact may spawn a whole new range of applications which utilize rich data sets which are not directly offered by the Substrate node itself. - Angular Explorer GUI Application
The purpose of the Explorer GUI Application is to make the data which is produced by the Harvester Application and disseminated by the Explorer API Application accessible to day-to-day end-user of the respective blockchains. The Explorer GUI Application provides a user interface to the Explorer API Application and intends to showcase what developers should be able to build on top of the Explorer API Application for a wide audience of day-to-day users. - Docker Compose
Deploying all artifacts with Docker containers leaves the stack hardware agnostic and provides the freedom to easily scale up service management requirements. For example early stage deployment on AWS and subsequent migration to self-managed hardware due to cost efficiencies. All listed artifacts will be Dockerized.
Challenges
We have solved a number of technical challenges when developing the Polkascan Open Source Explorer.
- Performance
Performance is a continuous challenge. In general, if you require the explorer to keep up with the chaintip of the blockchain, then you need to be able to perform all data processing and storage operations within the constraints of the blocktime. - Large databases
Managing very large database is a continuous challenge. In general, if you require the explorer to provide analytics over all data in a particular blockchain then your databases will grow very large. - Archive nodes
The explorer requires historical state data which is not necessarily found in a typical configuration of a Substrate node. Substrate nodes need to be configured as — so called — ‘archive’ nodes. Nodes with this configuration disable state pruning and keep historical states available. - Service management
A successful blockchain will be used extensively and thus will grow considerably. This leads to substantial amounts of data and thus high data storage requirements. In addition, a publicly served block explorer will have high data traffic requirements. - Substrate specific challenges
Substrate has relatively few RPC endpoints because it is a generalized low-level blockchain system. Most of the data that is provided by the RPC endpoints is encoded with the SCALE Codec and a number of cryptographic primitives and other codecs are applied by Substrate.
Milestones
The ecosystem grant allows us to develop key parts of the Polkascan Multi-chain Explorer’s architecture. Our immediate priorities have been broken down into four distinct milestones outlined below.
- Milestone 1: architecture
This milestone implements the base system-architecture for all artifacts of the Polkascan Open Source Explorer stack. With this milestone the explorer should provide a minimum set of harvested data to show how all components of the stack’s architecture work together. - Milestone 2: core data entities
This milestone extends the base system-architecture for all artifacts of the Polkascan Open Source Explorer stack with the data entities that are core to Substrate, such as: runtime metadata entities, extrinsics & events. With this milestone the explorer should provide a full set of harvested data, including fully decoded views of the following entities: blocks, runtime modules, runtime call functions, runtime storage functions, runtime events, runtime types, extrinsics and events. These entities should exist in [most] Substrate-based blockchains. Therefore support for these entities makes the Polkascan Open Source Explorer compatible with [most] Substrate-based blockchains. - Milestone 3: key runtime data entities
This milestone extends the base system-architecture for all artifacts of the Polkascan Open Source Explorer stack with a number of Substrate Runtime Module Libraries (SRML) that are key to the inner workings of the Polkadot Relay Chain, such as: timestamp, indices (accounts), balances, sessions, democracy, staking & parachains. With this milestone the explorer should provide a full set of harvested data for the following entities: timestamps (for blocks), accounts (balances), staking (validators, nominators, balances), sessions (rewards), democracy (proposals, votes), parachains. These entities may not exist in all Substrate-based blockchains. Therefore support for these entities should be optional and configurable. - Milestone 4: search, optimizations & usability
This milestone extends the base system-architecture for all artifacts of the Polkascan Open Source Explorer stack with search and filter functions. Furthermore this phase will implement performance optimizations and usability optimizations. With this milestone the explorer should efficiently provide a full set of harvested data for Substrate’s core data entities, Substrate’s key SRML entities and basic search/filter functions. Documentation and tutorials should allow Polkadot-ecosystem community members to run the Polkascan Open Source Explorer for the Polkadot relay chain (Alexander network) or for their own Substrate-based blockchain.
Project Updates
Our first milestone is due soon. You can expect frequent updates and we intend to finish this project before the end of summer 2019. The following public resources enable tracking of progress of the project: Medium, Twitter & GitHub.
Reach out to us
We encourage you to reach out if you would like to collaborate especially if you intend to be a Substrate implementer or ecosystem service provider. You can find us on the Riot channels on a daily basis. Come say hello and talk to us on how to get involved.
WEB3SCAN — the organization behind the Polkascan project — offers professional services around blockchain data and blockchain information management, including but not limited to providing fully serviced block explorers, consultancy services & systems integration services. Reach out to us if you would like to learn more.
