Polkascan Development Update #3
Milestone 3: key runtime data entities
With the Polkascan project we are working towards a multi-chain explorer. This development update is part of the work related to the Web3 Foundation Grant that ensures that the Polkadot-ecosystem has access to an open source block explorer from day one! Our project contributes to the Polkadot ecosystem by providing an MVP for a generalized open-source block explorer for any Substrate-based blockchain, such as the Polkadot relay chain.
This third development update is released together with the delivery of our third project milestones as announced here.
Milestone 3 | key runtime data entities: The third milestone extends the base system-architecture of the explorer with a number of Substrate Runtime Module Libraries (SRML) that are key to the inner workings of the Polkadot Relay Chain, such as: timestamp, indices (accounts), sessions, democracy & staking. With this milestone the explorer should provide basic data structures for the following entities: block timestamps, accounts, staking (validators), sessions (validators) & democracy (proposals, referenda).
1. Getting started
This development update will help you get started by running the third milestone release of Polkascan PRE yourself.
1.1. Polkascan PRE on Github
Repositories: The source code of Polkascan PRE can be found at our Github organization. This organization consists of a number of distinct repositories which collectively form Polkascan PRE. We apply a number of conventions for branches and versioned releases across these Polkascan PRE repositories.
Branches: Each repository has our most recent — but possibly unstable — work in the master-branch. The milestone 3 work can be found in the milestone3-branch.
Releases: The milestone 3 work is released as ‘v0.3.x’.
1.2. What makes Polkascan PRE?
Polkascan PRE consists of a number of distinct software artifacts which collectively orchestrate Polkascan PRE. The Harvester transforms a Substrate node’s raw data into relational data, with the help of Substrate Interface Library and the SCALE Codec Library. The produced relational data is disseminated by the Explorer API and in turn made accessible to end-users by the Explorer GUI. We have chosen to provide full Docker support for all our artifacts, hence all our repositories have Dockerfiles in their root.
Although these five distinct components of Polkascan PRE could be applied independently (and likely even in other projects), we offer a sixth piece of software, called: Polkascan PRE that glues together all these components with Docker Compose.
1.3. Requirements
- Recommended hardware: Memory: >8GB (more is likely better), Storage: >100GB (SSD is likely better), Processor: more and faster cores is better (Intel i7 quad core), Bandwidth: more and faster is better (>20 Mbps). These requirements will likely go down as the project matures.
- Software requirements: Git, Docker & Docker Compose.
- Deployment of Polkascan PRE has been tested on Mac, Linux and Windows.
- The Explorer GUI has been tested on Safari, Firefox, Chrome and Edge.
1.4. Running Polkascan PRE
This paragraph provides a step-by-step guide on running our milestone 3 release of Polkascan PRE on your own machine. The instructions outlined in this paragraph can be found in our Github repository.
Step 1: Clone the repository:
git clone https://github.com/polkascan/polkascan-pre.gitStep 2: Go to the new folder:
cd polkascan-preStep 3: Check available releases:
git tagStep 4: Checkout the latest release in the v0.3.x range (replace ‘x’ with highest number):
git checkout v0.3.xStep 5: Initialize and update submodules:
git submodule update --init --recursiveStep 6: Build and initialize the MySQL container:
docker-compose -p dev -f docker-compose.yml up -d mysqlStep 7: Build and initialize the other containers:
docker-compose -p dev -f docker-compose.yml up --buildResult: The Explorer GUI should now be available in a browser: http://127.0.0.1:8080
1.5. Cleaning up
The following [docker] commands should help you clean up.
- Stop all containers of the Docker Compose file.
docker-compose -p dev -f docker-compose.yml down- Remove all unused containers and images (caution!!).
docker system prune
[confirm] Y- Remove all containers and images (caution!!).
docker system prune -a
[confirm] Y- Remove all unused volumes (caution!!).
docker volume prune
[confirm] Y2. Exploring Polkascan PRE
The milestone 3 release of Polkascan PRE offers details on a number of new data entities, such as: logs, accounts, indices, transfers, proposals and referenda. These entities provide useful and meaningful data sets to the more abstract entities that were already supported by the milestone 2 release, such as: blocks, extrinsics, events and runtime specification entities. Basic definitions of these notions can be found in the Substrate’s official documentation.
2.1. Exploring the GUI
The Explorer GUI is an Angular (mobile friendly) application that consumes data from the Explorer API. The application’s landing page is accessible at the following link: http://127.0.0.1:8080. The top section of the landing page displays a navigation menu to the following sections: blocks, extrinsics, events, logs, accounts, indices, transfers, proposals, referenda and runtime specifications. The Polkascan logo returns you to this landing page.
The header sections shows you the name of the network you are on, optionally supported by a color-code if it is a known (pre-configured) network. Additionally this section allows you to search by block number, extrinsic hash or account address. The next section of the landing page lists a number of key indicators, such as: the most recent finalized blocks, the number of signed extrinsics, the number of module events, the number of active accounts and the number of distinct runtime specifications that have been harvested. Furthermore, the bottom section of the landing page lists some details of the most recently harvested blocks and of the most recently harvested balance transfers. Both these list have buttons that allow you to find more details. Below you can find a list of useful sections of the Explorer GUI:
Core blockchain entities
Blocks: http://127.0.0.1:8080/block
Extrinsics: http://127.0.0.1:8080/extrinsic
Events: http://127.0.0.1:8080/event
Logs: http://127.0.0.1:8080/log
Accounts: http://127.0.0.1:8080/account
Runtime specifications: http://127.0.0.1:8080/runtimeKey runtime module entities
Account Indices: http://127.0.0.1:8080/indices/account
Transfers: http://127.0.0.1:8080/balances/transfer
Democracy Proposals: http://127.0.0.1:8080/democracy/proposal
Democracy Referenda: http://127.0.0.1:8080/democracy/referendum2.2. Exploring the API
The Explorer API is a Falcon application that disseminates data from the relational database that is maintained by the Harvester. The Falcon-framework offers a fast RESTful API and we apply the JSON-API standard as message envelope.
Docker Compose uses NGINX proxy rules to embed the Explorer API in the routing schema of the Explorer GUI. Below you can find a list of useful API end-points of the Explorer API:
Core blockchain API endpoints
Block overview: http://127.0.0.1:8080/api/v1/block
Block detail: http://127.0.0.1:8080/api/v1/block/12345
Extrinsic overview: http://127.0.0.1:8080/api/v1/extrinsic
Extrinsic detail by id: http://127.0.0.1:8080/api/v1/extrinsic/12345-0
Extrinsic detail: http://127.0.0.1:8080/api/v1/extrinsic/0x123...def
Event overview: http://127.0.0.1:8080/api/v1/event
Event detail: http://127.0.0.1:8080/api/v1/event/12345-0
Log overview: http://127.0.0.1:8080/api/v1/log
Log detail: http://127.0.0.1:8080/api/v1/log/12345-0
Account overview: http://127.0.0.1:8080/api/v1/account
Account detail: http://127.0.0.1:8080/api/v1/account/5Cabdce..
Runtime specification overview: http://127.0.0.1:8080/api/v1/runtime
Runtime specification detail: http://127.0.0.1:8080/api/v1/runtime/112Key runtime module API endpoints
Account Index overview: http://127.0.0.1:8080/api/v1/indices/account
Account Index detail: http://127.0.0.1:8080/api/v1/indices/account/Fc12
Transfer overview: http://127.0.0.1:8080/api/v1/balances/transfer
Transfer detail: http://127.0.0.1:8080/api/v1/balances/transfer/0x123..def
Democracy Proposal overview: http://127.0.0.1:8080/api/v1/democracy/proposal
Democracy Proposal detail: http://127.0.0.1:8080/api/v1/democracy/proposal/1
Democracy Referendum overview: http://127.0.0.1:8080/api/v1/democracy/referendum
Democracy Referendum detail: http://127.0.0.1:8080/api/v1/democracy/referendum/12.3. Exploring the DB
Polkascan PRE is DBMS-agnostic. That said, we have set up the Docker Compose configuration for this milestone with recent version of MySQL. You can connect to the MySQL DBMS with you favorite tool. The default connection details are listed below:
Host: 127.0.0.1
Port: 33061
Database: polkascan
Username: root
Password: root
The database consists a number of tables which are outlined below:
alembic_version: version data to enforce ddl by SQLAlchemy.
data_account: account data.
data_account_audit: account audit data.
data_account_index: indices account data.
data_account_index_audit: indices account audit data.
data_block: block data.
data_block_total: additional block data.
data_democracy_proposal: democracy proposal data.
data_democracy_proposal_audit: democracy proposal audit data.
data_democracy_referendum: democracy referendum data.
data_democracy_referendum_audit: democracy referendum audit data.
data_event: event data.
data_extrinsic: extrinsic (and transaction and inherent) data.
data_log: digest log data.
data_session: session data.
data_session_total: additional session data.
data_session_validator: session validator data.
runtime: runtime specification data.
runtime_call: runtime specification data of call functions.
runtime_call_param: runtime specification data of call function parameters.
runtime_storage: runtime specification data of storage functions.
runtime_event: runtime specification data of events.
runtime_event_attribute: runtime specification data of event attributes.
runtime_module: runtime specification data of modules.
runtime_type: runtime specification data of types.3. Stateful data in state-machine
One of the key challenges we had to solve for our milestone 3 deliverables has been to devise a method to deal with Stateful entities in the explorer. Accounts, indices, democracy proposals and democracy referenda are examples of objects that actually live inside the blockchain and have a lifecycle in which they get their state transitioned by the runtime.
This, in contrast to the Stateless entities such as extrinsics, events and blocks which all get written to the blockchain once and remain static (or immutable).
Our milestone 3 harvester now has two distinct processes, which we call the Accumulator, which harvests stateless data and the Sequencer which builds stateful data based on the already available stateless data. The accumulator’s tasks scale really well because they can be parallelized and executed in any random order. Only very few tasks have to be executed by the sequencer.
In time our project may very well end up building a Python WASM-interpreter for handling the runtime binary or even building a full Python Substrate-client. However, for now, our method is very effective for our purposes.
4. Polkascan and the ecosystem
Our development activities are closely aligned with ecosystem partners (Web3 Foundation and Parity Technologies). We support various new networks and future parachains (Edgeware, Robonomics, Joystream and many more to come). And we are in ongoing dialog with other ecosystem service providers (wallets, faucets, clients) about integration with our platform.
5. Hacking Polkascan PRE
Our stack is mostly based on Python frameworks for our Harvester and API apps and the Angular framework for our GUI app. Our application is glued together with Docker and Docker-Compose which allow our stack to run virtually everywhere. Many cool hacks can be achieved by by hacking the docker compose file we used earlier in this document. We have documented many cool hacks in our previous development update, such as: customizing ports, running other networks, using your own Substrate-node and using your own database server. We will host a workshop at Web3 Summit 2019 to help you get started with hacking on Polkascan.
About Polkascan & WEB3SCAN
Our project contributes to the Polkadot ecosystem by providing an MVP for a generalized open-source ecosystem block explorer, called Polkascan PRE. Polkascan PRE offers a block explorer for any Substrate-based blockchain, such as the Polkadot relay chain (Alexander network). This block explorer harvests and decodes data from Substrate-nodes, stores the decoded data in a relational database and disseminates the data through an API to be used in a block explorer user interface. The Web3 Foundation Wave1 Grant ensures that the Polkadot-ecosystem has access to an open source block explorer from day one!
With the Polkascan project we are additionally working on a multi-chain explorer called Polkascan MC of which polkascan.io is an instance. This multi-chain explorer aims to make multi-chain data accessible and understandable. In order to further our goals with the Polkascan Multi-chain Explorer we are developing block explorers that works well with many individual chains. Block explorers for these individual chains are — in turn — aggregated into the Polkascan Multi-chain Explorer.
Polkascan project updates
Our fourth milestone is due soon and will provide various performance optimizations of our stack. You can expect frequent updates and we intend to finish this project before the end of summer 2019. The following public resources enable tracking of progress of the project: Medium, Twitter & GitHub. We encourage you to reach out if you would like to collaborate especially if you intend to be a Substrate implementer or ecosystem service provider. You can find us on the Riot channels on a daily basis. Come say hello and talk to us on how to get involved.
WEB3SCAN | Service Provider
WEB3SCAN — the organization behind the Polkascan project — offers professional services around blockchain data and blockchain information management, including but not limited to providing full-service block explorers, consultancy services & systems integration services. Reach out to us if you would like to learn more.
