Polkascan Development Update #2

Architecture (milestone 1) & core data entities (milestone 2)

With the Polkascan project we are working towards a multi-chain explorer. This development update is part of the work related to the Web3 Foundation Grant that ensures that the Polkadot-ecosystem has access to an open source block explorer from day one! Our project contributes to the Polkadot ecosystem by providing an MVP for a generalized open-source block explorer for any Substrate-based blockchain, such as the Polkadot relay chain (Alexander network).

This second development update is released together with the delivery of our first two project milestones as announced here.

1. Milestone 1 | architecture: The first milestones implements the basic system-architecture for all artifacts of our open source block explorer for Substrate-based blockchains, called Polkascan PRE. With this milestone Polkascan PRE provides a minimum set of harvested data for a number of Substrate-based blockchains. The first milestone release illustrates how all components of the architecture work together.
2. Milestone 2 | core data entities: The second milestone extends the base system-architecture with data entities that are core to Substrate, such as: runtime metadata entities, extrinsics & events. The second milestone release provides a full set of harvested data, including fully decoded views of: blocks, runtime modules, runtime call functions, runtime events, runtime types, extrinsics and events. These entities should exist in [most] Substrate-based blockchains. Therefore support for these entities makes Polkascan PRE compatible with [most] Substrate-based blockchains.

1. Getting started

This development update will help you get started by running the second milestone release of Polkascan PRE yourself.

1.1. Polkascan PRE on Github

Repositories: The source code of Polkascan PRE can be found at our Github organization. This organization consists of several distinct repositories which collectively form Polkascan PRE. We apply some conventions for branches and versioned releases across these Polkascan PRE repositories.

Image 1: Polkascan on Github: https://github.com/polkascan

Branches: Each repository has our most recent — but possibly unstable — work in the master-branch. We have defined a number of project milestones for our open-source block explorer. The source code and software releases of Polkascan PRE are aligned with our project milestones. Milestone 1 work can be found in the milestone1-branch, milestone 2 work can be found in the milestone2-branch, etc.

Releases: In addition to these distinct branches per milestone we align our software releases with our milestones. Milestone 1 work is released as version ‘0.1.x’, milestone 2 work is released as ‘v0.2.x’, etc.

Future support: Our milestones are cumulative in nature, this means for example that our milestone 2 work provides all features that were offered in the milestone 1 branch and releases. This also implies that the milestone1-branch is deprecated and new commits will no longer be back-ported to this branch. And additionally as soon as the milestone3-branch becomes available the milestone2-branch will be deprecated. This development update hence focuses on our most recent milestone 2 release.

1.2. What makes Polkascan PRE?

Polkascan PRE consists of a number of distinct software artifacts which collectively orchestrate Polkascan PRE. The Harvester transforms a Substrate node’s raw data into relational data, with the help of Substrate Interface Library and the SCALE Codec Library. The produced relational data is disseminated by the Explorer API and in turn made accessible to end-users by the Explorer GUI. We have chosen to provide full Docker support for all our artifacts, hence all our repositories have Dockerfiles in their root.

Although these five distinct components of Polkascan PRE could be applied independently (and likely even in other projects), we offer a sixth piece of software, called: Polkascan PRE that glues together all these components with Docker Compose.

1.3. Requirements

1. Recommended hardware: >8GB memory (more is likely better), >100GB storage: (SSD is better), multi-core processor (more and faster cores is better) & >20 Mbps bandwidth (more and faster is better). These requirements will likely go down as the project matures.
2. Software requirements: Git, Docker & Docker Compose.
3. Deployment of Polkascan PRE has been tested on Mac, Linux and Windows.
4. The Explorer GUI has been tested on Safari, Firefox, Chrome and Edge.

1.4. Running Polkascan PRE

This paragraph provides a step-by-step guide on running our milestone 2 release of Polkascan PRE on your own machine. The instructions outlined in this paragraph can be found in our Github repository.

Step 1: Clone the repository:

git clone https://github.com/polkascan/polkascan-pre.git

Step 2: Go to the new folder:

cd polkascan-pre

Step 3: Check available releases:

git tag

Step 4: Checkout the latest release in the v0.2.x range (replace ‘x’ with highest number):

git checkout v0.2.x

Step 5: Initialize and update submodules:

git submodule update --init --recursive

Step 6: Build and initialize the MySQL container:

docker-compose -p dev -f docker-compose.yml up -d mysql

Step 7: Build and initialize the other containers:

docker-compose -p dev -f docker-compose.yml up --build

Result: The Explorer GUI should now be available in a browser: http://127.0.0.1:8080

Image 2: A successful build of Polkascan PRE

1.5. Cleaning up

The following [docker] commands should help you clean up.

• Stop all containers of the Docker Compose file.

docker-compose -p dev -f docker-compose.yml down

• Remove all unused containers and images (caution!!).

docker system prune
[confirm] Y

• Remove all containers and images (caution!!).

docker system prune -a
[confirm] Y

• Remove all unused volumes (caution!!).

docker volume prune
[confirm] Y

2. Exploring Polkascan PRE

The milestone 2 release of Polkascan PRE offers details on a number of useful data entities, such as: blocks, extrinsics, events and runtime specifications. Basic definitions of these notions can be found in the Substrate’s official documentation.

2.1. Exploring the GUI

The Explorer GUI is an Angular (mobile friendly) application that consumes data from the Explorer API. The application’s landing page is accessible at the following URL when running the steps in the previous paragraph: http://127.0.0.1:8080. The top section of the landing page displays a navigation menu to the following sections: blocks, extrinsics , events and runtimes. The Polkascan logo returns you to this landing page.

The header sections shows you the name of the network you are on, optionally supported by a color-code if it is a known (preconfigured) network. Additionally this section allows you to search by block number, extrinsic hash or account address. The next section of the landing page lists a number of key indicators, such as: the most recent finalized blocks, the number of signed extrinsics, the number of events and the number of distinct runtimes that have been harvested. Furthermore bottom section of the landing page lists some details of the most recently harvested blocks and of the most recently harvested balance transfers. Both these list have buttons that allow you to find more details. Below you can find a list of useful sections of the Explorer GUI:

Blocks

1. Block overview page: http://127.0.0.1:8080/system/block
   Provides an overview of harvested blocks sorted by block number in descending order. The details button allows you to navigate to the block details page.
2. Block detail page: http://127.0.0.1:8080/system/block/12345
   Displays the harvested details of one particular block by block number.

Extrinsics

1. Extrinsic overview page: http://127.0.0.1:8080/system/extrinsic
   Provides an overview of harvested extrinsics sorted by block number in descending order and extrinsic index in ascending order. The details button allows you to navigate to the extrinsic details page.
2. Extrinsic detail page: http://127.0.0.1:8080/system/extrinsic/12345-0
   Displays the harvested details of one particular extrinsic by block number and extrinsic index.
3. Extrinsic detail page: http://127.0.0.1:8080/system/extrinsic/0x123...def
   Displays the harvested details of one particular extrinsic by extrinsic hash. Please note that the extrinsic hash is only available for signed extrinsics and is equal to the (32 byte) blake2b-hash of the extrinsic data itself.

Events

1. Event overview page: http://127.0.0.1:8080/system/event
   Provides an overview of harvested events sorted by block number in descending order and event index in ascending order. The details button allows you to navigate to the event details page.
2. Event detail page: http://127.0.0.1:8080/system/event/12345-0
   Displays the harvested details of one particular event by block number and event index.

Runtimes

1. Runtime landing page: http://127.0.0.1:8080/system/runtime
   Provides an overview of distinct runtime specifications for harvested blocks sorted by specification number in descending order. The details button allows you to navigate to the runtime details page.
2. Runtime detail page: http://127.0.0.1:8080/system/runtime/112
   Displays the harvested details of one particular runtime specification by specification number.

2.2. Exploring the API

The Explorer API is a Falcon (Python framework) application that disseminates data from the relational database that is maintained by the Harvester. The Falcon-framework offers a fast RESTful API and we apply the JSON-API standard as message envelope.

Image 3: Exploring the API

Docker Compose uses NGINX proxy rules to embed the Explorer API in the routing schema of the Explorer GUI. Below you can find a list of useful API end-points of the Explorer API:

Blocks

1. Block overview page: http://127.0.0.1:8080/api/v1/system/block
2. Block detail page: http://127.0.0.1:8080/api/v1/system/block/12345

Extrinsics

1. Extrinsic overview page: http://127.0.0.1:8080/api/v1/system/extrinsic
2. Extrinsic detail page: http://127.0.0.1:8080/api/v1/system/extrinsic/12345-0
3. Extrinsic detail page: http://127.0.0.1:8080/api/v1/system/extrinsic/0x123...def

Events

1. Event overview page: http://127.0.0.1:8080/api/v1/system/event
2. Event detail page: http://127.0.0.1:8080/api/v1/system/event/12345-0

Runtimes

1. Runtime landing page: http://127.0.0.1:8080/api/v1/system/runtime
2. Runtime detail page: http://127.0.0.1:8080/api/v1/system/runtime/112

2.3. Exploring the DB

Polkascan PRE is DBMS-agnostic. That said, we have set up the Docker Compose configuration for this milestone with recent version of MySQL. You can connect to the MySQL DBMS with you favorite tool. The default connection details are listed below:

Host: 127.0.0.1
Port: 33061
Database: polkascan
Username: root
Password: root

Image 4: Exploring the DB

The database consists a number of tables which are outlined below:

Blocks

data_block: provides decoded block data.
primary key: id
indexes: hash, parent_hash
columns: id, parent_id, hash, parent_hash, state_root, extrinsics_root, count_extrinsics, count_events, spec_version_id, debug_info

Extrinsics

data_extrinsic: provides decoded extrinsic data.
primary key: block_id, extrinsic_idx
indexes: block_id, extrinsic_idx, extrinsic_hash, call_id, signed, unsigned, account_index, address, module_id
columns: block_id, extrinsic_idx, extrinsic_length, extrinsic_version, signed, unsigned, address_length, address, account_index, signature, nonce, era, call, module_id, call_id, params, success, error, spec_version_id, codec_error, extrinsic_hash

Events

data_event: provides decoded event data.
primary key: block_id, event_idx
indexes: block_id, event_idx, extrinsic_idx, module, system, type, module_id, event_id
columns: block_id, event_idx, extrinsic_idx, type, spec_version_id, module_id, event_id, system, module, phase, attributes, codec_error

Runtimes

runtime: provides decoded runtime specification data.
runtime_call: provides decoded runtime specification data of call functions.
runtime_call_param: provides decoded runtime specification data of call function parameters.
runtime_event: provides decoded runtime specification data of events.
runtime_event_attribute: provides decoded runtime specification data of event attributes.
runtime_module: provides decoded runtime specification data of modules.
runtime_type: provides decoded runtime specification data of types.

Other

alembic_version: provides version data to enforce table definition version management by SQLAlchemy.

3. Hacking Polkascan PRE

It is recommended that you familiarize yourself with Docker and Docker-Compose. The following topics cover smart hacks to the Docker Compose file that has been used earlier in this document.

3.1. Customizing the exposed UI port

Problem: The Docker Compose file uses port 8080 in the default configuration to expose the Explorer GUI and the Explorer API and you would like to use a different port for that.

Solution: Simply substituting the occurrence of ‘8080’ before the colon (change 1) and substituting the occurrence of ‘8080’ in the URL (change 2) will do the job. (changes: 1 & 2)

3.2. Customizing the exposed DB port

Problem: The Docker Compose file uses port 33061 in the default configuration to expose the MySQL DBMS and you would like to use a different port for that.

Solution: Simply substituting the occurrence of ‘33061’ before the colon (change 1) with the required port in the Docker Compose file will do the job.

3.3. Customizing the exposed Substrate client’s RPC port

Problem: The Docker Compose file uses port 9933 in the default configuration to expose the Substrate node’s RPC endpoint and you would like to use a different port for that.

Solution: Simply substituting the occurrence of ‘9933’ before the colon (change 1) with the required port in the Docker Compose file will do the job.

3.4. Customizing the exposed Substrate client’s WS port

Problem: The Docker Compose file uses port 9944 in the default configuration to expose the Substrate node’s WS endpoint and you would like to use a different port for that.

Solution: Simply substituting the occurrence of ‘9944’ before the colon (change 1) with the required port in the Docker Compose file will do the job.

3.5. Running other networks

Problem: The Docker Compose file builds and runs a block explorer for a single node development network with the relay-chain client runtime.

Solution: Polkascan PRE comes with four different public network configurations for the Alexander testnet, the Edgeware testnet, the Robonomics testnet and the Joystream testnet. Please inspect these Docker Compose files to see how you could adjust to other networks. In addition it is recommended that you adjust the docker-compose command by changing the ‘project-name’ [‘-p’ or ‘ — project-name’-option]. I.e. you should substitute ‘-p dev’ with ‘-p [network-name]’ to avoid conflicts with instances of Polkascan PRE you have running for other networks. Example commands can be found here.

Image 5: Arjan Zijderveld demonstrates Polkascan PRE for the Alexander network at Sub0 in Berlin. (video)

3.6. Using your own Substrate-node

Problem: The Docker Compose file builds and runs a Substrate node for the selected network. You may already have a Substrate node configured and you may want to use this node.

Solution: Substitute the hostname and port in the Docker Compose file (change 1) with the connection settings (hostname and port) of another Substrate node. And additionally completely remove the service called ‘substrate-node’ from the Docker Compose file.

3.7. Using your own database server

Problem: The Docker Compose file builds and runs a MySQL DBMS. You may already have a MySQL DBMS configured and you may want to use that DBMS.

Solution: Substitute the (multiple) connection configuration setting environment variables in the Docker Compose file (DB_HOST, DB_PORT, DB_USER, DB_PASSWORD and DB_NAME) with the connection configuration settings of the external DBMS. Before building and running the Docker Compose command ensure that the database and user account work. And additionally remove the service called ‘mysql’ from the Docker Compose file.

3.8. Other off-scope topics

The Docker Compose files also contains multiple instances of the Harvester which are in turn glued together with a Redis messaging system. These topics and inner workings remain off-scope for this development update.

About Polkascan & WEB3SCAN

Our project contributes to the Polkadot ecosystem by providing an MVP for a generalized open-source ecosystem block explorer, called Polkascan PRE. Polkascan PRE offers a block explorer for any Substrate-based blockchain, such as the Polkadot relay chain (Alexander network). This block explorer harvests and decodes data from Substrate-nodes, stores the decoded data in a relational database and disseminates the data through an API to be used in a block explorer user interface. The Web3 Foundation Wave1 Grant ensures that the Polkadot-ecosystem has access to an open source block explorer from day one!

With the Polkascan project we are additionally working on a multi-chain explorer called Polkascan MC of which polkascan.io is an instance. This multi-chain explorer aims to make multi-chain data accessible and understandable. In order to further our goals with the Polkascan Multi-chain Explorer we are developing block explorers that works well with many individual chains. Block explorers for these individual chains are — in turn — aggregated into the Polkascan Multi-chain Explorer.

Image 6: Emiel Sebastiaan on the relation between Polkascan PRE & Polkascan MC at Sub0 in Berlin. (video)

Polkascan project updates

Our third milestone is due soon and will provide support for specific frequently used runtime modules. You can expect frequent updates and we intend to finish this project before the end of summer 2019. The following public resources enable tracking of progress of the project: Medium, Twitter & GitHub. We encourage you to reach out if you would like to collaborate especially if you intend to be a Substrate implementer or ecosystem service provider. You can find us on the Riot channels on a daily basis. Come say hello and talk to us on how to get involved.

WEB3SCAN | Service Provider

WEB3SCAN — the organization behind the Polkascan project — offers professional services around blockchain data and blockchain data analytics, including but not limited to providing full-service block explorers, consultancy services & systems integration services. Reach out to us if you would like to learn more.