Polkascan Development Update #3

Emiel Sebastiaan
Jul 11, 2019 · 9 min read
Image for post
Image for post

Milestone 3: key runtime data entities

With the Polkascan project we are working towards a multi-chain explorer. This development update is part of the work related to the Web3 Foundation Grant that ensures that the Polkadot-ecosystem has access to an open source block explorer from day one! Our project contributes to the Polkadot ecosystem by providing an MVP for a generalized open-source block explorer for any Substrate-based blockchain, such as the Polkadot relay chain.

This third development update is released together with the delivery of our third project milestones as announced here.

Milestone 3 | key runtime data entities: The third milestone extends the base system-architecture of the explorer with a number of Substrate Runtime Module Libraries (SRML) that are key to the inner workings of the Polkadot Relay Chain, such as: timestamp, indices (accounts), sessions, democracy & staking. With this milestone the explorer should provide basic data structures for the following entities: block timestamps, accounts, staking (validators), sessions (validators) & democracy (proposals, referenda).

1. Getting started

This development update will help you get started by running the third milestone release of Polkascan PRE yourself.

Repositories: The source code of Polkascan PRE can be found at our Github organization. This organization consists of a number of distinct repositories which collectively form Polkascan PRE. We apply a number of conventions for branches and versioned releases across these Polkascan PRE repositories.

Image for post
Image for post
Image 1: Polkascan on Github: https://github.com/polkascan

Branches: Each repository has our most recent — but possibly unstable — work in the master-branch. The milestone 3 work can be found in the milestone3-branch.

Releases: The milestone 3 work is released as ‘v0.3.x’.

Polkascan PRE consists of a number of distinct software artifacts which collectively orchestrate Polkascan PRE. The Harvester transforms a Substrate node’s raw data into relational data, with the help of Substrate Interface Library and the SCALE Codec Library. The produced relational data is disseminated by the Explorer API and in turn made accessible to end-users by the Explorer GUI. We have chosen to provide full Docker support for all our artifacts, hence all our repositories have Dockerfiles in their root.

Although these five distinct components of Polkascan PRE could be applied independently (and likely even in other projects), we offer a sixth piece of software, called: Polkascan PRE that glues together all these components with Docker Compose.

  • Recommended hardware: Memory: >8GB (more is likely better), Storage: >100GB (SSD is likely better), Processor: more and faster cores is better (Intel i7 quad core), Bandwidth: more and faster is better (>20 Mbps). These requirements will likely go down as the project matures.
  • Software requirements: Git, Docker & Docker Compose.
  • Deployment of Polkascan PRE has been tested on Mac, Linux and Windows.
  • The Explorer GUI has been tested on Safari, Firefox, Chrome and Edge.

This paragraph provides a step-by-step guide on running our milestone 3 release of Polkascan PRE on your own machine. The instructions outlined in this paragraph can be found in our Github repository.

Step 1: Clone the repository:

git clone https://github.com/polkascan/polkascan-pre.git

Step 2: Go to the new folder:

cd polkascan-pre

Step 3: Check available releases:

git tag

Step 4: Checkout the latest release in the v0.3.x range (replace ‘x’ with highest number):

git checkout v0.3.x

Step 5: Initialize and update submodules:

git submodule update --init --recursive

Step 6: Build and initialize the MySQL container:

docker-compose -p dev -f docker-compose.yml up -d mysql

Step 7: Build and initialize the other containers:

docker-compose -p dev -f docker-compose.yml up --build

Result: The Explorer GUI should now be available in a browser:

Image for post
Image for post
Image 2: A successful build of Polkascan PRE

The following [docker] commands should help you clean up.

  • Stop all containers of the Docker Compose file.
docker-compose -p dev -f docker-compose.yml down
  • Remove all unused containers and images (caution!!).
docker system prune
[confirm] Y
  • Remove all containers and images (caution!!).
docker system prune -a
[confirm] Y
  • Remove all unused volumes (caution!!).
docker volume prune
[confirm] Y

2. Exploring Polkascan PRE

The milestone 3 release of Polkascan PRE offers details on a number of new data entities, such as: logs, accounts, indices, transfers, proposals and referenda. These entities provide useful and meaningful data sets to the more abstract entities that were already supported by the milestone 2 release, such as: blocks, extrinsics, events and runtime specification entities. Basic definitions of these notions can be found in the Substrate’s official documentation.

The Explorer GUI is an Angular (mobile friendly) application that consumes data from the Explorer API. The application’s landing page is accessible at the following link: The top section of the landing page displays a navigation menu to the following sections: blocks, extrinsics, events, logs, accounts, indices, transfers, proposals, referenda and runtime specifications. The Polkascan logo returns you to this landing page.

The header sections shows you the name of the network you are on, optionally supported by a color-code if it is a known (pre-configured) network. Additionally this section allows you to search by block number, extrinsic hash or account address. The next section of the landing page lists a number of key indicators, such as: the most recent finalized blocks, the number of signed extrinsics, the number of module events, the number of active accounts and the number of distinct runtime specifications that have been harvested. Furthermore, the bottom section of the landing page lists some details of the most recently harvested blocks and of the most recently harvested balance transfers. Both these list have buttons that allow you to find more details. Below you can find a list of useful sections of the Explorer GUI:

Core blockchain entities

Runtime specifications:

Key runtime module entities

Account Indices:
Democracy Proposals:
Democracy Referenda:

The Explorer API is a Falcon application that disseminates data from the relational database that is maintained by the Harvester. The Falcon-framework offers a fast RESTful API and we apply the JSON-API standard as message envelope.

Image for post
Image for post
Image 3: Exploring the API

Docker Compose uses NGINX proxy rules to embed the Explorer API in the routing schema of the Explorer GUI. Below you can find a list of useful API end-points of the Explorer API:

Core blockchain API endpoints

Block overview:
Block detail:
Extrinsic overview:
Extrinsic detail by id:
Extrinsic detail:
Event overview:
Event detail:
Log overview:
Log detail:
Account overview:
Account detail:
Runtime specification overview:
Runtime specification detail:

Key runtime module API endpoints

Account Index overview:
Account Index detail:
Transfer overview:
Transfer detail:
Democracy Proposal overview:
Democracy Proposal detail:
Democracy Referendum overview:
Democracy Referendum detail:

Polkascan PRE is DBMS-agnostic. That said, we have set up the Docker Compose configuration for this milestone with recent version of MySQL. You can connect to the MySQL DBMS with you favorite tool. The default connection details are listed below:

Port: 33061
Database: polkascan
Username: root
Password: root
Image for post
Image for post
Image 4: Exploring the DB

The database consists a number of tables which are outlined below:

alembic_version: version data to enforce ddl by SQLAlchemy.
data_account: account data.
data_account_audit: account audit data.
data_account_index: indices account data.
data_account_index_audit: indices account audit data.
data_block: block data.
data_block_total: additional block data.
data_democracy_proposal: democracy proposal data.
data_democracy_proposal_audit: democracy proposal audit data.
data_democracy_referendum: democracy referendum data.
data_democracy_referendum_audit: democracy referendum audit data.
data_event: event data.
data_extrinsic: extrinsic (and transaction and inherent) data.
data_log: digest log data.
data_session: session data.
data_session_total: additional session data.
data_session_validator: session validator data.
runtime: runtime specification data.
runtime_call: runtime specification data of call functions.
runtime_call_param: runtime specification data of call function parameters.
runtime_storage: runtime specification data of storage functions.
runtime_event: runtime specification data of events.
runtime_event_attribute: runtime specification data of event attributes.
runtime_module: runtime specification data of modules.
runtime_type: runtime specification data of types.

3. Stateful data in state-machine

One of the key challenges we had to solve for our milestone 3 deliverables has been to devise a method to deal with Stateful entities in the explorer. Accounts, indices, democracy proposals and democracy referenda are examples of objects that actually live inside the blockchain and have a lifecycle in which they get their state transitioned by the runtime.

This, in contrast to the Stateless entities such as extrinsics, events and blocks which all get written to the blockchain once and remain static (or immutable).

Our milestone 3 harvester now has two distinct processes, which we call the Accumulator, which harvests stateless data and the Sequencer which builds stateful data based on the already available stateless data. The accumulator’s tasks scale really well because they can be parallelized and executed in any random order. Only very few tasks have to be executed by the sequencer.

In time our project may very well end up building a Python WASM-interpreter for handling the runtime binary or even building a full Python Substrate-client. However, for now, our method is very effective for our purposes.

4. Polkascan and the ecosystem

Our development activities are closely aligned with ecosystem partners (Web3 Foundation and Parity Technologies). We support various new networks and future parachains (Edgeware, Robonomics, Joystream and many more to come). And we are in ongoing dialog with other ecosystem service providers (wallets, faucets, clients) about integration with our platform.

Image for post
Image for post
Image 5: Integration between Polkadot-UI and Polkascan.io. (link)

5. Hacking Polkascan PRE

Our stack is mostly based on Python frameworks for our Harvester and API apps and the Angular framework for our GUI app. Our application is glued together with Docker and Docker-Compose which allow our stack to run virtually everywhere. Many cool hacks can be achieved by by hacking the docker compose file we used earlier in this document. We have documented many cool hacks in our previous development update, such as: customizing ports, running other networks, using your own Substrate-node and using your own database server. We will host a workshop at Web3 Summit 2019 to help you get started with hacking on Polkascan.

About Polkascan & WEB3SCAN

Our project contributes to the Polkadot ecosystem by providing an MVP for a generalized open-source ecosystem block explorer, called Polkascan PRE. Polkascan PRE offers a block explorer for any Substrate-based blockchain, such as the Polkadot relay chain (Alexander network). This block explorer harvests and decodes data from Substrate-nodes, stores the decoded data in a relational database and disseminates the data through an API to be used in a block explorer user interface. The Web3 Foundation Wave1 Grant ensures that the Polkadot-ecosystem has access to an open source block explorer from day one!

Image for post
Image for post
Image 6: Polkascan.io the multi-chain block explorer for the Polkadot ecosystem. (link)

With the Polkascan project we are additionally working on a multi-chain explorer called Polkascan MC of which polkascan.io is an instance. This multi-chain explorer aims to make multi-chain data accessible and understandable. In order to further our goals with the Polkascan Multi-chain Explorer we are developing block explorers that works well with many individual chains. Block explorers for these individual chains are — in turn — aggregated into the Polkascan Multi-chain Explorer.

Image for post
Image for post
Image 7: Polkascan’s Dave Hoogendoorn participating in the LongHash incubation program in Singapore. (tweet)

Our fourth milestone is due soon and will provide various performance optimizations of our stack. You can expect frequent updates and we intend to finish this project before the end of summer 2019. The following public resources enable tracking of progress of the project: Medium, Twitter & GitHub. We encourage you to reach out if you would like to collaborate especially if you intend to be a Substrate implementer or ecosystem service provider. You can find us on the Riot channels on a daily basis. Come say hello and talk to us on how to get involved.

Image for post
Image for post

WEB3SCAN — the organization behind the Polkascan project — offers professional services around blockchain data and blockchain information management, including but not limited to providing full-service block explorers, consultancy services & systems integration services. Reach out to us if you would like to learn more.

Polkadot Network

Medium account/ publication of the Polkadot Network

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store