Polkascan Development Update #7
Polkascan Signer Capabilities
Polkascan Foundation’s mission is to make multi-chain data accessible and understandable. With the Polkascan project we are working towards a multi-chain exploration and data analytic platform for the Polkadot-ecosystem.
This development update is part of our work that extends the Polkascan stack in general and extends Polkascan’s Python libraries in particular with key-generation, signing and signature verification capabilities. 1) Python Signer Libraries: for offline signing and offline signature verification and 2) Angular Parity Signer Components: for secure online signing.
Our organization is on an ongoing mission of building the Polkascan multi-chain explorer. Our fourth wave grant work left signing capabilities off-scope in our Python libraries. This project aims to provide full signing capabilities that can be used from within Python libraries and from user-facing Angular (web) applications.
The work covered in this development update contributes to the broader Polkadot & Substrate ecosystems by providing Python-based signing and signature verification capabilities and an independent Angular-based visual signer interface to Parity Signer. Both sets of deliverables provide signing and signature verification mechanisms which are independent of the currently dominant Javascript/Typescript mechanisms. Having independent implementations is good for the ecosystem as a whole. Needless to say, maturing and making available Python-based and Angular-based core components benefit the ecosystems.
Our team has been active in the Polkadot-ecosystem from day one, we believe Substrate and the Polkadot will solve many industry-wide problems. Additionally we are highly invested in Python. This project aims to lower the friction of using Python with Substrate-based chains and the completion of the Python-libraries will benefit many teams in the ecosystem. On top of that, this grant work will allow on-boarding of Python developers to integrate with existing or new Python-applications or applications in any other language with existing and new Substrate chains.
1. Getting started
This development update will help you get started by running a release of the Substrate Interface yourself.
1.1. Substrate Interface on Github
Repositories: The source code that extends the Polkascan stack with signer capabilities can be found at our Github organization. This organization consists of a number of distinct repositories which collectively form the Substrate Interface. We apply a number of conventions for branches and versioned releases across these repositories.
Branches: Each repository has our most recent work in the master-branch.
Releases: This development update’s work is released and tagged as ‘v0.10.x’.
1.2. What makes Substrate Interface?
Substrate Interface consists of a number of distinct software artifacts which collectively orchestrate the interface. The Substrate Interface API provides JSON-RPC end-points that allow for enriched interaction with Substrate’s standard JSON-RPC interface with the help of Python Substrate Interface Library, the Python SCALE Codec Library and various RUST-bindings for Python: the Python Bindings for Tiny-BIP39 RUST Crate, the Python Bindings for the ED25519-Dalek RUST Crate and the Python Bindings for the RUST Implementation of the SR25519 Cryptography Library.
Substrate Interface GUI is a basic user interface that showcases how Substrate Interface could be applied in front-end Angular Typescript applications. Of particular interest are the Angular Parity Signer Components. These components cover full compatibility with Parity Signer, through: 1) QR Address Creation, 2) QR Signing Payload, and 3) QR Signature.
We have chosen to provide full Docker support for all our artifacts, hence all relevant repositories have Dockerfiles in their root. Although these distinct components of Substrate Interface could be applied independently (in other projects), we offer an additional piece of software, called: Substrate Interface that glues together all these components with Docker Compose.
1.3. Requirements
- Recommended hardware: Memory: >2GB (more is better), Storage: >5GB (SSD is better), Processor: more and faster cores is better (Intel i3).
- Software requirements: Git, Docker & Docker Compose.
- Deployment of the Substrate Interface has been tested on Mac, Linux and Windows.1. Getting started
This development update will help you get started by running Substrate Interface yourself.
1.4. Building and running Substrate Interface
This paragraph provides a step-by-step guide on running our v0.10.* release of the Enriched Substrate Interface on your own machine. The instructions outlined in this paragraph can be found in our Github repository.
- Clone the repository:
git clone https://github.com/polkascan/substrate-interface.git - Go to the new folder:
cd substrate-interface - Check available releases:
git tag - Checkout the latest release in the 0.10.x range (replace ‘x’ with highest number):
git checkout 0.10.x - Initialize and update submodules:
git submodule update --init --recursive - Build and run Docker Compose for one of the following five configurations
##(a)## Kusama Network using a publicly hosted node:
docker-compose -p substrate-interface -f docker-compose.kusama-public.yml up --build --remove-orphans##(b)## Kusama Network using a local node that is part of the docker-compose configuration:
docker-compose -p substrate-interface -f docker-compose.kusama-local.yml up --build --remove-orphans##(c)## Polkadot Network using a publicly hosted node:
docker-compose -p substrate-interface -f docker-compose.polkadot-public.yml up --build --remove-orphans##(d)## Network using a local node that is part of the docker-compose configuration:
docker-compose -p substrate-interface -f docker-compose.polkadot-local.yml up --build --remove-orphans##(e)## Substrate Node Template Development Network using a local node. This step assumes that a Substrate node is running on your local machine. The Docker-Compose configuration points to your localhost from within the configuration:
docker-compose -p substrate-interface -f docker-compose.kusama-public.yml up --build --remove-orphans
1.5. Start Substrate Interface
The Substrate Interface GUI should now be available in a browser: http://127.0.0.1:8001
The Substrate Interface API should now be available for use in Postman.
The default JSON-RPC url: http://127.0.0.1:8000
1.6. Cleaning up
The following [docker] commands should help you clean up.
- Stop all containers of the Docker Compose file.
docker-compose -p substrate-interface -f docker-compose.yml down --remove-orphans - Remove all unused containers, volumes and images.
docker system prune
[confirm] Y - Remove all containers and images.
docker system prune -a
[confirm] Y - Remove all volumes.
docker volume prune
[confirm] Y
2. Substrate Interface GUI
After following the getting started instructions, the Substrate Interface GUI should be available in a browser through the following URI: http://127.0.0.1:8001
Substrate Interface GUI provides four generic sections which are described in this chapter’s sections. These sections replicate the similar respective functions of Polkadot JS Apps without any need for the Typescript/Javascript libraries used by the latter, effectively making Substrate Interface a fully independent stack.
2.1. Remote Procedure Calls
The method-control lists all remote procedure call methods the Substrate Interface offers. This includes all native Substrate-client methods and thus provides full backwards compatibility with any regular Substrate-client. Additionally a large number of new methods are available related to this project’s grant work and previous grant work.
The following list highlights a number of new RPC-methods that have been developed during current grant work.
- keypair_create: creates a new keypair
- keypair_inspect: provides details for a mnemonic phrase.
- keypair_sign: signs an arbitrary message with a mnemonic phrase.
- keypair_verify: verifies the signature of an arbitrary message.
- runtime_createSignaturePayload: creates a message payload for any runtime call, which can be signed by the keypair functions.
- runtime_createExternalSignerPayload: creates a message payload for any runtime call, which can be signed by an external signer (without any decoding context), such as Parity Signer.
- runtime_submitExtrinsic: submits a signed extrinsic to the blockchain. This method includes a valid signature for the payload as created by the internal Python signer or the external Parity Signer.
We’d like to stress that using keypair-methods in a user-facing application is not recommended. The respective components in Substrate Interface GUI are mainly used to showcase the new capabilities.
2.2. Chain State
The storage function control lists all storage functions the runtime offers. No significant changes have been made to this section of Substrate Interface. We refer to our previous development update #5 for further details.
2.3. Extrinsics with Secure Signing
A significant part of our recent grant work has resulted in full Parity Signer compatibility. Parity Signer functions as an external sandboxed signer interface that only requires verifiably secure visual communication through QR-codes. This required significant research and has resulted in an implementation of Universal Offline Signatures through QR-codes in the Polkascan stack. The remainder of this section provides a step by step instruction:
Step 1: Add account with Parity Signer
Use the ‘scan new address’-button to add the external account from the Parity Signer device. The example shows the composition of a system.remark extrinsic. This extrinsic will add an arbitrary text to the blockchain. Use the ‘sign extrinsic’-button to proceed to the next step.
Step 2: A QR-code is displayed which can be signed with an external offline signer, such as Parity Signer. When the external signer has created the QR-code with the signature, click on the ‘scan signature QR’-button to proceed to the next step.
Step 3: If the signature is valid, the extrinsic will be submitted to the blockchain. Substrate Interface GUI will display a reference to the transaction hash that has been submitted. The transaction will become available on Polkascan (or any other block explorer).
2.4. Secure Message Signing
A significant part of our recent grant work has resulted in full Parity Signer compatibility. Parity Signer functions as an external sandboxed signer interface that only requires verifiably secure visual communication through QR-codes. This section covers external signing of an arbitrary message in a step by step instruction:
Step 1: Add account with Parity Signer
Use the ‘scan new address’-button to add the external account from the Parity Signer device. The example shows how an arbitrary message can be signed with Parity Signer. Use the ‘sign message’-button to proceed to the next step.
Step 2: A QR-code is displayed which can be signed with an external offline signer, such as Parity Signer. When the external signer has created the QR-code with the signature, click on the ‘scan signature QR’-button to proceed to the next step.
Step 3: Substrate Interface GUI will display the signature that has been created by the external signer.
Step 4: The signature can be verified with external applications such as Polkadot JS Apps.
3. Substrate Interface API
3.1. Full Substrate node compatibility
Substrate Interface API provides a JSON-RPC interface to a Substrate-node and is fully compatible with JSON-RPC interface of a regular Substrate-node. E.g. the RPC-method `chain_getBlock` provides the details of a block at the current head of the chain. The RPC-method `rpc_methods` provides a discovery mechanism of the supported RPC-methods for all known groups, such as: `author`, `chain`, `state`, `system`, etc.
3.2. Keypair RPC group
Substrate Interface API extends the available RPC-methods with an entirely new group called: `keypair`, which exposes the JSON-RPC interfaces for keypair generation, keypair inspection, signing and signature verification capabilities. These capabilities were left off-scope in our previous grant work (as documented in development update #5).
An overview of these four new keypair functions can be found in this paragraph.
Keypair creation: keypair_create creates a new keypair.
Keypair inspection: keypair_inspect provides details for a mnemonic phrase.
Signing: keypair_sign signs an arbitrary message with a mnemonic phrase.
Verification: keypair_verify verifies the signature of an arbitrary message.
The Substrate Interface API currently has no support for derivation path features. The current grant work however does support a Python based solution that offers derivation path features. Please refer to section 4.6 of this development update, which describes the (deprecated) Python Subkey Wrapper.
3.3. Runtime RPC-group
Substrate Interface API extends the available RPC-methods with a group called: `runtime`, which enriches the JSON-RPC interface with all data particular to the current and previous runtimes of the Substrate-node. Most of the capabilities in this group were developed during previous grant work as documented in development update #5.
The current grant work offers a number of new RPC-methods related to signing transactions and arbitrary messages.
Create a Signature Payload: runtime_createSignaturePayload creates a message payload for any runtime call, which can be signed by the keypair functions. The example shows a simple balance transfer.
Create an External Signer Payload: runtime_createExternalSignerPayload creates a message payload for any runtime call, which can be signed by an external signer (without any decoding context), such as Parity Signer. The example shows a simple balance transfer.
Submit an extrinsic: runtime_submitExtrinsic submits a signed extrinsic to the blockchain. This method includes a valid signature for the payload as created by the internal Python signer or the external Parity Signer. The example shows a simple balance transfer. The example response provides an error message because the sending account has insufficient balance.
3.4. Documentation and code snippets
The current and extended RPC-methods have been fully documented through a Postman Collection and can be found here. The documentation provides tested examples for all current and extended RPC-methods and can easily be imported into your own Postman application by clicking on the ‘Run in Postman’ button at the top of the screen in the documented Postman Collection.
Substrate Interface API provides an enhanced JSON-RPC interface to a Substrate-node. The JSON-RPC standard is a general interface any developer can use regardless of their respective development environment. The documented Postman Collection provides code-snippets for up to 25 distinct development environments (C#, cURL, Go, Java, Javascript, NodeJs, OCaml, PHP, Python, Ruby) for all included RPC-methods. This should illustrate the broad applicability of our grant work.
The example in figure 3.4c illustrates the ease at which ‘Balances’ can be queried for a SS58-encoded account address through Postman.
4. Python Libraries
4.1. Python Substrate Interface
The Python Substrate Interface library specializes in interfacing with a Substrate node, providing additional convenience methods to deal with SCALE encoding/decoding (the default output and input format of the Substrate JSON-RPC), metadata parsing, type registry management and versioning of types. The Python Substrate Interface library is utilized by the Substrate Interface (and of course by our Polkascan-stack).
The grant work that has been completed with this development update has extended the Python Substrate Interface with key generation, signing and signature verification capabilities for general use and as an independent Python library.
Full Python library documentation has been made available through Pdoc.
The documentation illustrates how to use all available methods. These methods should look familiar given the topic covered throughout this development update.
The grant work provides a full stack of artifacts to illustrate how this library could be applied. That said, this library could be applied independently in any Python-based application. The readme in the root of the repository provides details on how to get started with using the Python Substrate Interface library in your own Python project.
Initialization: initialize interface and get head block hash of Kusama chain.
substrate = SubstrateInterface(
url=”wss://kusama-rpc.polkadot.io/”,
address_type=2,
type_registry_preset=’kusama’
)
substrate.get_chain_head()The readme contains additional examples that illustrate how to use Python Substrate Interface:
- Get extrinsics for a certain block
- Make a storage call
- Create and send signed extrinsics
- Keypair creation and signing
- Metadata and versioning
4.2. Python Bindings BIP39
The Python Substrate Interface library (4.1) utilizes a number of external cryptography libraries. Creating native Python implementations of such libraries is well beyond the scope of our current project and efforts. Python Bindings BIP39 is an interface that exposes methods from a RUST crate and makes them accessible to the Python Interpreter using PyO3.
On a high level Python Binding BIP39 allows Python applications to:
- Generate new mnemonic phrases
- Create mini-secrets from mnemonic phrases
- Create seeds from mnemonic phrases
- Validate mnemonic phrases
The RUST crate can be found on crates.io and the full library documentation can be found on docs.rs.
The readme contains detailed instructions on how to install and use these bindings from a Python application.
4.3. Python Bindings ED25519
The Python Substrate Interface library utilizes a number of external cryptography libraries. Creating native Python implementations of such libraries is well beyond the scope of our current project and efforts. Python Bindings ED25519 is an interface that exposes methods from a RUST crate and makes them accessible to the Python Interpreter using PyO3.
On a high level Python Binding ED25519 allows Python applications to:
- Return public and private keypairs from a given 32-byte seed
- Sign messages with the given keypair, returning the resulting signature
- Verify that signatures on given messages were generated by private keys corresponding to the specified public keys
The RUST crate can be found on crates.io and the full library documentation can be found on docs.rs.
The readme contains detailed instructions on how to install and use these bindings from a Python application.
4.4. Python Bindings SR25519
The Python Substrate Interface library utilizes a number of external cryptography libraries. Creating native Python implementations of such libraries is well beyond the scope of our current project and efforts. Python Bindings SR25519 is an interface that exposes methods from a RUST crate and makes them accessible to the Python Interpreter using PyO3.
On a high level Python Binding SR25519 allows Python applications to:
- Return public and private keypairs from a given 32-byte seed
- Sign messages with the given keypair, returning the resulting signature
- Verify that signatures on given messages were generated by private keys corresponding to the specified public keys
- Various low level functions regarding hard and soft derivation paths
The RUST crate can be found on crates.io and the full library documentation can be found on docs.rs.
The readme contains detailed instructions on how to install and use these bindings from a Python application.
4.5. Python SCALE Codec
The Python SCALE Codec library is a lightweight, efficient, binary serialization and deserialization codec. The Python SCALE Codec is utilized by the Python Substrate Interface (and of course by our Polkascan-stack).
No significant changes have been made to this section of Substrate Interface. We refer to our previous development update #5 for further details.
Full Python library documentation has been made available through Pdoc.
4.6. Python Subkey Wrapper
The Python Subkey Wrapper can be considered a fully functioning artifact. Our current grant work aims to develop feature compatibility of Parity Technologies’ Subkey CLI from within a Python library, in turn providing, keypair creation, signing and signature verification capabilities for native Python applications. Polkascan Foundation considers Python Subkey Wrapper deprecated code, because nearly all its features have been superseded by Python Bindings (section 4.2, 4.3 and 4.4 provides full details).
We have chosen to include this section of the development update. At time of writing using hard/soft derivation paths for keypair creation is not yet fully supported in Python Substrate Interface without using Python Subkey Wrapper.
The readme contains detailed instructions on how to install and use the wrapper from a Python application.
About Polkascan Foundation
Polkascan Foundation is a not-for-profit organization based in the Netherlands with a mission to make blockchain data accessible and understandable. Polkascan Foundation aims to achieve its mission through the following activities:
- Neutral Analytics Platform
Polkascan Foundation operates a neutral analytics platform: polkascan.io. This platform is and will remain free of commercial interests. - Governance
Polkascan Foundation actively participates in blockchain governance. Key focus of its efforts are to advocate novel methods to sustainably fund ecosystem infrastructure activities. - Maintenance & Community Support
Polkascan Foundation maintains a wide variety of repositories, some of which have a broad range of applications and users. Polkascan Foundation aims to increase adoption of its infrastructure libraries and components by actively maintaining these repositories and by providing community support to its users. - Research & Development
Technological innovations: Polkascan Foundation maintains Python infrastructure libraries, such as: Python Substrate Interface and Python Scale Codec. These libraries can be considered key building blocks for the broader Python ecosystem for interaction with Substrate-based systems.
Organizational & legal innovations: Polkascan Foundation’s Social Contract is a novel off-chain coordination mechanism between a blockchain, its treasury and its community on one hand and external service providers, such as Polkascan Foundation on the other hand. The Social Contract serves as a proving ground for effective coordination, developing the relationship of a service provider with an entity that exists outside of contemporary legal systems.
UX, design and analysis innovations: Polkascan Foundation continues its research activities to make multichain data accessible and understandable. One such innovations is Polkascan Foundation’s economic analysis methods for multichain data. By applying financial accounting practices on blockchains, its multi-chain data is made accessible and understandable for a wide and potentially new audience, beyond the tech-savvy.
The following public resources enable tracking of progress of the project: Medium, Twitter & GitHub.
