Deploying Substrate & Polkadot nodes with Polkasource

Polkadot is expected to be delivered in less than a year. As Polkadot approaches a feature-complete codebase, applications, services, and businesses can begin integrating with Polkadot. Polkasource is one of the first independent projects in the Polkadot ecosystem. This blog details our contribution to this great emerging ecosystem.

Motivation

We’ve identified a number of trends in relation to accessibility and reliability of blockchains that guide us in our approach to building Polkasource:

  1. The future will have many interoperable blockchains.
  2. Blockchain technology will become more mission-critical to organizations.
  3. Reliable blockchain infrastructure requires mature IT Service Management (ITSM).
  4. Specialized service providers will emerge to cater to blockchain infrastructure’s ITSM needs.

WEB3SOURCE — the organization behind the Polkasource project — intends to be one such service provider, specialized in ITSM for Blockchain Infrastructure.

Dave Hoogendoorn on the relevance of ITSM for Blockchain Infrastructure

Why Polkasource?

With Polkasource we intend to remove barriers to access blockchains. Providing reliable on-ramps to Substrate-based networks is one of the first step towards this objective. Such on-ramps should be able to seamlessly process fluctuating volumes of incoming and outgoing traffic beyond the limitations of single blockchain nodes. The Polkasource node-cluster gateway is our solution for such an on-ramp.

Run your own node

The ‘run your own node’-narrative has been strong since the invention of Bitcoin, and for good reasons. The security argument is the most important reason given to run your own node, although after careful inspection this is really a composite notion of: trustlessness, autonomy, safety, privacy, etc. Perhaps another good — though altruistic — reason for running your own node is that it helps the network. However beneficial, claims that running your own node is cheap or easy are exaggerations.

Whether you are an individual or an organization, running your own node can be quite a challenge. Running a full node or running an archive node has non-negligible resource requirements around storage, memory, bandwidth and processing power. Furthermore, anyone who has any experience running a node for an extensive period of time can attest to the system management challenges. These challenges relate to the requirements regarding the reliability of your node for your specific use-case, such as performance, availability, continuity, upgrade-ability, disaster recovery, backups, risk management and compliance.

Since ‘trade-off triangles’ are a constant debate in the blockchain space, (not without controversy), we figured that the ‘run your own node’-narrative deserves its own trade-off triangle, so get ready to ‘pick two!’ of either simplicity, security and reliability.

Figure 1: The ‘run your own node’ trade-off triangle: “pick two”

Light clients

One approach to making access to blockchains simple and secure is by using a light client. A well-balanced overview of what a light client is and what it does is provided by our friends at Parity Technologies. To paraphrase, “light clients are crucial elements in blockchain ecosystems. They help users access and interact with a blockchain in a secure and decentralized manner without having to sync the full blockchain.” Light clients are considered to work well for lightweight and intermittent blockchain interaction, and thus have relatively low requirements regarding reliability.

Reliability

What if the light client approach does not cut it for your use case? It is quite possible that your use case may require:

  • full nodes or archive nodes
  • nodes to operate continuously and without interruption
  • nodes with high capacity or high throughput beyond the limitations of the underlying hardware
  • multiple blockchains
  • multiple clients
  • multiple client versions
  • seamless version upgrades

These are just a few examples, but it is safe to say that not all blockchain use-cases have similar requirements regarding reliability and that meeting these requirements is a multifaceted problem.

Coinbase is an example of a company that has extreme requirements regarding reliability of their blockchain infrastructure. Custodian cryptocurrency exchanges are the first class of unicorn companies in the blockchain industry and should — for that reason alone — have the highest imaginable requirements regarding reliability of their blockchain infrastructure. These requirements regarding reliability form a spectrum with the intermittent lightweight use cases and low requirements regarding reliability on one end and the mission-critical heavyweight use cases with extremely high requirements regarding reliability on the other end.

Figure 2: Polkasource use case spectrum

Polkasource intends to add value and intends to be applicable to any use case not on the immediate edges of this spectrum. I.e. any use case where a light client is insufficient and where your risk-appetite allows you to — partially — outsource ITSM of your blockchain node infrastructure.

Risk management

When you go down the blockchain/Bitcoin rabbit-hole, you quickly become familiar with the notions of decentralization, anti-fragility and resilience. Although these concepts are great properties of blockchain networks as a whole, they are certainly not properties of individual nodes that compose these blockchain networks. Thus, it helps to start identifying and assessing the risks of your nodes failing. An organization’s risk-appetite really determines an organization’s willingness to mitigate these risks by deploying more resources.

One obvious way of mitigating single-node-failure risks is to simply not rely on a single node (figure 3a/b) in your blockchain infrastructure.

Figure 3: Increasing reliability of blockchain infrastructure

The more mature blockchain networks have multiple clients to mitigate the risks of systemic client failure, and any organization should take these risks into consideration when designing their blockchain infrastructure. You can read about the motivations for a multi-client system in W3F Grants announcement of a Polkadot Golang implementation. A node-cluster comprised of multiple instances of multiple clients with a single API gateway (figure 3c/d) behaves as a single node, without the reliability risks of a single node. The Polkasource node-cluster gateway is a solution for creating reliable blockchain infrastructure that behaves as a single node, thus making it easy to use while simultaneously maintaining security and increasing reliability of the infrastructure.

Node-cluster gateway architecture

The Polkasource node-cluster gateway architecture provides the features of a full node or archive node without the limitations of single blockchain nodes, around: performance, availability, continuity, upgrade-ability, disaster recovery, backups, risk management and compliance.

Figure 4: Polkasource: node-cluster gateway architecture

The node-cluster gateway architecture is our solution for creating reliable blockchain node infrastructures. The architecture is comprised of a number of components. Figure 4 provides a high-level overview of these components and their relationships. An API-key is mandatory for using the Polkasource node-cluster gateway. The user portal (1) enables end-users to request access (2) to authorized end-points of the gateway, resulting in an API-key (3). Using the API-key (4), requests can be forwarded to the authorized end-points. The reverse proxy (5) is responsible for interpreting the requests and seamlessly serving Substrate-based blockchain APIs; either by getting a response from the cache (6) or getting a response from one of the the node-clusters (7). The elastic manager (8) provides orchestration and management of components and resources within the node-cluster gateway architecture. By monitoring and analyzing live performance data the elastic manager is capable of scaling individual components up and down to meet fluctuating demand. The benchmark application (9) provides objective and unbiased metrics about the quality, conformity and performance of the reverse proxy as well as node-clusters and helps the elastic manager to learn how to act.

Polkadot & Substrate

We are anticipating a Cambrian explosion of innovation with Substrate, and consequently expect a market that will need to be serviced. One early example of the power of Substrate is that it will support light clients in an unprecedented way. We believe that enterprise and consortium use-cases will be served well by Substrate and Polkadot, as well as supporting middleware such as Polkasource. We are closely monitoring enterprise adoption of these technologies, including Energy Web Foundation’s efforts to implement an independent network of blockchains using Polkadot and Substrate.

What’s next

The ideas and the need for Polkasource started to make sense to us when we started working on our sister project Polkascan. A block explorer naturally requires a reliable blockchain node infrastructure. By now the paths of Polkascan and Polkasource have sufficiently diverged into two independent organizations with distinct focus. This divergence has allowed Polkascan to outsource its node-cluster management activities to Polkasource. Both projects are expected to maintain a close synergistic relationship.

We are proud to announce our collaboration with Speckle OS. Speckle is a Polkadot-ecosystem service provider and will provide account management, governance participation, secure signing, js object injection and parachain discovery/interactivity via their browser. Polkasource will in turn provide public access points to a number of supported Substrate-based networks. Speckle Browser will connect to these access points when interacting with the Polkadot network as a default (initially before browser embedded light-clients) and as a fallback method to query state and submit extrinsics.

These two service delivery solutions illustrate the diversity to which our services can be applied. Integration with the Polkascan block explorer provides an on-premise service delivery solution with private end-points to Substrate-based networks. Our collaboration with Speckle OS provides a full-service cloud delivery solution with public end-points to Substrate-based networks. We look forward to exploring all other hybrid solutions imaginable.

Reach out to us

We encourage you to reach out if you would like to collaborate, especially if you intend to be a Substrate implementer or service provider in the Polkadot-ecosystem. You can find us on the Riot channels on a daily basis. Come say hello and talk to us on how to get involved. Follow us on Twitter for updates on our progress and feel free to reach out if you have any further questions or comments.