KIRA Testnet Phase 0 | Milestone 2

Objective: KIRA Network deployment, state propagation, snapshots, networking and connectivity testing between nodes in local and public networks

Prerequisites:

• 4 vCPU cores (ARM64 or x64)
• Minimum 8GB of RAM
• Minimum 32GB+ of the free storage space (plus space required to persist blockchain state and snapshots)
• Disposable cloud instance or VM
• Ubuntu 20.04 LTS installed on the host instance or VM
• Stable internet connection with at least 10 Mbps Up/Dn speed
• Static IP address or dynamic DNS
• Access to router or otherwise your local network configuration

Source code:

• KIRA Management tool — kira
• Blockchain Application — sekai

Warning: Do not use your primary operating system to test any software releases as they can cause irreversible data loss. When using a personal computer and not a dedicated instance, virtualization software, such as VMWare, should be used. When using cloud instances, along SSH, ensure that you will not get locked out when firewall rules change — see list of exposed ports later in this article. While running tests in a home environment it is recommended to use VPN as certain firewall rules might prevent access to github and other dependencies that are hosted online.

Overview

Milestone 2 will be used to test node deployment software called KIRA Manager in the new “Validator Mode”. Two or more local or public validator nodes will be created by each milestone participant and interconnected together to form a network. Goal of this particular testnet exercise is to ensure a stable operation of deployed validator node software in the context of interconnecting it with other nodes as well as testing firewall and backup management tools. By precisely following all instructions in this article you will launch your own blockchain faster than Gavin Wood in his famous substrate presentation during 2018 Web3 summit :)

Software Stack (Validator Mode)

Registry

KIRA Manager uses Docker to isolate and manage multiple parts of the infrastructure such as docker image registry, validator nodes, sentries, private sentries, frontend and INTERX™ (interchain-nginx). Registry is a container, which stores and distributes images of containerized software that in turn speeds up the build process. Every part of the infrastructure is built locally, which helps to preserve integrity of the code and decreases bandwidth utilization.

Validator

Validator node is a container that produces “blocks” containing transactions submitted by users (client software). Validator node is a type of “full node” that preserves the blockchain state, executes state machine transitions and allows the decentralized ledger to achieve consensus on the state of that ledger. Each block must be signed by a minimum of ⅔ of all validators for it to be included on the blockchain. In the validator mode there will be multiple nodes deployed by each testnet participant and new blocks will be produced every few seconds regardless if new transactions are present or not — this will enable ease of fault tolerance testing. The validator node communicates with other nodes on the network though “sentries”, which are full nodes protecting the validator node from being reached by potentially malicious requests and transactions originating from the public internet.

Sentry

Sentry node is a type of “full node” which preserves, shares and synchronizes the state of the decentralized ledger with other full nodes on the network. Sentry node’s purpose is to enable query of the blockchain state by the client applications, while, at the same time, protecting validator nodes from potentially malicious queries that might cause software to halt, restart or otherwise fail. If the sentry node fails, the validator node running behind it will remain operational as invalid blocks or malicious messages will never be propagated to it. In the Validator Mode there will be two types of sentry nodes deployed: “public sentry” and “private sentry”. Public sentry is used to communicate with other full nodes exposed to the public internet, while the Private sentry nodes are intended for communication with other private sentries, such as those belonging to other validators. By separating communication channels between public and private networks KIRA can mitigate potential DoS and DDoS attacks.

Snapshot

Snapshot container is used to backup the blockchain state at the exact block height specified by the user, without the need to stop any active sentry nodes or otherwise interrupting operation of the validator node. Backups can be shared with full nodes (sentries) via INTERX to not only make it easy to recover from potential hardware faults but also, most importantly, to drastically decrease the time it takes to sync new full nodes joining the network.

INTERX

INTERX aka “Interchain Nginx” is a software that acts as an API situated between blockchain and frontend applications. INTERX not only enables the frontend to easily query the blockchain state but also secures the connection between user and blockchain nodes regardless of the communication protocol used (HTTP/HTTPS). INTERX communication channel by signing all responses with its unique private key verifiable on chain. Thanks to INTERX frontend applications can communicate with an uncapped number of parachains without requiring light clients. Furthermore, frontend users gain ability to fully verify the integrity of the “static” application they interact with. One of the most important features of the INTERX is the ability to cache client responses, host snapshots of the blockchain state and provide faucet with free test tokens. Snapshot hosting reduces time it takes to sync new nodes, while response caching greatly decreases resource utilization, as responses can be shared with hundreds of clients without the need to repeat a single RPC query.

Frontend

Frontend container is only available for testing purposes and will not be utilized in the Milestone 2. KIRA Frontend application is a static page enabling ease of interaction with the blockchain application through the publicly available INTERX nodes. In the production environment frontend will be statically hosted via the github pages, available as a mobile app or can be started locally. This means that users can verify the hash of an application interacting with the blockchain without fear of malicious code or potential DNS hijacking. Presence of the frontend container in the infra Demo Mode serves a purpose of making it easy to test the latest software releases and is not intended for use in production.

Network Layout (Validator Mode)

Docker networking facilitates connectivity between all parts of the software stack within dedicated subnets. The firewalld is used to protect the host machine from unauthorized access to deployed containers via configurable IP whitelist and blacklist. In the Validator Mode following ports are exposed by default through default network interface:

• 22 — Reserved for SSH, allows for remote communication with the host
• 80 — Allows for http access to KIRA Frontend
• 9090 — GRPC access to the public sentry
• 26656 — P2P port allowing public sentry to sync and propagate blocks
• 26657 — RPC port enabling blockchain state query
• 36656 — P2P port allowing private sentry to sync and propagate blocks
• 11000 — INTERX enabling easy blockchain state query & snapshot hosting

In the Validator Mode network is isolated into 4 individual subnets

• regnet — dedicated to the local docker registry
• kiranet — accessible by validator and sentries
• sentrynet — accessible by sentries and snapshot nodes
• servicenet — accessible by all public facing containers

Containers (individual software stacks such as validator, sentries, etc. running within virtualized OS) can only communicate with other containers if they are part of the same network. This solution greatly improves security of the validator and other nodes as they can’t be accessed or otherwise influenced by containers running within different subnets.

kira

KIRA Manager (kira) is a console tool that enables management and deployment of the entire software stack. In the Milestone 2 we will be testing the “Validator Mode” operation of this tool. The only supported operating system is Ubuntu 20.04 LTS and it has to be fully installed and updated before setup is attempted.

Setup

When setting up kira tool for the first time you should SSH or otherwise access your host machine, then open the terminal as sudo user (sudo -s) and execute the following instruction by copy-pasting it into your terminal. It is recommended that you use an SSH terminal supporting colors such as termius — otherwise the user interface might be distorted or entirely unusable.

cd /tmp && BRANCH="master" && rm -fv ./i.sh && wget https://raw.githubusercontent.com/KiraCore/kira/$BRANCH/workstation/init.sh -O ./i.sh && chmod 555 -v ./i.sh && H=$(sha256sum ./i.sh | awk '{ print $1 }') && read -p "Is '$H' a [V]alid SHA256 ?: " -n 1 V && [ "${V,,}" == "v" ] && ./i.sh "$BRANCH" || echo "Hash was NOT accepted by the user"

Instruction above will first navigate your console to the /tmp directory, download infrastructure initialization script, and check sha256 hash of that file. To make sure you are running a legitimate script you should approve the checksum by clicking the [V] option after verifying it (on the mainnet a signed hash will be provided via public communication channels).

Initialization script will proceed to install essential dependencies and afterwards you will be presented with TERMS & CONDITIONS. After reading it you can press any button to accept terms and continue:

You will be presented with a Demo & Validator Modes as the only two possible deployment options that you can use to initialize your working environment. (In future milestones “Sentry Mode” will also be made available).

Select option [2] (Validator Mode) to proceed, then Quick Setup [1] in the following submenu. The advanced setup [2] is not intended for the scope of Milestone 2, however might be required in certain cases where your computer uses a non-standard network interface to communicate via internet or if you would want to use a custom branch to build the infrastructure from.

Creating New Blockchain

When launching your first node you should create an entirely new network by selecting option [N] and then naming your new chain according to the format <short_name>-<identifier>, for example “bobchain-1” (max 12 characters).

At the end of the setup you will be prompted with a final configuration which you can verify and approve by selecting option [A].

Depending on your internet connection the very first setup might take between 30 minutes and several hours. It is possible that due to the network connectivity issues the process might fail or be otherwise halted. In such case you should reboot your machine and start again proceeding with first step of the setup instructions. Every time setup is executed it should be faster as dependencies are installed only once and their changes tracked. If you can’t go past the installation step you should try again from a clean, updated Ubuntu 20.04.

During the setup you will be prompted to reboot your machine. After restart open your terminal again, enter sudo mode by typing ”sudo -s” command followed by a simple 4 letter command ”kira” to let the setup process continue.

If the final result is a “FINISHED: LAUNCH SCRIPT” notification followed by exit code 0, then the setup was successful and you just launched your private blockchain entirely on your own! Click any key and KIRA Manager will appear (it might take a couple of seconds to load). If you accidentally close the window you will always be able to access the manager by simply typing “kira” in the window of your terminal.

KIRA Tools & Features

Network Manager

First section of the home screen provides information such as resource utilization (CPU, RAM, DISK), network name, current blockchain height along with the genesis file checksum (4adf…80a7 in this example), local IP address, public IP address, name of the default network adapter (eth0 in our example), snapshot name including its checksum (02dd…80a7 in this example) and a color coded (green/red) global status of the local infrastructure (healthy/unhealthy).

Second section presents a list of deployed containers, their names, status, block height and health. Selecting a numeric option [0], [1] … [n] will automatically open KIRA Container Manager, allowing to manage, inspect and preview detailed information about individual containers.

Third section allows to interact with all containers simultaneously - stopping them, staring, pausing (freezing) and re-starting. If by chance any part of your infrastructure fails it is recommended to use the [R]ESTART All Containers option.

Final section of kira manager, along with the [B]ACKUP option, allows to create blockchain state snapshots. By using option [E]XPOSE (visible after the snapshot is created) your latest snapshot can be shared (exposed) with other nodes to enable them instant sync. By choosing [D]UMP option all logs and configurations can be saved onto a single ZIP file which can then be used for the purpose of debugging (You should never share these logs with untrusted parties as they can contain sensitive information about your machine such as configuration files, IP addresses and potentially passwords and other private info). Option Re-[I]NITALIZE has the same effect as executing the setup command and makes it simpler to rebuild your entire setup if something goes wrong and you manage to irreversibly break the infrastructure.

Container Manager

Container manager is a tool that appears after selecting a numeric option [0], [1] … [n] within the Network manager tool. Its purpose is to provide access and detailed information in regards to individual containers.

First section provides information such as container name, including prefix and suffix of the container ID, repository and branch relevant to the software stack deployed within the container, port map which details how internal ports are mapped to the external ports (accessible from the host), and IP addresses of the container in each subnet it belongs to.

Second section provides detailed status, such as host name, node identifier, block height, state of the container and the health information.

Third section enables control of the individual container — restart, start, stop, pausing, unpausing as well as killing processes that are running within the container. If faults occur, containers automatically attempt a restart, however if processes are killed, restart will not occur and you will be able to freely [I]NSPECT the container to debug and report potential issues. Inspecting containers can be treated as OS inspection, that is entering the operating system from within the operating system. By selecting that option you will find yourself within a virtualized Ubuntu shell that allows to access the file system and all applications installed within that isolated OS.

Final section enables dumping of all logs and debugging. In the case where a node fails, [L]OGS and [H]EALTHCHECK options enable previewing of the docker log without need to enter (inspect) the virtual machine, thus making it very easy to report or identify any potential faults without the need to [I]NSPECT the container and to manually look for potential issues.

Networking Manager

By selecting option [N] in the kira tool main menu you can access networking manager that enables you to fully control the firewall as well as to configure private and public seed and peer nodes that your validator nodes connect to.

First section lists services along with corresponding ports, which your validator node exposes to public networks via your default (or predefined) network interface. By selecting option [0], [1], … [n] you can navigate to the individual port configuration menu where independent ports can be managed — enabled, disabled from any sort of access or made accessible in the custom manner though the whitelist/blacklist of IP addresses.

Second section of the networking manager is responsible for defining state of all ports at once, that is exposing them though a predefined network [I]NTERFACE or enforcing [C]USTOM (blacklist/whitelist), [E]NABLED or [D]ISABLED configuration.

Final section of the networking manager, apart from enabling [F]IREWALL and docker network [R]ELOAD, allows for configuration of peers and seeds list. [S]EEDS option contains a configurable list of public and private nodes, which your validator does not expect to keep a constant connection with — for example, a random IP address provided to you by a friend that would allow you to join the network and discover other nodes. [P]EERS option contains a configurable list of nodes with which your validator will always try to keep a constant connection with, such as full nodes deployed by you or addresses of nodes, which you trust to be highly reliable. Both seeds and peers lists are configurable in such a manner, that when you change [ P]ublic IP addresses they will only be relevant to the “sentry” container and its P2P port 26656, while pri[V]ate IP addresses be relevant to the “priv_sentry” container and its corresponding P2P port 36656. This means that you can isolate and fine-tune network traffic reaching your validator node.

Joining Existing Network

After your first validator node (A) is deployed and you familiarize yourself with the KIRA Network Manager, you can proceed to start a second one (B) and join the existing network you just created (for example “bobchain-1” in our case). To launch a new node you will need a second, separate and clean instance or virtual machine, where you will install your KIRA Network Manager again.

Before proceeding with installation of the kira tool you must ensure that your two instances (or VM’s) are on the same network OR otherwise fully exposed to the public internet, so that they can communicate bidirectionally with each other. To test if there is connectivity between your machines use a ping command followed by the IP address or DNS name associated with your server.

If you do not know your Local IP (L.IP) address, because your kira manager is not installed yet, you can use following commands:

apt install -y net-tools dnsutils
IFACE=$(netstat -rn | grep -m 1 UG | awk '{print $8}' | xargs)
echo -e "\e[31;1mYOUR LOCAL IP: $(/sbin/ifconfig $IFACE | grep -i mask | awk '{print $2}' | cut -f2)\e[0m"

If you do not know what is your Public IP (PUB.IP) address you can use following command:

apt install -y net-tools dnsutils
echo -e "\e[31;1mYOUR PUBLIC IP: $(dig TXT +short o-o.myaddr.l.google.com @ns1.google.com +time=5 +tries=1 | awk -F'"' '{ print $2}')\e[0m"

Your ping commands must pass using both local IP addresses between both machines or using both public IP addresses. It’s fine if there is no local network connectivity, however, if there is no (bidirectional) connectivity over the public internet you will not be able to become a validator on the mainnet or public testnet. There might be many reasons why your machine might not be available over the internet and ping commands to public IP addresses fail (from outside of your local network). If your machine can not be reached over the public internet, you will need admin access to the router and “Google” its model number to find relevant instructions to safely expose your machine/s. You might also consider setting up a DDNS service. While experimenting with your router settings, make absolutely sure you are not exposing other machines in your local network to many, many, many potential attack vectors.

After ensuring connectivity between both of your machines, run the initialization command on your new machine (B) and accept TERMS & CONDITIONS. Continue setup by selecting Validator Mode option [2] and Quick Setup [1], then choose option to [J]oin existing network. You will be immediately prompted to input the IP address of the first validator node you deployed. If your machines have local network connectivity you should input local IP (L.IP) of your first validator node. If your machines have connectivity with each other over the public internet, then you should input public IP (PUB.IP) of your first validator node (A).

After providing the IP address of the already existing node you will be prompted to input the minimum block height at which your new node should start. Your new node will await for that block height, before continuing the setup process and attempting to participate in the consensus. You can skip this option by clicking [ENTER] so that the latest block is auto discovered, however in real life scenario it is recommended to manually set this number to the value higher than the last block which your validator produced in the past, otherwise you might risk double signing blocks (create a fork), which might result in your node becoming jailed or slashed.

At the end you will be prompted with a final configuration, which you can verify and approve by selecting option [A]. This time you will also find that the node automatically downloaded the genesis file from your first validator node along with the snapshot file. To ensure integrity of the files you should verify the checksums. It is important that during setup you should never connect to a node, which you do not trust or without knowing those checksums beforehand (in real life scenario they should be signed and verifiable by every node operator).

Depending on your internet connection the very first setup will take likely longer then the first one as your new validator node (B) will have to not only install all dependencies, but also sync all blocks (this process takes considerably shorter if up to date snapshot is exposed on your first node A). It is possible that due to network connectivity issues the process might fail or be otherwise halted. In such a case you should reboot your machine and start again proceeding with the step 1 of the setup instructions. Every time setup is executed it should be faster as dependencies are installed only once and their changes tracked. If you find out that a node awaits too long syncing new blocks you can use [B]ACKUP option on your first node to generate a new snapshot.

Once setup finalizes and all containers change status to healthy you will notice notification bar still implying that the validator node is NOT operational. To make your node operational and start producing blocks you will have to add it to the validator set first.

Joining Validator Set

After creating your first validator node (A) and the new network (e.g. bobchain-1) and after deploying a second node (B) by connecting it with the first node (A) you should see that new blocks are being produced on your node (A) and synced on node (B). You can also notice that if all containers on node B are [S]TOPPED or [P]AUSEd, node A will continue producing new blocks, however if all containers on node A are stopped, no more blocks will be produced until they are [R]ESTARTED. The rationale is that when you created a new network your node A created a new genesis file, which included node A as the first and only validator on your network, while node B only connected to an already existing network and is not participating in the consensus yet. If both nodes A and B were a part of the validator set then the network should stop producing blocks if either of them stopped (KIRA uses Byzantine Fault Tolerant consensus which requires ⅔ of all nodes to be always online). In this section we will use the governance process to add node B as the real validator on your new network. Since KIRA uses a governance-permissioned consensus, here are a few steps we need to follow in order to add a new node to the validator set:

• Learn the public kira address of the node (B) you want to add to the validator set
• Assign permission enabling your account (A) to create gov proposal to add a new validator
• Create & pass a gov proposal to add a new validator (B) to the validator set
• Submit claim validator transaction to join the validator set with your node (B)

When a new network was created your node A validator account automatically became a “sudo” — that is an account, which can modify roles and permissions of other accounts. To add a new validator (node B) to the validator set of your new network (e.g. bobchain-1) we need to find out what the public address of your node B validator is, so that we can submit a governance proposal and include it. To get the public address inspect the validator container on node B and execute the following command:

echo $(sekaid keys show -a validator --keyring-backend=test)

Result of the above command execution should be a Bech32 public key of your new validator node B. You should also be able to find Val.ADD displayed in the first section of the kira container manager ( kira1qq…9w0 in our example)

Now that you know the public address of the node B validator, you can enter your node A and inspect the validator container hosted there. You will have to ensure that the account of your new validator (B) has funds otherwise it will not be registered on-chain. To check if new validator account has funds, type:

read -p "INPUT ADDRESS: " ADDR && sekaid query bank balances $ADDR

If the account does not have funds you can send to it 1 KEX from the faucet account:

read -p "INPUT ADDRESS: " ADDR && sekaid tx bank send faucet $ADDR 1000000ukex --keyring-backend=test --chain-id=$NETWORK_NAME --fees 100ukex --yes

Now that your node B account is fueled with funds, proceed by assigning permissions, which will enable your node A to submit a governance proposal, allowing node A to change permissions of node B to enable that node B to claim a validator seat.

sekaid tx customgov permission whitelist-permission --from validator --keyring-backend=test --permission=$PermCreateSetPermissionsProposal --addr=$(sekaid keys show -a validator --keyring-backend=test) --chain-id=$NETWORK_NAME --fees=100ukex --yes | jq
sekaid tx customgov permission whitelist-permission --from validator --keyring-backend=test --permission=$PermVoteSetPermissionProposal --addr=$(sekaid keys show -a validator --keyring-backend=test) --chain-id=$NETWORK_NAME --fees=100ukex --yes | jq

Now that your node A has appropriate permissions to create proposals and vote on it, you can submit a proposal to add “claim validator seat” permission to node B. Use the following command and input your new validator node B address when prompted:

read -p "INPUT ADDRESS OF YOUR NEW VALIDATOR: " ADDR && sekaid tx customgov proposal assign-permission $PermClaimValidator --addr=$ADDR --from=validator --description="" --keyring-backend=test --chain-id=$NETWORK_NAME --fees=100ukex --yes | jq

After governance proposal was created you will have 10 minutes (default proposal expiration time) to accept it by casting YES vote (1) :

LAST_PROPOSAL=$(sekaid query customgov proposals --output json | jq -cr '.proposals | last | .proposal_id') && sekaid tx customgov proposal vote $LAST_PROPOSAL 1 --from=validator --chain-id=$NETWORK_NAME --keyring-backend=test  --fees=100ukex --yes | jq

After your vote was casted, use the following command to preview/await enactment_end_time of the proposal and VOTE_RESULT_PASSED result.

LAST_PROPOSAL=$(sekaid query customgov proposals --output json | jq -cr '.proposals | last | .proposal_id') && sekaid query customgov votes $LAST_PROPOSAL --output json | jq && sekaid query customgov proposal $LAST_PROPOSAL --output json | jq && echo "Time now: $(date '+%Y-%m-%dT%H:%M:%S')"

Now that your proposal passed successfully you can switch machines and inspect your node B validator container and submit a “create validator” transaction, which will make your node a part of the consensus:

sekaid tx customstaking claim-validator-seat --from validator --keyring-backend=test --home=$SEKAID_HOME --moniker="NODE-B" --chain-id=$NETWORK_NAME --fees=100ukex --yes

You can now check if your new validator is an active part of the validator set:

sekaid query validator --addr=$VALIDATOR_ADDR --output=json | jq

If the status is ACTIVE you have successfully joined the validator set and completed the basic Milestone 2 setup, now it’s time to explore.

Milestone Challenges

We are presenting here a list of potential test cases which can be performed by the Selected Phase 0 Participants and their successful or unsuccessful execution can be reported via the submission form here. Completing any of the test case suggestions presented below is optional for the phase 0 participants. The scope of your report can widely differ from any of the mentioned test cases as long as it is valuable and insightful in the context of improving the software and future experience.

There are no limitations in terms of how many reports you can submit. All of your findings must be submitted via the report submission form before February 26, 2021. No other channels of communication will be taken into account. Helping other testnet participants in solving their issues is allowed, however will not result in higher scores.

Test Case Suggestions

• Install kira using the Validator Mode and deploy a new private network
• Install kira using the Validator Mode and join existing network which you created
• Join existing network created by another testnet participant
• Try to create individually, or with friends, the largest possible network (by validator count)
• Attempt to stop the validator node though tx spam, DoS or DDoS
• Test all available options in the Network, Networking and Container managers
• Attempt to cause total or partial infrastructure failure, while interacting with the machine remotely (from outside of the host)
• Test maximum duration of the infrastructure operation without failure
• Provide insight and suggestions in regards to user experience and how it can be improved
• Provide insight and suggestions in regards of improving security
• Create simple bash-shell script/s enabling to test tx spam and other malicious network conditions
• Create PR to kira repository which improves performance, user experience, console interface, security or other aspects of the infrastructure operation

Telegram Channel: https://tg.kira.network
Announcements: https://ann.kira.network
Twitter: https://twitter.kira.network
GitHub: https://github.kira.network
Blog: https://medium.kira.network