Idena Sibling: Deploying and managing Idena Shared and Regular node

Using Ansible Playbooks and Roles collection

Italiano (🇮🇹) | Magyar (🇭🇺)

The main goal of this guide is to assist newbies and potential shared node operators in deploying an Idena Node or Shared Node quickly and securely. With the help of these playbooks and roles, you can configure all parameters of your node or shared node, and easily update your shared node API keys after the deployment process.

If you choose to deploy a shared node, you can opt to use your own SSL certificate or a certificate from Let’s Encrypt. If you choose Let’s Encrypt, the certificate will be automatically updated via a special crontab task. As a result, you will have a production-ready shared node with almost no need for any manual tasks. This can be particularly useful if you are deploying your shared node server on a more powerful droplet before the validation session.

Additionally, a special playbook will be provided to you, which will enable you to quickly perform actions within your already-deployed Idena Node. These actions include switching your mining status on/off, adding/removing API keys, starting/stopping your Idena Node daemon, and changing your node key (private key).

Table of contents

• The Shared Node Deployment Scheme
• Getting Your System Ready for Ansible Playbooks
  ✧ Installing Ubuntu on WSL2 for Windows Users
  ✦ Independent WSL2 Linux Distribution in Parallel with the Main One (not necessarily)
  ✧ Some important things about WSL to keep in mind
  ✦ Prepare VS Code to set up Idena Sibling and deploy your node
• Requirements for the local computer and for the remote droplets
  ✧ Destination droplets
  ✦ Master node
• Preparing your system and installing Ansible on the master node
• Let’s talk about the generation and management of Shared Node API keys
• Two Types of Authorization for the Remote Droplets That Ansible Supports
  ✧ Authorization via SSH Key
  ✦ Authorization via SSH Password
• Your own SSL certificate versus a Let’s Encrypt one
• Preparing Your Cloned Repository for Deploying and Managing Your Node
• Some optional preparations that could be useful and some potential troubles that you could encounter
• Using Idena Sibling Playbooks to Deploy and Manage Your Node
• Bottom line

The Shared Node Deployment Scheme

There are various shared node deployment schemes, but for the purpose of this article, we will be using the following scheme:

( idena-go ↔ idena-node-proxy ) ⇄ HAProxy (via SSL certificate layer)

Getting Your System Ready for Ansible Playbooks

Installing Ubuntu on WSL2 for Windows Users

In my example, I would use WSL2 on Windows 10 with the installed Ubuntu 20.04 image on it, but all of the provided instructions could be used in any Debian-based Linux distribution. If you don’t know what WSL2 is or how to set it up, this article will help you familiarize yourself with it.

Install everything you need to run WSL with a single command in PowerShell or Windows Command Prompt with administrator privileges, then restart your machine.

wsl --install

This command will enable the features necessary to run WSL and install the Ubuntu distribution of Linux.

To run Ubuntu, simply locate it in the list of your installed programs.

Icon of WSL2 Ubuntu Linux Distribution After Installation

How to Install an Independent WSL2 Linux Distribution in Parallel with the Main One (not necessarily)

If you’re running Windows 10 Build 18305 or later and want to use a different Linux distribution than the one installed by default, simply run commands like this within the folder where you want to deploy your Ubuntu distribution. Just make sure that you have enough free space:

1. Download the root tarball containing the Ubuntu 20.04 server distribution for WSL. The complete list of available distributions can be found here. In my case, I used the latest AMD64 Ubuntu 20.04 distribution.
2. Within the folder where you downloaded the tarball, enter the command:

wsl.exe --import <Distribution_Name> <Install Folder> ubuntu-20.04-server-cloudimg-amd64-wsl.rootfs.tar.gz

Please note that the folder Install Folder name and Distribution Name must be different.

3. You will see that a directory with the same name as the one you used for the Install Folder will appear in the folder where you ran the above-mentioned command. Now, to run your distribution, simply enter the command:

wsl.exe -d Distribution_Name

4. In addition, there are several other wsl commands that can help you manage your distributions:

To see the list of all distributions that have been installed in your system, you can use the command:

wsl --list

To remove a previously installed distribution, you can use the command:

wsl --unregister <Distribution_Name>

After unregistering a specific distribution, you can easily remove its folder.

Some important things about WSL to keep in mind

1. All your local disks after running your distribution would be mounted inside the /mnt directory.
2. Please do not use the regular Windows file system folder to clone repository. It must be cloned into a folder inside your WSL system, preferably in your home directory. The default command to enter your current user’s home folder is cd ~. If you try to clone and run the Ansible playbook within a regular Windows folder (any location in /mnt/), you will encounter an error:
   [WARNING]: Ansible is being run in a world writable directory (/mnt/...), ignoring it as an ansible.cfg source.
3. After running your distribution, you will enter the Linux subsystem, which is different from the Windows command-line interface. One of the main differences is that in the Linux file system, the prefix before your cursor will look like this:

There is a difference between the console output in the WSL (left picture) and the standard Windows command prompt (right picture).

3. To exit from the WSL subsystem, simply type the exit command.

4. All future commands related to using our Ansible repository that we intend to run should be executed under the WSL subsystem.

Prepare VS Code to set up Idena Sibling and deploy your node

We will need to work with the terminal and edit some configuration files, including the encrypted ones. If you’re familiar with Linux, it won’t be a problem, but if you’re a Windows user, I would suggest installing VS Code with a special WSL extension.

Developing in the Windows Subsystem for Linux with Visual Studio Code

Requirements for the local computer and for the remote droplets

Destination droplets

• Python version 3.9 or higher is required. Some cloud providers offer droplets with Python already installed, while others do not. If the destination droplet does not have Python installed, Ansible cannot perform any actions on it. Please be aware of this.
• The script has been tested on Ubuntu 20.04 LTS and may work on other Debian-based distributions, but they have not been explicitly tested.

Master node

• Python version 3.9 or higher and pip3 must be installed.
• The latest version of Ansible must be installed on the machine.
• If you are using a Windows machine, you can run this playbook by using WSL2 with an Ubuntu distribution.

Preparing your system and installing Ansible on the master node

Update your packages to the latest version

sudo apt update && sudo apt upgrade

How to install Python 3 in case it is not installed

Ubuntu 20.04 LTS already has the preinstalled python3 package. You can simply check it by typing python3 --version. In case it is not installed, you will need to install it using the following command:

sudo apt install python3

Adding ansible repository, installing Ansible using Pip and the Required Packages

sudo apt install python3-pip
sudo pip3 install pywinrm
sudo pip3 install pyvmomi
sudo apt-add-repository ppa:ansible/ansible
sudo pip3 install ansible
sudo pip3 install ansible[azure]
python3 -m pip install --upgrade --user ansible

Cloning Idena Sibling Repository

cd ~
git clone https://github.com/ltraveler/idena-sibling.git

Getting inside the idena-sibling repository folder

cd idena-sibling

Install required collections and python libraries on your local machine

Before starting to use these playbooks, please install the required Python packages and Ansible-galaxy modules.

• Installing dnspython library pip3 install dnspython
• Installing required ansible community modules ansible-galaxy install -r requirements.yml

Let’s talk about the generation and management of Shared Node API keys.

API keys generation

To import node keys to your shared node, you need to generate them first. The best method I have found for myself is using the service Online Hash Tools.

Generate Random MD5 Hashes - Online Hash Tools

Creating an Array from the generated API keys

The next step is to put the keys into an array format, as shown in the provided examples inside the api_keys.yaml file of the repository. The easiest way to do this is by using a special Google Sheets table template. The sheet is publicly available here:

IDENA Shared Node Array Preparation Tool by LTraveler

Two Types of Authorization for the Remote Droplets That Ansible Supports

Authorization via SSH Key (Recommended)

• If you don’t have a SSH key pair you could generate it using the ssh-keygen -t rsa command it will save your SSH private key as ~/.ssh/id_rsa and your SSH public key as ~/.ssh/id_rsa.pub.
• After that, you would need to set the correct values in the private variables: ansible_ssh_private_key_file = ~/.ssh/id_rsa and public_key_file = ~/.ssh/id_rsa.pub in the ./group_vars/main/vault file. For more information on editing and creating this file, you can find it here.
• Your public SSH key must be added to the authorized_keys file on the remote droplet. You can use the following command: ssh-copy-id -i ~/.ssh/my_public_key droplet_sudouser@droplet_ipaddress.

Authorization via SSH Password (Less Secure and Not Recommended)

Alternatively, you may use password authentication as a less secure option.

• To change the authentication method, please modify the value of the sshd_PasswordAuthentication variable to yes inside the ./group_vars/main/vars file and uncomment the ansible_connection variable inside the host file located in the root of the repository folder.
• If you don’t want to type your SSH password every time you run your Ansible playbook, you could add it to the ansible_ssh_pass: ”your_droplet_ssh_password” variable in the ./group_vars/main/vault file.
• Also, you would need to remove or make the variables ansible_ssh_private_key_file and public_key_file empty by setting their values as '' inside the host file located in the root of the repository folder.

Your own SSL certificate versus a Let’s Encrypt one

There are 2 options to issue an SSL certificate. You can buy one using one of the SSL certificate providers or you can issue it for free from Let’s Encrypt provider. The main difference between them is that the Let’s Encrypt certificate would be valid only for 3 months and must be prolonged. However, Idena Sibling will prepare a special script to run the task and make that process completely automatic. In case you don’t need any special level of validation, and you are okay to automatically prolong your certificate every 3 months, the Let’s Encrypt certificate would be more than enough for the purposes of the shared node. More information about the main differences between commercial and ACME issued certificates can be read here. Note that Let’s Encrypt now supports wildcard certificates, but in our case, the certificate that this playbook would issue would be the regular one related to the specified in droplet_domain: “shared.node.domain.com” variable in the ./group_vars/main/vault file.

In Case of Using the Let’s Encrypt SSL-Certificate Provider (the variable letsencrypt: “yes”)

Make sure that you have correctly set the A records for your shared node domain. If the IP address of your domain is linked to the wrong destination, you will not be able to pass the challenge and obtain the SSL certificate.

The Let’s Encrypt service has a limit on how many times you can request a certificate. You can find all the possible limits described here.

To prevent the situation where you have passed the available threshold and do not have any certificate, you will get a copy of it in the folder ./node/remote_ssl_clone of your local repository after successfully issuing an SSL certificate.

In case of using your own custom SSL certificate ( the variable letsencrypt: “no” )

Your own SSL certificate would be imported from ./node/domain_pem file.

To generate a Privacy Enhanced Mail (PEM) file, you typically need to concatenate the two files using the following command inside the folder that contains the crt and key files:
cat server.crt server.key > domain_pem

After creating your final PEM file called domain_pem, you would need to copy it (overwrite the existed one) to the ./idena-sibling/node/ folder of the repository and encrypt it using the command ansible-vault encrypt domain_pem.
If you want to change the contents of your certificate vault storage in the future, you can use the command ansible-vault edit domain_pem.

Preparing Your Cloned Repository for Deploying and Managing Your Node

Before starting to use the playbooks, you will have to change some variables inside those files:

./hosts

contains variables related to your destination droplet that you will use to deploy your node.

[main]
1.2.3.4

[all:vars]
ansible_ssh_private_key_file = ~/.ssh/id_rsa
ansible_user = root
idena_group = Olga
username = Olga
public_key_file = ~/.ssh/id_rsa.pub
#ansible_connection=ssh
#ansible_ssh_port=10222

1.2.3.4 - IP address of your droplet
ansible_ssh_private_key_file = ~/.ssh/id_rsa - the path to your SSH private key is needed in case you are willing to connect to your droplet via an SSH private key. If you prefer to use an SSH password, you can remove this variable or set it as ansible_ssh_private_key_file = ''
ansible_user = root - normally, it should stay as it is, but in case your root user has a different non-standard username, you have to change this variable
idena_group = Olga and username = Olga - these two variables represent the username and user group to be used for installing Idena-go daemon and Idena-node-proxy instance. Normally, the user group is equal to the username
public_key_file = ~/.ssh/id_rsa.pub - path to your SSH public key, in case you are deploying your droplet via SSH password and want to import your public key to your droplet’s authorized_keys file for future connections via SSH private key. In other words, if this variable is set and you are connecting to your droplet via SSH password, the script will perform the same action as the ssh-copy-id command.

If you prefer to use root password authentication instead of SSH key authentication, you will need to set your root password in the vault data storage under ansible_ssh_pass variable as described earlier and uncomment ansible_connection=ssh variable in the hosts file.

#ansible_ssh_port=10222- if you are using a non-standard SSH port, you would need to uncomment and set it here.

./group_vars/main/vars

contains public variables related to the configuration of your idena-go, idena-node-proxy, domain, required package versions, and so on. I will mention only important variables here to make the script do its job. If you don’t know the meaning of one or another variable, keep it at its default value. Just change the most critical variables, which are:

---
idena_go_ver: "1.0.3"
sshd_PasswordAuthentication: "no"
letsencrypt: "yes"
idenachain_bootstrap: "yes"
api_keys_path: "./group_vars/main/api_keys.yaml"
blockpinthreshold: 0.3
flippinthreshold: 0.5
shared: true

idena_go_ver: “1.0.3” - the version of the idena-go that you would like to use in your droplet. To install the latest version, please set it to latest.
sshd_PasswordAuthentication: “no” - in case you want to use an SSH private key, you can also restrict the connection to your droplet only through the SSH private key
letsencrypt: “yes” - in case of a shared node installation, you have to choose between using Let’s Encrypt SSL certificate (yes) or your own custom SSL certificate (no)
idenachain_bootstrap: “yes” - if you prefer not to download the Idena blockchain bootstrap archive idenachain.db, you will need to set the idenachain_bootstrap variable to no
api_keys_path: “./group_vars/main/api_keys.yaml”` - path to the file that contains API keys to be imported into the Idena-node-proxy of your droplet
blockpinthreshold: 0.3 - by default, it is set to 0.3. However, if you are deploying an Idena shared node, this parameter should be set to 1
flippinthreshold: 0.5 - by default, it is set to 0.5. However, if you are deploying an Idena shared node, this parameter should be set to 1
shared: true - specify whether the idena-go client should be run with or without the --profile=shared flag. Set this value to true if you are deploying a shared node, and false if you are deploying a regular node.

./group_vars/main/vault

contains private variables such as:

---
vault_api_key: e288dab99a8357371f527b41fe0439ba
vault_node_key: 6XCh2EqlfhHjOmb82Z386XhyJ8abeeKORBTi9r2joVhz9594y2138H9xqv8PS2cXts1bYTDaBlu
userpass: "eo4fejVM5jpA9Z"
letsencrypt_email: "email@for_lets_encrypt.com"
droplet_domain: "shared.node.domain.com"
#ansible_ssh_pass: "your_droplet_ssh_password"
#ansible_sudo_pass: "your_droplet_sudo_pass"

vault_api_key: e288dab99a8357371f527b41fe0439ba - desired API key for your node
vault_node_key: 6XCh2EqlfhHjOmb82Z386X - if the vault_node_key (private key) variable is not set, it will be automatically generated for your node
userpass: “eo4fejVM5jpA9Z” - the password for the user running your idena-go daemon and idena-node-proxy instance
letsencrypt_email: “email@for_lets_encrypt.com” - email that would be used in case if you are willing to generate your certificate via Let’s Encrypt
droplet_domain: “shared.node.domain.com” - your shared node domain address. In case you are willing to use Let’s Encrypt SSL certificate provider, the A DNS record of your domain must be set to this IP address before you run the playbook. Please keep in mind that updating DNS records could take some time.
ansible_ssh_pass: ”your_droplet_ssh_password” - in case you want to use SSH password authentication, you can set your droplet SSH password here if you don’t want Ansible to prompt you for it every time you run your playbook.

ansible_sudo_pass: “your_droplet_sudo_pass” - In most cases, the password for ansible_sudo_pass is the same as the one used for the ansible_ssh_pass variable. The password would only be different if the sudo username used for SSH connection is different from the one used for sudo privileges during playbook execution. For this playbook, the password should be the same as the one set for the ansible_ssh_pass variable.

Please keep in mind that you will need to overwrite the existing encrypted vault file (./group_vars/main/vault) with the unencrypted copy from ./node/ex/vault. After editing all required variables, you would need to encrypt it using the ansible-vault encrypt ./group_vars/main/vault command from the root folder of the repository. In the future, to edit this file you could use the command ansible-vault edit ./group_vars/main/vault entered from the root folder of the repository.

To avoid having to re-enter your vault storage password every time you want to edit it, you could save it in the .vault_pass plain text file in the root folder of the repository.

You have to use the same password for all files that you want to encrypt. In the case of our repository, we have 2 encrypted files:

1. ./group_vars/main/vault- storage for private variables.
2. ./node/domain_pem - Privacy Enhanced Mail (PEM) file that contains your own SSL certificate.

./group_vars/main/api_keys.yaml

contains API keys that are supposed to be imported during the shared node deployment process via the idena_shared.yaml playbook.

---
api_keys: ["key1", "key2", "key3"]

./group_vars/main/api_mgmt.yaml

contains the API keys that need to be added or removed from the already deployed shared node by using the idena_node_mgmt.yaml playbook with the tag IdenaApiKeysUpdate. After adding and/or removing keys to your shared node, all key duplicates are automatically removed, and the final output overwrites the original one (./group_vars/main/api_keys.yaml) in your local repository folder.

---
api_keys_add: ["key_to_add1", "key_to_add2", "key_to_add3"]
api_keys_remove: ["key_to_remove1", "key_to_remove2", "key_to_remove3"]

./.vault_pass

you could create and add your vault password to this file if you don’t want to enter it every time you want to edit your encrypted vault variable storage that you have created earlier.

We use two encrypted vault storages: one for the private variables file located at ./group_vars/main/vault, and the other for the Privacy Enhanced Mail (PEM) file at ./node/domain_pem. Ensure that you use the exact same password to encrypt both of them.

Some optional preparations that could be useful and some potential troubles that you could encounter

Preventing Prompt for SSH Passphrase on Remote Host

If you are using a password-protected SSH private key, the system will prompt you to enter your private key’s SSH password each time you want to run your playbook. To avoid this, you can use ssh-agent and add your private key to it using the command ssh-add, as described below:

Start ssh agent:

eval `ssh-agent -s`

To add your private key, use the command ssh-add with the path to your SSH private key (e.g., ~/.ssh/id_rsa_key):

ssh-add ~/.ssh/id_rsa_key

Ansible Remote Host Identification Has Changed

The Host Identification Has Changed error typically occurs when you try to connect to a remote server using SSH (Secure Shell) and the server’s identification key has changed since the last time you connected to it.

Typically, this happens when you are rebuilding your droplet from scratch on the same IP address.

The ‘REMOTE HOST IDENTIFICATION HAS CHANGED’ error in the Ansible terminal layout

To fix this problem, you simply need to remove the line associated with your remote droplet server from the ~/.ssh/known_hosts file. If you don't have any other connections apart from your remote droplet, it should be the last line in the file. You can use the nano editor by typing the command sudo nano ~/.ssh/known_hosts.

Using Idena Sibling Playbooks to Deploy and Manage Your Node

After configuring all the required variables, you can use three playbooks depending on what kind of node, shared or regular, you would like to deploy. Additionally, there is a special playbook to manage an already deployed node.

Deploying Idena regular node via idena_node.yaml

ansible-playbook -i hosts idena_node.yaml

Deploying Idena shared node via idena_shared.yaml

ansible-playbook -i hosts idena_shared.yaml

Managing already deployed Idena regular or shared node via idena_node_mgmt.yaml

ansible-playbook -i hosts idena_node_mgmt.yaml --tags operation_tag_name

Possible values of operation_tag_name to use with idena_node_mgmt.yaml playbook:

• IdenaNodekeyUpdate: This task takes a new nodekey (private key) value from the vault_node_key variable and overwrites the existing nodekey file on your droplet.
• IdenaNodeStart: Starts the Idena Node.
• IdenaNodeStop: Stops the Idena Node.
• IdenaMiningOn: Changes mining status to ON.
• IdenaMiningOff: Changes mining status to OFF.
• IdenaApiKeysUpdate: Updates the Shared Node API keys after changing the variables inside the ./group_vars/main/api_mgmt.yaml file responsible for adding and removing API keys on your droplet.
  Please be aware that the final list of keys, after making all manipulations, will also overwrite your local ./group_vars/main/api_keys.yaml file. As a result, you will always have an updated copy of your API keys in the local api_keys.yaml file.
• IdenaGoUpdate: Updates your idena-go node client to the version specified in the idena_go_ver variable. If idena_go_ver is set to latest, it will upgrade the idena-go node client to the latest version.
• IdenaProxyUpdate: Updates your idena-node-proxy to the latest version.
• IdenaChainRefresh: Initiates a complete resync of the Idena blockchain.
• IPFSRefresh: Initiates a complete resync of the IPFS.

Bottom line

Once the configuration is complete, you can easily deploy and manage your Idena node/shared node. If you have any questions, feel free to contact me via Telegram or simply open an issue in the Idena Sibling GitHub repository. This project was funded through the community wallet; the entire proposal can be found here.

💻 LTraveler:
💬 Telegram: https://t.me/ltrvlr
🌐 WWW: https://ltraveler.github.io
👛 0xf041640788910fc89a211cd5bcbf518f4f14d831