Key Management, TMKMS

by

6 min readMay 16, 2022

TMKMS overview
Why using a Key Management System
TMKMS
KMS to avoid double signing
Handshake diagram
HW & SW Requeriments
Initial setup
Config file
GENERATE SIGNING KEY
Systemd integracion
Validator setup for remote signing
Appendix and FAQ
Further reads

TMKMS overview

TMKMS is the solution for managing validator keys in Tendermint-based networks. It was initially developed by the Cosmos team and it is currently maintained by Iqlusioninc.

TMKMS is compatible with two signing backends:

Soft-sign: the common software signing with a plain private key. The advantages between this and not using TMKMS are the double-signing protection and having the keys and the validator in separated servers. Softsign backend which uses ed25519-dalek.
Hardware security modules (HSM): provided by devices like Ledger Nano S and YubiHSM, that signs without exposing the private keys. These methods are more secure but you need to have physical access to the server.

TMKMS is Licensed under the Apache License, Version 2.0. You may obtain a copy of the License at

https://www.apache.org/licenses/LICENSE-2.0

Why using a Key Management System

Key Management System for Tendermint applications such as Evmos Validators.

Provides isolated, optionally HSM-backed signing key management for Tendermint applications including validators, oracles, IBC relayers, and other transaction signing applications.

This key management service is intended to be deployed in conjunction with Tendermint applications, like Evmos, (ideally on separate physical hosts) which provides the following:

High-availability access to validator signing keys
Double-signing prevention even in the event the validator process is compromised
Hardware security module storage for validator keys which can survive host compromise

We will only provide advice for software-based signing. Hardware security modules (Ledgers / Yubikeys) are outside of the scope of this document.

TMKMS, how it works

KMS to avoid double signing

Double signatures lead to conflicting blocks and votes, which may lead to fork of chain or double-spend. Key Management Service, or KMS, is a service which attempts to prevent validator from double-signing by making the KMS aware of the current block height.

Hardware Security Module, or HSM is an implementation of KMS, which can store a private key in some hardware and ensure in a secure setting that it won’t sign the same block twice.

The signing process is deterministic state machine whose responsibility is signing protocol message and ensuring that the validator does not double sign

Thanks to Sophie Huang for the knowledge @kidinamoto

Handshake diagram

Signer — Validator

HW & SW Requeriments

1 vCPU
2 GB of ram
20 GB of free space (for signing logs)
Tested in Ubuntu Server 22.04 / Rocky Linux 8 x64

Should also work in GNU/Linux aarch64/riscv64 and Freebsd/Openbsd.

This signing system should be as secure as possible: only accesible via vpn / fixed administrator address. No services should be available from the public Internet.

If this is your first time approaching this kind of proffesional signing service, we do encourage you to do it at the testnet. For this purpose we have compiled the following steps on a secure testnet scenario.

Initial setup

The following steps will configure a separate machine which role will be the “Remote Signer” or TMKMS server.

Install Ubuntu Server 22.04 minimal. Use LVM+XFS for better i/o performance and future storage expansion.

Once installed the operating system, upgrade all packages

sudo apt update && sudo apt full-upgrade -y

Restart system ‘just in case’

System should only be restarted if packages like kernel/glibc/openssl/zlib are updated. The old-grey-bearded-unix uptime lover

After restart, install Rust and deps:

sudo apt install rustc curl build-essential gcc make

We will use Cargo to install tmkms software. Cargo is the built-in package manager of Rust lang.

Do not run services as root if not needed. We will use Cargo as a non-privileged user:

cargo install tmkms --features=softsign

This should take some time. Enjoy your prefered beverage.

Once installed, edit $HOME/.profile to the execution search path:

PATH=$HOME/.cargo/bin:$PATH

Logout and login again for the changes to take effect.

Most people use bash nowadays. For zsh/csh/tcsh user, adjust to your enviroment The old-grey-bearded-unix alt shells lover

Now, let’s create the directory structure & initialize for evmos-testnet (not mainnet!!)

mdkir -p $HOME/tmkms/evmos-testnet
tmkms init $HOME/tmkms/evmos-tesnet

Config file

File name: $HOME/tmkms/evmos-tesnet/tmkms.toml

# Tendermint KMS configuration file
## Chain Configuration
### Evmos Testnet[[chain]]
id = "evmos_9000-4"
key_format = { type = "bech32", account_key_prefix = "evmospub", consensus_key_prefix = "evmosvalconspub" }
state_file = "/home/ubuntu/tmkms/evmos-testnet/state/evmos-testnet-state.json"## Signing Provider Configuration### Software-based Signer Configuration[[providers.softsign]]
chain_ids = ["evmos_9000-4"]
key_type = "consensus"
path = "/home/ubuntu/tmkms/evmos-testnet/secrets/evmos-testnet-consensus.key"## Validator Configuration[[validator]]
chain_id = "evmos_9000-4"
#addr = "tcp://123.123.123.123:26658
# VPN
addr = "tcp://10.10.10.2:26658"
secret_key = "/home/ubuntu/tmkms/evmos-testnet/secrets/kms-identity.key"
protocol_version = "v0.34"
reconnect = true

In this example we are using Wireguard to secure network traffic between the signer<->validator. It is not mandatory, but recommended. Configuring an VPN is out of the scope of this manual. You could start reading at Wireguard official site: https://www.wireguard.com/

GENERATE SIGNING KEY

Copy priv_validator_key.json to the signer system and run this command:

tmkms softsign import $HOME/priv_validator_key.json ~/tmkms/evmos-tesnet/secrets/evmos-testnet-consensus.key

After this you could safely remove priv_validator_key.json from signer

If not using wireguard, allow incoming traffic to the 26658/tcp port of the validator from the ip address of the signer.

Tip: use netcat to check that traffic from signer can reach port 26658/tcp of the validator The old-grey-bearded-unix firewall lover

Systemd integracion

Let’s create (as root) a systemd unit to run the signer for testnet, file /etc/systemd/system/tmkms-evmos-testnet.service

[Unit]
Description=tmkms evmos-testnet service
After=network.target
StartLimitIntervalSec=0[Service]
Type=simple
Restart=always
RestartSec=10
User=ubuntu
ExecStart=/home/ubuntu/.cargo/bin/tmkms start -c /home/ubuntu/tmkms/evmos-testnet/tmkms.toml
LimitNOFILE=1024[Install]
WantedBy=multi-user.target

Now, let’s reload systemd and start the service tmkms-evmos-testnet

sudo systemctl daemon-reload
sudo systemctl start tmkms-evmos-testnet
sudo systemctl enable mkms-evmos-testnet
sudo systemctl status tmkms-evmos-testnet

Double check that there are not errors.

Validator setup for remote signing

If not using wireguard, make sure that port 26658/tcp of the validator is only available from the signer address (iptables/nftables/isp firewall)

Stop the validator and edit config.toml

priv_validator_laddr = tcp://ip_address_of_validator:26658

If your validator has a public ip address, use that ip. If it has a private address and nat has been setup elsewhere, put the private address. The old-grey-bearded-unix firewall lover

Launch again the validator and review logs to check that everything is working as intended and remote signing is being used.

You can safely remove priv_validator_key.json from validator.

If you reached this point, congratulations, you have a remote signing setup working!

Appendix and FAQ

1. What is the right order to run this setup?

Always start first the signer. Check signer logs before starting the validator

2. I am sold! I want to try this in mainnet

Test firtly and always in testnet. When you are confortable, create a new directory structure/config file/system unit for mainnet.

Remember to change the chain_id to match mainnet.

3. Why 0.34 in protocol_version in config file of signer?

Because reasons!

4. I am not using wireguard. When I use tcpdump in the validator, I can see json traffic coming from the signer in clear-text!

Use wireguard/openvpn/stunnel/ssh tunnel for additional security.

Find this document incomplete? Leave a comment!

Further reads

https://github.com/iqlusioninc/tmkms https://docs.tendermint.com/master/nodes/remote-signer.html