Key Management, TMKMS
Table of Contents
TMKMS overview
Why using a Key Management System
TMKMS
KMS to avoid double signing
Handshake diagram
HW & SW Requeriments
Initial setup
Config file
GENERATE SIGNING KEY
Systemd integracion
Validator setup for remote signing
Appendix and FAQ
Further reads
TMKMS overview
TMKMS is the solution for managing validator keys in Tendermint-based networks. It was initially developed by the Cosmos team and it is currently maintained by Iqlusioninc.
TMKMS is compatible with two signing backends:
- Soft-sign: the common software signing with a plain private key. The advantages between this and not using TMKMS are the double-signing protection and having the keys and the validator in separated servers. Softsign backend which uses ed25519-dalek.
- Hardware security modules (HSM): provided by devices like Ledger Nano S and YubiHSM, that signs without exposing the private keys. These methods are more secure but you need to have physical access to the server.
TMKMS is Licensed under the Apache License, Version 2.0. You may obtain a copy of the License at
https://www.apache.org/licenses/LICENSE-2.0
Why using a Key Management System
Key Management System for Tendermint applications such as Evmos Validators.
Provides isolated, optionally HSM-backed signing key management for Tendermint applications including validators, oracles, IBC relayers, and other transaction signing applications.
This key management service is intended to be deployed in conjunction with Tendermint applications, like Evmos, (ideally on separate physical hosts) which provides the following:
- High-availability access to validator signing keys
- Double-signing prevention even in the event the validator process is compromised
- Hardware security module storage for validator keys which can survive host compromise
We will only provide advice for software-based signing. Hardware security modules (Ledgers / Yubikeys) are outside of the scope of this document.
TMKMS, how it works
KMS to avoid double signing
Double signatures lead to conflicting blocks and votes, which may lead to fork of chain or double-spend. Key Management Service, or KMS, is a service which attempts to prevent validator from double-signing by making the KMS aware of the current block height.
Hardware Security Module, or HSM is an implementation of KMS, which can store a private key in some hardware and ensure in a secure setting that it won’t sign the same block twice.
The signing process is deterministic state machine whose responsibility is signing protocol message and ensuring that the validator does not double sign
Thanks to Sophie Huang for the knowledge @kidinamoto
Handshake diagram
Signer — Validator
HW & SW Requeriments
- 1 vCPU
- 2 GB of ram
- 20 GB of free space (for signing logs)
- Tested in Ubuntu Server 22.04 / Rocky Linux 8 x64
Should also work in GNU/Linux aarch64/riscv64 and Freebsd/Openbsd.
This signing system should be as secure as possible: only accesible via vpn / fixed administrator address. No services should be available from the public Internet.
If this is your first time approaching this kind of proffesional signing service, we do encourage you to do it at the testnet. For this purpose we have compiled the following steps on a secure testnet scenario.
Initial setup
The following steps will configure a separate machine which role will be the “Remote Signer” or TMKMS server.
Install Ubuntu Server 22.04 minimal. Use LVM+XFS for better i/o performance and future storage expansion.
Once installed the operating system, upgrade all packages
sudo apt update && sudo apt full-upgrade -y
Restart system ‘just in case’
System should only be restarted if packages like kernel/glibc/openssl/zlib are updated. The old-grey-bearded-unix uptime lover
After restart, install Rust and deps:
sudo apt install rustc curl build-essential gcc make
We will use Cargo to install tmkms software. Cargo is the built-in package manager of Rust lang.
Do not run services as root if not needed. We will use Cargo as a non-privileged user:
cargo install tmkms --features=softsign
This should take some time. Enjoy your prefered beverage.
Once installed, edit $HOME/.profile to the execution search path:
PATH=$HOME/.cargo/bin:$PATH
Logout and login again for the changes to take effect.
Most people use bash nowadays. For zsh/csh/tcsh user, adjust to your enviroment The old-grey-bearded-unix alt shells lover
Now, let’s create the directory structure & initialize for evmos-testnet (not mainnet!!)
mdkir -p $HOME/tmkms/evmos-testnet
tmkms init $HOME/tmkms/evmos-tesnet
Config file
File name: $HOME/tmkms/evmos-tesnet/tmkms.toml
# Tendermint KMS configuration file
## Chain Configuration
### Evmos Testnet[[chain]]
id = "evmos_9000-4"
key_format = { type = "bech32", account_key_prefix = "evmospub", consensus_key_prefix = "evmosvalconspub" }
state_file = "/home/ubuntu/tmkms/evmos-testnet/state/evmos-testnet-state.json"## Signing Provider Configuration### Software-based Signer Configuration[[providers.softsign]]
chain_ids = ["evmos_9000-4"]
key_type = "consensus"
path = "/home/ubuntu/tmkms/evmos-testnet/secrets/evmos-testnet-consensus.key"## Validator Configuration[[validator]]
chain_id = "evmos_9000-4"
#addr = "tcp://123.123.123.123:26658
# VPN
addr = "tcp://10.10.10.2:26658"
secret_key = "/home/ubuntu/tmkms/evmos-testnet/secrets/kms-identity.key"
protocol_version = "v0.34"
reconnect = true
In this example we are using Wireguard to secure network traffic between the signer<->validator. It is not mandatory, but recommended. Configuring an VPN is out of the scope of this manual. You could start reading at Wireguard official site: https://www.wireguard.com/
GENERATE SIGNING KEY
Copy priv_validator_key.json to the signer system and run this command:
tmkms softsign import $HOME/priv_validator_key.json ~/tmkms/evmos-tesnet/secrets/evmos-testnet-consensus.key
After this you could safely remove priv_validator_key.json from signer
If not using wireguard, allow incoming traffic to the 26658/tcp port of the validator from the ip address of the signer.
Tip: use netcat to check that traffic from signer can reach port 26658/tcp of the validator The old-grey-bearded-unix firewall lover
Systemd integracion
Let’s create (as root) a systemd unit to run the signer for testnet, file /etc/systemd/system/tmkms-evmos-testnet.service
[Unit]
Description=tmkms evmos-testnet service
After=network.target
StartLimitIntervalSec=0[Service]
Type=simple
Restart=always
RestartSec=10
User=ubuntu
ExecStart=/home/ubuntu/.cargo/bin/tmkms start -c /home/ubuntu/tmkms/evmos-testnet/tmkms.toml
LimitNOFILE=1024[Install]
WantedBy=multi-user.target
Now, let’s reload systemd and start the service tmkms-evmos-testnet
sudo systemctl daemon-reload
sudo systemctl start tmkms-evmos-testnet
sudo systemctl enable mkms-evmos-testnet
sudo systemctl status tmkms-evmos-testnet
Double check that there are not errors.
Validator setup for remote signing
If not using wireguard, make sure that port 26658/tcp of the validator is only available from the signer address (iptables/nftables/isp firewall)
Stop the validator and edit config.toml
priv_validator_laddr = tcp://ip_address_of_validator:26658
If your validator has a public ip address, use that ip. If it has a private address and nat has been setup elsewhere, put the private address. The old-grey-bearded-unix firewall lover
Launch again the validator and review logs to check that everything is working as intended and remote signing is being used.
You can safely remove priv_validator_key.json from validator.
If you reached this point, congratulations, you have a remote signing setup working!
Appendix and FAQ
1. What is the right order to run this setup?
Always start first the signer. Check signer logs before starting the validator
2. I am sold! I want to try this in mainnet
Test firtly and always in testnet. When you are confortable, create a new directory structure/config file/system unit for mainnet.
Remember to change the chain_id to match mainnet.
3. Why 0.34 in protocol_version in config file of signer?
Because reasons!
4. I am not using wireguard. When I use tcpdump in the validator, I can see json traffic coming from the signer in clear-text!
Use wireguard/openvpn/stunnel/ssh tunnel for additional security.
Find this document incomplete? Leave a comment!
Further reads
https://github.com/iqlusioninc/tmkms https://docs.tendermint.com/master/nodes/remote-signer.html