Tendermint: The Ultimate Guide for Faster Consensus

12 Angry Men (1957)

While the original Ethan Buchman’s thesis says Tendermint can handle ~ 10,000 transactions per second, you may not necessarily get the same results when measuring your network performance. That’s because the outcome depends on many parameters such as Tendermint’s configuration, whenever your ABCI application is built-in or not, hardware used, network speed and topology, and even a way clients send their transactions.

1) Tendermint’s configuration

Part 1. Single node

Let’s start with the single node network and built-in kvstore application

$ tendermint init
$ tendermint node --proxy_app=kvstore
$ tm-bench -T 30 -r 1000 -c 10 localhost:26657
Stats Avg StdDev Max Total
Txs/sec 2420 2457 5000 72600
Blocks/sec 0.533 0.499 1 16

By default, Tendermint is configured for public permissionless networks like Cosmos. If you’re building a private ledger, then you can significantly boost the performance by increasing some params and lowering others.

Mempool size is limited to 5000. Let’s change it to 50,000.

# ~/.tendermint/config/config.toml
[mempool]
...
# size of the mempool
size = 50000
Stats          Avg       StdDev     Max       Total
Txs/sec 7000 11696 43775 210000
Blocks/sec 0.300 0.458 1 9

As of 0.27, Tendermint logs infomessages produced by the state package. For other packages, it only logs errors. This, of course, can be changed. It might be a good idea to redirect the output to syslog and later analyse it using something like Logstash. Here, let’s try something simple like redirect the output to a file:

tendermint node --proxy_app=kvstore 2&>1 > /var/tm.log
Stats          Avg       StdDev     Max       Total
Txs/sec 7012 11926 42994 210361
Blocks/sec 0.300 0.458 1 9

A slight increase, but it could’ve been bigger if the log level was *:info , which means log info messages produced by all packages.

Currently, Tendermint is usinggoleveldb (LevelDB written in Golang) to store state, which is slow comparing to cleveldb (LevelDB written in C). What if we try to switch to the latter one? All we need is to install LevelDB, recompile Tendermint with gcc flag and change the db backend setting. Please visit this guide for complete steps.

# ~/.tendermint/config/config.toml
db_backend = "cleveldb"
Stats          Avg       StdDev     Max       Total
Txs/sec 9022 8647 20000 270665
Blocks/sec 0.533 0.499 1 16

Since for kvstore application there’s no need to recheck transactions after Tendermint has committed a block, we can safely disable it:

# ~/.tendermint/config/config.toml
[mempool]
recheck = false
$ tm-bench -T 30 -r 1000 -c 20 localhost:26657
Stats Avg StdDev Max Total
Txs/sec 10401 14633 43563 312042
Blocks/sec 0.367 0.482 1 11

Finally, suppose we’re building a private ledger and don’t really care how many votes proposer receives (their number just needs to be sufficient for making a block):

[consensus]
...
skip_timeout_commit = true
create_empty_blocks = false

Stats Avg StdDev Max Total
Txs/sec 11164 12418 31965 334929
Blocks/sec 0.567 0.496 1 17

Notice that we also configure Tendermint to not create empty blocks. Otherwise, it will be baking them like crazy 🍪

built-in kvstore application

These tests were performed on a Digital Ocean droplet: 

ubuntu 18.04 x64 8GB RAM, 4 vCPU, 160 GB SSD
go 1.11.2 linux/amd64
leveldb 1.20
tendermint 0.27.0-d9a1aad5

Part 2. Four node cluster

TODO

2) ABCI application

If you write your application in Golang, the same programming language Tendermint is written in, you can bundle them together and ship as a single binary. Performing local calls is obviously faster than communicating via a socket. Missing: Guide to building ABCI app with Tendermint

# the last results for a single node
Stats Avg StdDev Max Total
Txs/sec 11164 12418 31965 334929
Blocks/sec 0.567 0.496 1 17
# same setup, but kvstore ABCI application is communicating with Tendermint via a Unix domain socket
Stats Avg StdDev Max Total
Txs/sec 10437 12143 29935 313124
Blocks/sec 0.500 0.500 1 15
# same setup, but kvstore ABCI application is communicating with Tendermint via a TCP socket
Stats Avg StdDev Max Total
Txs/sec 10389 11948 30862 311684
Blocks/sec 0.500 0.500 1 15

Communicating via gRPC is even slower and should not be considered if you want to archive the highest possible performance.

Caught panic while benchmarking gRPC kvstore app:

E[4126-12-04|14:14:41.953] CONSENSUS FAILURE!!!                         module=consensus err="runtime error: invalid memory address or nil pointer dereference" stack="goroutine 68 [running]:\nruntime/debug.Stack(0xc0034283c0, 0xd363c0, 0x19284d0)\n\t/usr/local/go/src/runtime/debug/stack.go:24 +0xa7\ngithub.com/tendermint/tendermint/consensus.(*ConsensusState).receiveRoutine.func2(0xc000888380, 0xfab6d0)\n\t/root/go/src/github.com/tendermint/tendermint/consensus/state.go:576 +0x57\npanic(0xd363c0, 0x19284d0)\n\t/usr/local/go/src/runtime/panic.go:513 +0x1b9\ngithub.com/tendermint/tendermint/types.NewResultFromResponse(...)\n\t/root/go/src/github.com/tendermint/tendermint/types/results.go:44\ngithub.com/tendermint/tendermint/types.NewResults(0xc002da0000, 0x183a, 0x183a, 0xc003918dc0, 0xc001d6c2b8, 0xc0027dc2d0)\n\t/root/go/src/github.com/tendermint/tendermint/types/results.go:36 +0x6b\ngithub.com/tendermint/tendermint/state.(*ABCIResponses).ResultsHash(0xc004013710, 0xc0027dc5a0, 0xb602ec, 0xc0034285c8)\n\t/root/go/src/github.com/tendermint/tendermint/state/store.go:133 +0x40\ngithub.com/tendermint/tendermint/state.updateState(0x8, 0x1, 0xc00095286a, 0x6, 0xc000877640, 0x11, 0xd, 0x5a57, 0xc0026f3840, 0x20, ...)\n\t/root/go/src/github.com/tendermint/tendermint/state/execution.go:469 +0x3c6\ngithub.com/tendermint/tendermint/state.(*BlockExecutor).ApplyBlock(0xc000954380, 0x8, 0x1, 0xc00095286a, 0x6, 0xc000877640, 0x11, 0xd, 0x5a57, 0xc0026f3840, ...)\n\t/root/go/src/github.com/tendermint/tendermint/state/execution.go:125 +0x50b\ngithub.com/tendermint/tendermint/consensus.(*ConsensusState).finalizeCommit(0xc000888380, 0xe)\n\t/root/go/src/github.com/tendermint/tendermint/consensus/state.go:1290 +0xa8c\ngithub.com/tendermint/tendermint/consensus.(*ConsensusState).tryFinalizeCommit(0xc000888380, 0xe)\n\t/root/go/src/github.com/tendermint/tendermint/consensus/state.go:1221 +0x451\ngithub.com/tendermint/tendermint/consensus.(*ConsensusState).enterCommit.func1(0xc000888380, 0x0, 0xe)\n\t/root/go/src/github.com/tendermint/tendermint/consensus/state.go:1167 +0x90\ngithub.com/tendermint/tendermint/consensus.(*ConsensusState).enterCommit(0xc000888380, 0xe, 0x0)\n\t/root/go/src/github.com/tendermint/tendermint/consensus/state.go:1198 +0x6b8\ngithub.com/tendermint/tendermint/consensus.(*ConsensusState).addVote(0xc000888380, 0xc0022117c0, 0x0, 0x0, 0x488edd, 0xc0014e7bc0, 0xc000a64100)\n\t/root/go/src/github.com/tendermint/tendermint/consensus/state.go:1622 +0xc03\ngithub.com/tendermint/tendermint/consensus.(*ConsensusState).tryAddVote(0xc000888380, 0xc0022117c0, 0x0, 0x0, 0x42f272, 0xc000000008, 0xc003429b80)\n\t/root/go/src/github.com/tendermint/tendermint/consensus/state.go:1469 +0x59\ngithub.com/tendermint/tendermint/consensus.(*ConsensusState).handleMsg(0xc000888380, 0x10673a0, 0xc001d6c0d0, 0x0, 0x0)\n\t/root/go/src/github.com/tendermint/tendermint/consensus/state.go:650 +0x696\ngithub.com/tendermint/tendermint/consensus.(*ConsensusState).receiveRoutine(0xc000888380, 0x0)\n\t/root/go/src/github.com/tendermint/tendermint/consensus/state.go:611 +0x555\ncreated by github.com/tendermint/tendermint/consensus.(*ConsensusState).OnStart\n\t/root/go/src/github.com/tendermint/tendermint/consensus/state.go:300 +0x132\n"

3) Hardware

TODO

4) Network

TODO

5) Clients

TODO

Next steps: