Using etcd raftexample for Go

Raft Consensus Algorithm is most widely used algorithm after Paxos for designing fault-tolerant & replicated state machines. There are 2 good Raft implementations in Go available online HashiCorp Raft & etcd Raft. etcd Raft is used widely in production by many popular vendors such as CockroachDB and others.

etcd Raft has an example binary called raftexample which is a minimal implementation for creating and understanding replicated state machines. A better understanding of Raft implementation example such as with raftexample will help you better use Raft in your applications. Let’s see raftexample in action with a simple use case.

Build raftexample binary

Clone the etcd repository and checkout the source code:

git clone https://github.com/etcd-io/etcd
cd etcd/contrib/raftexample

Set GOPATH and build binary:

export GOPATH=$(pwd)
go build -o raftexample

Optional: Move newly created raftexample binary to /usr/local/bin with execute permissions

Forming a Dynamic Replicated Cluster

I can use tools like goreman to automate common system tasks. But let’s keep things simple. Open-up 3 terminal windows and in the first window, run:

raftexample --id 1 --cluster http://127.0.0.1:12379 --port 12380

Consider processes as machines for now

This will create a one node(instance) by the id 1 in the cluster. Each has a KV (Key-Value) stored associated with listening at port defined by — port, 12380 in our case as defined above. The KV has a REST interface to work with the store. We will use curl to PUT and GET values from the KV store. Let’s PUT a value in the KV store with:

curl -L http://127.0.0.1:12380/key -XPUT -d foo

Try GET:

curl -L http://127.0.0.1:12380/key
foo

We can see the data. Since we have stored a KV data in one node. Let’s spin-up another machine in the second terminal by the id 2. To join second node with the already created cluster, we have to submit a POST request to the existing cluster with the second node port:

curl -L http://127.0.0.1:12380/2 -XPOST -d http://127.0.0.1:22379

Then add the second machine in the cluster:

raftexample --id 2 --cluster http://127.0.0.1:12379,http://127.0.0.1:22379 --port 22380 --join

You may see errors like:

2019-11-10 00:18:31.545755 I | replaying WAL of member 2
2019-11-10 00:18:31.759367 I | loading WAL at term 0 and index 0
panic: no peers given; use RestartNode instead
goroutine 51 [running]:
go.etcd.io/etcd/raft.StartNode(0xc0001bff20, 0x0, 0x0, 0x0, 0x0, 0xc00019a0c0)
 /home/kjanshair/Practice/raft/etcd/raft/node.go:216 +0x421
main.(*raftNode).startRaft(0xc0001a8000)
 /home/kjanshair/Practice/raft/etcd/contrib/raftexample/raft.go:293 +0x6a8
created by main.newRaftNode
 /home/kjanshair/Practice/raft/etcd/contrib/raftexample/raft.go:106 +0x35f

When adding the second node in the cluster. This might be due to not having the raftexample-2 folder on local directory. Simply re-run the command if that’s the case and the second node will join the cluster. Now here is the interesting thing: We have 2 machines 1 & 2, we put the data in 1 now try to retrieve the data from 2 with:

curl -L http://127.0.0.1:22380/key
foo

We got the data from the second node. This is because first machine has replicated its state to the second machine. Now add the 3rd machine with the same curl request in the 3rd terminal window and try to get value from the 3rd node:

curl -L http://127.0.0.1:12380/3 -XPOST -d http://127.0.0.1:32379
raftexample --id 3 --cluster http://127.0.0.1:12379,http://127.0.0.1:22379,http://127.0.0.1:32379 --port 32380 --join
curl -L http://127.0.0.1:32380/key
foo

You will get the value from the 3rd node as well. This is how etcd’s Raft implementation works. You can find the source code for raftexample here.

Deleting a Node

Try to stop the 2nd node (or any) and run the HTTP DELETE:

curl -L http://127.0.0.1:12380/2 -XDELETE

To remove the node from the cluster.