Herd Effect in Watch

ZooKeeper, Chubby, etcd are all able to provide distributed lock service. And mostly, they have been used to do leader election.

Consider the following case: a cluster would like replication to provide fault tolerant service. There is a primary replicating data on other backup machines. The primary is holding a lock (create a node) in ZooKeeper, and backups are watching the lock (watch node delete).

The Problem

However, there is a herd effect as described in ZooKeeper Recipes: if the primary crashed and disconnected, there would be a workload surge for notifying each client.

How To Solve It In ZooKeeper

As described in ZooKeeper Recipes:

To avoid the herd effect, it is sufficient to watch for the next znode down on the sequence of znodes. If a client receives a notification that the znode it is watching is gone, then it becomes the new leader in the case that there is no smaller znode. Note that this avoids the herd effect by not having all clients watching the same znode.

Another way to solve it

If you read the paper of Chubby, it talks a lot about cache on client side. Well, if a client has a copy of nodes for the “lock” directory, the client knows whether a service server on it has the smallest number.