Secure Redis
José Coelho
102

Just aheads up on your HAproxy check: it allows split-brain to happen. A certain condition is thst If you have a failover and bring the original master back online (eg. you reboot the node it is on) there will be a window where you will have two masters and your HAProxy will send traffic to the old one.

This happens because until Sentinel demotes the old-master it will report itself as a master. Any data modification commands sent to the old-master will be lost when it is demoted. Other standard split-brain scenarios are likewise possible here.

If you want to use HAproxy to dynamically do it, you’ll want to have it query all Sentinels instead and only use a backend as master if at least #QUORUM sentinels agree it is the master. There are possible issues with some use cases due to the way HAProxy closes connections when it changes the backend but there isn’t a way around that when using HAProxy.

The better way is to use a client-reconfigure script on your Sentinel nodes to reconfigure & restart HAProxy (or use consul-template to do it). That way you get a clean reconnect on each kind of failover.

Show your support

Clapping shows how much you appreciated Bill Anderson’s story.